How to resolve KafkaConnectionError: Socket EVENT_READ without in-flight-requests

35 Views Asked by At

I am using kafka for job processing, Jobs usually take from somewhere around 2 min to 30,40 minutes. For that I have increased the kafka poll time out and also increased some other timeouts in the configuration. But with this below configuration I am getting this error exactly after 10 minutes of message processing

KafkaConnectionError: Socket EVENT_READ without in-flight-requests

WARNING, "func": "_conn_state_change", "lineno": "331", "msg": "Node 2 connection failed -- refreshing metadata"

WARNING, "func": "_run_once", "lineno": "992", "msg": "Heartbeat poll expired, leaving group"}

INFO "func": "maybe_leave_group", "lineno": "773", "msg": "Leaving consumer group

Here is my kafka consumer config

kafka_consumer = KafkaConsumer(
        bootstrap_servers=[env.kafka_broker_address],
        auto_offset_reset="earliest",
        group_id=env.kafka_consumer_group,
        value_deserializer=lambda x: json.loads(x.decode("utf-8")),
        enable_auto_commit=False,
        max_poll_interval_ms=2500000,
        max_poll_records=2,
        heartbeat_interval_ms=1000,
        request_timeout_ms=3000000,
        session_timeout_ms=1000000,
        connections_max_idle_ms=3300000,
    )

Is there something I am missing in kafka consumer configs?

0

There are 0 best solutions below