keep kafka connection alive even if it is idle for long time

Question

I am making a microservice application and using AWS MSK for Kafka. Currently, I am facing an issue in keeping the connection alive for a long time. As we are not in production mode, currently there is less traffic in inter-service communication, so Kafka closes the connection after a certain time, (default 9 min). I want to keep the connection alive so that I can produce and consume messages even after a long idle time. Does anybody have a solution?

Note: we can only set connections.max.idle.ms upto 1800000 ms.

Answer 1

connections.max.idle.ms can be a configuration of a broker, producer, consumer, admin, etc... But let's agree it can be broker side or client side.

While client side is indeed 9 mins default (except for admin, which is 5 mins), broker side defaults to 10 mins, to give chances to the client gracefully react on exceeding max idle before a broker kicks a client out.

If you would like to increase the idling, you would need to increase first of all the broker side idling. In MSK, you need to create (or update existing) cluster configuration and set this param to desired value with some extra, and then also client's configuration. as you have already done.

For example:

1. broker configuration: connections.max.idle.ms=1860000, which is 31 mins
2. consumer configuration: connections.max.idle.ms=1800000, which is 30 mins, as mentioned in the question

If this doesn't help, please add a comment to this answer