Kafka producers failing to produce message due to LeaderNotAvailableError

Question

I'm having some issues with kafka and zookeeper, I have 3x kafka brokers and 3x zookeepers. I'm getting several kafka issues from all over my containers, some have this:

time="2023-12-26T09:46:00Z" level=error msg="Failed to produce message {low_priority[-1]@0 [123 34 99 111 109 112 111 ......] 0001-01-01 00:00:00 +0000 UTC NotAvailable <nil> []} with producer &{rdkafka#producer-1 x}"

others have this:

future: <Future finished exception=LeaderNotAvailableError()>
kafka.errors.LeaderNotAvailableError: [Error 5] LeaderNotAvailableError

Looking at kafka logs nothing seems out of the ordinary. Zookeeper has these logs:

[2023-12-25 13:23:14,197] WARN Close of session 0x0 (org.apache.zookeeper.server.NIOServerCnxn)
java.io.IOException: ZooKeeperServer not running
    at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:544)
    at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:332)
    at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
    at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

I'm failing to understand the issue. Is zookeeper running out of memory, perhaps? something similar? Any pointers would be appreciated. Thanks

Answer 1

I recommend you migrate to the new Kafka versions and use a new KRaft stands for Kafka Raft Metadata mode and replace zookeeper consensus protocol. https://developer.confluent.io/learn/kraft/