I have an application that uses fs2-kafka for reading business events from a kafka cluster. In that application, I have multiple fs2-kafka consumers, each subscribed to a different topic. But one of the consumers seems to be stuck, as it does not consume any events.
Checking the consumer group's offsets yielded the following results:
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
consumer topic 0 - 5 - consumer-consumer-1-99c1c19a-faaf-40e6-a3dc-75b7d04e96f9 /10.0.3.2 consumer-consumer-1
(edited this slightly cause privacy)
I have also managed to get the CURRENT-OFFSET to be 1 (though, it seems like no actual consuming happened because none of my logs were triggered), but regardless - the group does not seem to want to move its offset.
The topic has just one partition and there's only one consumer/consumer group reading from that topic. There is no reason I can see for kafka to hold consumers from consuming. If it matters - that topic, as well as any other topic in this cluster, is created automatically, using kafka's "AUTO_CREATE_TOPICS". (this is a dev environment, it's simply more convinient than creating topics by hand)
The strangest thing is this - the same code, working on a different topic, works. Also, as it is always the case with these things, on my laptop the issue does not reproduce. There's barely any differences between my local kafka and the kafka in our dev cluster.
Originally, I had just one consumer group for the entire application. I have now tried multiple consumer groups per consumer and even sharing a consumer for reading from multiple topics. The only topic that's stuck is this one, every other topic works.
I have also tried:
- Restarting kafka and the app, updating kafka to a newer version
- Manually resetting the consumer group's offsets
- Deleting the topic
Apart from deleting all the data of kafka, I believe I have tried everything on my and kafka's side.