Reading kafka messages in batch by using conn.ReadBatch in kafka-go and manual offset management

154 Views Asked by At

I'm working on implementing batch message reading using kafka-go. You can find the context for this task in the following discussion: Seeking a Third Opinion on Kafka Consumer Implementation and Architectural Disagreements.

I'm considering using conn.ReadBatch from kafka-go, which is primarily used internally, as discussed in this GitHub issueWay to get batch messages and commit if the batch is successful. It seems that I'll need to manually manage the offset with this method. The ReadBatch description suggests potential complexity when used concurrently:

While it is safe to call ReadBatch concurrently from multiple goroutines, the unpredictable results due to multiple goroutines reading and writing the connection offset can lead to duplicates or inconsistent message visibility.

To implement this, my plan is to:

  1. Manually create a Conn object for each partition.
  2. Use the Conn object to read a batch of messages.
  3. Process the messages, calculate the offset, and commit it manually.

My questions:

  1. Does this flow seem valid?
  2. How can I manually commit the offset, as I didn't find specific documentation on this?
  3. Is there a more user-friendly way to read batch messages from Kafka? I'm currently unsure, but I thought it's worth asking again.

Thanks for your assistance.

0

There are 0 best solutions below