I have two consumer groups for one kafka topic I want setup such a method that there is no duplication

Question

I have two consumer groups for one kafka topic I want setup such a method that if one consumer group gets down then 2nd comsumer group starts consuming from the offset left by the 1st one to avoid duplication

I am exploring ideas that are industry level

Answer 1

This isn't possible, nor necessary. Offsets are unique per consumer group.

You'd have to explicitly seek and commit the offsets for the second group, or just not use Kafka for group management altogether, and instead store and query from an external system.

Restarting any instance of one group would pickup from the last committed position. This would cause at-least once processing. You'd additionally require transactions to handle exactly once processing.

Answer 2

What you want to achieve can be achieved only by 1 consumer group but if you have 2 different consumer groups, they behave independently and both maintains there own offset.

Also when you say one consumer group is down, do you mean all the consumers within that consumer is down? in which case you bring up those consumers to start consuming from where it left.

Answer 3

When you mean two consumer groups, I believe they are two different consumer applications hosted in two different regions. If your intention is to handle DR scenario by brining up secondary consumer app when your primary is down and thereby you should avoid duplicates, it is not something new to the industry. First of all, we must aim at building consumer applications which are capable of handling duplicates. If this is not possible, during a DR scenario, when primary goes down, you would know the timestamp when it went down. Use this time to get the offset in secondary consumer using getOffsetsByTimes method. Once you have the offset, start consuming from this offset to avoid duplicates.