We're using MirrorMaker2 to replicate some topics from one kafka cluster to another kafka cluster (unidirectional) for DR purpose.
As a poc, I have created a sample fast producer producing to a topic and a slow consumer consuming it, to simulate lag and wanted to ensure that latest consumer offset in source cluster is getting copied to target cluster periodically. The objective here is when the primary region down, the DR region should continue near to the latest offset from source cluster.
However the consumer offset is not getting copied periodically. During the initial start of mirrormaker2 the consumer offset 70095 is copied to target cluster, although the source cluster at that time had consumer offset 89065.
mm2.properties
clusters=clusterA, clusterB
clusterA.bootstrap.servers=10.0.0.1:9092
clusterB.bootstrap.servers=10.0.0.2:9092
clusterA.config.storage.replication.factor=1
clusterA.offset.storage.replication.factor=1
clusterA.status.storage.replication.factor=1
clusterB.config.storage.replication.factor=1
clusterB.offset.storage.replication.factor=1
clusterB.status.storage.replication.factor=1
clusterA->clusterB.enabled=true
offset-syncs.topic.replication.factor=1
heartbeats.topic.replication.factor=1
checkpoints.topic.replication.factor=1
topics=IFLIGHT-RTU-DB-RW-TOPIC
groups=iflight_rtudbrwtopic
tasks.max=2
replication.factor=1
refresh.topics.enabled=true
sync.topic.configs.enabled=true
sync.group.offsets.enabled=true
refresh.topics.interval.seconds=60
refresh.groups.interval.seconds=5
emit.checkpoints.enabled=true
emit.checkpoints.interval.seconds=10
sync.group.offsets.interval.seconds=5
topics.blacklist=.*[\-\.]internal, .*\.replica, __consumer_offsets
groups.blacklist=console-consumer-.*, connect-.*, __.*
clusterA->clusterB.emit.heartbeats.enabled=true
clusterA->clusterB.emit.checkpoints.enabled=true
clusterA->clusterB.sync.group.offsets.enabled=true
source cluster
target cluster
- What might be the reason for the target cluster has consumer offset as 70095 instead of 89065 during the first time mirrormaker2 started.
- When new messages have received in source cluster and consumer offset changes, its not getting periodically synced to target cluster, although the mirrormaker2 is running.
Note: Kafka version: 3.5.1

