I'm after some clarification on how Kafka standby replicas work for some piece of mind!
Given this scenario;
A Kafka streams application;
- reading from a source topic with 10 partitions
- writing to a sink topic with 6 partitions.
- rocksdb session state changelog
- hosted in docker with two replicas running num.standby.replicas set to 1
The idea is that if one pod does down the other pod will take over processing immediately...as least that is what I thought having the num.standby.replicas set did.
What I am seeing is when initially starting up, only one pod is processing traffic. This is what I'd expect, one pod processing traffic and the other pod just keeping state in case of fail over.
However, if I restart the a pod, both pods start processing traffic equally.
Is there any information online which could clarify how standby replicas are meant to work?..or is this expected behaviour?
J
What you observe is by design. Both pods will do 50% of processing and the standby tasks are assigned "reversed". Ie, if pod A host active task 0_0, the corresponding standby task 0_0 will be hosted by pod B.
Thus, whenever one pods fails, only 50% of tasks are "migrated", ie, if pod A fails, all standbys of pod B will be "promoted" as active, and pod B will now do all processing.