We currently have a Kafka partition with 3 and 3 Kubernetes pods for each Kafka partition. We observed that there is a performance lag because of less number of pods. We are planning to increase the pods. We observed that lag is happening during specific timing, so we thought of using Kubernetes HPA.
- What are the drawbacks if we just increase the partition to 6 and implement Kubernetes HPA
- Whats the advantage of using Keda/knative for the same ?
reference : https://www.confluent.io/events/kafka-summit-americas-2021/intelligent-auto-scaling-of-kafka-consumers-with-workload-prediction/, https://itnext.io/kafka-consumer-autoscaling-with-keda-41310f80a62a
Increasing the partition count will definitely help with throughput. HPA will work on metrics like CPU and memory using which you could still scale. But if you could keep replicas equal to number of partition that will work like charm eg if you have 6 partition have 6 replica always running.
Keda works in event-driven fashion so it will monitor the consumer lag (you could configure that) and it will do the autoscaling better than k8s HPA.