I've recently deployed locust 2.15.1 into a kubernetes cluster v1.25.10 via locust helm chart version 0.31.5. I have HPA enabled and configured like so:
hpa:
enabled: true
minReplicas: 5
maxReplicas: 100
targetCPUUtilizationPercentage: 50
During my load tests I'm seeing this unwanted and unexpected behaviour whereby some minutes after locust has hit the specified number of users, the horizontal pod autoscaler starts to gradually downscale by removing locust worker pods, in accordance with the configuration above. This is fine except with each worker pod that HPA removes, the user load is also reduced leading to a gradual step down of the load until the minimum number of locust workers is reached i.e. the minReplicas value above.
Whenever a worker is removed by HPA something like the following is logged by the locust master:
[2024-01-05 15:15:11,778] locust-master-54b7d9b998-wwjtq/INFO/locust.runners: Worker 'locust-worker-79fbc94fbf-ntqqj_0c9efb47081c463c8ae94258ce52d5bc' (index 2) quit. 0 workers ready.
Does anyone have any advice on how to avoid this unwanted ramp down in load that coincides with locust worker pod scale down?