Kubernetes HPA doesnt scale up properly

Question

I have app-1 deployment and I have configured HPA for it. It seems that HPA is not working as expected because even though the memory usage (108%) exceeds the target, HPA does not add any more replicas.

$ kubectl get hpa                               
NAME              REFERENCE                    TARGETS             MINPODS   MAXPODS   REPLICAS   AGE
app-1          Deployment/app-1                109%/100%, 5%/70%   2         4         3           3d

$ kubectl describe hpa app-1
...
Metrics:                                                  ( current / target )
  resource memory on pods  (as a percentage of request):  109% (5827813376) / 100%
  resource cpu on pods  (as a percentage of request):     18% (361m) / 70%
Min replicas:                                             2
Max replicas:                                             4
Behavior:
  Scale Up:
    Stabilization Window: 400 seconds
    Select Policy: Max
    Policies:
      - Type: Pods  Value: 1  Period: 60 seconds
  Scale Down:
    Stabilization Window: 400 seconds
    Select Policy: Min
    Policies:
      - Type: Pods  Value: 1  Period: 60 seconds
Deployment pods:    3 current / 3 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:           <none>

Can someone help me and explain what is wrong?

Answer 1

Based at the ‘kubectl describe hpa’ output, we see the following information:

‘AbleToScale: True ReadyForNewScale recommended size matches current size’ ‘ScalingLimited: False DesiredWithinRange the desired count is within the acceptable range’

This implies that the HPA thinks it's at the right scale, despite the memory utilization being over the target.

You need to dig deeper by monitoring the HPA and the associated metrics over a longer period, considering your 400-second stabilization window.That means the HPA will not react immediately to metrics but will instead observe them over this window before making scaling decisions. Also check the cluster and HPA events for any signs of issues. Lastly, make sure that your cluster has enough resources to schedule new pods if they are scaled. Attached is a good read regarding Kubernetes HPA that might be helpful on your use case. [1]

[1] https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

Answer 2

Based on the provided information, it has already scaled to 3 pods and is now waiting to scale more. Time can be adjusted here:

  Scale Up:
    Stabilization Window: 400 seconds
    Select Policy: Max
    Policies:
      - Type: Pods  Value: 1  Period: 60 seconds