How do I use Horizontal Pod Autoscaling in Kubernetes with an external/custom metric?

Question

I'm trying to implement HorizontalPodAutoscaler (HPA) for my EKS cluster using an external application metric. However, my HPA doesn't seem to be able to detect this metric from my Prometheus configuration, even though I'm able to scrape the metric successfully via Prometheus.

I was expecting HPA to detect my custom metric after defining it.

The YAML files are as follows:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-connection-based
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: eamm-deployment-v1
  minReplicas: 1
  maxReplicas: 4
  metrics:
  - type: External
    external:
      metric:
        name: active_connections_total
      target:
        type: Value
        averageValue: 1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: adapter-config
  namespace: prometheus
data:
  config.yaml: |-
    rules:
    - seriesQuery: |
        active_connections_total
      resources:
        template: pod
      name:
        matches: "^(.*)_total"
        as: "$1"
      metricsQuery: |
        sum by (app) (
          active_connections_total{app="eamm"}
        )

The error I'm getting:

Warning  FailedComputeMetricsReplicas  18m (x12 over 21m)             horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get active_connections_total external metric value: failed to get active_connections_total external metric: unable to get external metric default/active_connections_total/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get active_connections_total.external.metrics.k8s.io)
Warning  FailedGetExternalMetric       94s (x81 over 21m)             horizontal-pod-autoscaler  unable to get external metric default/active_connections_total/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get active_connections_total.external.metrics.k8s.io)
Warning  FailedGetExternalMetric       <invalid> (x2 over <invalid>)  horizontal-pod-autoscaler  unable to get external metric default/active_connections_total/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get active_connections_total.external.metrics.k8s.io)
Warning  FailedComputeMetricsReplicas  <invalid> (x2 over <invalid>)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get active_connections_total external metric value: failed to get active_connections_total external metric: unable to get external metric default/active_connections_total/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get active_connections_total.external.metrics.k8s.io)

Any help is much appreciated

Answer 1

You're using the RegEx ^(.*)_total to match the key and using $1 as the RegEx group selector. Given the metric active_connections_total, the key selected by $1 would be active_connections, NOT active_connections_total.

Try updating

      metric:
        name: active_connections_total

to

      metric:
        name: active_connections

See: Metrics Discovery and Presentation Configuration -- Prometheus Adapter.