Can't access default k3s metrics server from Prometheus-Adapter

Question

I have installed Prometheus-adapter along with the default metrics-server that comes with k3s securely on port 443.

Unfortunately, I get no resources when I query custom.metrics.k8s.io

$  kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": []
}

When I look at the logs of Prometheus-adapter I get unable to update list of all metrics: unable to fetch metrics for query ...: x509: certificate is valid for localhost, localhost, not metrics-server.kube-system

How can I resolve this issue?

Answer 1

To solve this issue, I had to create separate certificate for both metrics-server and adapter. Adapter also has an issue about adding capability to ignore cert validation which wasn't merged.

For metrics-server and cert request I used the following:

{
  "hosts": [
    "prometheus-adapter",
    "prometheus-adapter.monitoring",
    "prometheus-adapter.monitoring.svc",
    "prometheus-adapter.monitoring.pod",
    "prometheus-adapter.monitoring.svc.cluster.local",
    "prometheus-adapter.monitoring.pod.cluster.local",
    "<pod ip>",
    "<service ip>"
  ],
  "CN": "prometheus-adapter.monitoring.pod.cluster.local",
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
}

{
  "hosts": [
    "metrics-server",
    "metrics-server.kube-system",
    "metrics-server.kube-system.svc",
    "metrics-server.kube-system.pod",
    "metrics-server.kube-system.svc.cluster.local",
    "metrics-server.kube-system.pod.cluster.local",
    "<service ip>",
    "<pod ip>"
  ],
  "CN": "metrics-server.kube-system",
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
}

For ca, you can create your certificate authority or use Kubernetes signers as indicated here The only point worth noting here is that if you use either of signers, you should mount the ca bundle yourself to your deployments.

Finally, mount tls keys and ca bundle to your deployment.

  extraArguments:
    - --tls-cert-file=/var/run/serving-cert/tls.crt
    - --tls-private-key-file=/var/run/serving-cert/tls.key
    - --client-ca-file=/etc/ssl/certs/ca.crt

Answer 2

You have two main options here:

1. Configure Prometheus to ignore ssl, setting to true the skip verify

tls_config:
insecure_skip_verify: true

2. Add the certificate to the scrape_configs under tls_config

tls_config: ca_file: /path/to/prometheus.crt

UPDATE:

On metrics-server-deployment.yaml

1. Leave internalip only, on the --kubelet-preferred-address-type

--kubelet-preferred-address-types=InternalIP