configure prometheus service monitor for external services like confluent cloud kafka

Question

I'm trying to configure prometheus service monitor to fetch confluent cloud(kafka) metrics. According to their documentation (https://api.telemetry.confluent.cloud/docs?&_ga=2.215665047.1225578079.1655110920-914939436.1654166692#tag/Version-2/paths/~1v2~1metrics~1{dataset}~1export/get), they have configured using the scrape_config.

But I wanted to scrape using Service monitor.

I created a k8s external service:

kind: Service
apiVersion: v1
metadata:
  name: kafka-metric-api-service
  namespace: monitoring
  labels:
    app: kafka-metric-api
spec:
  type: ExternalName
  externalName: api.telemetry.confluent.cloud

And I created service monitor like below:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kafka-metric-api
  namespace: monitoring
  labels:
    app: kafka-metric-api
spec:
  endpoints: 
  - basicAuth:
      password:
        name: kafka-basic-auth
        key: password
      username:
        name: kafka-basic-auth
        key: username
    path: /v2/metrics/cloud/export
    interval: 60s
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
    params:
      resource.kafka.id:
        - lkc-xxxx
  selector:
    matchLabels:
      app: kafka-metric-api

Since service doesn't have any ports, port names, I don't know what to put in spec.endpoint. current setup not working and I see below status:

could someone help me on how to configure service monitors for external services like confluent kafka which doesn't have ports and how to scrape metrics? In internet I see examples for services which has ports defined.

Answer 1

This worked for me

apiVersion: v1
kind: Endpoints
metadata:
  name: confluent-cloud-telemetry-svc
  namespace: monitoring
  labels:
    app: confluent-cloud-telemetry
    release: prometheus-operator
subsets:
- addresses:
  - ip: 54.149.69.190
  ports:
  - name: confluent-cloud-telemetry-port
    protocol: TCP
    port: 443

---
apiVersion: v1
kind: Service
metadata:
  name: confluent-cloud-telemetry-svc
  namespace: monitoring
  labels:
    app: confluent-cloud-telemetry
    release: prometheus-operator
spec:
  type: ExternalName
  externalName: api.telemetry.confluent.cloud
  ports:
  - name: confluent-cloud-telemetry-port
    protocol: TCP
    port: 443
    targetPort: 443

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: confluent-cloud-telemetry-sm
  namespace: monitoring
  labels:
    app: confluent-cloud-telemetry
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app: confluent-cloud-telemetry
      release: prometheus-operator
  namespaceSelector:
    matchNames: 
    - monitoring
  endpoints: 
  - basicAuth:
      password:
        name: kafka-basic-auth
        key: password
      username:
        name: kafka-basic-auth
        key: user
    port: confluent-cloud-telemetry-port
    interval: 60s
    honorLabels: true
    scheme: https
    path: /v2/metrics/cloud/export
    tlsConfig:
      insecureSkipVerify: true
    relabelings:
      - sourceLabels: [__address__]
        targetLabel: __address__
        regex: (.*)
        replacement: "api.telemetry.confluent.cloud:443"
        action: replace
    params:
      resource.kafka.id:
        - <YOUR_CLUSTER_ID>

---
apiVersion: v1
kind: Secret
metadata:
  name: kafka-basic-auth
data:
  password: <YOUR_SECRET_BASE64>
  user: <YOUR_KEY_BASE64>
type: Opaque