cAdvisor docker service doesn't publish name o image tag to use in Prometheus

178 Views Asked by At

I'm an absolute beginner in the monitoring world but I've deployed some services using docker-compose:

  • cAdvisor
  • Prometheus
  • Grafana

My hope was to be able to show metrics on Grafana querying Prometheus and show the containers up or the state of service scraping the metrics of cAdvisor but in many examples I've followed they showed how you can query for example:

  • container_last_seen{image="blah"}
  • container_last_seen{name="blah"}

But I'm not able because cAdvisor is not publishing metrics with these labels.

My docker-compose.yml is like:

version: "3.9"

services:
  backend:
    [... hide this code for privacy]
  frontend:
    [... hide this code for privacy]
  db:
    image: arm64v8/mysql:8.0
    [... hide this code for privacy]

  prometheus:
    image: prom/prometheus:v2.46.0
    container_name: prometheus
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.console.libraries=/etc/prometheus/console_libraries"
      - "--web.console.templates=/etc/prometheus/consoles"
      - "--storage.tsdb.retention.time=200h"
      - "--web.enable-lifecycle"
      - "--web.enable-remote-write-receiver"
      - "--enable-feature=native-histograms"
    restart: unless-stopped
    expose:
      - 9090
    networks:
      - monitor-net
    labels:
      org.label-schema.group: "monitoring"
    user: root

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.47.1
    container_name: cadvisor
    privileged: true
    devices:
      - /dev/kmsg:/dev/kmsg
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
      #- /cgroup:/cgroup:ro #doesn't work on MacOS only for Linux
    restart: unless-stopped
    expose:
      - 8080
    networks:
      - monitor-net
    labels:
      org.label-schema.group: "monitoring"

  grafana:
    image: grafana/grafana-enterprise
    container_name: grafana
    ports:
      - "3100:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
      - GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
      - GF_USERS_ALLOW_SIGN_UP=false
    restart: unless-stopped
    expose:
      - 3100
    networks:
      - monitor-net

networks:
  monitor-net:
    driver: bridge

volumes:
  dev-db-data:
  dev-static-data:
  grafana_data: {}
  prometheus_data: {}

And my prometheus.yml:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'docker-host-alpha'

# A scrape configuration containing exactly one endpoint to scrape.
scrape_configs:
  - job_name: 'nodeexporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['nodeexporter:9100']

  - job_name: 'cadvisor'
    scrape_interval: 5s
    static_configs:
      - targets: ['cadvisor:8080']

  - job_name: 'prometheus'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'pushgateway'
    scrape_interval: 10s
    honor_labels: true
    static_configs:
      - targets: ['pushgateway:9091']

I tried: docker-compose up checked all my services were up and running check the cAdvisor /metrics

# HELP cadvisor_version_info A metric with a constant '1' value labeled by kernel version, OS version, docker version, cadvisor version & cadvisor revision.
# TYPE cadvisor_version_info gauge
cadvisor_version_info{cadvisorRevision="7dfeea7d",cadvisorVersion="v0.47.1",dockerVersion="",kernelVersion="5.15.49-linuxkit-pr",osVersion="Alpine Linux v3.16"} 1
# HELP container_blkio_device_usage_total Blkio Device bytes usage
# TYPE container_blkio_device_usage_total counter
container_blkio_device_usage_total{device="",id="/013-mount-services",major="7",minor="0",operation="Read"} 53760 1695050571700
container_blkio_device_usage_total{device="",id="/013-mount-services",major="7",minor="0",operation="Write"} 0 1695050571700
container_blkio_device_usage_total{device="",id="/artifactory",major="7",minor="0",operation="Read"} 1.250304e+07 1695050574896
container_blkio_device_usage_total{device="",id="/artifactory",major="7",minor="0",operation="Write"} 0 1695050574896
container_blkio_device_usage_total{device="",id="/binfmt",major="7",minor="0",operation="Read"} 2.6136576e+07 1695050572835
container_blkio_device_usage_total{device="",id="/binfmt",major="7",minor="0",operation="Write"} 0 1695050572835
container_blkio_device_usage_total{device="",id="/container-filesystem",major="7",minor="0",operation="Read"} 1.4364672e+07 1695050549973
container_blkio_device_usage_total{device="",id="/container-filesystem",major="7",minor="0",operation="Write"} 0 1695050549973
container_blkio_device_usage_total{device="",id="/devenv-service",major="7",minor="0",operation="Read"} 5.885952e+06 1695050574243
container_blkio_device_usage_total{device="",id="/devenv-service",major="7",minor="0",operation="Write"} 0 1695050574243
container_blkio_device_usage_total{device="",id="/dhcpcd",major="7",minor="0",operation="Read"} 1.83296e+06 1695050559325

But no tags name or image were found.

Any help, any clue would be much appreciate.

P.S.: my first question here

0

There are 0 best solutions below