I'm an absolute beginner in the monitoring world but I've deployed some services using docker-compose:
- cAdvisor
- Prometheus
- Grafana
My hope was to be able to show metrics on Grafana querying Prometheus and show the containers up or the state of service scraping the metrics of cAdvisor but in many examples I've followed they showed how you can query for example:
- container_last_seen{image="blah"}
- container_last_seen{name="blah"}
But I'm not able because cAdvisor is not publishing metrics with these labels.
My docker-compose.yml is like:
version: "3.9"
services:
backend:
[... hide this code for privacy]
frontend:
[... hide this code for privacy]
db:
image: arm64v8/mysql:8.0
[... hide this code for privacy]
prometheus:
image: prom/prometheus:v2.46.0
container_name: prometheus
volumes:
- ./prometheus:/etc/prometheus
- prometheus_data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--web.console.libraries=/etc/prometheus/console_libraries"
- "--web.console.templates=/etc/prometheus/consoles"
- "--storage.tsdb.retention.time=200h"
- "--web.enable-lifecycle"
- "--web.enable-remote-write-receiver"
- "--enable-feature=native-histograms"
restart: unless-stopped
expose:
- 9090
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
user: root
cadvisor:
image: gcr.io/cadvisor/cadvisor:v0.47.1
container_name: cadvisor
privileged: true
devices:
- /dev/kmsg:/dev/kmsg
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker:/var/lib/docker:ro
#- /cgroup:/cgroup:ro #doesn't work on MacOS only for Linux
restart: unless-stopped
expose:
- 8080
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
grafana:
image: grafana/grafana-enterprise
container_name: grafana
ports:
- "3100:3000"
volumes:
- grafana_data:/var/lib/grafana
environment:
- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
- GF_USERS_ALLOW_SIGN_UP=false
restart: unless-stopped
expose:
- 3100
networks:
- monitor-net
networks:
monitor-net:
driver: bridge
volumes:
dev-db-data:
dev-static-data:
grafana_data: {}
prometheus_data: {}
And my prometheus.yml:
global:
scrape_interval: 15s
evaluation_interval: 15s
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'docker-host-alpha'
# A scrape configuration containing exactly one endpoint to scrape.
scrape_configs:
- job_name: 'nodeexporter'
scrape_interval: 5s
static_configs:
- targets: ['nodeexporter:9100']
- job_name: 'cadvisor'
scrape_interval: 5s
static_configs:
- targets: ['cadvisor:8080']
- job_name: 'prometheus'
scrape_interval: 10s
static_configs:
- targets: ['localhost:9090']
- job_name: 'pushgateway'
scrape_interval: 10s
honor_labels: true
static_configs:
- targets: ['pushgateway:9091']
I tried: docker-compose up checked all my services were up and running check the cAdvisor /metrics
# HELP cadvisor_version_info A metric with a constant '1' value labeled by kernel version, OS version, docker version, cadvisor version & cadvisor revision.
# TYPE cadvisor_version_info gauge
cadvisor_version_info{cadvisorRevision="7dfeea7d",cadvisorVersion="v0.47.1",dockerVersion="",kernelVersion="5.15.49-linuxkit-pr",osVersion="Alpine Linux v3.16"} 1
# HELP container_blkio_device_usage_total Blkio Device bytes usage
# TYPE container_blkio_device_usage_total counter
container_blkio_device_usage_total{device="",id="/013-mount-services",major="7",minor="0",operation="Read"} 53760 1695050571700
container_blkio_device_usage_total{device="",id="/013-mount-services",major="7",minor="0",operation="Write"} 0 1695050571700
container_blkio_device_usage_total{device="",id="/artifactory",major="7",minor="0",operation="Read"} 1.250304e+07 1695050574896
container_blkio_device_usage_total{device="",id="/artifactory",major="7",minor="0",operation="Write"} 0 1695050574896
container_blkio_device_usage_total{device="",id="/binfmt",major="7",minor="0",operation="Read"} 2.6136576e+07 1695050572835
container_blkio_device_usage_total{device="",id="/binfmt",major="7",minor="0",operation="Write"} 0 1695050572835
container_blkio_device_usage_total{device="",id="/container-filesystem",major="7",minor="0",operation="Read"} 1.4364672e+07 1695050549973
container_blkio_device_usage_total{device="",id="/container-filesystem",major="7",minor="0",operation="Write"} 0 1695050549973
container_blkio_device_usage_total{device="",id="/devenv-service",major="7",minor="0",operation="Read"} 5.885952e+06 1695050574243
container_blkio_device_usage_total{device="",id="/devenv-service",major="7",minor="0",operation="Write"} 0 1695050574243
container_blkio_device_usage_total{device="",id="/dhcpcd",major="7",minor="0",operation="Read"} 1.83296e+06 1695050559325
But no tags name or image were found.
Any help, any clue would be much appreciate.
P.S.: my first question here