First time user of cAdvisor here - I'm trying to set up a monitoring stack to get metrics on containers running a series of Python Flask apps. This is all on an M2 Mac, with intent to eventually deploy alongside my app to a Linux host with hopefully zero to minimal edits required to development configuration.
Here's my docker compose for cAdvisor with Prometheus, PromTail, Loki, and Grafana:
version: '3'
networks:
monitoring:
services:
cadvisor:
image: gcr.io/cadvisor/cadvisor:v0.47.2
platform: linux/aarch64 # Necessary for running on macOS, all other archs return more errors
privileged: true
container_name: cadvisor
ports:
- "8080:8080"
volumes:
- /etc/machine-id:/etc/machine-id:ro
- /var/lib/dbus/machine-id:/var/lib/dbus/machine-id:ro
- /:/rootfs:ro
- /var/run/docker.sock:/var/run/docker.sock
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
networks:
- monitoring
promtail:
image: grafana/promtail:2.4.0
container_name: promtail
volumes:
- /var/log:/var/log
- ./config/promtail:/etc/promtail
command: --config.file=/etc/promtail/promtail-config.yml
networks:
- monitoring
prometheus:
image: prom/prometheus:latest
container_name: prometheus
volumes:
- ./prometheus-data:/prometheus
- ./config/prometheus:/config
ports:
- "9090:9090"
command: --config.file=/config/prometheus.yml
networks:
- monitoring
depends_on:
- cadvisor
loki:
image: grafana/loki:latest
container_name: loki
volumes:
- ./config/loki:/etc/loki
ports:
- "3100:3100"
command: --config.file=/etc/loki/loki-config.yml
networks:
- monitoring
depends_on:
- promtail
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
volumes:
- ./config/grafana:/var/lib/grafana
networks:
- monitoring
depends_on:
- loki
- prometheus
And in case it's relevant, here's my prometheus config:
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
labels:
alias: 'prometheus'
- job_name: 'cadvisor'
scrape_interval: 10s
metrics_path: '/metrics'
static_configs:
- targets: ['cadvisor:8080']
labels:
group: 'cadvisor'
alias: 'cadvisor'
The issue: I'm not getting detailed information about my containers to be able to filter them in Prometheus/Grafana.
Here's what I see in Prometheus:
{alias="cadvisor", cpu="total", group="cadvisor", id="/docker/00bb79f8411136638874dab49f1f374d6f57a481913c874fc5a655744735538b", instance="cadvisor:8080", job="cadvisor"}
{alias="cadvisor", cpu="total", group="cadvisor", id="/docker/02af865e22c71c93118888998655177ace03e92d7ddd23e03126c46685522246", instance="cadvisor:8080", job="cadvisor"}
{alias="cadvisor", cpu="total", group="cadvisor", id="/docker/04be086a146a18efd1ebf150f28cd670e024c4018b4fc5fefa19a7b82642c35d", instance="cadvisor:8080", job="cadvisor"}
... so on and so forth. There are far more of these entries than containers that I'm running. I'm not even sure if these are related to my containers at all.
In cAdvisor, screenshots I see suggest that there should be a "Subcontainers" section on the cAdvisor /docker/ page. This doesn't appear for me - the entire page I see at http://localhost:8080/docker/ is pictured here.
The only logs I see from cAdvisor are:
cadvisor | W0211 11:25:10.725909 1 sysinfo.go:203] Nodes topology is not available, providing CPU topology
cadvisor | W0211 11:25:10.726651 1 machine.go:65] Cannot read vendor id correctly, set empty.
I believe cAdvisor is the issue, as the rest of the stack works fine. I'm not sure what I'm doing wrong though, most of my research suggests that what I'm doing should be working, but it isn't.