Dashboard visualizing CPU usage of kafka container chopped up

Question

I want to monitor the cpu usage of kafka container, but the graph is chopped up into different pieces. There seem to be gaps in the graph and after each gap a different colored line follows. The time range is last 30 days. For the exporter we use danielqsj/kafka-exporter:v1.4.2

The promql query used to create this graph is:

rate(container_cpu_usage_seconds_total{container="cp-kafka-broker"}[1m])

Can I merge these lines into one continual? If so, with what promql expression/dashboard configuration?

Answer 1

This happens when at least 1 of the labels that are attached to the metric changes. The rate function keeps all the original labels from the underline time series. In Prometheus, each time series is uniquely identified by the metric name container_cpu_usage_seconds_total and any labels (key-value pairs) attached to the metric (container, for instance). This is why Grafana uses different colors because they are different time series.

If you want to get a single series in Grafana you can aggregate using the sum operator:

sum(rate(container_cpu_usage_seconds_total{container="cp-kafka-broker"}[1m]))

which by default will not keep any of the original labels.