Micrometer Gauge with Spring boot - value not updating

Question

I have this block of Java code that uses micrometer library to report metrics regarding failed jobs in each tenant. It reports failed jobs for each tenant correctly because I verified it on my Grafana, but the problem is that when the number of failed jobs for a tenant increases or decreases, the update is not getting reflected on Grafana. What do I do so that each update is reflected on Grafana?

@Component
@RequiredArgsConstructor
public class FailedJobsGaugeReporter {
    private final ManagementService managementService;

    @Scheduled(fixedDelayString = "3000", initialDelayString = "1000")
    public void reportFailedJobGauge() {
        List<Job> list = managementService.createFailedJobQuery().listPage(0, 999);

        Map<String, Long> countByTenantId = list.stream()
                .collect(Collectors.groupingBy(Job::getTenantId, Collectors.counting()));

        String name = "rc_job_failed_gauge";

        MeterRegistry registry = ApplicationContextHolder.getBean(PrometheusMeterRegistry.class);

        for (Map.Entry<String, Long> entry : countByTenantId.entrySet()) {
                Tag tag = Tag.of("tenant", entry.getKey());
                Gauge.builder(name, entry::getValue).tags(List.of(tag)).strongReference(true).register(registry);
            }
    }
}

Answer 1

You should not try to recreate a Gauge every time but you should update the value of the Gauge. Please read the docs first: https://micrometer.io/docs/concepts#_gauges

Please also check the MultiGauge: https://micrometer.io/docs/concepts#_multi_gauge

in your ctor:

MultiGauge jobs = MultiGauge.builder("jobs")
    .tag("failed", "true")
    .register(registry);

Then in your scheduled method:

jobs.register(
    countByTenantId.entrySet().stream()
        .map(entry -> Row.of(Tags.of("tenant", entry.getKey()), entry.getValue()))
        .collect(toList())
);

I also think that you should use a Counter instead not a Gauge` for this (if it is a monotonic count) where the jobs are failing.

Also, please do not do this:

MeterRegistry registry = ApplicationContextHolder.getBean(PrometheusMeterRegistry.class);

1. You can inject the MeterRegistry the same way you injected ManagementService.
2. This will not use the composite registry, only prometheus so if you have multiple registries (e.g.: actuator metrics endpoint) those will not be updated.

The separator in the name of the meter should be ., Micrometer will convert that for you to _ in case of Prometheus.