Prometheus returning "Empty query result" even if the scrape target is answering

Question

I have a Prometheus instance which is configured to scrape 4 targets with a custom metric (xinetd).

When I look at the definition of the targets, they appear as Up (green) with a scrape duration varying between 150ms a 8s.

From the machine hosting the Prometheus instance, I can successfully scrape the endpoints:

# curl http://myserver:11111
# HELP migrate_to_pure_finished_jobs Number of finished jobs
# TYPE migrate_to_pure_finished_jobs counter
migrate_to_pure_finished_jobs 0

Note that for other targets, the counter value is actually greater than 0.

When I go to the web interface of Prometheus, the autocomplete proposes me the name of the metric correctly:

However, when I run the query migrate_to_pure_finished_jobs returns:

Empty query result

I fail to understand how an endpoint can be scraped every 15 seconds successfully, get the metrics correctly registered in Prometheus, but not have values.

Answer 1

The disk holding the Prometheus data was actually full. Prometheus was actually not returning any data from any query. Even the up query was not returning anything.