Disclaimer: I'm new to PromQL, and might have constructed a query that is completely wrong. I'm using PromQL as the built-in monitoring of Google Cloud does not produce expected results (see this post).
When I use PromQL for a custom metric in Google Cloud I'm facing several issues:
- my query generates too many data points (most likely something wrong with the query below?)
- when I zoom out, all data disappears, or some aggregates are gone. The counts go from 1k to 50 for some reason
- when I use a 1m interval, the data disappears
- it is unclear how user-based distribution metrics should be queried, the docs are missing examples (it seems 3 submetrics are generated: bucket, count and sum, but I have no clue on how to use them)
I mainly want to have a sum of log-based metrics that contains a number of API hits like:
hits: 14
hits: 56
...
an interval should just show the sum of these numbers.
The query:
sum by (dag_id) (
sum_over_time(logging_googleapis_com:user_metric_sum[30m])
)
Example
Zoomed in version:
Data is gone when zooming out:
Multiple data points when zoomed in:
Sums are wrong when zooming out (note the y-axis changing):
