I was wondering if I can set up custom(variable) labels to an alert in Grafana. Basically, We have one alert, that covers multiple environments(at least that's my understanding for the alert), which the metric is collected from Prometheus in the query below:
sum(increase(statement_failed[5m]) or vector(0) ) by (taskqueue,cluster)
+
sum(statement_failed unless statement_failed offset 5m or vector(0)) by(taskqueue,cluster)
Whenever the statement execution fails, I'd like, for example, to add in the labels, the value that comes from cluster.
That is, in the Notification section of the alert, where we can add labels I would like to have something like:
Environment -> cluster
Is that achievable?