PromQL join two metrics with identic label names but different label values

Question

I have following PromQL metric structures

metric1{label_1="a",label_2="b",...,status="running"} value = 0
metric1{label_1="a",label_2="b",...,status="running"} value = 1
metric2{label_1="a",label_2="b",...,status="healthy"} value = 0
metric2{label_1="a",label_2="b",...,status="healthy"} value = 1

I want to select only metrics with specific value and merge these metrics into this structure

resultmetric{label_1="a",label_2="b",...,metric1_status="running", metric2_status="healthy"}

What is the correct PromQL query? I tried queries with group_left() and on() but no luck.

Answer 1

The main roadblock you are probably hitting is the fact that the in both metrics you have the status label which is different AND the one you want to replicate. Prometheus won't be able resolve it while the label names collide.

The first thing you want to do is generate a new timeseries with a different label name for the second status. You can do this by using label_replace. For example:

label_replace(metric2{...}, "metric2_status", "$1", "status", "(.*)")

This will keep the status label but will also add a metric2_status with the value of the status label.

Now you can proceed to use on() and group_left(). These need to be used with an arithmetic or group function. If you are using this to add more labels to metric1, remember to use an arithmetic that will not alter the metric value.

For example - assuming metric2 has always a value of 1 - this is what you could use:

metric1{label_1="a", ...} * 
    on (label_a, label_b) 
    group_left(metric2_status)
label_replace(metric2{...}, "metric2_status", "$1", "status", "(.*)")

the resulting metric should have metric1's labels plus the metric2_status label.

Answer 2

When using group modifiers such as

ignoring()
on()
group_left()
group_right()

the label(s) used in on() modifier must be unique at least in one vector.

Consider metric1, metric2 vectors and common label_a=... label used in on(). These cases may occur

If metric1 has unique label_a=... and metric2 has unique label_a=... we have one-to-one match.

If metric1 has duplicate label_a=... and metric2 has unique label_a=... we have many-to-one match.

If metric1 has unique label_a=... and metric2 has duplicate label_a=... we have one-to-many match.

If metric1 has duplicate label_a=... and metric2 has duplicate label_a=... we have many-to-many match and vectors are not joinable.

Additionally the result vector must have unique set of label keys. That's why we are renaming the label in the right-hand side vector and joining it as a label with different name.

Working query is

( metric1{...} == 1 ) *
    on(label_a)
    group_left(metric2_status)
label_replace((metric2{...} == 1), "metric2_status", "$1", "status", "(.*)")

( metric1{...} == 1 )

Filter metric1 so that the left-hand side of the query contains only data with unique label_a=....

on(label_a)

join two metrics on label_a=....

group_left(metric2_status)

add metric2_status label from right-hand metric2.

label_replace((metric2{...} == 1), "metric2_status", "$1", "status", "(.*)")

create new label metric2_status in metrics2 from the status label.