enter image description hereI have created a dashboard for the DCGM metrics to see the notebooks gpu usage and all. For that I have the following query
sum(DCGM_FI_DEV_FB_TOTAL) by (exported_pod)
* on (exported_pod) group_left()
label_replace(
kube_pod_labels{label_notebook_name="gen-ai-solutions"},
"exported_pod", "$1",
"pod", "(.*)")
so this will give me the total frame buffer memory available for the specific notebook by using the label (label_notebook_name="gen-ai-solutions"). So I have so many number of notebooks with different label keys and values.
So, what I have been trying is creating the dashboard variables. For that I have created two drop downs for the dashboards with names label_key and label_value.
Variable Definition
label_key label_values(kube_pod_labels , __name__)
label_value label_values(kube_pod_labels{label_key="$label_key"}, label_value)
so I added like this, but I am not able to see the related label keys and values in the dropdown. It's showing none.
I am using Grafana 7.5.8 app version and data source is Prometheus.