combining the gpu queries in prometheus

[kumar yele]

hi,

i have a gke setup with monitoring setup. so i have the requirement of using gpu nodes so i have configured the prometheus to get the gpu metrics. but when i plot the query using the gpu metrics i got empty query result. when i do the queries individually they are showing the data. 

here are the result 

i am using this gpu query ----->DCGM_FI_DEV_GPU_UTIL

when i combine with my deployment  ----->  sum by (namespace, pod, label_seldon_deployment_id) (DCGM_FI_DEV_GPU_T) * on(namespace, pod) group_left(label_seldon_deployment_id) sum by (namespace, pod, label_seldon_deployment_id) (DCGM_FI_DEV_GPU_UTIL) * on(namespace, pod) group_left(label_seldon_deployment_id) kube_pod_labels{label_seldon_deployment_id="llm-testing-1"}

its not giving the result. showing empty result.

[Алексей Марковский]

It is not clear what are you trying to accomplish with on(namespace, pod) group_left(label_seldon_deployment_id) clauses. It seems like all three of your metrics are expected to have all three of these labels.

Nevertheless, if all three sums produce output, and all three outputs contain same labels values you should get some result.

четверг, 24 августа 2023 г. в 20:17:59 UTC+3, kumar yele: