Querying Kubernetes metrics by pod label

[tomer.l...@gmail.com]

Hi,

I'm trying to query my pods metrics by their label instead of the pod name.
I've tried to use the following query:

sum

    (max(kube_pod_labels{label_app="query"}) by (label_app, pod)

    *

    on(pod)

    group_right(label_app)

    label_replace(

          sum by(pod) (

                   rate(container_cpu_usage_seconds_total{namespace="default"}[1m])

          ), "pod", "$1", "pod", "(.+)"

   )

) by (label_app)

The result here was: 0.005788935803884404

While runng the following query by pod name:

sum by (pod) (rate(container_cpu_usage_seconds_total{job="kubelet", pod="query-75576b478c-ckcqk"}[1m]))

Resulted with: 0.006083688360254887

Any idea why there's such a difference in the values between the two queries and how to avoid that?

Thanks in advance