Issue with the deployment specific usage metrics in Prometheus

[Moksh]

Hi Team,

I've a problem while fetching the metrics for the dedicated deployment. I don't see any deployment related metrics in Prometheus to get the deployment specific resource usage with kube-state-metrics. 

For Example: When I use the regex for a pod_name on container specific metric its populating all #thanos related metrics including sub components like; thanos-receiver, thanos-frontend, etc. My current PromQL is as follows. It should show thanos deployment metrics only, please look into this issue and help me out!

sum(container_memory_working_set_bytes{container_name!="", container_name!="POD", pod_name=~"thanos-(([a-z0-9]*)|([a-z0-9]*-[a-z0-9]*)|[0-9])", namespace="$namespace", k8s_cluster="$cluster_name"}) by (id,pod_name)

Current output:

{pod_name="thanos-6c4d9c7bf7-ggnem"} {pod_name="thanos-6c4d9c7bf7-h6tld"} {pod_name="thanos-receiver-0"} {pod_name="thanos-receiver-1"}

Expected output:
{pod_name="thanos-6c4d9c7bf7-ggnem"} {pod_name="thanos-6c4d9c7bf7-h6tld"}

Looking forward to hearing from you, thanks!

Best Regards,

Moksha

[Brian Candler]

Your query is asking for metrics at the *pod* level - container_memory_working_set_bytes - and that's what you're getting.

If you want to see which pods were started by specific deployments, you'll need to find a metric which gives this relationship.  What does kube_pod_info show? (In particular, the labels created_by_kind, created_by_name)

Then you should be able to use a one-to-many relationship in your query to filter the pods.

https://prometheus.io/docs/prometheus/latest/querying/operators/#many-to-one-and-one-to-many-vector-matches

https://www.robustperception.io/how-to-have-labels-for-machine-roles

https://www.robustperception.io/exposing-the-software-version-to-prometheus

[Ben Kochie]

For this, container_ metrics are what they want. Summed by pod.

The problem is they are using a regexp to match the pod name, rather than using a label like "job", "deployment" or some other higher-level matching. The regexp is too fragile.

Without knowing what other labels they have available, it's hard to say. They may also need to improve their relabel configs to better map Kubernetes metadata onto the container metrics.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bc09a69a-3b97-4b20-8b34-21cb0992ed87n%40googlegroups.com.