Relabeling assistance

[Jared]

I’m not 100% sure what best practice is for this, but I’m trying to keep metrics merged through pod updates. Because kubernetes adds the hash after the pod name, whenever I update an app the grafana dashboards become a mess because it’s pulling historic data. I want it to just keeps this as a consistent stream. This is something I would like to add globally and not 100% sure the correct path to take. I’m using the latest kube-prometheus-stack chart. 

I’m thinking I need to add a relabeling for kubernetes_sd_configs and have pod name relabeled to the service name, but looking for advice here. 

--

                                                                                                                                                                                    

Jared Wechsler

[Ian Billett]

Hey Jared,

I'm afraid I don't fully understand your question. Please could you provide an example of what you have now, and what your desired end state is? That way we can help provide the best path between those two points.

Ian

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAAca8K%2B0y%2B8rXzsMrh%3Ds8Qq_c48A7sjAsrUNTOZPErLr5Rn8NQ%40mail.gmail.com.

[Jared Wechsler]

On kubernetes every time a pod gets upgraded the hash changes for example nut-exporter-7cd9bc9669-2hzkf. So what happens is that as this hash changes through updates or just testing deployments is as grafana pulls up historical data it becomes messy. I’m just not sure the best approach to take to handle this properly. I have attached a screenshot displaying the issue I’m trying to address. 

--Jared

On Jul 26, 2021, at 2:34 PM, Ian Billett <ibil...@redhat.com> wrote: