Prometheus appending metric_name to timeseries

[ryan woods]

Hey guys,

I seem to be running into a strange issue where Prometheus is adding a 'metric_name' label to a timeseries but instead of updating the existing one it creates a new timeseries (See attached screenshot). This is causing issues in Grafana as single stat panels can only understand a single timeseries and not multiple.

What would be causing prometheus to do this?

Thanks,

Ryan

[Ben Kochie]

I don't think this is anything that Prometheus does automatically. My guess is that it is a relabeling issue, can you share the relevant portion of your prometheus.yml?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/26212b98-2166-4fb7-bb08-a5d47e80a209o%40googlegroups.com.

[Brian Candler]

On Monday, 29 June 2020 12:40:43 UTC+1, ryan woods wrote:

I seem to be running into a strange issue where Prometheus is adding a 'metric_name' label to a timeseries but instead of updating the existing one it creates a new timeseries

That's standard behaviour.  A timeseries is uniquely identified by its set of labels: e.g. these are two different timeseries

foo{bar="xyz"}

foo{bar="xyz",baz="123"}

Therefore, if you change the labels, then you have made a new timeseries, by definition.  Over time, the old one will age out.