Modify labels of existing metrics

[makl...@gmail.com]

I was asked if there is a way to modify labels without loosing old data and I could not yet find an answer to it.

Let's take this usecase:

I have collected a few months worth of data.

To fix a problem, I analyse it using the collected metrics.

As it turns out, I need to query the data grouped by a label my exporters did not export yet, let's say the network card model the server uses.

It is easy for me to determine which metric belongs to which server and which network card model the server uses.

Of couse I need to export the new labelset from now on.

Prometheus uses a hash over the labelset and uses that as an identifier to store the data.

Another labelset means another hash, so I would end up generating new metrics instead of appending to the existing data.

A query grouped over the new label would only return data collected after I modified the export, so my old data appears lost and cannot be used for analysis.

From my understanding prometheus would need to rename the series file to match the new fingerprint, move the memorySeries inside of seriesMap, remove the old labelset/fingerprint from the indices and add the new labelset. Would this even be enough?

Is there a label modification feature I have not yet discovered?

[Brian Brazil]

There's no way to do this.

Generally I'd advise not worrying about things like this. Metric breakage due to architectural and code changes is a regular occurrence, and you'll usually spend a lot of time trying to get everything perfect for rare cases where you're looking back in time. After a week, it likely won't be relevant for most metrics.

--

Brian Brazil

www.robustperception.io