Add new labels to metrics

[Kishore Chopne]

Hi,

    I want to add new labels to metrics via recording rules.  Is it possible to run and get resul

of a script which has a curl command to fetch some meta data.  Input will be an existing label value and in return enrich the metric the existing metric from result of the script.

/Kishore

[Brian Candler]

No.

The only way that I know to do this is to ingest the enrichment data as *additional timeseries*, which you combine in your recording rule PromQL expression.

To do this, those additional timeseries could be stored in a flat text file served by a static webserver (e.g. Apache) that you scrape using a prometheus scrape job; or they could be picked up by the node_exporter textfile collector. Or you could write your own webserver and return these enrichment timeseries dynamically, in response to every HTTP request.

This creates additional timeseries in the database - but timeseries are very cheap in Prometheus, especially ones whose metric value doesn't change, so you shouldn't worry about that.

Then you join these additional labels using a many-to-one join as I showed you before.  You need some set of labels which is common to both the original metric and the enrichment data, to match them up.

There's a more concrete example here:

https://www.robustperception.io/using-time-series-as-alert-thresholds

This is for using extra timeseries as alerting thresholds, but you could just as well use these external timeseries as sources of extra label data. This would generally be "many to one", with the "many" side being the metrics you want to enrich, and the "one" being a metric with additional labels, as in this post.

If you don't like this approach, then I guess you could do it at scrape time instead.  You'd have to write a proxy exporter which accepts a scrape request (and the target address as a URL parameter), scrapes the target, and modifies the response before returning it.