How to add a new label in all metrics?

[Anoop]

Hi,

Is there anyway to add a new label to all metrics? Please share the sample configuration details, if there is any.

Thanks

[Anoop Mohan]

Hi,

I know that it is not easy to add the new label to old data. Please let me know, if there is any way to add a new label to new data?

Thank you

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d0aba3e9-fc4e-4ced-944c-dcfeacd6c315n%40googlegroups.com.

[Brian Candler]

If you want to add a label unconditionally to every metric returned from a specific scrape job, then just add the label in the service discovery (e.g. "file_sd_configs" or "static_configs").  Different labels can be applied for different groups of targets, e.g.

- targets:

    - 1.2.3.4

    - 5.6.7.8

  labels:

    foo: bar

- targets:

    - 9.10.11.12

  labels:
    baz: qux

    aaa: bbb

If you want to add labels more dynamically, but using only information known before the scrape (e.g. target address), then use relabelling.

If you want to add a label only for certain metrics returned by a scrape, or to drop certain metrics, then use metric relabelling.

[Anoop Mohan]

Thanks Brian for your suggestion.

I am using "kubernetes_sd_configs" to discover the targets dynamically. So I just tried the below configuration, since I don't want to specify any specific targets under "static_configs". But, this is not working as expected as I can't see the custom label in the query results.

static_configs:
  - targets:
    - .*
    labels:
      foo: bar

Not sure if this implementation is correct or not. Please share your suggestions.

Thanks

[Brian Candler]

If you're using kubernetes_sd_configs then I don't think you can do it this way.

Instead, you'll need to use target relabelling:

  - job_name: foo
    kubernetes_sd_configs:

      ...

    relabel_configs:

      - target_label: foo
        replacement: bar

This should add {foo="bar"} to every target, and hence to every scraped metric.

Note that the SD mechanism sets lots of __meta_XXX labels on each target, so there's lots of opportunity here to use these values and copy them into "real" labels as required.  (Labels which start with double-underscore are removed after relabelling).

[Anoop Mohan]

Thank you so much Brian. The "relabel_configs" worked for me.

I tried "metric_relabel_configs" also with the below configuration and this is also adding the new label with all metrics. Not sure if this is the correct method though :)

metric_relabel_configs:
  - source_labels: [__name__]
    target_label: foo
    replacement: bar

I am going to use "relabel_configs" anyway. However, I'm just curious whether we can use the "metric_relabel_configs" method also? Will it make any issue?

Thanks,

Thanks & Regards,

Anoop Mohan

Mob# +91-7293009486

E-mail : anoopmo...@gmail.com

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e6cc71e1-dbd0-477a-8d57-392a27d38bb8n%40googlegroups.com.

[Brian Candler]

Target relabelling occurs *before* the scrape takes place; metrics relabelling occurs *after* the scrape has taken place.

If you're just adding fixed labels to every metric, then either way will work.  However I'd expect target relabelling to be more efficient, since it's only run once for each target, whereas metric relabelling is run once for every individual timeseries returned in the scrape.

Otherwise:

* If you want to set any labels which affect how the scrape is done - for example changing the metrics_path or adding query parameters when communicating with the exporter, or changing the address or port scraped - then this has to be done in target relabelling.

* If you want to label some returned timeseries differently to others, or selectively drop timeseries, then this has to be done in metric relabelling.

[Anoop Mohan]

Thanks Brian for a good explanation. Will go with target relabelling.

Thanks,

Anoop

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/09f08de8-ff05-4d78-bf91-ef131d3e8854n%40googlegroups.com.

[Anoop Mohan]

Hi,

Is it possible to have a common relabel_config configuration (may be under global configuration) for all jobs instead of duplicating the same relabel_config under each job?

Thanks,

Anoop

[Brian Candler]

Depending on your use case, there is "external_labels" in global config options:

global: # The labels to add to any time series or alerts when communicating with # external systems (federation, remote storage, Alertmanager). external_labels: [ <labelname>: <labelvalue> ... ]

But if you want the labels to be added to the data in the TSDB, and hence returned by a PromQL query, then AFAIK you need to duplicate those relabelling rules.  The whole config is just YAML, so you can pre-process it with a macro language of your choice.

[Anoop Mohan]

Thanks for your reply Brian.

In my case, I want to query the data directly from prometheus using this new label. Then, I will need to duplicate the relabelling rule under all job configurations.

Thanks,

Anoop

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2fa661b9-10de-4db9-a4da-e12c3da6ae14n%40googlegroups.com.