Adding extra labels to metrics

[Juan Rosero]

Hello,

I'm trying to add location labels to my Apache Exporter metrics and other exporters that don't expose them when Prometheus scrapes them from the server nodes. Here is my Prometheus job and the labels (ad, dom, reg) that I'm looking to add depending on the server location. Please let me know if it's possible to be added on the exporter end or maybe in Prometheus while it ingest those metrics. Many thanks!

- job_name: apache_server
  static_configs:
  - targets: ["server1:9117", "server2:9117", "server3:9117", "server4:9117", "server5:9117", "server6:9117",]
    labels:
      env: "int"
      app: "apache"
      version: "2.4.43"

      ad: ""

      dom: ""

      reg: ""

Here how I'm looking for the metrics to look like:

{env="int", ad="ad-1", dom="domain-1", reg="emea-1", instance="server1:9117", app="apache", version="2.4.43"}
(env="int", ad="ad-2", dom="domain-2", reg="emea-1", instance="server2:9117", app="apache", version="2.4.43"}
(env="int", ad="ad-3", dom="domain-1", reg="emea-1", instance="server3:9117", app="apache", version="2.4.43"}
(env="int", ad="ad-3", dom="domain-1", reg="emea-1", instance="server4:9117", app="apache", version="2.4.43"}
(env="int", ad="ad-3", dom="domain-2", reg="emea-1", instance="server5:9117", app="apache", version="2.4.43"}
(env="int", ad="ad-3", dom="domain-1", reg="emea-1", instance="server6:9117", app="apache", version="2.4.43"}

[Julien Pivotto]

Hi,

On 21 May 14:00, Juan Rosero wrote:
> Hello,
>
> I'm trying to add location labels to my Apache Exporter metrics and other
> exporters that don't expose them when Prometheus scrapes them from the
> server nodes. Here is my Prometheus job and the labels (ad, dom, reg) that
> I'm looking to add depending on the server location. Please let me know if
> it's possible to be added on the exporter end or maybe in Prometheus while
> it ingest those metrics. Many thanks!
>
>

version is a metric of the exporter: apache_version
app should be the job name, then the job label

> - job_name: apache_server

> - targets: ["server1:9117", "server2:9117", "server3:9117",
> "server4:9117", "server5:9117", "server6:9117",]
> labels:
> env: "int"
> app: "apache"
> version: "2.4.43"
> ad: ""
> dom: ""
> reg: ""

then, static_configs is a list:

static_configs:
- targets: ["server1:9117", "server2:9117"]
labels:
env: "int"
dom: "aaaaaaa"
- targets: ["server2:9117", "server3:9117"]
labels:
env: "int"
dom: "bbbbbb"

>
> Here how I'm looking for the metrics to look like:
> {env="int", ad="ad-1", dom="domain-1", reg="emea-1",
> instance="server1:9117", app="apache", version="2.4.43"}
> (env="int", ad="ad-2", dom="domain-2", reg="emea-1",
> instance="server2:9117", app="apache", version="2.4.43"}
> (env="int", ad="ad-3", dom="domain-1", reg="emea-1",
> instance="server3:9117", app="apache", version="2.4.43"}
> (env="int", ad="ad-3", dom="domain-1", reg="emea-1",
> instance="server4:9117", app="apache", version="2.4.43"}
> (env="int", ad="ad-3", dom="domain-2", reg="emea-1",
> instance="server5:9117", app="apache", version="2.4.43"}
> (env="int", ad="ad-3", dom="domain-1", reg="emea-1",
> instance="server6:9117", app="apache", version="2.4.43"}
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c2a3ab04-405c-4e79-a332-0ac2f02aa1d9%40googlegroups.com.

regards,

--
Julien Pivotto
@roidelapluie

[Juan Rosero]

Thanks, Julien. I'm wondering if there is way to do this with relabeling. Thank you.

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Brian Candler]

It depends where the 'ad', 'dom', 'reg' information comes from.  You can embed that information in the target, e.g.

- targets:

    - server1:9117/ad-1/domain-1/emea-1

    - server2:9117/ad-2/domain-2/emea-1

and then use rewriting rules to extract each of the parts in turn into separate labels.  But to my eye that's less clear than doing the simple thing already suggested.

There's a more compact way of writing YAML if you prefer:

- targets: [server1:9117]
  labels: {ad: ad-1, dom: domain-1, reg: emea-1}

- targets: [server2:9117]

  labels: {ad: ad-2, dom: domain-2, reg: emea-1}

[Juan Rosero]

Thanks, Brian. I'm spinning my wheels here and getting all sorts of errors in Prometheus with different relabeling configurations. I've been reading on the topic but relabeling is still a bit obscure to me.

  - targets:

      - server1:9117;ad-1;domain-1;emea-1

      - server2:9117;ad-2;domain-2;emea-1

    relabel_configs:

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: 'instance'

      replacement: '$1'

    metric_relabel_configs:

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: 'ad'

      replacement: '$2'

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: 'dom'

      replacement: '$3'

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: 'reg'

      replacement: '$4'

[Brian Candler]

That's almost right.  Firstly, all these settings go under "relabel_configs", none under "metric_relabel_configs"

Secondly, you need a final rule to set the __address__ to the actual target you want to scrape:

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: '__address__'

      replacement: '$1'

A useful pattern is that you can take the port out of the 'targets' file, and apply it here - this gives you meaningful instance labels.

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: '__address__'

      replacement: '$1:9117'

[Juan Rosero]

Awesome! That worked! Thanks for everything. Leaving the solution here.

  - targets:

      - server1;ad-1;domain-1;emea-1

      - server2;ad-2;domain-2;emea-1

    relabel_configs:

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: 'ad'

      replacement: '$2'

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: 'dom'

      replacement: '$3'

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: 'reg'

      replacement: '$4'

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: '__address__'

      replacement: '$1:9117'

[Brian Candler]

Glad it's working!

Note that if you add an extra step to set the "instance" label explicitly (which you had originally), then this disables prometheus' default behaviour of copying __address__ to instance.  Hence you can keep the port number out of the instance label, whilst keeping it in the __address__ that is scraped.

[Juan Rosero]

I see what you mean so I added it back into the mix. Thanks for the tip. Really appreciated.

  - targets:

      - server1;ad-1;domain-1;emea-1

      - server2;ad-2;domain-2;emea-1

    relabel_configs:

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: 'instance'

      replacement: '$1'

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: 'ad'

      replacement: '$2'

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: 'dom'

      replacement: '$3'

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: 'reg'

      replacement: '$4'

    - source_labels: ['__address__']

      regex: '(.*);(.*);(.*);(.*)'

      target_label: '__address__'

      replacement: '$1:9117'