relabelling metrics in prometheus

[sunil sagar]

Hi All, 

I need additional metrics in prometheus based on the instance name . My current instance name is as below : 

servername1.doman.com:port1

servername1.doman.com:port2

servername1.doman.com:port3

I want to relabel instance name so that I can draw dashboard by instance (servername1 - combined)

I tried below relabelling in yml file but it didn't worked out . Please help . Thanks 

  metric_relabel_configs:
  - source_labels: [instance]
    regex: '/system.slice/var-lib-docker-containers.*-shm.mount'
    action: replace    replace: $1    target_labels: host

[Christian Hoffmann]

Hi,

On 12/7/19 10:27 PM, sunil sagar wrote:
> I need additional metrics in prometheus based on the instance name . My
> current instance name is as below : 
>
> servername1.doman.com:port1
> servername1.doman.com:port2
> servername1.doman.com:port3
>
> I want to relabel instance name so that I can draw dashboard by instance
> (servername1 - combined)

What exactly do you want to do? Do you want to strip the port, i.e. turn
instance="servername1.doman.com:port1" into
instance="servername1.doman.com"?

(I assume you checked that your metrics will remain unique despite this
change)

You can accomplish that by overriding the instance label in your service
discovery or by using relabeling.

> I tried below relabelling in yml file but it didn't worked out . Please
> help . Thanks 
>
>
> metric_relabel_configs:
> - source_labels: [instance]
> regex: '/system.slice/var-lib-docker-containers.*-shm.mount'
> action: replacereplace: $1target_labels: host

I don't know what your metrics look like, but your regex sounds like
they could come from cadvisor and you are trying to extract container
names from cgroup names.

Your example looks almost good. However, in order to be able to
reference the $1 variable, you have to tell Prometheus what this should
refer to. You use this by placing brackets around the part you are
interested in. E.g.:

regex: '/system.slice/var-lib-docker-containers(.*)-shm.mount'

However, to provide semantically more thorough advice, we would need
some more context.

Kind regards,
Christian

[Brian Candler]

Aside: it's "replacement: $1" not "replace: $1" (although you can omit it entirely: "action: replace" and "replacement: $1" are defaults)

> servername1.doman.com:port1
> servername1.doman.com:port2
> servername1.doman.com:port3
>
> I want to relabel instance name so that I can draw dashboard by instance
> (servername1 - combined)

Rather than relabelling the metrics after you scrape them, you can relabel *before* scraping - leave the port out of the "instance" label, and using "job" label to distinguish the different metrics.  It's the __address__ which is used to scrape, but this doesn't have to be the same as the instance label. More here.

Here's an example:

  - job_name: node

    file_sd_configs:

      - files:

        - /etc/prometheus/targets.d/node_targets.yml

    relabel_configs:

      - source_labels: [__address__]

        regex: '([^/]+)'  # name or address only

        target_label: instance

      - source_labels: [__address__]

        regex: '(.+)/(.+)'  # name/address

        target_label: instance

        replacement: '${1}'

      - source_labels: [__address__]

        regex: '(.+)/(.+)'  # name/address

        target_label: __address__

        replacement: '${2}'

      - source_labels: [__address__]

        regex: '([^:]+)'

        target_label: __address__

        replacement: '${1}:9100'

This allows different formats in the targets file:

"foo" => instance="foo", __address__="foo:9100"

"1.2.3.4" => instance="1.2.3.4", __address__="1.2.3.4:9100"

"foo/1.2.3.4" => instance="foo", __address__="1.2.3.4:9100"
"foo/foo.bar.com" => instance="foo", __address__="foo.bar.com:9100"
"foo/1.2.3.4:5555" => instance="foo", __address__="1.2.3.4:5555"

Note how the instance label doesn't have to be the same as the address or domain name which is scraped.

Then you can make similar scrape configs for other jobs; they'll get the same instance label but scrape a different port.

[Sagar]

Hi Brian, 

Thank you so much for sharing examples , I will try them . 

from the below scraps , I just wanted hostname=servername1 . It looks like from your example , I will be able to the scraps . Thanks 

> servername1.doman.com:port1
> servername1.doman.com:port2
> servername1.doman.com:port3
>
> I want to relabel instance name so that I can draw dashboard by instance
> (servername1 - combined)

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a052cdda-c983-465a-bc36-ee39ca5499f8%40googlegroups.com.

[Sagar]

Hi Brian , 

Thank you so much for help on this . Below regex solved my issue . 

      - source_labels: [__address__]

        regex: '(.+)/(.+)'  # name/address

        target_label: __address__

        replacement: '${2}'

Thanks

Sunil Sagar