file_cd_config - label for each target

[Kolja Krückmann]

Hi y'all I'm trying to get a label for each target in my file_sd_config

my prom.yml is:

(redacted)

# my global config
global:
  scrape_interval: 1m # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 30s # Evaluate rules every 15 seconds. The default is every 1 minute.
  scrape_timeout: 30s

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'node'
    file_sd_configs:
      - files:
        - C:/Prometheus/prometheus-2.41.0.windows-amd64/target_cluster_b.yml

my target_cluster_b.yml:

- targets:
    - 172.25.X0.XXX:9182
    labels:
        instance: "SVR-DS01"
    - 172.21.X1.XXX:9182  
    - 172.25.X2.XXX:9182
    - 172.25.X3.XXX:9182

as seen in the target.yml I want to try to get a label to each target - to see in the prom dashboard which ip is which targetname

Thanks in advance.

Kind regards Kolja

[Julius Volz]

The labels in the SD file can only be provided for each target group (the top-level list item type of the YAML file), so you'd have to do something like this:

- targets:

  - [...first list of targets...]

  labels:

    instance: "A"

- targets:

  - [...second list of targets...]

  labels:

    instance: "B"

- targets:

  - [...third list of targets...]

  labels:

    instance: "C"

  

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fa4e5c9e-807b-4680-9640-592c8172e568n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Brian Candler]

Alternatively, you use rewriting rules.

- targets:

  - SVR-DS01 172.25.X0.XXX
  - SRV-DS02 172.21.X1.XXX

... etc

and the corresponding relabel_configs in your scrape job could be something like this:

    relabel_configs:
      # When __address__ consists of just a name or IP address,
      # copy it to the "instance" label.  This keeps the port
      # number out of the instance label.
      - source_labels: [__address__]
        regex: '([^ ]+)'
        target_label: instance

      # When __address__ is of the form "name address", extract
      # name to "instance" label and address to "__address__"
      - source_labels: [__address__]
        regex: '(.+) (.+)'
        target_label: instance
        replacement: '${1}'
      - source_labels: [__address__]
        regex: '(.+) (.+)'
        target_label: __address__
        replacement: '${2}'

      # Append port number to __address__ so that scrape gets
      # sent to the right port
      - source_labels: [__address__]
        target_label: __address__
        replacement: '${1}:9182'

[Kolja Krückmann]

Hi Brian,

thank you very much for your response!

I just wanted to clarify, if it's better to "regex" the port to the end of the address, or if it is equally fine to just add the port in the target.yml to each target? (That's what im currently doing) Is it just best practice with the regex expression or is there actually a technical reason behind that?

Kind regards

Kolja

[Brian Candler]

Well, there is a good reason to keep the port out of the "instance" label: it's very awkward to make a query to join two different different metrics collected from two different exporters on the same host, if one has (say) instance="foo:9100" and another has instance="foo:9104".  A second reason is that it's nicer in dashboards to see "foo" rather than "foo:9100"

See https://www.robustperception.io/controlling-the-instance-label

IMO the cleanest way to achieve this is to leave the port out of the targets file, copy __address__ to instance, and then append the exporter port statically to __address__.

However, you could leave the port number in the targets file, and strip it out to create the instance label:

      - source_labels: [__address__]
        regex: '(.*):[0-9]+'

        target_label: instance
        replacement: '${1}'

This would be useful if the same exporter were running on different ports on different hosts (fairly uncommon I think).

Also, you risk a collision of metrics if targets list includes the same host twice on two different ports in the same scrape job (which should be very unlikely).

[Brian Candler]

So to be clear, if you want your targets file to look like this:

- targets:

  - SVR-DS01 172.25.X0.XXX:9182
  - SRV-DS02 172.21.X1.XXX:9182

you can do something like this:

   relabel_configs:

      # When __address__ is a single item, set the instance

      # label to the part without the port

      - source_labels: [__address__]
        regex: '([^ ]+):[0-9]+'

        replacement: '${1}'
        target_label: instance

      # When __address__ is space-separated "name address:port",

      # put the first part in the "instance" label and leave the second part

      # in "__address__"

      - source_labels: [__address__]
        regex: '(.+) (.+)'
        target_label: instance
        replacement: '${1}'
      - source_labels: [__address__]
        regex: '(.+) (.+)'
        target_label: __address__
        replacement: '${2}'

If your target entries *always* consist of two items separated by a space, then you don't need the first rule. It's only there in case you have entries in the old format, i.e.

- targets:

  - 172.25.X0.XXX:9182

(in which case, the instance is set to "172.25.X0.XXX")

[Kolja Krückmann]

Thanks so much!

Can you tell me where I could kinda "learn" regex? I find it very difficult to get known to regex because it's nothing im using on daily basis ^^

[Kolja Krückmann]

nevermind - sorry for the question, should have just googled learn regular expression.

I'm fine for now.

[Kolja Krückmann]

Hi y'all, now i got my metrics running (Thanks Brian)

For the next step I want to use the Blackbox_exporter with the same targets (where the target.yml is build like "- targetname targetIP")

I want to have the same labeling as in the question above where each target has a name lable but I'm not quite getting there. Either the Endpoint URL is missing the target IP or the label is missing the name of the endpoint.

[Brian Candler]

Do you want to show what you've done so far?

You can read these:

https://nsrc.org/workshops/2022/rwnog/nmm/netmgmt/en/prometheus/ex-blackbox-exporter.html   # simple config

https://nsrc.org/workshops/2022/rwnog/nmm/netmgmt/en/prometheus/ex-relabeling.html   # node_exporter with relabelling

If you combine the two, you could end up with something like this:

  - job_name: blackbox
    file_sd_configs:
      - files:
          - /etc/prometheus/blackbox.d/*.yml
    metrics_path: /probe
    relabel_configs:
      - source_labels: [__address__]
        regex: '([^ ]+)'    # name or address only
        target_label: instance
      - source_labels: [__address__]
        regex: '([^ ]+)'    # name or address only
        target_label: __param_target
      - source_labels: [__address__]
        regex: '(.+) (.+)'  # name address

        target_label: instance
        replacement: '${1}'
      - source_labels: [__address__]

        regex: '(.+) (.+)'  # name address
        target_label: __param_target
        replacement: '${2}'
      - source_labels: [module]
        target_label: __param_module
      - target_label: __address__
        replacement: 127.0.0.1:9115  # Blackbox exporter

Where /etc/prometheus/blackbox.d/ping.yml contains, for example:

- labels:
    module: icmp
  targets:
    - google-pri 8.8.8.8
    - google-sec 8.8.4.4

Then you should get metrics like this:

probe_success{job="blackbox",instance="google-pri",module="icmp"} 1

[Kolja Krückmann]

Hi Brian, thanks for your quick response.

Actually I just pasted the relabeling you attached and it immediately worked :) 

[Kolja Krückmann]

For context and maybe upcomming errors here is my prom.yml so far:

# my global config
global:
  scrape_interval: 1m # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 30s # Evaluate rules every 15 seconds. The default is every 1 minute.
  scrape_timeout: 30s

  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
           - localhost:9093


# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
    - rules.yml

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

  - job_name: 'Windows-Exporter Host-Server'
    file_sd_configs:
      - files:
        - C:\Prometheus\targets\hosts.yml
    relabel_configs:

      # When __address__ is of the form "name address", extract
      # name to "instance" label and address to "__address__"
      - source_labels: [__address__]
        regex: '(.+) (.+)'
        target_label: instance
        replacement: '${1}'
      - source_labels: [__address__]
        regex: '(.+) (.+)'
        target_label: __address__
        replacement: '${2}'
       

[...]
       
  - job_name: 'Ping (ICMP)'
    metrics_path: /probe
    params:
      module: [icmp_ttl5]
    file_sd_configs:
      - files:
        - C:\Prometheus\targets\hosts.yml
        - C:\Prometheus\targets\netcomponents.yml
        - C:\Prometheus\targets\vmhost-05.yml
        - C:\Prometheus\targets\vmhost-06.yml
        - C:\Prometheus\targets\vmhost-08.yml
        - C:\Prometheus\targets\vmhost-09.yml
        - C:\Prometheus\targets\vmhost-10.yml
        - C:\Prometheus\targets\vmopheo-01.yml
        - C:\Prometheus\targets\vmopheo-02.yml
        - C:\Prometheus\targets\vmopheo-04.yml
        - C:\Prometheus\targets\vmopheo-06.yml
        - C:\Prometheus\targets\vmopheo-07.yml
        - C:\Prometheus\targets\vmopheo-08.yml

       
    relabel_configs:
      - source_labels: [__address__]
        regex: '([^ ]+)'    # name or address only
        target_label: instance
      - source_labels: [__address__]
        regex: '([^ ]+)'    # name or address only
        target_label: __param_target
      - source_labels: [__address__]
        regex: '(.+) (.+)'  # name address

        target_label: instance
        replacement: '${1}'
      - source_labels: [__address__]
        regex: '(.+) (.+)'  # name address
        target_label: __param_target
        replacement: '${2}'
      - source_labels: [module]
        target_label: __param_module
      - target_label: __address__
        replacement: 127.0.0.1:9115  # Blackbox exporter

[Brian Candler]

On Friday, 28 April 2023 at 13:13:22 UTC+1 Kolja Krückmann wrote:

  - job_name: 'Ping (ICMP)'
    metrics_path: /probe
    params:
      module: [icmp_ttl5]

Incidentally, you don't need to specify that here, because with the rewriting rules given, it is set from the "module" label in the targets file. (However, it will act as a default in case you don't specify it there)

 

    file_sd_configs:
      - files:
        - C:\Prometheus\targets\hosts.yml
        - C:\Prometheus\targets\netcomponents.yml
        - C:\Prometheus\targets\vmhost-05.yml
        - C:\Prometheus\targets\vmhost-06.yml
        - C:\Prometheus\targets\vmhost-08.yml
        - C:\Prometheus\targets\vmhost-09.yml
        - C:\Prometheus\targets\vmhost-10.yml
        - C:\Prometheus\targets\vmopheo-01.yml
        - C:\Prometheus\targets\vmopheo-02.yml
        - C:\Prometheus\targets\vmopheo-04.yml
        - C:\Prometheus\targets\vmopheo-06.yml
        - C:\Prometheus\targets\vmopheo-07.yml
        - C:\Prometheus\targets\vmopheo-08.yml

That's fine but it means you've hard-coded a list of files in your prometheus.yml.  You should be able to simplify it to this (works under Linux anyway):

    file_sd_configs:
      - files:
        - C:\Prometheus\targets\*.yml

[Kolja Krückmann]

Thanks for the hint with the wildcard!

What I don't yet understand is what you mean by "it is set from the "module" label in the targets file." My target files are built like:

- targets:

      - "name" "ip-adress"

      - "name" "ip-adress"

      - "name" "ip-adress"

where does the label actually come from?

[Brian Candler]

Each group of targets in your SD file can have both 'targets' and 'labels' keys. The 'labels' are applied to all targets in that particular group. For example:

- targets:

    - foo 1.2.2.2

    - bar 2.2.2.2

  labels:

    module: icmp

- targets:

    - baz http://3.3.3.3

    - qux http://4.4.4.4

  labels:

    module: http_2xx

Then, in the rewriting rules I gave you, one of the rules was this:

      - source_labels: [module]
        target_label: __param_module

This copies the "module" label to "__param_module", which is used as the HTTP query parameter in the scrape of blackbox_exporter.

Doing it this way means you only need a single scrape job, even when using different blackbox modules with different targets. Also, you can set other labels to categorise your targets, like "env: prod" say, which can be used in queries, alerting rules etc.