Blackbox relabel_configs

[Kolja Krückmann]

Hi there, hope y'all doing fine.

I'm fairly new to prom and having a bit of trouble with all of the configs.

Currently I want to use the blackbox_exporter to ping the endpoints. I just want to know if they are up and healthy or down/off.

But I just can't figure out how to relabel correctly in order to get the correct url.

the correct one should look like this: http://1st_client_ip:9115/probe?module=icmp&target=1st_client_ip

my prometheus.yml:

# my global config
global:
  scrape_interval: 30s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  scrape_timeout: 15s
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
           - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
    - rules.yml

[...]

static_configs:

  - job_name: 'blackbox-icmp'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
        - 1st_client_ip:9115
        -  2nd_client_ip:9182

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance

With this relabel_config I at least have the target and module in the probe-url. But attached at the end is some kind of modulo and I don't really know where it's coming from: http://1st_client_ip:9115/probe?module=icmp&target=1st_client_ip%3A9115 and with that all values in the icmp probe is 0. when deleting the modulo manually from url - everything works just fine.

Can someone help me to get it right?

Kind regards - Kolja

[Brian Candler]

I already replied in your other thread, but you seem to have changed your requirements a bit here:

    static_configs:
      - targets:
        - 1st_client_ip:9115
        -  2nd_client_ip:9182

Note that port 9182 is windows_exporter, not blackbox_exporter, so you can't scrape it as if it were blackbox_exporter (in particular, the path "/probe" won't work)

You should have a separate scrape job for all your windows_exporter hosts, scraping port 9182 with path "/metrics". Once you've done this, you just use the "up" metric to determine whether the scrape was successful or not.  There is no need for blackbox_exporter at all, to determine whether the host is up or down.

(Well, not unless you want to distinguish the case of "host is down" from "windows_exporter has crashed", but in my experience, these exporters are very reliable)

I still recommend you do some relabelling:

static_configs:

  - job_name: 'windows'
    static_configs:
      - targets:
        -  2nd_client_ip

        # ... etc

    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - source_labels: [__address__]
        replacement: '${1}:9182'
        target_label: __address__

The simpler version would be just

static_configs:

  - job_name: 'windows'
    static_configs:
      - targets:
        -  2nd_client_ip:9182

        # ... etc

...but then you get the port number in your instance labels (which can make things awkward if you are joining metrics from different exporters on the same host).

Also: once you get beyond two or three targets, you'll be much better off with file_sd_configs than static_configs - i.e. you put the list of targets in a separate file.  You can then modify this file without reloading prometheus. It will notice changes and pick them up automatically.