Need example of per-target parameter overrides in blackbox-exporter

[Sebastian N.]

Hey guys,

In the following discussion about the blackbox exporter Julius Volz mentioned the following:

The Blackbox targets to probe (like "https://google.com") can come from file_sd (or any other SD), no problem. You still need a scrape config in the Prometheus config that actually scrapes the Blackbox exporter, but it can get targets to probe from file_sd. You can even vary the Blackbox Exporter probe module to use for each target by setting a "__param_module" hidden label on each target to override any "module" HTTP param that is set in the scrape config itself.

During my research I have not found an example that shows me how I can override module parameters on a per-target basis. My use case is that I'd like to set a different user agent or toggle the follow_redirects flag for some hosts without having to create another scraping job for that.  

[Brian Candler]

This should work:

  - job_name: blackbox

    file_sd_configs:

      - files:

        - /etc/prometheus/blackbox.d/*.yml

    metrics_path: /probe

    relabel_configs:

      - source_labels: [__address__]

        target_label: __param_target

      - source_labels: [module]

        target_label: __param_module

      - target_label: __address__

        replacement: 127.0.0.1:9115  # Blackbox exporter

That is, you set a label "module" in your targets file, and it gets copied to __param_module.  Your targets files /etc/prometheus/blackbox.d/foo.yml look like this:

- labels:

    module: icmp

  targets:

    - 1.1.1.1

    - 2.2.2.2

- labels:

    module: dns

  targets:

    - 3.3.3.3

... etc

All your blackbox scrapes are done in a single job.

The actual config I use is a little more complex than that:

  - job_name: blackbox

    file_sd_configs:

      - files:

        - /etc/prometheus/blackbox.d/*.yml

    metrics_path: /probe

    relabel_configs:

      - source_labels: [__address__]

        regex: '([^/]+)'  # name or address only

        target_label: instance

      - source_labels: [__address__]

        regex: '([^/]+)'  # name or address only

        target_label: __param_target

      - source_labels: [__address__]

        regex: '(.+)/(.+)'  # name/address

        target_label: instance

        replacement: '${1}'

      - source_labels: [__address__]

        regex: '(.+)/(.+)'  # name/address

        target_label: __param_target

        replacement: '${2}'

      - source_labels: [module]

        target_label: __param_module

      - target_label: __address__

        replacement: 127.0.0.1:9115  # Blackbox exporter

This lets me use target entries of the form "<instance-label>/<address-or-hostname>", e.g.

- labels:

    module: icmp

  targets:

    - google-primary/8.8.8.8

    - google-secondary/8.8.4.4

so my metrics have meaningful instance labels.

[Sebastian N.]

Hey Brian,

thanks for the quick and detailed answer. It's really insightful so I'll be sure to adapt that to our needs. However, I'm still a bit lost how I can solve my requirement (maybe my approach is wrong or it's simply not possible).

Given this blackbox-exporter module definition...

modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      follow_redirects: false
      headers:
        User-Agent: "BlackBox-Exporter/EO"
      preferred_ip_protocol: "ip4"

and this Prometheus scraping config...

prometheus_scrape_configs:
  scrape_configs:
    - job_name: 'blackbox-http'
      scrape_interval: 5m
      metrics_path: /probe
      params:
        module: [http_2xx]
      static_configs:
        - targets:
            - https://foo.dev.contoso.com/
            - https://foo.dev.contoso.com/cms/
            - https://foo.dev.contoso.com/dealer-portal/
            - https://bar.dev.contoso.com/
            - https://bar.dev.contoso.com/cms/
            - https://bar.dev.contoso.com/dealer-portal/
          labels:
            env: dev
        - targets:
            - https://foo.int.contoso.com/
            - https://foo.int.contoso.com/cms/
            - https://foo.int.contoso.com/dealer-portal/
            - https://bar.int.contoso.com/
            - https://bar.int.contoso.com/cms/
            - https://bar.int.contoso.com/dealer-portal/
          labels:
            env: int
        - targets:
            - https://foo.contoso.com/
            - https://foo.contoso.com/cms/
            - https://foo.contoso.com/dealer-portal/
            - https://bar.contoso.com/
            - https://bar.contoso.com/cms/
            - https://bar.contoso.com/dealer-portal/
          labels:
            env: prod

      relabel_configs:
        - source_labels: [__address__]
          target_label: __param_target

        - source_labels: [__param_target]
          target_label: instance
        - target_label: __address__
          replacement: blackbox-exporter:9115 # Docker service name
        - target_label: project
          replacement: contoso

...is there any way to override the module parameters on a per-target basis? E.g. change user agent, enable/disable certificate trust, enable/disable following redirects for a single target?

Kind regards,

Seb

[Brian Candler]

You need to define different modules with the different sets of parameters you require, e.g.

modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      follow_redirects: false
      headers:
        User-Agent: "BlackBox-Exporter/EO"
      preferred_ip_protocol: "ip4"

  http_2xx_ie:

    prober: http
    timeout: 10s
    http:
      follow_redirects: false
      headers:

        User-Agent: "InternetExplorer 1.0"
      preferred_ip_protocol: "ip4"

Then you need to set __param_module to point to that (named) module.  The config I provided before copies the "module" label (set in the targets list) to the "__param_module" label.

The reason you should do this, rather than just set "__param_module" directly in the targets list, is because if you scrape the same target with multiple modules, you need the timeseries to be distinct; and the timeseries is defined by its set of labels.  Target labels which start with double-underscore are used to control scraping but are removed from the final label set.

BTW, I recommend using file_sd_configs rather than static_configs.  The format is exactly the same (a list of targets+labels), but you can change the targets file without having to HUP prometheus.