Multiple blackbox exporters as one job

[Roman]

Setting up multiple targets with individual labels is simple:

  - job_name: 'cloudwatch'
    static_configs:

      - targets:
        - localhost:9110
        labels:
          instance: eu
      - targets:
        - localhost:9111
        labels:
          instance: us

However blackbox exporter requires `metric_relabel_configs` setup and setting `__address__` label didn't work for me:

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_text_html]  # Look for a HTTP 200 response.
    static_configs:

      - targets: ['example.com']
        labels:
          src:    eu
          env:   dev
          __address__: blackbox-eu:9449

      - targets: ['example.com']
        labels:
          src:    us
          env:   dev
          __address__: blackbox-us:9449

So I had to set up a separate job for each blackbox instance but I'm not sure if that's the right way to solve this. Any ideas?

[Brian Candler]

> blackbox exporter requires `metric_relabel_configs` setup

You don't need metric_relabel_configs, but relabel_configs is what you need.  relabel_configs is done *before* scraping, metric_relabel_configs is *after* scraping.

>  setting `__address__` label didn't work for me:

No it won't.  "targets" is what sets the __address__.

What are you trying to do precisely - I am guessing you are trying to probe the same site, "example.com", via two different exporters?

A simple way is to use two different jobs:

  - job_name: blackbox_eu

    metrics_path: /probe
    params:

      module: [http_text_html]

    static_configs:

      - targets:

          - example.com

          - example.net

        labels:

          src: eu

          env: dev

    relabel_configs:

      - source_labels: [__address__]

        target_label: instance

      - source_labels: [__address__]

        target_label: __param_target

      - target_label: __address__

        replacement: blackbox-eu:9449  # The blackbox exporter's real hostname:port.

  - job_name: blackbox_us

    metrics_path: /probe

    params:

      module: [http_text_html]

    static_configs:

      - targets:

          - example.com

          - example.net

        labels:

          src: us

          env: dev

    relabel_configs:

      - source_labels: [__address__]

        target_label: instance

      - source_labels: [__address__]

        target_label: __param_target

      - target_label: __address__

        replacement: blackbox-us:9449  # The blackbox exporter's real hostname:port.

I recommend making an explicit "module" label:

  - job_name: blackbox_eu

    metrics_path: /probe
    static_configs:

      - targets:

          - example.com

          - example.net

        labels:

          src: eu

          env: dev

          module: http_text_html

    relabel_configs:

      - source_labels: [__address__]

        target_label: instance

      - source_labels: [__address__]

        target_label: __param_target

      - source_labels: [module]

        target_label: __param_module

      - target_label: __address__

        replacement: blackbox-eu:9449  # The blackbox exporter's real hostname:port.

This lets you probe the same host with multiple modules (you need a "module" label to distinguish the timeseries), and you can have different targets with different modules all in the same job.

      - targets:

          - example.com

        labels:

          module: http_text_html

      - targets:

          - example.net

        labels:

          module: icmp

Finally, unless you only ever have one or two targets, I suggest you use file_sd_configs rather than static_configs.  This means you can change the targets without touching the main config file - and prometheus will pick up the changes automatically, without you having to send a HUP signal.

[Roman]

Nah, you didn't get it. Your example shows how to track multiple sites with 1 blackbox exporter. However I use multiple blackbox exporters to track one/multiple sites. So `replacement: blackbox-eu:9449` doesn't work in my case.

[Brian Candler]

Perhaps I confused you by putting both example.com and example.net, but it was to show that you that your multiple exporters could poll multiple hosts.

There are two jobs: one does replacement: blackbox-eu:9449 and one does replacement: blackbox-us: 9449.  If you only have two exporters then this is a reasonable way to do it.

If that's not what you're looking for, then you can get cleverer with the rewrite logic.  For example:

- job_name: blackbox

    metrics_path: /probe
    static_configs:

      - targets:

          - example.com,http_text_html,blackbox-eu:9449

          - example.com,http_text_html,blackbox-us:9449

    relabel_configs:

      - source_labels: [__address__]

        regex: '^(.*),(.*),(.*)$'

        replacement: '$1'

        target_label: instance

      - source_labels: [__address__]

        regex: '^(.*),(.*),(.*)$'

        replacement: '$1'

        target_label: __param_target

      - source_labels: [__address__]

        regex: '^(.*),(.*),(.*)$'

        replacement: '$2'

        target_label: module

      - source_labels: [__address__]

        regex: '^(.*),(.*),(.*)$'

        replacement: '$2'

        target_label: __param_module

      - source_labels: [__address__]

        regex: '^(.*),(.*),(.*)$'

        replacement: '$3'

        target_label: exporter

      # This one must be last

      - source_labels: [__address__]

        regex: '^(.*),(.*),(.*)$'

        replacement: '$3'

        target_label: __address__

You need the "exporter" label so that you get different timeseries for the different probes with the same target.

[Roman]

Wow, that's complicated. Looks legit, I'll give it a try, thanks!