blackbox exporter - scraping with grafana cloud agent - using consul sd for blackbox address

[david goodine]

Hi,

We are running blackbox exporter as a Nomad allocation, so its ip:port is dynamic.  We had been running a prometheus instance, also in Nomad, and it was configured with consul-template to dynamically set the __address__:

- target_label: __address__
          replacement: [% range  $i, $s := service "blackbox-exporter" %][% if eq $i 0 %][%.Address%]:[%.Port%][% end %][% end %]

Now, we have migrated to using grafana cloud agent instead of a standalone prometheus.  We run the agent on every Nomad client, to collect logs and metrics with its built in integrations.  I want to update the config so that it will spawn a prometheus instance to scrape any local blackbox exporter services.  However, instead of probing the URL that I expect, I get these targets: (from curl of http://<grafana agent ip>:53000/agent/api/v1/targets)

    {
      "instance": "f5d8368b42ae35f965c73d1be161364c",
      "target_group": "blackbox-ba-dev",
      "endpoint": "http://<bbexip>:21699/probe?target=<bbexip>%3A21699",
      "state": "up",
      "labels": {
        "instance": "10.25.194.212:21699",
        "job": "blackbox-ba-dev"
      },
      "last_scrape": "2020-11-11T21:31:30.45482873Z",
      "scrape_duration_ms": 1,
      "scrape_error": ""
    },
    {
      "instance": "f5d8368b42ae35f965c73d1be161364c",
      "target_group": "blackbox-ba-dev",
      "endpoint": "http://:/probe?target=mineralsoft.dev.ba.drillinginfo.com",
      "state": "down",
      "labels": {
        "instance": "mineralsoft.dev.ba.drillinginfo.com",
        "job": "blackbox-ba-dev"
      },
      "last_scrape": "2020-11-11T21:31:29.843730608Z",
      "scrape_duration_ms": 0,
      "scrape_error": "Get \"http:///probe?target=mineralsoft.dev.ba.drillinginfo.com\": http: no Host in request URL"
    }

What I need is a single target, with endpoint = http://<bbexip>:<bbex port>/probe?target=mineralsoft.dev.ba.drillinginfo.com

Here is my prometheus config:

      prometheus:
        wal_directory: /tmp/wal
        global:
          scrape_interval: 60s
        configs:
          - name: 'ba-[% key "global/config/env" %]-prometheus-gca'
            host_filter: false
            remote_write:
              - url: https://prometheus-us-central1.grafana.net/api/prom/push
                basic_auth:
                  [%with secret "di-secrets/prometheus/credentials"%]
                  username: [%.Data.data.USERNAME%]
                  password: [%.Data.data.APIKEY%]
                  [%end%]
            scrape_configs:
              - job_name: 'blackbox-ba-[% key "global/config/env" %]'
                metrics_path: /probe
                static_configs:
                  - targets:
                    - mineralsoft.dev.ba.drillinginfo.com    # Target to probe with http.
                consul_sd_configs:
                - services: ['blackbox-exporter']
                relabel_configs:
                  - source_labels: [__address__]
                    target_label: __param_target
                  - source_labels: [__param_target]
                    target_label: instance
                  - source_labels: ['__meta_consul_address','__meta_consul_service_port']
                    separator: ':'
                    target_label: __address__

Does anyone see what I am doing wrong?

[Brian Brazil]

Job labels should not vary by environment, you should use external labels for this.

 

                metrics_path: /probe
                static_configs:
                  - targets:
                    - mineralsoft.dev.ba.drillinginfo.com    # Target to probe with http.

                consul_sd_configs:
                - services: ['blackbox-exporter']

You can't mix SDs like this, they get all merged together. Use this like a normal exporter, remove the static_configs and put the single URL to be probed in as a params.

Brian

 

                relabel_configs:
                  - source_labels: [__address__]
                    target_label: __param_target
                  - source_labels: [__param_target]
                    target_label: instance
                  - source_labels: ['__meta_consul_address','__meta_consul_service_port']
                    separator: ':'
                    target_label: __address__

Does anyone see what I am doing wrong?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f4a4b496-3c1a-4cd9-890d-0496ca09ee98n%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

[david goodine]

Brian,

Thank you, that was very helpful - this is the config I ended up with (copy/paste breaks the yaml spacing):

scrape_configs:

- job_name: 'blackbox-ba-[% key "global/config/env" %]'

metrics_path: /probe

params:

target: ['mineralsoft.dev.ba.drillinginfo.com']

consul_sd_configs:

- services: ['blackbox-exporter']

relabel_configs:

- source_labels: [__param_target]

target_label: instance

- source_labels: ['__meta_consul_address','__meta_consul_service_port']

separator: ':'

target_label: __address__