Service discovery for multiple blackbox instances

1,317 views
Skip to first unread message

Carl Pettersson (BN)

unread,
Sep 27, 2017, 3:17:36 PM9/27/17
to Prometheus Users

Hi,

We’re looking into deploying a large-ish set of blackbox exporters. The plan was to use the dns service discovery mechanism to find these as they come and go, but I just realized that is not how the blackbox jobs are configured... We already do config generation for the actual targets we want probed, but we’ve always just had a single blackbox instance to point to when doing the relabelling.

Is there some way to use service discovery for the actual blackbox? With the config from the docs

 

scrape_configs:

  - job_name: 'blackbox'

    metrics_path: /probe

    params:

      module: [http_2xx]  # Look for a HTTP 200 response.

    static_configs:

      - targets:

        - http://prometheus.io    # Target to probe with http.

        - https://prometheus.io   # Target to probe with https.

        - http://example.com:8080 # Target to probe with http on port 8080.

    relabel_configs:

      - source_labels: [__address__]

        target_label: __param_target

      - source_labels: [__param_target]

        target_label: instance

      - target_label: __address__

        replacement: 127.0.0.1:9115  # Blackbox exporter.

 

Can we do something to have the __address__ replacement served by dns_sd? This gets a bit odd since there are two “dynamic” parts.

 

Best,

Carl

wcooley...@gmail.com

unread,
Sep 27, 2017, 4:33:19 PM9/27/17
to Prometheus Users
You might want to have a read through this page https://latency.at/configuration# to see how their relabeling rules work

Brian Brazil

unread,
Sep 27, 2017, 7:26:44 PM9/27/17
to Carl Pettersson (BN), Prometheus Users
Yes you can, things were designed to work this way. While the blackbox exporter itself must be static, the targets can come from any SD with relabelling.
 

 

Best,

Carl

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/AM2PR05MB12336D4CC9A691F26A9873848F780%40AM2PR05MB1233.eurprd05.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.



--

Carl Pettersson (BN)

unread,
Sep 28, 2017, 5:48:05 AM9/28/17
to Brian Brazil, Prometheus Users

I think I may have been a bit unclear. The case is that I have multiple blackboxes coming and going, all registering themselves under a SRV record. I also have a dynamic list of targets, currently populated via consul-template.

That is, the configuration is dynamic in multiple dimensions. If I add a new blackbox instance, I want it to be used for all targets, and if I add a new target, I want it scraped from all instances.

 

But thinking about it, it probably cannot work like that. I will need to either have one job per target, or one job per blackbox, right?

 

Carl

 

Best,

Carl

--

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.



 

--

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLriHbcthwUJgZcA9a57oc-5BH2U1fMYgxWoC9X%2BC7SDmg%40mail.gmail.com.

Brian Brazil

unread,
Sep 28, 2017, 5:53:38 AM9/28/17
to Carl Pettersson (BN), Prometheus Users
On 28 September 2017 at 10:48, Carl Pettersson (BN) <carl.pe...@bonniernews.se> wrote:

I think I may have been a bit unclear. The case is that I have multiple blackboxes coming and going, all registering themselves under a SRV record. I also have a dynamic list of targets, currently populated via consul-template.

That is, the configuration is dynamic in multiple dimensions. If I add a new blackbox instance, I want it to be used for all targets, and if I add a new target, I want it scraped from all instances.

 

But thinking about it, it probably cannot work like that. I will need to either have one job per target, or one job per blackbox, right?


Yes, that's how to go about it. You'll likely need to use file_sd here.

Brian
 

 

Carl

 

Best,

Carl

--

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.



 

--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.



--
Reply all
Reply to author
Forward
0 new messages