multiple blackbox_exporter modules with different targets

[Barry S.]

Hi,

Prometheus blackbox_exporter documentation suggests below config:

scrape_configs:

  - job_name: 'blackbox'

    metrics_path: /probe

    params:

      module: [http_2xx]  # Look for a HTTP 200 response.

    static_configs:

      - targets:

        - http://prometheus.io    # Target to probe with http.

In this config we can define multiple target for a single module. But what I need, I wanna assign some targets for ICMP probe and some others for http_2xx. I haven't seen anything about this on google.

What is the syntax for multiple modules with different targets?

Thanks.

Barry.

[Matthias Rampke]

In that case, create separate scrape_configs. If you are monitoring different things with different target sets, then you can no longer subsume this under one "blackbox" bucket. The job_name of the example is a bit misleading in this; it should be something like job_name: 'blackbox-http-prometheus-io' I suppose?

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/02f46533-cc60-4306-afaf-e06babba829a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Barry S.]

yes, I realized that.

I created two different job like blackbox_icmp, blackbox_http and it works.

What kind of metrics does it provide? I don't see anything on prometheus console related to blackbox.

thanks

[Matthias Rampke]

Does it actually scrape without error? Check the target page. Your configuration snippet does not include the relabeling rules, these are important to scrape the exporter instead of the actual target directly.

/MR

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d8efe789-7aca-4fbf-a5b7-f7d22853f51e%40googlegroups.com.

[Baris Simsek]

yes it is working with alertmanager. it doesn't provide any metric, but gets status of end point, if it is down or not. Is there any metric related to blackbox? and where cn i get their list?

[Matthias Rampke]

We're not at the stage where we can or should worry about Alertmanager yet. Let's get the interaction between your service, Prometheus, and the blackbox exporter right.

Find the target on the Prometheus target page, and click the link. It will take you to the blackbox exporter (this only works if it's at an address that you can reach as well – otherwise, substitute the correct address in the URL). This will cause your browser to make a request to the blackbox exporter, which causes it to send a probe, and show you all the metrics you get from that.

/MR

[Barry S.]

Thanks. I found it.

Metrics that would have been returned:
# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 0.001244391
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 0.010763475
# HELP probe_failed_due_to_regex Indicates if probe failed due to regex
# TYPE probe_failed_due_to_regex gauge
probe_failed_due_to_regex 0
# HELP probe_http_content_length Length of http content response
# TYPE probe_http_content_length gauge
probe_http_content_length -1
# HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirects
# TYPE probe_http_duration_seconds gauge
probe_http_duration_seconds{phase="connect"} 0.001968657
probe_http_duration_seconds{phase="processing"} 0.00250948
probe_http_duration_seconds{phase="resolve"} 0.0017022220000000002
probe_http_duration_seconds{phase="tls"} 0.005056645
probe_http_duration_seconds{phase="transfer"} 6.9843e-05
# HELP probe_http_redirects The number of redirects
# TYPE probe_http_redirects gauge
probe_http_redirects 1
# HELP probe_http_ssl Indicates if SSL was used for the final redirect
# TYPE probe_http_ssl gauge
probe_http_ssl 1
# HELP probe_http_status_code Response HTTP status code
# TYPE probe_http_status_code gauge
probe_http_status_code 200
# HELP probe_http_version Returns the version of HTTP of the probe response
# TYPE probe_http_version gauge
probe_http_version 1.1
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 4
# HELP probe_ssl_earliest_cert_expiry Returns earliest SSL cert expiry in unixtime
# TYPE probe_ssl_earliest_cert_expiry gauge
probe_ssl_earliest_cert_expiry 1.5888528e+09
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 1

[trn...@gmail.com]

You can rely on relabel_configs to do some processing.

Example:

- job_name: 'upstream'

metrics_path: /probe

static_configs:

- targets:

- tcp_connect:upstream1.dev:80

- http_2xx:http://upstream2.stage

relabel_configs:

- source_labels: [__address__]

regex: (\w+):(.+)

target_label: __param_module # Tokenize on ':' and use 1st element as 'module' param to blackbox

replacement: ${1}

- source_labels: [__param_module]

target_label: module # Copy module into label for further analysis in metrics

- source_labels: [__address__]

regex: (\w+):(.+)

target_label: __param_target # Tokenize on ':' and use 2nd part of the original target as 'target' param to blackbox

replacement: ${2}

- target_label: __address__

replacement: blackbox-exporter.tools # Replace __address__ with Blackbox exporter incluster address

- source_labels: [__param_target]

target_label: instance

- source_labels: [__param_target]

regex: (\w+\:\/\/)?([\w-]+)\.(\w+)(\:\d+)?([^:\n]+)?

target_label: service

replacement: ${2} # Do some additional processing for further analysis in metrics

- source_labels: [__param_target]

regex: (\w+\:\/\/)?([\w-]+)\.(\w+)(\:\d+)?([^:\n]+)?

target_label: kubernetes_namespace

replacement: ${3} # Do some additional processing for further analysis in metrics.

Let me know if that's what you are looking for.

-vova

[mpur...@gmail.com]

To see the metrics from the blackbox checks, use probe_success()... e.g. this query will show you all the jobs with name "blackbox.*" that are currently failing...

probe_success{job=~"blackbox.*"} == 0