Using alertmanager with blackbox_exporter

[csana...@worldsensing.com]

Hello,

I've been using prometheus for 2 month already. My setup actually monitors nodes, mysql and cassandra. Next step is monitoring http response codes with blackbox_exporter.

I am trying to blackbox some urls and place alerts on failure but I cannot find any documentation on how to setup prometheus and alertmanager to alert me. Can someone help me?

Thank you.

c.

[Conor Broderick]

See the bottom of this sample prometheus configuration for for an example of connecting Prometheus with the Alertmanager.

You'd then need to have rules defined in Prometheus to alert on based on the metrics being scraped from the Blackbox exporter from Prometheus.

This blogpost gives an example of how to go about doing this.

On 14 September 2017 at 10:57, Conor Broderick <conor.b...@robustperception.io> wrote:

See the bottom of this sample prometheus configuration for for an example of connecting Prometheus with the Alertmanager.

You'd then need to have rules defined in Prometheus to alert on based on the metrics being scraped from the Blackbox exporter from Prometheus.

This blogpost gives an example of how to go about doing this.

BARCELONA Viriat, 47, Edificio Numancia 1, 10th floor, 08014 Barcelona, Spain LONDON 9-10 Carlos Place, Mayfair - London W1K 3AT, United Kingdom - +44 (0) 203 807 2495

This email may contain confidential information and may be subject to professional secrecy. Its dissemination, copying, distribution, reproduction, forwarding or any other unauthorized use is strictly prohibited. If you receive this email in error, please notify us immediately and arrange for its deletion, together with any attached documents.

Este correo electrónico puede contener información estrictamente confidencial, y puede estar sometido al secreto profesional. Está prohibida su difusión, copia, distribución, reproducción, reenvío o cualquier otro uso no autorizado. En caso de recibir este correo por error, rogamos nos lo comunique de inmediato y proceda a su eliminación, así como de cualquier documento que se adjunte.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d0156977-83e3-41f0-8b25-fad9a26fb96d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Conor Broderick

www.robustperception.io

--

Conor Broderick

www.robustperception.io

[Conor Broderick]

Note that Prometheus releases before 2.0 use the -alertmanager.url command line flag to define the running Alertmanager's url.

On 14 September 2017 at 10:58, Conor Broderick <conor.b...@robustperception.io> wrote:

See the bottom of this sample prometheus configuration for for an example of connecting Prometheus with the Alertmanager.

You'd then need to have rules defined in Prometheus to alert on based on the metrics being scraped from the Blackbox exporter from Prometheus.

This blogpost gives an example of how to go about doing this.

--

Conor Broderick

www.robustperception.io

[csana...@worldsensing.com]

OK thank you! I'll give it a try but what probes can I use apart of probe_ssl_earliest_cert_expiry ?

I don't see any documentation about probes to check.

Thank you.

c.

On Thursday, September 14, 2017 at 12:06:50 PM UTC+2, Conor Broderick wrote:

Note that Prometheus releases before 2.0 use the -alertmanager.url command line flag to define the running Alertmanager's url.

On 14 September 2017 at 10:58, Conor Broderick <conor.b...@robustperception.io> wrote:

See the bottom of this sample prometheus configuration for for an example of connecting Prometheus with the Alertmanager.

You'd then need to have rules defined in Prometheus to alert on based on the metrics being scraped from the Blackbox exporter from Prometheus.

This blogpost gives an example of how to go about doing this.

On 14 September 2017 at 10:57, Conor Broderick <conor.b...@robustperception.io> wrote:

See the bottom of this sample prometheus configuration for for an example of connecting Prometheus with the Alertmanager.

You'd then need to have rules defined in Prometheus to alert on based on the metrics being scraped from the Blackbox exporter from Prometheus.

This blogpost gives an example of how to go about doing this.

On 13 September 2017 at 17:13, <csana...@worldsensing.com> wrote:

Hello,

I've been using prometheus for 2 month already. My setup actually monitors nodes, mysql and cassandra. Next step is monitoring http response codes with blackbox_exporter.

I am trying to blackbox some urls and place alerts on failure but I cannot find any documentation on how to setup prometheus and alertmanager to alert me. Can someone help me?

Thank you.

c.

BARCELONA Viriat, 47, Edificio Numancia 1, 10th floor, 08014 Barcelona, Spain LONDON 9-10 Carlos Place, Mayfair - London W1K 3AT, United Kingdom - +44 (0) 203 807 2495

This email may contain confidential information and may be subject to professional secrecy. Its dissemination, copying, distribution, reproduction, forwarding or any other unauthorized use is strictly prohibited. If you receive this email in error, please notify us immediately and arrange for its deletion, together with any attached documents.

Este correo electrónico puede contener información estrictamente confidencial, y puede estar sometido al secreto profesional. Está prohibida su difusión, copia, distribución, reproducción, reenvío o cualquier otro uso no autorizado. En caso de recibir este correo por error, rogamos nos lo comunique de inmediato y proceda a su eliminación, así como de cualquier documento que se adjunte.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d0156977-83e3-41f0-8b25-fad9a26fb96d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Conor Broderick

www.robustperception.io

--

Conor Broderick

www.robustperception.io

--

Conor Broderick

www.robustperception.io

[csana...@worldsensing.com]

I am still confused by the lack of documentation. Correct me if I am wrong but....

I have 3 files to write in order to get alerts by http response code check:

• prometheus configuration
  
• blackbox_exporter configuration
  
• prometheus rules
  

I am trying this:

@ prometheus_config.yml (prometheus configuration)

following: https://github.com/prometheus/blackbox_exporter

rule_files:
  - "http.rules"
scrape_configs:
  - job_name: 'blackbox_test'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - http://www.google.com         # 200 - OK
        - http://www.google.com/qwerty  # 404 - Alert
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_container:9115  # pointing to linked blackbox_exporter container

@ blackbox_exporter_config.yml (blackbox_exporter configuration)

following: https://github.com/prometheus/blackbox_exporter/blob/master/example.yml

modules:
  http_post_ok:
    prober: http
    http:
      method: POST
      headers:
        Host: http://www.google.com
  http_post_ko:
    prober: http
    http:
      method: POST
      headers:
        Host: http://www.google.com/qwerty

questions

• What is the purpose of this file if I specify targets at prometheus configuration file?

@ http.rules (prometheus rules)

following: https://www.robustperception.io/get-alerted-before-your-ssl-certificates-expire/

ALERT SSLCertExpiringSoon
 IF probe_ssl_earliest_cert_expiry{job="blackbox"} - time() < 86400 * 30
 FOR 10m

questions

• Are there more probe_XXX parameters? What other parameters can I test?

[Conor Broderick]

What is the purpose of this file if I specify targets at prometheus configuration file?

 

The purpose of the blackbox configuration file is to configure the probes themselves.

For instance in your example above you're selecting HTTP as the probe, the HTTP method as POST, and the headers of the probe. 

You can find documentation for the Blackbox exporter's configuration here.

In your Prometheus configuration, you're simply selecting which probes to use against which targets. In your case you're using the previously defined http_2xx probe against your two google targets. Targets are not set in the Blackbox exporter's configuration file.

​Are there more probe_XXX parameters? What other parameters can I test?

Yes.

The HTTP probe for instance will return :

content_length 
probe_dns_lookup_time_seconds 

probe_duration_seconds 
probe_failed_due_to_regex 
probe_http_redirects 
probe_http_ssl 
probe_http_status_code 
probe_http_version 
probe_ip_protocol 
probe_success  
You can see a full list of the probes returned by your Blackbox exporter at <blackboxURL>:<port>/probe?target=<your-target>&module=<your-module>

So in your case it would be something like <blackboxURL>:<port>/probe?target=google.com&module=http_2xx

[csana...@worldsensing.com]

Hi Conor,

By doing: 

curl http://localhost:9115/probe?target=google.com&module=http_post

I get:

# HELP content_length Length of http content response
# TYPE content_length gauge
content_length 1589
# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 4.97e-07
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 0.014915568
# HELP probe_failed_due_to_regex Indicates if probe failed due to regex
# TYPE probe_failed_due_to_regex gauge
probe_failed_due_to_regex 0
# HELP probe_http_redirects The number of redirects
# TYPE probe_http_redirects gauge
probe_http_redirects 0
# HELP probe_http_ssl Indicates if SSL was used for the final redirect
# TYPE probe_http_ssl gauge
probe_http_ssl 0
# HELP probe_http_status_code Response HTTP status code
# TYPE probe_http_status_code gauge
probe_http_status_code 405
# HELP probe_http_version Returns the version of HTTP of the probe response
# TYPE probe_http_version gauge
probe_http_version 1.1
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 6
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 0

But I don't see any appearance of (http.rules)

probe_ssl_earliest_cert_expiry

Are there more probe outputs? Thank you.

Regards,

c.