[blackbox_exporter] No probe metrics on Prometheus

[Vladimir Shushkov]

Hello guys,

Kindly asking for your help. I am new here, so if I am breaking some rules in this group please excuse me and let know.

I am trying to bind blackbox_exporter with prometheus. 

Blackbox_exporter is launched as docker container using guidance on the service web-site. 

Config is almost default:

[ec2-user@ip-10-10-10-10 prometheus]$ cat ~/blackbox.yml

modules:

  http_2xx:

    prober: http

    http:

      preferred_ip_protocol: "ip4"

  http_post_2xx:

    prober: http

    http:

      method: POST

  tcp_connect:

    prober: tcp

  pop3s_banner:

    prober: tcp

    tcp:

      query_response:

      - expect: "^+OK"

      tls: true

      tls_config:

        insecure_skip_verify: false

  ssh_banner:

    prober: tcp

    tcp:

      query_response:

      - expect: "^SSH-2.0-"

  irc_banner:

    prober: tcp

    tcp:

      query_response:

      - send: "NICK prober"

      - send: "USER prober prober prober :prober"

      - expect: "PING :([^ ]+)"

        send: "PONG ${1}"

      - expect: "^:[^ ]+ 001"

  icmp:

    prober: icmp

Blackbox launched as

docker run --rm -d -p 9115:9115 --name blackbox_exporter -v `pwd`:/config prom/blackbox-exporter:master --config.file=/config/blackbox.yml

Everything looks fine from first glance and I am able to get metrics directly from blackbox using web-browser when I am using the URL - http://10.10.10.10:9115/probe?target=prometheus.io&module=http_2xx

# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 0.00107878
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 0.276252422
# HELP probe_failed_due_to_regex Indicates if probe failed due to regex
# TYPE probe_failed_due_to_regex gauge
probe_failed_due_to_regex 0
# HELP probe_http_content_length Length of http content response
# TYPE probe_http_content_length gauge
probe_http_content_length -1
# HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirects
# TYPE probe_http_duration_seconds gauge
probe_http_duration_seconds{phase="connect"} 0.018540241
probe_http_duration_seconds{phase="processing"} 0.242696764
probe_http_duration_seconds{phase="resolve"} 0.001804118
probe_http_duration_seconds{phase="tls"} 0.022516492
probe_http_duration_seconds{phase="transfer"} 0.000253678
# HELP probe_http_redirects The number of redirects
# TYPE probe_http_redirects gauge
probe_http_redirects 1
# HELP probe_http_ssl Indicates if SSL was used for the final redirect
# TYPE probe_http_ssl gauge
probe_http_ssl 1
# HELP probe_http_status_code Response HTTP status code
# TYPE probe_http_status_code gauge
probe_http_status_code 200
# HELP probe_http_uncompressed_body_length Length of uncompressed response body
# TYPE probe_http_uncompressed_body_length gauge
probe_http_uncompressed_body_length 15908
# HELP probe_http_version Returns the version of HTTP of the probe response
# TYPE probe_http_version gauge
probe_http_version 2
# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
# TYPE probe_ip_addr_hash gauge
probe_ip_addr_hash 2.601109704e+09
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 4
# HELP probe_ssl_earliest_cert_expiry Returns earliest SSL cert expiry in unixtime
# TYPE probe_ssl_earliest_cert_expiry gauge
probe_ssl_earliest_cert_expiry 1.6022448e+09
# HELP probe_ssl_last_chain_expiry_timestamp_seconds Returns last SSL chain expiry in timestamp seconds
# TYPE probe_ssl_last_chain_expiry_timestamp_seconds gauge
probe_ssl_last_chain_expiry_timestamp_seconds 1.6022448e+09
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 1
# HELP probe_tls_version_info Contains the TLS version used
# TYPE probe_tls_version_info gauge
probe_tls_version_info{version="TLS 1.3"} 1

Problem appears when I want to have these metrics on Prometheus. It shows nothing when I am searching for these metrics on it

Prometheus config:

[ec2-user@ip-10-10-10-10 prometheus]$ cat prometheus.yml

global:

  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

alerting:

  alertmanagers:

  - static_configs:

    - targets:

      - localhost:9093

rule_files:

  - rules.yml

global:

  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

alerting:

  alertmanagers:

  - static_configs:

    - targets:

      - localhost:9093

rule_files:

  - rules.yml

scrape_configs:

  - job_name: 'blackbox'

  metrics_path: /probe

  params:

    module: [http_2xx]  # Look for a HTTP 200 response.

  static_configs:

  - targets:

    - http://prometheus.io

    relabel_configs:

  - source_labels: [__address__]

    target_label: __param_target

  - source_labels: [__param_target]

    target_label: instance

  - target_label: __address__

    replacement: 127.0.0.1:9115

The job is in Up state on prometheus but search for probe, http or any other Blackbox metrics related keywords returns nothing. 

I tried to make some investigation and noticed a difference in this exporter comparing for instance with snmp_exporter or vmware_exporter where we have everything works correctly. Not sure if it's important or not for this issue:

if I am using curl for blackbox I am getting the metrics but not final carriage return that's why I have to push Enter to get my CLI back:

With VMware_exporter the behavior is different and I am getting metrics with my CLI immediately.

Could you please advise where I should look into? 

Thank you very much!

Regards,

Vlad

[Vladimir Shushkov]

Sorry guys, problem solved. Not sure what exactly solved it but I made  everything from scratch and used freshest version of blackbox binary with my own Dockerfile. Looks like there was some stupid mistake somewhere.

Topic is closed