blackbox exporter docker - socket connection error

[Amit Das]

Hi everyone. 

Anyone faced the issues with blackbox exporter (docker) 

"icmp socker connection error".   I tried the solutions by  building the image with the permission as on the thread.  https://github.com/prometheus/blackbox_exporter/issues/689

Out of three ip address  (cmd ping works fine) two ip address probe_status==0 and  still fails  now this time timeout reading error from socket  err="read ip4 0.0.0.0 i/o timeout".

 Also my second issue is my ssl certificate (openssl) is nearly more than 6 months to expire but its showing as expired and expiring in 30 days.

probe_ssl_earliest_cert_expiry - time() < 86400 * 30
probe_ssl_earliest_cert_expiry - time() <= 0

Any suggestions please.  

[Brian Candler]

> Out of three ip address  (cmd ping works fine) two ip address probe_status==0 and  still fails  now this time timeout reading error from socket  err="read ip4 0.0.0.0 i/o timeout".

Test it using curl:

curl 'a.a.a.a:9115/probe?module=XXX&target=Y.Y.Y.Y'

where a.a.a.a is docker container address.  Alternatively if you can docker exec into the container, and there is curl inside the container, then you can do

curl '127.0.0.1:9115/probe?module=XXX&target=Y.Y.Y.Y'

Here's what I get when I probe an address that is not reachable (1.2.3.4):

$ curl '127.0.0.1:9115/probe?module=icmp&target=1.2.3.4'

# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds

# TYPE probe_dns_lookup_time_seconds gauge

probe_dns_lookup_time_seconds 5.2838e-05

# HELP probe_duration_seconds Returns how long the probe took to complete in seconds

# TYPE probe_duration_seconds gauge

probe_duration_seconds 3.000263382

# HELP probe_icmp_duration_seconds Duration of icmp request by phase

# TYPE probe_icmp_duration_seconds gauge

probe_icmp_duration_seconds{phase="resolve"} 5.2838e-05

probe_icmp_duration_seconds{phase="rtt"} 0

probe_icmp_duration_seconds{phase="setup"} 0.000278196

# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.

# TYPE probe_ip_addr_hash gauge

probe_ip_addr_hash 3.268949123e+09

# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6

# TYPE probe_ip_protocol gauge

probe_ip_protocol 4

# HELP probe_success Displays whether or not the probe was a success

# TYPE probe_success gauge

probe_success 0

This is with the following blackbox.yml config:

modules:

  icmp:

    prober: icmp

    timeout: 3s

> Also my second issue is my ssl certificate (openssl) is nearly more than 6 months to expire but its showing as expired and expiring in 30 days.

Go to the PromQL browser in Prometheus web interface, and enter the expression

probe_ssl_earliest_cert_expiry - time()

What does it show?  If you get a negative number then it sounds to me like one of the certificates in the chain has expired, but unless you tell us the domain name (and the server is reachable over the Internet), we can't check.

[Amit Das]

Hi thanks for the quick response. Please see the details below.

1. icmp socker error for testing run from inside container 

  $ curl -s '127.0.0.1:9115/probe?target=172.0.16.90&module=icmp&debug=true'

Logs for the probe:

ts=2020-10-17T10:06:07.696205726Z caller=main.go:304 module=icmp target=172.0.16.90 level=info msg="Beginning probe" probe=icmp timeout_seconds=5

ts=2020-10-17T10:06:07.696325226Z caller=icmp.go:84 module=icmp target=172.0.16.90 level=info msg="Resolving target address" ip_protocol=ip4

ts=2020-10-17T10:06:07.696348246Z caller=icmp.go:84 module=icmp target=172.0.16.90 level=info msg="Resolved target address" ip=172.0.16.90

ts=2020-10-17T10:06:07.696361525Z caller=main.go:119 module=icmp target=172.0.16.90 level=info msg="Using source address" srcIP=0.0.0.0

ts=2020-10-17T10:06:07.696377468Z caller=main.go:119 module=icmp target=172.0.16.90 level=info msg="Creating socket"

ts=2020-10-17T10:06:07.696400989Z caller=main.go:119 module=icmp target=172.0.16.90 level=debug msg="Unable to do unprivileged listen on socket, will attempt privileged" err="socket: permission denied"

ts=2020-10-17T10:06:07.696444967Z caller=main.go:119 module=icmp target=172.0.16.90 level=info msg="Creating ICMP packet" seq=45267 id=5018

ts=2020-10-17T10:06:07.696464706Z caller=main.go:119 module=icmp target=172.0.16.90 level=info msg="Writing out packet"

ts=2020-10-17T10:06:07.696590348Z caller=main.go:119 module=icmp target=172.0.16.90 level=info msg="Waiting for reply packets"

ts=2020-10-17T10:06:12.696498118Z caller=main.go:119 module=icmp target=172.0.16.90 level=warn msg="Timeout reading from socket" err="read ip4 0.0.0.0: i/o timeout"

ts=2020-10-17T10:06:12.696736957Z caller=main.go:304 module=icmp target=172.0.16.90 level=error msg="Probe failed" duration_seconds=5.000476818

Metrics that would have been returned:

# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds

# TYPE probe_dns_lookup_time_seconds gauge

probe_dns_lookup_time_seconds 1.451e-05

# HELP probe_duration_seconds Returns how long the probe took to complete in seconds

# TYPE probe_duration_seconds gauge

probe_duration_seconds 5.000476818

# HELP probe_icmp_duration_seconds Duration of icmp request by phase

# TYPE probe_icmp_duration_seconds gauge

probe_icmp_duration_seconds{phase="resolve"} 1.451e-05

probe_icmp_duration_seconds{phase="rtt"} 0

probe_icmp_duration_seconds{phase="setup"} 8.7043e-05

# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.

# TYPE probe_ip_addr_hash gauge

probe_ip_addr_hash 2.606393893e+09

# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6

# TYPE probe_ip_protocol gauge

probe_ip_protocol 4

# HELP probe_success Displays whether or not the probe was a success

# TYPE probe_success gauge

probe_success 0

blackbox.yml

modules:

  http_2xx:

    http:

      fail_if_not_ssl: false

      fail_if_ssl: false

      method: GET

      no_follow_redirects: false

      preferred_ip_protocol: ipv4

    prober: http

    timeout: 5s

  https_2xx:

    http:

      fail_if_not_ssl: false

      fail_if_ssl: false

      method: GET

      no_follow_redirects: false

      preferred_ip_protocol: ipv4

    prober: http

    timeout: 5s

  icmp:

    icmp:

      preferred_ip_protocol: ip4

      source_ip_address: "0.0.0.0"

    prober: icmp

    timeout: 5s

  irc_banner:

    prober: tcp

    tcp:

      query_response:

        - send: "NICK prober"

        - send: "USER prober prober prober :prober"

        - expect: "PING :([^ ]+)"

          send: "PONG ${1}"

        - expect: "^:[^ ]+ 001"

    timeout: 5s 

prometheus.yml 

 

- job_name: blackbox

    metrics_path: /probe

    params:

      module: [http_2xx]

    static_configs:

      - targets:

        - https://www.robustperception.io/

        - http://prometheus.io/blog

        - https://devconnected.com/

        - https://github.com/

        - https://www.bogotobogo.com/

        - http://www.columbia.edu/

        - http://www.http2demo.io/

        - https://www.w3schools.com/

        - https://xxxxxx1.com/

        - http://xxxxxx2.com/

    relabel_configs:

      - source_labels: [__address__]

        target_label: __param_target

      - source_labels: [__param_target]

        target_label: instance

      - target_label: __address__

        replacement: blackbox_exporter:9115 # The blackbox exporter.

 

  - job_name: 'blackbox-ping'

    metrics_path: /probe

    params:

      module: [icmp]

    static_configs:

      - targets:

        - localhost

        - prometheus.io

        - robustperception.io

        - google.com

        - hotmail.com

        - 172.0.16.90    # internal can ping via icmp

               

    relabel_configs:

      - source_labels: [__address__]

        target_label: __param_target

      - source_labels: [__param_target]

        target_label: instance

      - target_label: __address__

        replacement: blackbox_exporter:9115  # This is your blackbox exporter.    

2. ssl_certificate negative values. Config files above added for ref.

[Brian Candler]

On Saturday, 17 October 2020 11:49:36 UTC+1, Amit Das wrote:

ts=2020-10-17T10:06:07.696400989Z caller=main.go:119 module=icmp target=172.0.16.90 level=debug msg="Unable to do unprivileged listen on socket, will attempt privileged" err="socket: permission denied"

If you were running blackbox_exporter standalone, there would be three possible solutions:

1. run it as root

2. setcap cap_net_raw+ep /path/to/blackbox_exporter

3. use the rootless ping settings (net.ipv4.ping_group_range):

https://github.com/prometheus/blackbox_exporter/pull/642/files
https://github.com/prometheus/blackbox_exporter/issues/147

In docker things will be different, because you'll also be constrained by what the container environment permits.  You may need to run the container in privileged mode.

Re "ssl_certificate negative values": you posted a blank white box.  Please DON'T post images anyway.  Just select the text, copy it, and paste it.

If the values are negative, then one of the certificates in the chain has expired.

You can retrieve the certificates like this:

openssl s_client -connect www.google.com:443 -name www.google.com -showcerts   # replace www.google.com with your server

Then copy-paste each of the certificates, from -----BEGIN CERTIFICATE----- to -----END CERTIFICATE----- (including those lines), into separate text files.

Then run this command:

openssl x509 -in filename.pem -noout -text

to display all of the fields in the certificate, or

openssl x509 -in filename.pem -noout -enddate

to see just the expiry date.

[Brian Candler]

On Saturday, 17 October 2020 11:49:36 UTC+1, Amit Das wrote:

ts=2020-10-17T10:06:07.696444967Z caller=main.go:119 module=icmp target=172.0.16.90 level=info msg="Creating ICMP packet" seq=45267 id=5018

ts=2020-10-17T10:06:07.696464706Z caller=main.go:119 module=icmp target=172.0.16.90 level=info msg="Writing out packet"

ts=2020-10-17T10:06:07.696590348Z caller=main.go:119 module=icmp target=172.0.16.90 level=info msg="Waiting for reply packets"

ts=2020-10-17T10:06:12.696498118Z caller=main.go:119 module=icmp target=172.0.16.90 level=warn msg="Timeout reading from socket" err="read ip4 0.0.0.0: i/o timeout"

ts=2020-10-17T10:06:12.696736957Z caller=main.go:304 module=icmp target=172.0.16.90 level=error msg="Probe failed" duration_seconds=5.000476818

Sorry, scrub that.  It looks like your machine *did* send the packet after all.  Use tcpdump inside the container to confirm it:

tcpdump -i eth0 -nn host x.x.x.x

If a echo-request packet is sent but an echo-reply doesn't come back, then that's a networking problem, nothing to do with blackbox_exporter.

If a packet comes back according to tcpdump but blackbox exporter doesn't see it, then that requires further investigation.  There was something weird observed here:

https://github.com/prometheus/blackbox_exporter/issues/360

[Amit Das]

I have created a new docker image with  setcap cap_net_raw+ep /path/to/blackbox_exporter  as mentioned here  https://github.com/prometheus/blackbox_exporter/issues/689.

Tried as root as well as  setcap cap_net_raw+ep . Most of the public urls or Ip i dont have issues except  my internal   3 server ip  only one works fine, rest two ips i can ping  publicly. 

After running the tcpdump from inside docker container for the two public ip address i am not getting echo-reply doesn't come back. Ping works fine from inside container but why not the blackbox!! 

As mentioned bt multiple threads if tcpdump fails then its network issue probably.

[Amit Das]

I have run the above commands for openssl  inside the container  and i am getting the correct results as expected but in Prometheus why i am getting negative values.  

As mentioned maybe the network problem here too!!!