Prometheus blackbox-exporter ICMP ping results for all targets are showing up

[Chenna Kesava]

HI Team, 

Could you please help me below issue.

I am working on configuring a monitor set up for N number of systems UP/Down status with Prometheus Blackbox-exporter ICMP ping response.

Host version:

 [root@vmfarm-infra-monitaring Monitering-beta]# uname -a
Linux vmfarm-infra-monitaring.private 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Container version:

/ # uname -a
Linux 65c1d0470331 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 GNU/Lin

blackbox_exporter version: output of blackbox_exporter -version
/ # blackbox_exporter --version
blackbox_exporter, version 0.16.0 (branch: HEAD, revision: 991f89846ae10db22a3933356a7d196642fcb9a9)
  build user:       root@64f600555645
  build date:       20191111-16:27:24
  go version:       go1.13.4
What is the blackbox.yml module config.
modules:
  icmp:                                                                                                                                                                                       
    prober: icmp                                                                                                                                                                              
    timeout: 5s                                                                                                                                                                               
    icmp:                                                                                                                                                                                     
      #protocol: icmp                                                                                                                                                                        
      preferred_ip_protocol: "ip4"
What is the prometheus.yml scrape config.
global:                                                                                                                                                                                       
  scrape_interval:     15s                                                                                                                                                                                                  
  external_labels:                                                                          
    monitor: 'codelab-monitor'                                                                                                                                                                
scrape_configs:                                                                                                                                                                               
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['prometheus:9090']
  - job_name: 'blackbox'                                                                                                                                                                      
    metrics_path: /probe                                                                                                                                                                      
    params:                                                                                                                                                                                   
      module: [icmp]
    scrape_interval: 5s
    scrape_timeout: 2s      #ping                                                                                                                                                                    
    static_configs:                                                                                                                                                                           
      - targets: ['192.168.1.29', '987.234.121.1']                                                                                                                                                            
        labels:                                                                                                                                                                               
          group: 'Build'                                                                                                                                                                   
      - targets: ['161.92.248.21', '161.92.3.185', '10.10.4.18']                                           
        labels:                                                                                                                                                                               
          group: 'RND' 
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackboxexporter:9115
What logging output did you get from adding &debug=true to the probe URL?

Module | Target | Result | Debug
-- | -- | -- | --
icmp | 192.168.1.29 | Failure | Logs
icmp | 192.168.3.185 | Failure | Logs
icmp | 161.92.248.21 | Success | Logs
Prometheus targets results are not accurate it's always and for all the targets UP only.   
Prometheus I am getting all the targets are up
Endpoint | State | Labels | Last Scrape | Scrape Duration | Error
-- | -- | -- | -- | -- | --
http://blackboxexporter:9115/probemodule="icmp" target="161.92.248.21" | UP | group="RND" instance="161.92.248.21" job="blackbox" | 1.43s ago | 1.522ms |  
http://blackboxexporter:9115/probemodule="icmp" target="192.168.1.29" | UP | group="Build" instance="192.168.1.29" job="blackbox" | 5.548s ago | 1.501s |  
http://blackboxexporter:9115/probemodule="icmp" target="192.168.3.185" | UP | group="RND" instance="192.168.3.185" job="blackbox" | 1.944s ago | 1.501s |  
http://blackboxexporter:9115/probemodule="icmp" target="192.168.4.185" | UP | group="RND" instance="192.168.4.185" job="blackbox" | 3.09s ago | 1.501s |  
http://blackboxexporter:9115/probemodule="icmp" target="987.234.121.1" | UP | group="Build" instance="987.234.121.1" job="blackbox" | 2.796s ago | 1.506ms

What I am expect to see?
I am expecting in Prometheus target "192.168.3.185" Down. because it's not up and it's showing down in blackbox-exporter probes results.
What I see instead?
I saw all the targets are up in Prometheus targets status.

 

[Brian Candler]

There are two different metrics you are looking at.

(1) "up" says whether the scrape was successful - i.e. prometheus was able to communicate with blackbox_exporter and read the response

(2) "probe_success" is one of the metrics returned from blackbox_exporter, saying whether it was able to talk to the target or not

So if blackbox_exporter is working, but the target is down, you will get up=1 and probe_success=0.  If you kill blackbox_exporter, then you will get up=0

In terms of the two GUIs you showed: the blackbox_exporter is showing you "probe_success", but prometheus is showing you "up".

"up" is a special metric for prometheus: its one that it generates itself, so the prometheus GUI knows that it exists and gives it special treatment. 

"probe_success" is a metric which happens to be returned by blackbox_exporter.  The prometheus GUI doesn't give this any special treatment because it's just a metric like any other.  But if you go into the query page of prometheus and type "probe_success" as the promQL query, you'll see what you are looking for.

[Chenna Kesava]

Thank you @brian Candler. I got this. As you mention Prometheus query "probe_success" giving accurate result. Much appreciate for your help full replay.