Blackbox explorter cofig parameter "insecure_skip_verify" doesn't seem to be effective

[Bostin Hoo]

Hi

This is my first time asking a question in the group, and I may not be very clear on some of the rules, but I will do my best to describe my question clearly. Below is the description of my issue:

•  I deployed Prometheus, Node Exporter, Grafana, Alertmanager, Blackbox Exporter, and Webhook-WeChat on the server with IP 10.1.1.161 using Docker, and all the ports are set to their default values. These services are running normally. I want to use Blackbox Exporter to monitor whether the Tableau service installed on a Windows system with IP 10.101.1.112 returns a status code of 200. I am monitoring port 8850 (the Tableau port).I used curl https://10.101.1.112:8850 on 10.1.1.161 and received the following response:

------------------------------------------------------------------------------------------------------------------------------------------

curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.

-------------------------------------------------------------------------------------------------------------------------------------------

• when I use "curl -k https://10.101.1.112:8850", I can successfully retrieve the response body. Therefore, I added insecure_skip_verify: true in the 'config/blackbox.yml' file to skip certificate verification.
• I restarted Prometheus, Blackbox Exporter, and the Blackbox container, but after the restart, the Webhook-WeChat service triggered an alert: 'https://10.101.1.112:8850 responded with code 0' instead of 200. However, when I access https://10.101.1.112:8850 in the browser, it works, though it warns that the connection is not secure. At the same time, when I use the browser to access http://10.1.1.161:9115/probemodule=http_2xx&target=https://10.101.1.112:8850, it returns a status in a format like this, which seems to indicate that the request was unsuccessful(for example：probe_success 0,probe_http_status_code 0):

-------------------------------------------------------------------------------------------------------------------------------------------

# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 1.6356e-05
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 0.022416452
# HELP probe_failed_due_to_regex Indicates if probe failed due to regex
# TYPE probe_failed_due_to_regex gauge
probe_failed_due_to_regex 0
# HELP probe_http_content_length Length of http content response
# TYPE probe_http_content_length gauge
probe_http_content_length 0
# HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirects
# TYPE probe_http_duration_seconds gauge
probe_http_duration_seconds{phase="connect"} 0
probe_http_duration_seconds{phase="processing"} 0
probe_http_duration_seconds{phase="resolve"} 1.6356e-05
probe_http_duration_seconds{phase="tls"} 0
probe_http_duration_seconds{phase="transfer"} 0
# HELP probe_http_redirects The number of redirects
# TYPE probe_http_redirects gauge
probe_http_redirects 0
# HELP probe_http_ssl Indicates if SSL was used for the final redirect
# TYPE probe_http_ssl gauge
probe_http_ssl 0
# HELP probe_http_status_code Response HTTP status code
# TYPE probe_http_status_code gauge
probe_http_status_code 0
# HELP probe_http_uncompressed_body_length Length of uncompressed response body
# TYPE probe_http_uncompressed_body_length gauge
probe_http_uncompressed_body_length 0
# HELP probe_http_version Returns the version of HTTP of the probe response
# TYPE probe_http_version gauge
probe_http_version 0
# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
# TYPE probe_ip_addr_hash gauge
probe_ip_addr_hash 3.959139489e+09
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 4
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 0

------------------------------------------------------------------------------------------------------------------------------------------

Here is my Prometheus prometheus.yml configuration:
---
global:
  scrape_interval: 15s
  evaluation_interval: 15s

# config alertmanagers
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 10.1.1.161:9093
rule_files:
  - "/etc/prometheus/rules/*.yml"
scrape_configs:
  # Prometheus port 9090
  - job_name: 'prometheus'
    static_configs:
      - targets: ['10.1.1.161:9090']
        labels:
          instance: 'prometheus'
  - job_name: 'tableau_blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://10.101.1.112:8850
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        # This is Blackbox Exporter
        replacement: 10.1.1.161:9115
      - source_labels: [__param_target]
        target_label: service
        replacement: 'new pro environment'  # https://10.101.1.112:8850
        regex: 'https://10\.101\.1\.112:8850'
---
Here is the Prometheus rule configuration:
---
groups:
  - name: tableau_alerts
    rules:
      # 1. HTTP/HTTPS
      - alert: Tableau service problem
        expr: probe_http_status_code{job="tableau_blackbox",service="new pro environment"} != 200
        for: 10s
        labels:
          severity: critical
        annotations:
          summary: "tableau code exception"
          description: "Target service {{ $labels.instance }} response code {{ $value }}，should be 200!"
---
Here is the Blackbox Exporter blackbox.yml configuration:
---
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      method: GET
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: false
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      tls_config:
        insecure_skip_verify: true
---

Here is my Docker startup command:

Prometheus：
docker run -d --name prometheus -p 9090:9090 -v /data/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
  -v /data/prometheus/rules/:/etc/prometheus/rules/  prom/prometheus

blackbox explorter:
docker run -d -p 9115:9115 -v /data/blackbox_exporter/config/:/config/  --name blackbox_exporter  quay.io/prometheus/blackbox-exporter