Blackbox error

[sri L]

Hi all,

We are trying to use blackbox exporter to monitor one of the URL using http_2xx module and bearer token at job level for authorization, we are receiving following error message and probe failed

level=info msg="Address does not match first address, not sending TLS ServerName" first=IP address=<hostname>

level=info msg="Invalid HTTP response status code, wanted 2xx" status_code=403

level=error msg="Probe failed" duration_seconds=0.231352246

Connectivity from prometheus pod

/prometheus $ wget -S <URL>
Connecting to <URL> (xx.x.xx.x:80)
  HTTP/1.1 302 Moved Temporarily
  Date: Fri, 23 Jun 2023 09:07:18 GMT
  Content-Type: text/html
  Content-Length: 138
  Connection: close
  Location: <URL>
Connecting to  <URL> (xx.x.xx.x:80)
  HTTP/1.1 308 Permanent Redirect
  Date: Fri, 23 Jun 2023 09:07:18 GMT
  Content-Type: text/html
  Content-Length: 164
  Connection: close
  Location: <URL>
Connecting to  <URL> (xx.x.xx.x:443)
wget: note: TLS certificate validation not implemented
  HTTP/1.1 403 Forbidden
wget: server returned error: HTTP/1.1 403 Forbidden

[Brian Candler]

You can see from wget that your original request is redirected to a second URL, which is redirected to a third.  Therefore, you have to decide what you want to test: do you want to test that the original URL returns a redirect?  Or do you want to test that the target URL itself is working?  Those would be different configurations in blackbox_exporter.

> level=info msg="Address does not match first address, not sending TLS ServerName" first=IP address=<hostname>

That comes from here:

https://github.com/prometheus/blackbox_exporter/blob/v0.24.0/prober/http.go#L167-L172

Since the original request caused a redirect, the ServerName isn't set on the redirected target.

> wget: server returned error: HTTP/1.1 403 Forbidden

This means you're not authenticating to the target host - which is the same as you got from prometheus.  You can add extra flags to wget or curl to set your bearer token, to check it works.

Once that's working, you said you are setting "bearer token at job level". How are you doing this? Please show both the prometheus scrape job config and the blackbox exporter config (with the token itself blanked out)?

You shouldn't be setting the bearer token header in the prometheus scrape job, because that would be authenticating prometheus to blackbox_exporter. Instead you should configure blackbox_exporter to authenticate to the target host.  This was asked and answered recently:

https://groups.google.com/g/prometheus-users/c/1Es9Ok9tK-s/m/tuBnU3OPBAAJ

(you'll need to change the blackbox_exporter config slightly to set a bearer token header instead of basic auth)

[sri L]

Thanks Brian Candler for your reply.

We want to test the target url itself working. Here is my job and  Blackbox configuration. Please suggest

      - job_name: 'blackbox'

       honor_timestamps: true
       scrape_interval: 1m
       scrape_timeout: 10s
       metrics_path: "/probe"
       scheme: http
       tls_config:
         insecure_skip_verify: true
       params:
          module: [http_2xx]
       static_configs:
         - targets: [ '<url>' ]
       bearer_token: "xxxxxxxxxx"
       relabel_configs:
         - source_labels: [__address__]
           target_label: __param_target
         - source_labels: [__param_target]
           target_label: instance
         - target_label: __address__
           replacement: xxxxxx:9115


   http_2xx:
      prober: http
      timeout: 5s
      http:
        tls_config:
          insecure_skip_verify: True
        valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
        follow_redirects: true
        preferred_ip_protocol: "ip4"

[Brian Candler]

> We want to test the target url itself working.

In the chain where URL1 redirects to URL2 redirects to URL3, do you want to test URL1 or URL3?  If the latter, then I'd suggest you put URL3 in your static_targets, instead of URL1.  However URL1 may work well enough with "follow_redirects"; you could end up with certificate validation errors, but you're already throwing away all certificate problems.

Apart from this: the problem with authentication is that you've put bearer_token in your scrape job.  This is only passed from prometheus to blackbox_exporter - which ignores it (unless you have blackbox_exporter itself setup to require authentication from clients).

Instead, you need to configure a blackbox_exporter module to send authentication to the target, with a fixed bearer token. I'd suggest you create a new module:

   http_2xx_auth:

      prober: http
      timeout: 5s
      http:

        headers:
          Authorization: "Bearer xxxxxxxx"

        tls_config:
          insecure_skip_verify: True
        valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
        follow_redirects: true
        preferred_ip_protocol: "ip4"

(first check with curl or wget what exact header you need to send to get a successful result)

The other thing to note is that with your current config, the module name is hard-coded within the scrape job in "params", meaning that each blackbox_exporter module you use needs a new scrape job creating.  But there's a better way: instead of setting params directly, you can set the target label "__param_module". Change:

       params:
          module: [http_2xx]
       static_configs:
         - targets: [ '<url>' ]

to:

       static_configs:
         - targets: [ '<url>' ]
           labels: { '__param_module': 'http_2xx_auth'}

You can then add other targets using different modules under the same static_configs section, e.g.

         - targets: [ '8.8.8.8', '8.8.4.4' ]
           labels: { '__param_module': 'icmp'}

etc.