401 Error from https target

[Manu K]

Hi All,

I have been facing an issue , as My https service api is returning me 401 response on grafana dashboard. I have provided correct username and password, but not giving me success. Please let me know if any one has idea. 

- job_name: blackbox-http
  metrics_path: /xxx
  params:
    module: [http_2xx]
  static_configs:
    - targets: ['https://xxxxxx']
  basic_auth:
   username: username
   password: password
  tls_config:
   insecure_skip_verify: true
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: xxxx.xx.xx.xx

Regards,

Manoj

[Brian Candler]

"basic_auth" in prometheus.yml provides credentials for talking to the exporter (i.e. for prometheus to authenticate itself to blackbox_exporter) - not for blackbox_exporter to authenticate to the target service.

To get blackbox_exporter to use http basic auth when talking to the target, you need to configure it to do so, as in the example here:

modules:
  http_basic_auth_example:
    prober: http
    timeout: 5s
    http:
      method: POST
      headers:
        Host: "login.example.com"
      basic_auth:
        username: "username"
        password: "mysecret"

This means creating a new module in your blackbox.yml, and then using that module instead of http_2xx in your scrape job.

Personally, I like to use rewriting rules to set the module as well:

  - job_name: blackbox
    file_sd_configs:
      - files:
          - /etc/prometheus/blackbox.d/*.yml
    metrics_path: /probe
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

      - source_labels: [__address__]
        target_label: __param_target

      - source_labels: [module]
        target_label: __param_module
      - target_label: __address__
        replacement: 127.0.0.1:9115  # Blackbox exporter

With targets files like this:

- labels:
    module: icmp
  targets:
    - 8.8.8.8
    - 8.8.4.4

- labels:
    module: certificate
  targets:

    - https://google.com/

This allows a single scrape job in prometheus to use many different blackbox exporter modules.