Related to blackbox exporter

[dikshant rai]

Hi,

I tried to configure blackbox exporter for checking api status with prometheus on kubernetes. When I am using static config then it shows fine but when I am using service discovery for status of service or pod some are coming up some are down. I used telnet to check connectivity as well that is fine but still don't know where it is failing.

Please help.

[Brian Candler]

> When I am using static config then it shows fine but when I am using service discovery for status of service or pod some are coming up some are down

Then clearly you're not scraping/probing the same endpoints as you did with the static config.

The easiest way to check is to go to the Prometheus web interface and look at the "Status > Targets" page and see what *actual* targets you're probing: first with the static config, and then secondly with the SD config (actually you can do both at the same time, but configure them as separate jobs so you can distinguish them by the "job" label).  The first column shows the URL being scraped and the URL params being passed ("module" and "target" are the ones that need to be set for blackbox_exporter, of course).

If you can show your static config, and the service discovery config, the problem might be clearer.  My guess is the SD discovery isn't returning data in the form that you expect, or that your rewrite rules aren't working in the way that you expect.

[dikshant rai]

Hi Brian,

Thanks for the response.

I have configured static and sd both as different job and when I am hitting SD job, I am getting targets as all the services inside the cluster but they are throwing 400 or 404 error but when I check the logs, the status is 200.

I am attaching job for your reference. Let me know if I am missing something. I also tried to check connection by going into blackbox-exporter pod but it is still throwing same error. I have put all under same namespace to avoid any communication failure.

    - job_name: "blackbox-kubernetes-services"
      metrics_path: /probe
      params:
        module: [http_2xx]
      kubernetes_sd_configs:
      - role: service
      relabel_configs:
      # Example relabel to probe only some services that have "example.io/should_be_probed = true" annotation
      #  - source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_probed]
      #    action: keep
      #    regex: true      
        - source_labels: [__address__]
          target_label: __param_target
        - target_label: __address__
          replacement:  prometheus-blackbox-exporter:9115
        - source_labels: [__param_target]
          target_label: instance
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          target_label: kubernetes_service_name

[Brian Candler]

What you need to do, as I suggested before, is go to the PromQL web interface and look at Status > Targets to see exactly the scrape URL and its query parameters, e.g.

Then you'll see what's different between the static_configs job and the kubernetes_sd_configs job, in terms of the generated scrape URL and params.  (And the total number of endpoints being probed, of course).

Looking at your config, my guess is that the __address__ that you're getting from kubernetes SD is either a plain address or an address:port pair, but the target for your http_2xx module requires a complete URL like http://address:port or https://address:port (maybe with an additional path too).  You'll be able to see if the http:// is missing under Status > Targets.

If that's true, then you need to modify your rewriting rules to build the target appropriately, for example:

        - source_labels: [__address__]
          target_label: __param_target

          replacement: "http://${1}"

That's just a starting point. You need to work on it as appropriate until your kubernetes SD config gives "target" params which match your static SD config.

[dikshant rai]

Hi Brian,

Thanks for responding. I am using grafana-agent because we are using mimir on top of prometheus. so at first, I thought the same so I went to grafanaa-agent pod using exec command and tried to connect  other running services using telnet command. For ex. telnet develop.static-service.dev.svc.cluster.local:8080, it throws 400 error and the same error we are getting from sd-config jobs. Either we are missing something or there may be one more reason that we are using istio for communication that is listening on 15020 port but we tried to connect using that port also but it throws error.

[Brian Candler]

If it's the same blackbox exporter you're talking to, and it is returning "success" to one probe request (originating from your static_sd_configs) but "fail" to another probe request (from your kubernetes_sd_configs), then ISTM that the requests that you are sending to blackbox_exporter are different in those two cases.

[dikshant rai]

Hi Brian,

I hope there is some confusion for static config, we are not using any SD, instead we are adding targets manually that is exposed to internet and that is working fine but when we want to monitor pods or services using sd_config, we are able to list out those with their name along with port number assigned. The only issue is that they are down.

[Brian Candler]

Inside a scrape job, either you are using

static_configs:

   ...

or

file_sd_configs:

   ...

or

kubernetes_sd_configs:

   ...

(or others). These are all forms of service discovery.  The result of all of these is the same: a list of targets to scrape as __address__, and other labels to associate with those targets.

I will say this for the final time: if it works with static_configs but doesn't work with kubernetes_sd_configs, it's because you're generating a different __address__ and/or labels.  And I've already told you twice how to view the list of targets and labels which the scrape job is using.