blackbox-operator config to scrape multiple static dns targets

[Morné Kruger]

Hi there!

I have been busting my head for 2 days now and hope somebody has a pointer or two on what I am missing here.

I have a Prometheus setup running on AKS 1.19.9

bitnami/prometheus-operator:0.48.1

bitnami/prometheus:2.25.2

blackbox-exporter:0.18.0

Use case:

I want to check when a DNS is no longer responding.

10.25.22.15

10.25.33.15

10.105.12.21

Problem:

I have the job running and can see metrics being scraped. 

I cannot seem to figure out how to distinguish between the metrics of the 3 dns listed.

I was expecting 3 separate time series for each, in order to determine when one has a probe that is failing. At this stage, I can only see 1 time series.

I am suspecting that my skills at relabeling are letting me down...

Config

I have the following definition in my prometheus-config.yaml:

apiVersion: monitoring.coreos.com/v1

kind: Probe

metadata:

  name: bbexporterdns

  namespace: monitoring

  labels:

    prometheus: prometheus

    role: probes

spec:

  jobName: bbexporter_dns

  prober:

    url: bbexporter.monitoring.svc:9115

    scheme: http

    path: /probe

  module: dns_tcp   

  targets:

    staticConfig:

      static:      

        - '10.25.22.15'

        - '10.25.33.15'

        - '10.105.12.21'

      relabelingConfigs:

      - sourceLabels: [__address__]

        targetLabel: __param_target

      - sourceLabels: [__param_target]

        targetLabel: instance

---

In the bbexporter.yaml I have the following configured:

apiVersion: v1

kind: ConfigMap

metadata:

  name: bbexporter-cfg

  namespace: monitoring

data:

  bbexporter.yml: |

    modules:

      dns_tcp:

        prober: dns

        dns:

          preferred_ip_protocol: "ip4"

          transport_protocol: "tcp"

          query_name: "www.robustperception.io"

          tls_config:

            insecure_skip_verify: true

Current results:

After setting up a port forward to the service, and running the following command in my browser: http://localhost:9115/probe?module=dns_tcp&target=10.25.22.15

 I can see the following metrics:

# HELP probe_dns_additional_rrs Returns number of entries in the additional resource record list

# TYPE probe_dns_additional_rrs gauge

probe_dns_additional_rrs 0

# HELP probe_dns_answer_rrs Returns number of entries in the answer resource record list

# TYPE probe_dns_answer_rrs gauge

probe_dns_answer_rrs 1

# HELP probe_dns_authority_rrs Returns number of entries in the authority resource record list

# TYPE probe_dns_authority_rrs gauge

probe_dns_authority_rrs 0

# HELP probe_dns_duration_seconds Duration of DNS request by phase

# TYPE probe_dns_duration_seconds gauge

probe_dns_duration_seconds{phase="connect"} 0.018158141

probe_dns_duration_seconds{phase="request"} 0.017735813

probe_dns_duration_seconds{phase="resolve"} 1.3401e-05

# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds

# TYPE probe_dns_lookup_time_seconds gauge

probe_dns_lookup_time_seconds 1.3401e-05

# HELP probe_duration_seconds Returns how long the probe took to complete in seconds

# TYPE probe_duration_seconds gauge

probe_duration_seconds 0.036181472

# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.

# TYPE probe_ip_addr_hash gauge

probe_ip_addr_hash 2.764160084e+09

# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6

# TYPE probe_ip_protocol gauge

probe_ip_protocol 4

# HELP probe_success Displays whether or not the probe was a success

# TYPE probe_success gauge

probe_success 1

In the Prometheus browser, I see the following metric: probe_dns_duration_seconds (see the attachment)

I just have no idea to which of the dns it applies - i.e. which dns was probed for this?

I would greatly appreciate some help!

Many thanks :-)

Morné Kruger

[Brian Candler]

If you were using pure prometheus, this would be a simple bit of target relabelling:

  - job_name: bbexporter_dns

    static_configs:

      - targets:

        - ...

    metrics_path: /probe
    params:
      module: [dns_tcp]

    relabel_configs:

      - source_labels: [__address__]

        target_label: instance

      - source_labels: [__address__]

        target_label: __param_target

      - target_label: __address__

        replacement: bbexporter.monitoring.svc:9115  # Blackbox exporter

The "instance" label would then distinguish between the different probe targets.

However since you're using some third-party operator to write the prometheus config, I've no idea what you'd have to do to make it work sensibly.  Perhaps start by looking at the prometheus config which the operator has generated, then work backwards to see what you need to tweak?  Otherwise, ask this question on the github repo of bitnami/prometheus-operator (since it's not part of prometheus).

[Morné Kruger]

Hi there Brian,

Thank you for taking the time to answer the question, much appreciated.

Your hunch in terms of the operator (bitnami/prometheus-operator:0.48.1) was indeed corect!

The suggested relabeling I have mentioned here (and what you find in most books) will work on a normal Prometheus i.e. the non-operator flavor.

The weirdness that I have witnessed here was indeed caused by the operator I use - the operator turns out to be very polite and handles the relabeling for you. 

It is just really humble and does not mention that it will handle this for us - or I was not able to find it so quickly anyway : )

This means that when you omit this section completely you should see each target in your metrics.

    relabel_configs:

      - source_labels: [__address__]

        target_label: instance

      - source_labels: [__address__]

        target_label: __param_target

      - target_label: __address__

        replacement: bbexporter.monitoring.svc:9115 

Adding the relabeling will mangle the good work the operator did beyond repair.

Kind regards,

Morné