Blackbox not probing services

[sthorn...@gmail.com]

I've setup blackbox exporter as one part of our monitoring/alerting stack for applications deployed into our k8s cluster.  I have yet to get a working blackbox exporter that is successfully probing the service endpoints that we've specified with the appropriate annotation.  Below, you'll find the configs I've setup for local use with minikube.  For the sake of simplicity, I've only deployed Prometheus, Grafana, and Blackbox-exporter, along with their corresponding services.

blackbox-exporter.yaml

apiVersion: v1

kind: ReplicationController

metadata:

  name: blackbox-exporter

spec:

  replicas: 1

  selector:

    app: blackbox-exporter

  template:

    metadata:

      name: blackbox-exporter

      labels:

        app: blackbox-exporter

    spec:

      containers:

      - image: prom/blackbox-exporter:latest

        name: blackbox-exporter

        ports:

        - containerPort: 9115

          name: blackbox     

prometheus-service.yaml

apiVersion: v1

kind: Service

metadata:

  name: prometheus

  namespace: default

  annotations:

    prometheus.io/probe: "true"

spec:

  ports:

  - port: 9090

    protocol: TCP

    targetPort: 9090

  selector:

    name: prometheus

  type: NodePort

grafana-service.yaml

apiVersion: v1

kind: Service

metadata:

  name: grafana

  namespace: default

  annotations:

    prometheus.io/probe: "true"

spec:

  ports:

  - port: 3000

    protocol: TCP

    targetPort: 3000

  selector:

    name: grafana

  type: NodePort

NOTE: In addition to specifying the probe annotation for each service, I've also updated our prometheus configmap with the updated blackbox-exporter endpoint and port 9115.

prometheus-config.yaml

    # Example scrape config for probing services via the Blackbox Exporter.

    #

    # The relabeling allows the actual service scrape endpoint to be configured

    # via the following annotations:

    #

    # * `prometheus.io/probe`: Only probe services that have a value of `true`

    - job_name: 'kubernetes-services'

      metrics_path: /probe

      params:

        module: [http_2xx]

      tls_config:

        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token  

      kubernetes_sd_configs:

      - role: service

      relabel_configs:

      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]

        action: keep

        regex: true

      - source_labels: [__address__]

        target_label: __param_target

      - target_label: __address__

        replacement: blackbox-exporter:9115

      - source_labels: [__param_target]

        target_label: instance

      - action: labelmap

        regex: __meta_kubernetes_service_label_(.+)

      - source_labels: [__meta_kubernetes_namespace]

        target_label: kubernetes_namespace

      - source_labels: [__meta_kubernetes_service_name]

        target_label: kubernetes_name

When I go to the Prometheus UI targets page, I see the kubernetes-services section, however it shows the grafana and prometheus services as down.

I've spent hours looking through documentations, this user group, google, etc. and I've found nothing that's helped me resolve this issue.  I'm nearing the point where we may just get rid of blackbox exporter entirely and figure out an alternate route because of the amount of time that has been sunk into this failure of a task.

Any help would be greatly appreciate.

Thanks,

Sean

[sthorn...@gmail.com]

One other note - I've gone into the blackbox container and attempted to ping one of the services from there.

/ # ping grafana:3000

PING grafana:3000 (10.109.98.118): 56 data bytes

^C

--- grafana:3000 ping statistics ---

78 packets transmitted, 0 packets received, 100% packet los

There seems to be a network related issue since it can't reach the grafana endpoint, however it does resolve the IP address correctly.  I'm also not seeing the probe_success metric at all in Prometheus.