Kubernetes-service job is not displayed in Prometheus-target (blackbox installed)

427 views
Skip to first unread message

Gayathri Ravichandran

unread,
Dec 28, 2017, 8:28:49 AM12/28/17
to Prometheus Users
I have a prometheus pod running on Kubernetes cluster.

I have blackbox installed and module http_2xx running in the same box.

<html>
            <head><title>Blackbox Exporter</title></head>
            <body>
            <h1>Blackbox Exporter</h1>
            <p><a href="/probe?target=prometheus.io&module=http_2xx">Probe prometheus.io for http_2xx</a></p>
            <p><a href="/metrics">Metrics</a></p>
            </body>

I have the following endpoint in prometheus configmap

      - job_name: 'kubernetes-services'
        metrics_path: /probe
        params:
          module: [http_2xx]
        kubernetes_sd_configs:
          - role: service
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
            action: keep
            regex: true
          - source_labels: [__address__]
            target_label: __param_target
          - target_label: __address__
            replacement: 10.45.35.25:9115
          - source_labels: [__param_target]
            target_label: instance
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            target_label: kubernetes_name.

In the target page in Prometheus 'kubernetes-services' is not getting reflected. Please help.

Regards
Gayatri

Brian Brazil

unread,
Dec 28, 2017, 8:51:05 AM12/28/17
to Gayathri Ravichandran, Prometheus Users
On 28 December 2017 at 13:28, Gayathri Ravichandran <gaya3c...@gmail.com> wrote:
I have a prometheus pod running on Kubernetes cluster.

I have blackbox installed and module http_2xx running in the same box.

<html>
            <head><title>Blackbox Exporter</title></head>
            <body>
            <h1>Blackbox Exporter</h1>
            <p><a href="/probe?target=prometheus.io&module=http_2xx">Probe prometheus.io for http_2xx</a></p>
            <p><a href="/metrics">Metrics</a></p>
            </body>

I have the following endpoint in prometheus configmap

      - job_name: 'kubernetes-services'
        metrics_path: /probe
        params:
          module: [http_2xx]
        kubernetes_sd_configs:
          - role: service
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
            action: keep
            regex: true

Do your services have this annotation?

Brian
 
          - source_labels: [__address__]
            target_label: __param_target
          - target_label: __address__
            replacement: 10.45.35.25:9115
          - source_labels: [__param_target]
            target_label: instance
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            target_label: kubernetes_name.

In the target page in Prometheus 'kubernetes-services' is not getting reflected. Please help.

Regards
Gayatri

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/47ad437d-0ed4-4c3f-a1fb-a081e21278da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

Gayathri Ravichandran

unread,
Dec 28, 2017, 9:00:33 AM12/28/17
to Brian Brazil, Prometheus Users
sorry I don't get it. I set up blackbox as linux process and mentioned the IP:port in the service endpoint in Kubernetes. Am I missing out that something essential?

Regards
Gayatri

Gayathri Ravichandran

unread,
Dec 28, 2017, 9:13:12 AM12/28/17
to Brian Brazil, Prometheus Users
My requirement is to monitor all services across Kubernetes namespaces. For HTTP URLs my requirement is to check for 200 status code. I am not sure how to probe TCP endpoints.

On Thu, Dec 28, 2017 at 7:30 PM, Gayathri Ravichandran <gaya3c...@gmail.com> wrote:
sorry I don't get it. I set up blackbox as linux process and mentioned the IP:port in the service endpoint in Kubernetes. Am I missing out that something essential?

Regards
Gayatri

Gayathri Ravichandran

unread,
Jan 3, 2018, 9:24:02 AM1/3/18
to Prometheus Users
I have added 

metadata:
  annotations:
    prometheus.io/probe: "true"  in the service I want to monitor and I could see it is getting reflected in 'targets' Prometheus under the job for blackbox.
My requirement is to trigger alert if the service is deleted accidentally. Once the service is deleted ,it is not shown in probe_success. Please let me know if there is any optimum way to monitor this.

Regards
Gayatri

Brian Brazil

unread,
Jan 3, 2018, 9:50:36 AM1/3/18
to Gayathri Ravichandran, Prometheus Users
On 3 January 2018 at 14:24, Gayathri Ravichandran <gaya3c...@gmail.com> wrote:
I have added 

metadata:
  annotations:
    prometheus.io/probe: "true"  in the service I want to monitor and I could see it is getting reflected in 'targets' Prometheus under the job for blackbox.
My requirement is to trigger alert if the service is deleted accidentally. Once the service is deleted ,it is not shown in probe_success. Please let me know if there is any optimum way to monitor this.

You are looking to add an alert using the absent() function to detect it if the time series no longer exists.

Brian
 

For more options, visit https://groups.google.com/d/optout.



--
Reply all
Reply to author
Forward
0 new messages