probe_success is returning 0 everytime
1,954 views
Skip to first unread message
vteja...@gmail.com
Jan 4, 2018, 12:10:47 PM1/4/18
to Prometheus Users
Hi,
I have installed Prometheus, BlackBox Exporter and a Prometheus Python Client Application on K8s.
Blackbox running on:
curl -l http://136.225.241.11:32466
<html>
<head><title>Blackbox Exporter</title></head>
<body>
<h1>Blackbox Exporter</h1>
<p><a href="/probe?target=prometheus.io&module=http_2xx">Probe prometheus.io for http_2xx</a></p>
<p><a href="/probe?target=prometheus.io&module=http_2xx&debug=true">Debug probe prometheus.io for http_2xx</a></p>
<p><a href="/metrics">Metrics</a></p>
<p><a href="/config">Configuration</a></p>
<h2>Recent Probes</h2>
<table border='1'><tr><th>Module</th><th>Target</th><th>Result</th><th>Debug</th><tr><td>http_2xx</td><td>searchapp1.staging.svc:9001</td><td><strong>Failure</strong></td><td><a href='logs?id=4'>Logs</a></td></td><tr><td>http_2xx</td><td>searchapp1.staging.svc:9001</td><td><strong>Failure</strong></td><td><a href='logs?id=3'>Logs</a></td></td><tr><td>http_2xx</td><td>blackbox.monitoring.svc:80</td><td>Success</td><td><a href='logs?id=2'>Logs</a></td></td><tr><td>http_2xx</td><td>searchapp1.staging.svc:9001</td><td><strong>Failure</strong></td><td><a href='logs?id=1'>Logs</a></td></td><tr><td>http_2xx</td><td>searchapp1.staging.svc:9001</td><td><strong>Failure</strong></td><td><a href='logs?id=0'>Logs</a></td></td></table></body>
K8s ConfigMAp is like:
- job_name: 'kubernetes-services'
metrics_path: /metrics
params:
module: [http_2xx]
kubernetes_sd_configs:
- role: service
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
action: keep
regex: true
- source_labels: [__address__]
target_label: __param_target
- target_label: __address__
replacement: '136.225.241.11:32466'
- source_labels: [__param_target]
target_label: instance
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
target_label: kubernetes_name
But my Client Application is getting probe_success value as 0
probe_success{instance="blackbox.monitoring.svc:80",job="kubernetes-services",kubernetes_name="blackbox",kubernetes_namespace="monitoring",name="blackbox"} 1
probe_success{instance="searchapp1.staging.svc:9001",job="kubernetes-services",kubernetes_name="searchapp1",kubernetes_namespace="staging",version="1.0"} 0
My test application Service yaml:
kind: Service
apiVersion: v1
metadata:
name: searchapp1
namespace: staging
labels:
version: "1.0"
annotations:
prometheus.io/probe: 'true'
prometheus.io/scrape: 'true'
prometheus.io/path: '/metrics'
spec:
type: NodePort
selector:
app: searchapp1
ports:
- name: http
protocol: TCP
port: 9001
targetPort: 9001
nodePort: 32498
I am not sure why my probe success is not 1. Could you please help me with this
Message has been deleted
Message has been deleted
kun...@shenzhou-ucar.com
Jan 5, 2018, 4:43:17 AM1/5/18
to Prometheus Users
try to debug it with cmdline, like that:
Ben Kochie
Jan 5, 2018, 11:45:07 AM1/5/18
to kun...@shenzhou-ucar.com, Prometheus Users
Try with &debug=true
On Fri, Jan 5, 2018 at 10:43 AM, <kun...@shenzhou-ucar.com> wrote:
try to debug it with cmdline, like that:
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/06990111-b6af-466a-bcdb-2acdbdf5587f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
vteja...@gmail.com
Jan 7, 2018, 4:18:43 PM1/7/18
to Prometheus Users
Hi,
It is working with one modification:
Instead of target=ABCservice.XYZnamespace.svc:9001 If I relabel it to target=ABCservice.XYZnamespace.svc:9001/metrics
for the base URL address, my application doesn't respond. It responds only for /metrics endpoint.
Just curious to know, how this can be handled efficiently here. Not all the applications give some output to GET method on base address(ABCservice.XYZnamespace.svc:9001)
Also, I didn't get the point why my blackbox target address is taken as blackbox:80 by default
Many Thanks :)
On Friday, January 5, 2018 at 5:45:07 PM UTC+1, Ben Kochie wrote:
Try with &debug=true
On Fri, Jan 5, 2018 at 10:43 AM, <kun...@shenzhou-ucar.com> wrote:
try to debug it with cmdline, like that:
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
kun...@shenzhou-ucar.com
Jan 7, 2018, 10:26:12 PM1/7/18
to Prometheus Users
Just curious to know, how this can be handled efficiently here. Not all the applications give some output to GET method on base address(ABCservice.XYZnamespace.svc:9001)
if it doesn't suport http GET method at all, it should be probed by tcp module. also you can use relabel to handle this:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
regex: (.+):(?:\d+);(.+)
replacement: $1:$2
target_label: __param_target
- source_labels: [__param_target, __meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
regex: (.+);(.+)
replacement: ${1}/${2}
target_label: __param_target
Then , in your service yaml define something like:
metadata
annotations:
prometheus.io/probe: "true"
prometheus.io/port: "9001"
prometheus.io/path: "/metrics"
Also, I didn't get the point why my blackbox target address is taken as blackbox:80 by default
it's should come from the service yaml, check it with
kubectl get svc -nmonitoring
vteja...@gmail.com
Jan 7, 2018, 11:14:26 PM1/7/18
to Prometheus Users
Hi,
Thanks for the answer.
HTTP GET is not supported on the base URL instead GET on baseURL/metrics is working.
In this case do I need relabel config or probe on tcp port?
BR,
Teja
kun...@shenzhou-ucar.com
Jan 8, 2018, 12:59:19 AM1/8/18
to Prometheus Users
You can handle this with relabel.
Reply all
Reply to author
Forward
0 new messages