Blackbox Scrap - multi target pattern - get ip:9115 from K8S pod
190 views
Skip to first unread message
SebV
Dec 11, 2022, 12:50:38 PM12/11/22
to Prometheus Users
Hi,
I've been pulling my hair out for hours !
I try to probe some IPs with ICMP , I try to build a URL like :
Here is my scrap:
- job_name: blackbox-icmp1
kubernetes_sd_configs:
- namespaces:
names:
role: pod
selectors:
role: pod
metrics_path: /probe
params:
module: [icmp]
static_configs:
- targets:
- 10.129.100.213
- 10.129.100.214
- 10.129.100.215
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- action: replace
source_labels: [__meta_kubernetes_pod_ip,__meta_kubernetes_pod_container_port_number]
separator: ":"
target_label: __address__
The targets I can see in prom gui:
It's a completely mess, the endpoint http://10.233.65.239:9115/probe is well formatted, it is the pod ip , but the target it's himself, it should be :
- 10.129.100.213
- 10.129.100.214
- 10.129.100.215
if I set the final target ip like explained on blackbox git, it works: #https://github.com/prometheus/blackbox_exporter:
- target_label: __address__
replacement: 127.0.0.1:9115 # The blackbox exporter's real hostname:port.
what am I missing ?
Thx
Brian Candler
Dec 11, 2022, 1:20:51 PM12/11/22
to Prometheus Users
On Sunday, 11 December 2022 at 17:50:38 UTC SebV wrote:
what am I missing ?
Nothing is setting the labels __meta_kubernetes_pod_ip or __meta_kubernetes_pod_container_port_number. Therefore, the target __address__ you are constructing is simply ":" - which is invalid, as the error says.
This in turn is because the service discovery mechanism you've chosen to use is static_configs. This only sets the __address__ label to the values you give, and nothing else.
I think you first need to be clear about which address to send the scrape to (which is the final value of __address__ after relabeling, and will be the address:port where blackbox_exporter is listening), and the address you want blackbox_exporter to probe (which is __param_target). What are 10.129.100.213, 10.129.100.214, 10.129.100.215? Are these three instances of blackbox_exporter, or three targets that you want blackbox_exporter to probe?
If you want the to use the labels __meta_kubernetes_pod_ip and __meta_kubernetes_pod_container_port_number then you need to use kubernetes_sd_configs instead. You'd need to configure this block to talk to the Kubernetes API, and it will return a list of targets (e.g. pods or services, whatever you ask for) with those meta labels, from which you can construct your scraping targets. This might make sense if what you're trying to do is to scrape or probe each pod. It could also be used to find your blackbox_exporter pod but it would be inefficient, unless you can apply some sort of label to your blackbox_exporter service and tell the API you're only interested in services with that label.
It sounds to me, though, that what you want is simply:
- target_label: __address__
replacement: my_service_ip:9115 # The blackbox exporter's real hostname:port.
That is, create blackbox_exporter in kubernetes with an attached service, and give the service either a static port or a static IP address. Then if the pod is recreated on a different IP, the service IP:port is unchanged. (And: if you have multiple blackbox_exporter pods, the load should be distributed between them)
SebV
Dec 12, 2022, 5:08:17 AM12/12/22
to Prometheus Users
Hi Brian,
Yes, I want to probe
10.129.100.213, 10.129.100.214, 10.129.100.215, so it is not blackbox address.
And I don't want to user a network service, because my scrap must be handled by a blackbox exporter on a specific node.
This is why, it try to find a way to set dynamicly the last step :
- target_label: __address__
replacement: my blackbox pod ip on a specific node
Is there a way to mix the 2 discovery services (kubernetes and static ) ?
Thank you in advance.
Brian Candler
Dec 12, 2022, 5:49:28 AM12/12/22
to Prometheus Users
> And I don't want to user a network service, because my scrap must be handled by a blackbox exporter on a specific node.
So how are you going to identify which particular node or pod you want to talk to? If it's always the same physical node, then just scrape that node's IP address, and bind the service to a fixed port on that node. Otherwise, what are you going to do to identify the pod - search for it by name? By a specific label?
But really, this is exactly what a "service" is for in Kubernetes: to provide an external access point for a service. This would allow k8s to move that pod to a different node, and keep it accessible on the same IP/port. Your blackbox_exporter pod definitely provides a "service" like this, in my opinion. A k8s Service will identify the pod in the same way as you'd do manually anyway (e.g. by linking to a pod with a particular label). All it does then is forward the traffic to that pod.
> Is there a way to mix the 2 discovery services (kubernetes and static ) ?
Not really. In label rewriting rules, a single __address__ cannot be rewritten to multiple __address__es - it's a 1:1 mapping (or 1:0 if you drop the target). Therefore, if you use kubernetes SD to select one specific pod, then that scrape job can only scrape a single blackbox target.
What I think you'd have to do is:
1. Write a separate program to determine the IP/port of the blackbox exporter. Say it finds 1.2.3.4:9115
2. From this program, write out a YAML file which merges this with a list of targets you want to probe:
- labels:
__param_target: 10.129.100.213
targets:
- labels:
__param_target: 10.129.100.214
targets:
(note: you must also use rewriting rules to copy __param_target to some other persistent label, e.g. "instance" or "target", to ensure each timeseries has a unique set of labels)
3. Use file_sd_configs to read this file
But why go to all that trouble, when you could create a single Service resource definition for your blackbox_exporter pod and be done with it?
Reply all
Reply to author
Forward
0 new messages