Configure blackbox as daemon set to check multiple external services avaliability

[Skaven]

Hello, folks.

The story.

In my org we have several k8s clusters and quite unreliable security department, who has control over firewall and have a habbit of corrupting the rules on said firewall.

The confusion is immense. The issue is that at any point in time one or several nodes can lose access to one or several external resources.

So, we had a brilliant idea to mitigate the confusion by deploying a blackbox daemonset and configure Prometheus to query external resources. But! We have many-to-many relationship here. We want to query multiple resources from all of the worker nodes in cluster.

This means, that we need auto discovery (to dynamically get instances of daemon set) and multiple targets.

In docs we found an example of multi target requests though static configs. Unfortunately, as the name suggest, it is a static config and we can't get the metadata of node that lost connection or reliably determine the fact that connection was lost at all (no way to properly configure alert threshold, so it wouldn't misfire).

On the other hand - there is Kubernetes service discovery. But the only way we found is to have a separate job for each of the remote resources.

Is there, maybe, a better way to approach the problem?

[Brian Brazil]

Yes, that's the way to do it. Use the blackbox exporter as though it was a normal exporter, and then have a scrape config for each of your external resources.

Brian

 

Is there, maybe, a better way to approach the problem?

--

Brian Brazil

www.robustperception.io

[Skaven]

Thanks, Brian.

пятница, 18 декабря 2020 г. в 13:54:06 UTC+3, Brian Brazil: