Forced to use the Pushgateway as a workaround?

[a...@binoklo.com]

Hello,

I'd like to use Prometheus to monitor my (wrapped) k8s services, but unfortunately the admin won't let me connect directly to individual pods, I can just access them via load balancing.

In this case, I guess I have to use the Pushgateway. It seems to be working, however timeseries for old pods persist (the "instance" label is a random string). I am thinking of creating a program to periodically delete them from the Pushgateway.

This is not an ideal situation but I guess this is the best I can do. Or am I missing something?

Thanks,

--

Adriano

[Matthias Rampke]

If you can, deploy (a) Prometheus into the cluster itself. The easiest way to manage that is using the Prometheus operator, but if that is not possible, you can configure it directly using relabeling, as in this example[0].

This Prometheus can scrape the various targets. You have a few options from there:

You can use this directly, reaching it through the load balancer, or through a Grafana deployed to the same cluster.

Or use remote write to push to another Prometheus or other metric store. In this case you can run the in-cluster Prometheus in the pared down agent mode. This also works if you run e.g. one Prometheus per namespace.

On the more complex but full featured end, you can use Thanos to tie multiple servers in multiple clusters together with a long term store.

What is appropriate for you depends on the size of your setup, what you want monitor, and the restrictions that your admins impose. I hope this gives you some pointers to discuss with them!

Best,

Matthias

[0]: https://github.com/prometheus/prometheus/blob/main/documentation/examples/prometheus-kubernetes.yml

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e009b2af-53e3-4c43-8d3c-64b3cc29041bn%40googlegroups.com.

[a...@binoklo.com]

Hello Matthias,

Many thanks for the suggestion. I will give it a try, although I have the impression that the k8s API will also be off-limits. At the moment I just have a couple of services / pods, but it would be nice to experiment with agent-mode.

Cheers,

--
Adriano