Best practice for gathering metrics from client sites

[Patrick Macdonald]

This might be more of a general networking question than a Prometheus question, but I thought I'd ask anyway.

If we have a Prometheus server on site-A and an arbitrary number of client sites, each with hardware we might want monitor, is there best practice for how to achieve this? 
I'm assuming pushgateway isn't the correct use-case here?  Is the only way to create a tunnel in to each client site so that the Prometheus server can access each target directly? 
I'm hoping there might be some way to achieve this without much work on the client IT side. 

Thanks

p.
 

[Stuart Clark]

Generally you'd want Prometheus servers at each client site. The reason
is that you want to be able to monitor the end devices/services
themselves, rather than also be monitoring/affected by the network. For
example, if the connection is down you'd lose all visibility for the
whole period, even once the connection is back.

Another useful tool is Push Prox:
https://github.com/prometheus-community/PushProx

But you are otherwise correct that the Push Gateway isn't something that
you should be using at all for this use case (unless you also have short
lived cron style jobs that you are wanting to monitor).

--
Stuart Clark

[Christian Hoffmann]

Hi,

On 2020-12-14 11:24, Patrick Macdonald wrote:
> If we have a Prometheus server on site-A and an arbitrary number of
> client sites, each with hardware we might want monitor, is there best
> practice for how to achieve this?
> I'm assuming pushgateway isn't the correct use-case here?  Is the only
> way to create a tunnel in to each client site so that the Prometheus
> server can access each target directly?
> I'm hoping there might be some way to achieve this without much work on
> the client IT side.

Do you have SSH access? Then sshified may help for creating dynamical
tunnels:

https://github.com/hoffie/sshified

Kind regards,
Christian

[Ben Kochie]

I agree, the best practice here is to have Prometheus as close to the targets as possible. This way any WAN links, VPNs, or proxies are not monitored as a side effect.

The remote write feature is designed for this. It can stream data to a central service like Cortex or Thanos. Or a hosted service like Grafana Cloud.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a1ff2424c9506887e9bb3cf29a77dd15%40Jahingo.com.