Prometheus - Push Metrics - Multiple Datacenters

[Rafael Lima Joia]

Hi, I'm working on a project that developed a solution (including several services/containers) that run on Docker (Swarm).

This solution is being installed in several third part private datacenters, with firewall rules to avoid inbound connections (Inbound: traffic initiate from external). In other words, none, from outside, can started a connection with the containers, but the opposite is possible.

I created a docker monitoring solution (based on Prometheus, cAdvisor, Node-Exporter, Dockerd-Exporter and Grafana) to each datacenter to have the alternative to monitor locally the Docker Environment.

Now, we need to have a consolidated vision of all environments that solution is running for. For example, we need to know the state of each node from each datacenter. This consolidated vision will run only in our datacenter. The main objective is give to us some idea about the use of the solution on these datacenters.

I studied some subjects, like "Federate Prometheus", "Grafana Multiple Datasources" and "Prometheus PushGateway", but I didn't find a way (or a solution) to admit a "push approach". On other words, a way to receive the data or metrics from each datacenters and store in our Prometheus or Grafana, without violate the firewall rules from datacenters of our clients. This requisite necessarily require that the "leaf" nodes start a connection with our environment.

Anyone have some idea to resolve this problem?

Thanks,

[Jakub Jakubik]

Try thanos.io - push data from the local prometei (prometheuses?) to s3 and use thanos to get a global view.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bb108f7d-4b43-4df8-97f0-eec3b4d0dd1a%40googlegroups.com.

--

Kuba Jakubik SRE Tech Lead Netguru - Building software for world changers

jakub....@netguru.com netguru.com

[Stuart Clark]

You could look at one of the central metrics storage systems which use the remote write system, such as Thanos or Cortex

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Aliaksandr Valialkin]

On Tue, Nov 12, 2019 at 5:21 PM Jakub Jakubik <jakub....@netguru.com> wrote:

Try thanos.io - push data from the local prometei (prometheuses?) to s3 and use thanos to get a global view.

How to connect Thanos Query to Thanos Sidecars in private datacenters in order to get global query view? Otherwise Thanos Query won't have access to last 2 hours of scraped data, which isn't uploaded to S3 yet.

I'd recommend setting up Prometheus instances to write to remote storage. Prometheus instantly replicates scraped data to the configured remote storage, so it can be queried later from there.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAD5xb_E2bQZbR_7ca7Xqf04nukwG-Jf_-eSQeWHjgeXkjFvbfA%40mail.gmail.com.

--

Best Regards,

Aliaksandr

[Jakub Jakubik]

well you can do some magic with reverse tunnels, stunnel for example to access the last 2h :) but I might be overcomplicating things...

[Rafael Lima Joia]

So, someone heard about this project?

https://github.com/openshift/telemeter/blob/master/README.md

I think it can be useful.

[]’,

Rafael Lima Joia

[Rafael Lima Joia]

For clarification about Telemeter (see below). 

I think the best idea is like Aliaksandr Valialkin said.

Thanks everybody,

---------- Forwarded message ---------
De: Sergiusz Urbaniak <sergiusz...@gmail.com>
Date: qui, 14 de nov de 2019 às 15:09
Subject: Re: [prometheus-users] Prometheus - Push Metrics - Multiple Datacenters
To: Rafael Lima Joia <rlj...@gmail.com>

Hey Rafael,

Telemeter is a part of OpenShift monitoring and not meant for general use.

Best,

Sergiusz

On Wed, Nov 13, 2019 at 6:53 PM Rafael Lima Joia <rlj...@gmail.com> wrote:

Hi Sergiusz,

I'm contacting you because I saw that you are a great contributor for the project "Telemeter". I'm discussing with other developers a challenge that I'm having in project that I'm working for. I had some doubts about the scope of Telemeter:

   - Is it a generic and standalone project that can be used to implement a prometheus federation using PUSH or is a component from Openshift project?

   - How can I setup Telemeter? The GitHub instructions are too generic. For  example, I didn´t find a way to see the telemeter in action.

  

Thanks for your time,

Rafael Lima Joia

--

Rafael Lima Joia

--

Rafael Lima Joia