Setting up a Prometheus environment with a push mechanism

[robbe vaes]

Hi,

I am trying to setup a monitoring environment with Prometheus, but it has to be using a push mechanism instead of the standard pull mechanism Prometheus uses. I was wondering what options there are to create an environment like this. It would also have to perfom data deduplication. The main issue is that I don't want Prometheus to scrape the clients itself, but rather that it scrapes a certain location for all the metrics and that the clients push their metrics to that location automatically.

Suggestions are very welcome!

[Ben Kochie]

Can you describe more about what your network topology is exactly? There are a number of solutions for dealing with distributed monitoring.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com.

[robbe vaes]

Okay so, we want to have an environment using Prometheus, where we can monitor our servers etc with a push method rather then pull due to network security aspects. As of now, we managed to set up collectd together with collectd exporter for prometheus. This way we can have the clients or servers push their data to the exporter and have prometheus scrape the data from the exporter to minimize the amount of pulling that has to be done. The issue now is, we want to have multiple collectd exporters to improve HA, but the problem now is that in prometheus, when scraping both exporters, it takes the data from both. It shows for example for server A the cpu load but it shows it twice instead of only once because its being pulled from both exporters. We want to have something like this but it needs to have a form of deduplication so we dont have the same data twice.

Op donderdag 25 maart 2021 om 13:27:44 UTC+1 schreef sup...@gmail.com:

[Julien Pivotto]

The way this is usually solved is by duplicating prometheus - it seems
that now you have moved the SPOF from the exporter to prometheus.

Regards,

> >> <https://groups.google.com/d/msgid/prometheus-users/44d081b2-90c1-451f-ac94-efb143fdd0c0n%40googlegroups.com?utm_medium=email&utm_source=footer>
> >> .

> >>
> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3325df03-3ad7-4d24-be81-f0e6c4c5b07fn%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

[robbe vaes]

The thing is we can implement multiple prometheus instances as well, thats no issue and will probably happen anyway. The thing is, we tried using Thanos to manage multiple prometheus servers and do deduplication, but the deduplication does not work for the collectd exporters. The problem its having i think is that we need to filter out duplicate data by exported instance, but this is not possible for Thanos since it needs predefined external labels within prometheus. 

Regards,

Op donderdag 25 maart 2021 om 14:15:24 UTC+1 schreef Julien Pivotto:

[Ben Kochie]

Deduplication works fine with Thanos, you need to make sure you setup replica labels in your Prometheus external labels.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com.

[Ben Kochie]

If you're monitoring within a single network, but can't make any connections to your targets, PushProx might be a good idea

https://github.com/prometheus-community/PushProx

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3325df03-3ad7-4d24-be81-f0e6c4c5b07fn%40googlegroups.com.

[Aliaksandr Valialkin]

Probably VictoriaMetrics could help with this setup. It supports data push via various protocols, it supports data deduplication and it is compatible with Prometheus datasource in Grafana.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/634b223b-467e-4356-a556-513675d79b9fn%40googlegroups.com.

--

Best Regards,

Aliaksandr Valialkin, CTO VictoriaMetrics

[robbe vaes]

It is correct that Thanos' deduplication works when using just 2 or more Prometheus servers. The thing is, Thanos cannot use existing labels to do deduplication. I want it to deduplicate using the label called exported_instance, but i cannot predefine this label in the external labels for thanos because it is not from prometheus but from the collectd exporter. We are now looking into Prometheus Proxy, since we already tried PushProx and it did not do what we wanted it to do. VictoriaMetrics might be interesting to look into as well!

Regards,

Robbe

Op donderdag 25 maart 2021 om 14:25:17 UTC+1 schreef val...@gmail.com:

[Julius Volz]

What should be noted when bringing up VictoriaMetrics in the Prometheus context is that it is deliberately incompatible with Prometheus in multiple ways:

- VM's MetricsQL behaves differently from PromQL in a multitude of ways and is *not* backwards-compatible with PromQL (see also https://promlabs.com/promql-compliance-tests).

- The VMAgent does remote_write without the necessary pre-processing of series staleness handling, and it throws away datapoints like special float values (like NaN). At least last time I checked.

- The TSDB has lossy compression, so it throws away part of your data even further. Prometheus and Thanos store and retrieve float sample values exactly as sent, which depending on the use case, may matter.

All that is to say that you can use VictoriaMetrics, but you are "on your own" then in terms of proper Prometheus ecosystem support and compatibility.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9f28afd6-2516-4329-b8df-dc08d5bb84f9n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[robbe vaes]

Alright, did not know all of that. We are now setting up the Prometheus Proxy and are hoping that it does what we want it to do. VictoriaMetrics would be a last case scenario anyway

Op donderdag 25 maart 2021 om 14:54:22 UTC+1 schreef juliu...@promlabs.com:

[Ben Kochie]

Another option if your security will allow opening one port to Prometheus data, there is the "exporter_exporter"

https://github.com/QubitProducts/exporter_exporter

On the subject of external labels, you should have those anyway on your Prometheus instances otherwise you can't identify them properly.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e717b6cf-775c-4662-a1d4-e45157805a1bn%40googlegroups.com.

[Aliaksandr Valialkin]

On Thu, Mar 25, 2021 at 3:54 PM Julius Volz <juliu...@promlabs.com> wrote:

What should be noted when bringing up VictoriaMetrics in the Prometheus context is that it is deliberately incompatible with Prometheus in multiple ways:

- VM's MetricsQL behaves differently from PromQL in a multitude of ways and is *not* backwards-compatible with PromQL (see also https://promlabs.com/promql-compliance-tests).

MetricsQL implements some functions differently compared to PromQL, because users struggle with the existing behavior in PromQL. This is explained in the beginning of https://victoriametrics.github.io/MetricsQL.html .

 

- The VMAgent does remote_write without the necessary pre-processing of series staleness handling, and it throws away datapoints like special float values (like NaN). At least last time I checked.

vmagent proxies all the incoming floating-point values, including infinity values and NaN values.

 

- The TSDB has lossy compression, so it throws away part of your data even further. Prometheus and Thanos store and retrieve float sample values exactly as sent, which depending on the use case, may matter.

These concerns are addressed at https://valyala.medium.com/evaluating-performance-and-correctness-victoriametrics-response-e27315627e87 .

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5x%2BM9i%3DuDS45tGgVSmERv8fjD56p0qj4VSM2dQf6mEoYg%40mail.gmail.com.

[Julius Volz]

On Thu, Mar 25, 2021 at 3:15 PM Aliaksandr Valialkin <val...@gmail.com> wrote:

On Thu, Mar 25, 2021 at 3:54 PM Julius Volz <juliu...@promlabs.com> wrote:

What should be noted when bringing up VictoriaMetrics in the Prometheus context is that it is deliberately incompatible with Prometheus in multiple ways:

- VM's MetricsQL behaves differently from PromQL in a multitude of ways and is *not* backwards-compatible with PromQL (see also https://promlabs.com/promql-compliance-tests).

MetricsQL implements some functions differently compared to PromQL, because users struggle with the existing behavior in PromQL. This is explained in the beginning of https://victoriametrics.github.io/MetricsQL.html .

 

- The VMAgent does remote_write without the necessary pre-processing of series staleness handling, and it throws away datapoints like special float values (like NaN). At least last time I checked.

vmagent proxies all the incoming floating-point values, including infinity values and NaN values.

Right, thanks for the correction. I remembered this incorrectly / confused it with the inability to do staleness handling, which is also done via a special NaN sample value (so staleness-related series gaps are missing in VM).

[Julius Volz]

Regarding the other points: however you judge the differences, they are still significant departures from the rest of the Prometheus ecosystem (including compatible third-party providers). It's ok to build something incompatible, it's just IMO not ok to superficially make it sound everywhere like it's compatible with Prometheus (like in your email earlier in this thread saying "compatible with Prometheus datasource in Grafana"), while only the fine print points out that it is not. This also leaves surprises up to the users. I think it's doubly problematic to use Prometheus community forums to push for deliberately incompatible solutions in this way.

On Thu, Mar 25, 2021 at 3:15 PM Aliaksandr Valialkin <val...@gmail.com> wrote: