Prometheus agent mode HA pushing to remote write

[Navid Shakibapour]

We are evaluating using Prometheus agent mode in a highly available manner to forward metrics to a 3rd party remote storage using remote write API.

We need to have multiple shards and multiple replicas per shard. Our 3rd party remote storage does not handle deduplication on their side.

Could someone point me to different options for metrics deduplication on ingestion?

[Ben Kochie]

There is no option without using a single instance. If your 3rd party solution does not handle HA, you can not use HA.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1b274d3d-77a9-4de3-9b63-61a533b84140n%40googlegroups.com.

[Navid Shakibapour]

Is there a way to set up multiple Prometheus servers in an active/passive way? Meaning only 1 server scrapes targets at a time and other ones only scrape if the active one doesn't "work" after x seconds/minutes.

[Julien Pivotto]

You would need additional tools like corosync or keepalived to achieve
this. In prometheus terms, all the prometheis servers are active and
deduplication is done at the remote write end if needed.

> >> <https://groups.google.com/d/msgid/prometheus-users/1b274d3d-77a9-4de3-9b63-61a533b84140n%40googlegroups.com?utm_medium=email&utm_source=footer>
> >> .

> >>
> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/91c8b0d4-67e2-40d3-8314-bd58409cf937n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

[sayf.eddi...@gmail.com]

I faced this issue in the design phase too, found this https://github.com/beorn-/highlander but it looks a bit dead now...

else you can push duplicated metrics to thanos (while setting a unique label for every prometheus instance) and then configure thanos querier to deduplicate when serving metrics

[Navid Shakibapour]

> You would need additional tools like corosync or keepalived to achieve this.

Thanks for pointing me to those tools. I did a quick search and didn't see much info on setting up Prometheus HA with corosync/keepalived. Except that it looks like AWS managed Prometheus uses that approach [1]:
When you set up deduplication, Amazon Managed Service for Prometheus makes one Prometheus instance a leader replica and ingests data samples only from that replica. If the leader replica stops sending data samples to Amazon Managed Service for Prometheus for 30 seconds, Amazon Managed Service for Prometheus automatically makes another Prometheus instance a leader replica and ingests data from the new leader.

I'm a bit surprised that this approach isn't used more.

Re using multiple replicas, my concern is the cost in bigger clusters. The cost will add up to having multiple replicas per shard knowing the data would be deduplicated downstream.

[tejaswini vadlamudi]

Very good question, have the same concern as Navid. 

Any ideas on how to get this work in a DB that doesn't provide native support for de-dupe?

[Ben Kochie]

It is used all over the place. AWS managed prometheus is basically just Cortex[0], which is what Grafana Mimir[1] is based on. The solution has been around for quite a while.

[0]: https://github.com/cortexproject/cortex

[1]: https://github.com/grafana/mimir

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/01470303-ceac-4b6a-b837-463fd5440db5n%40googlegroups.com.

[Ben Kochie]

Prometheus itself doesn't provide any native deduplication because when you get to larger scale systems that need this Prometheus becomes just another small component in an overall system.

Remember, Prometheus is not a database, it's a monitoring system that just happens to have a database.

The slightly offset, duplicated data, is an intentional design choice to make it more robust against partial network outages and other degraded system scenarios. Lots of other metrics systems work great when everything is fine. Prometheus is designed to keep working with minimal network dependencies when everything is on fire.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f3f21dd9-9aed-43e1-ac24-031331efc6fen%40googlegroups.com.

[Bryan Boreham]

Grafana Agent has a "cluster mode" which will shard targets pick a single sender client-side.

This is currently in beta.

https://grafana.com/docs/agent/latest/flow/concepts/clustering/

[disclosure: I work for Grafana Labs]

Bryan