Spreading single alertmanager cluster nodes over multiple geographical regions

[Al]

The alertmanager documentation states that each Prometheus instance should send the alerts to each AM instance in a cluster:

https://github.com/prometheus/alertmanager/blob/main/README.md#high-availability

but from what I can see, these no explicit mention of distributing nodes over a large geographical region (WAN instead of LAN)

Brian Candler also mentions in this post that we shouldn't attempt any gossip or other network communications across regions :

https://groups.google.com/g/prometheus-users/c/vyHn-727Vp0

Unfortunately I can't seem to find any documentation clearly stating that an alertmanager cluster spread over multiple regions (for example, 2 nodes in a DC in North America and 2 other nodes in a DC in Europe) will not work due to specific reasons.  If ia relatively high speed network exists between birth regions and t's acceptable to potentially have a slightly higher latency, wouldn't it be feasible to have a cluster distributed this way?   Considering the eventually consistent nature of Gossip, why doesn't this type of AM cluster more common?  I understand that the added latency could potentially lead to duplicate alerts being sent to the destination receiver, but given receivers such ss victorops would have the incident triggered with the same ID, these should be essentially unaffected based on my understanding?

The main purpose of this kind of configuration would be to adress the following :

- have a single cluster to which silences to be managed 

- to ensure global redundancy if one region should become unavailable 

I would appreciate any feedback or advice on this topic.

Thank you 

[Brian Candler]

On Thursday, 27 February 2025 at 13:44:07 UTC Al wrote:

The main purpose of this kind of configuration would be to adress the following :

- have a single cluster to which silences to be managed 

Another way to address this is to have a front-end which manages all the alertmanager clusters. karma is one, alerta.io is another. You can manage silences globally from one panel (at least, I know karma can; I have not tried alerta).

 

- to ensure global redundancy if one region should become unavailable 

If the region has completely failed, then presumably there's nothing within that region that is worth alerting on anyway. You can monitor the alertmanager cluster in one region from a prometheus in another region, to get an alert of that failure mode.

However, the simplest solution would be to have a single alertmanager cluster, spread across AZs in a single region; all the other prometheuses send their alerts to this cluster. Alerting is low traffic and I don't see a particular reason to have a separate alertmanager cluster in every region.  You can test that you can reach prometheus in every other region, and then you have high confidence that prometheus in those regions will be able to contact the central alertmanager.

[hartfordfive]

Thank you for the quick reply Brian, much appreciated!

 

If the region has completely failed, then presumably there's nothing within that region that is worth alerting on anyway. You can monitor the alertmanager cluster in one region from a prometheus in another region, to get an alert of that failure mode.

That's assuming that everything within that region is actually down.   In some cases, some companies may have private network links (usually redundant) they use for network communication between regions.  In the situation both redundant links to and from a given region go down,  all systems and services within the region itself may still be fully functional.   They just won't be visible to users who are located in another region (think an office location in America and an office location in Europe).  In this case, we still want to continue monitoring all services in each region.   Considering receivers such as Victorops will then send their payloads to an internet facing address, in theory this should continue working even in the case where a company's internal private cross region network link goes down.

 

However, the simplest solution would be to have a single alertmanager cluster, spread across AZs in a single region; all the other prometheuses send their alerts to this cluster. Alerting is low traffic and I don't see a particular reason to have a separate alertmanager cluster in every region.  You can test that you can reach prometheus in every other region, and then you have high confidence that prometheus in those regions will be able to contact the central alertmanager.

In this context, i'm mostly referring to  a setup where a company may be managing all of it's infrastructures across multiple private datacenters.   With this approach, multiple AZ which are typically each hosted within a single DC, still run the risk of being inaccessible should the link to the DC go down.   So let's say you have datacenters in 3 regions (AMER, EMEA and APAC) and you've chosen to have a single AM cluster in EMEA, should the link between AMER and EMEA and/or EMEA and APAC go down , then Prometheus instances located in AMER or APAC won't be able to send alert notifications.   If you instead of 2 or 3 alertmanager instances in each of these regions, wouldn't that still allow alerts to be received and actioned within each of those regions?    

I realize the other option could be just to have 3 separate AM clusters in each region and have all Prometheus instances in each region send to all existing 6 to 9 Alertmanager servers (2 or 3 in of AMER, EMEA and APAC regions).   I realize we could centralize silence management with something like Karma or alerta.io and have a single-pane of glass view on the global list of alerts although I'm just trying to see if we can eliminate a 3rd party application to accomplish this and stick with a single globally-distributed alertmanager cluster?

[Brian Candler]

On Thursday, 27 February 2025 at 15:37:54 UTC hartfordfive wrote:

With this approach, multiple AZ which are typically each hosted within a single DC, still run the risk of being inaccessible should the link to the DC go down.   So let's say you have datacenters in 3 regions (AMER, EMEA and APAC) and you've chosen to have a single AM cluster in EMEA, should the link between AMER and EMEA and/or EMEA and APAC go down , then Prometheus instances located in AMER or APAC won't be able to send alert notifications.   If you instead of 2 or 3 alertmanager instances in each of these regions, wouldn't that still allow alerts to be received and actioned within each of those regions?    

Only you know what the meaningful failure modes are for your environment. It seems to me that you expect key DC-to-DC connectivity to go down, but you are still able to send alerts (presumably via Internet or some other out-of-band means).  You could get Prometheus to talk to alertmanager over the Internet too, using https, if you felt that was more reliable.

Also, if DC-to-DC communication is unreliable, then personally I would not want to run any sort of distributed application across it (alertmanager or otherwise), due to problems with partitioning / split brain.

However, you need to make your own call as to what works best for you, and what is the optimum tradeoff between cost, complexity, and reliability.  My gut feeling is towards simplicity and reliability, which for me means either a single global alertmanager cluster, or a separate AM cluster per region, but you can build whatever you're comfortable with.

[Ben Kochie]

Part of the Prometheus/Alertmanager design is to better survive WAN split-brain.

IMO, running a wide Alertmanager cluster is a good idea when you have a wide network. The AM gossip protocol and deduplication is designed to fail open in the event of a split brain.

The only thing you have to be aware of is that Prometheus-to-Alertmanager is an all-all communication. All Prometheus instances need to send to all Alertmanagers.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/ec7b1e1f-d1af-4e0c-ad59-1f238e661737n%40googlegroups.com.

[hartfordfive]

Great, thank you for the feedback.   Should any of the flags be set to custom values when deploying across a wide WAN or should the default values still suffice? 

[Ben Kochie]

I don't think any flag changes are needed.

To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/e0d30be0-0dfb-421a-a457-ebef81b4d1d9n%40googlegroups.com.