Why not to load balance traffic between Prometheus and its Alertmanagers?

[赵金鑫]

I'm seeing this on alertmanager's github:

Important: Do not load balance traffic between Prometheus and its Alertmanagers, but instead point Prometheus to a list of all Alertmanagers. The Alertmanager implementation expects all alerts to be sent to all Alertmanagers to ensure high availability.

Can somebody tell me why not to do this, and what would happen if a load balance traffic?

[Stuart Clark]

As it says it is important that all Alertmanagers can talk to each  other and that Prometheus keeps them all informed about alerts for the HA clustering system to properly operate. If this isn't the case you could have alerts being resolved incorrectly or notifications sent at the wrong points.

-- 
Stuart Clark

[赵金鑫]

Thanks, so if i push an alert to two alertmanagers(ex. A and B), and stop push alert to A, then i will receive a 'resolved' notification sent from A.

Is that right?

[Julien Pivotto]

On 07 Jun 20:24, 赵金鑫 wrote:
> Thanks, so if i push an alert to two alertmanagers(ex. A and B), and stop
> push alert to A, then i will receive a 'resolved' notification sent from A.
>
> Is that right?

Yes.

>
> 在2021年6月7日星期一 UTC+8 下午2:10:46<Stuart Clark> 写道：
>
> > On 07/06/2021 04:24, 赵金鑫 wrote:
> >
> > I'm seeing this on alertmanager's github:
> >

> > *Important: Do not load balance traffic between Prometheus and its

> > Alertmanagers, but instead point Prometheus to a list of all Alertmanagers.
> > The Alertmanager implementation expects all alerts to be sent to all

> > Alertmanagers to ensure high availability.*

> >
> > Can somebody tell me why not to do this, and what would happen if a load
> > balance traffic?
> >
> >
> > As it says it is important that all Alertmanagers can talk to each other
> > and that Prometheus keeps them all informed about alerts for the HA
> > clustering system to properly operate. If this isn't the case you could
> > have alerts being resolved incorrectly or notifications sent at the wrong
> > points.
> >
> > --
> > Stuart Clark
> >
> >
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b84dbc61-6098-4de9-860e-ddf5834feed5n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

[Harald Koch]

On Tue, Jun 8, 2021, at 02:25, Julien Pivotto wrote:
> On 07 Jun 20:24, 赵金鑫 wrote:
> > Thanks, so if i push an alert to two alertmanagers(ex. A and B), and stop
> > push alert to A, then i will receive a 'resolved' notification sent from A.
> >
> > Is that right?
>
>
> Yes.

That sounds like a high availability defect?

It's possible for prometheus to be up, and lose network connectivity to alertmanager A, while still sending alerts to alertmanager B. Shouldn't the alertmangers' internal clustering handle this case, and keep the alert active?

--
Harald