AlertManager notifying other AlertManager / AM hierarchy?

[Marcel May]

Hi,

has anyone tried to route an alert from one AlertManager to another one?

It would be nice if one AM could route (certain) alerts to another AM as a receiver .

I don't mean HA but different AM groups, organised like a hierarchy:

- one or more "local" Prometheus/AM setup (maintained by projects)

- one "central" / "global" AM setup (maintained by eg central monitoring group)

Motivation is that the "global" AM has a rather complicated / centrally controlled

AM integration with a (legacy) alert system, in a large company.

So routing from all the local AMs directly to the legacy alert system is not really an option.

An AM-2-AM bridge component between the AMs (eg via WebHooks) is possible I guess, but complex and error prone.

So I would rather prefer a direct AM-2-AM integration.

Are there any better options/thoughts?

Cheers,

Marcel

[Simon Pasquier]

This isn't supported out of the box. If you really need it, you'd have to write a webhook receiver that will act as a client for the global AlertManager.

Can you describe more precisely the use case? For instance, which notifications you want to route?

Without knowing more, I would recommend that all your "local" Prometheus servers send their alerts to the central AM cluster as having multiple hops reduces the reliability of the monitoring toolchain.

If a team needs to manage and receive their own notifications, it is also possible to have the Prometheus server send alerts to a local AM in addition to the central one.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/db3ba829-1b4b-45e7-a716-04e1b0bdadc6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.