Prometheus send alerts to different alertmanager based on alert labels

[Ben Ye]

Hi community,

We have a use case that a Prometheus instance is shared by multiple teams and each team maintains their own Alertmanager.

Now the requirement is to send alerts to different Alertmanager instances based on alert labels. For example, alerts with team=A go to Alertmanager A and alerts with team=B go to Alertmanager B.

At the beginning I wanted to use alert_relabel_configs for this requirement but then I found out that alert_relabel_configs is global to all alertmanagers. But my requirement is actually have alert_relabel_config per alertmanager destination.

There are other ways to go but are not very ideal to us:

1. Maintain separate Prometheus for each team

2. Filter the alerts label on Alertmanager routing configurations

Please let me know if there are other better ways to achieve this. Also, does it make more sense to support alert_relabel_configs per alertmanager?

Thanks and regards,

Ben Ye