How to add label for alert sent to specified Alertmanager
843 views
Skip to first unread message
Morven Cao
May 13, 2021, 3:29:33 AM5/13/21
to Prometheus Users
I have a case in which one prometheus configured with two Alertmanagers(A and B).
I want to add additional label for alert send to Alertmanager A, but not for alert sent to Alertmanager B.
I know I can add external_labels to prometheus configuration, but the label wil be added to alerts for both Alertmanagers.
Maybe alert_relabel_configs helps, but I don't know how to distinguish the two Alertmanagers in alert_relabel_configs.
Appreciate it if someone can help on this.
I want to add additional label for alert send to Alertmanager A, but not for alert sent to Alertmanager B.
I know I can add external_labels to prometheus configuration, but the label wil be added to alerts for both Alertmanagers.
Maybe alert_relabel_configs helps, but I don't know how to distinguish the two Alertmanagers in alert_relabel_configs.
Appreciate it if someone can help on this.
Brian Brazil
May 13, 2021, 3:34:21 AM5/13/21
to Morven Cao, Prometheus Users
On Thu, 13 May 2021 at 08:29, Morven Cao <lc...@redhat.com> wrote:
I have a case in which one prometheus configured with two Alertmanagers(A and B).
I want to add additional label for alert send to Alertmanager A, but not for alert sent to Alertmanager B.
Can you explain more about what you're trying to do with your setup?
Things are designed for all the alertmanagers you are sending to to be part of the same cluster, and thus it's not considered a failure if you can only talk to one alertmanager. Thus the setup you describe is unsafe. I'd recommend combining these alertmanagers into one.
Brian
I know I can add external_labels to prometheus configuration, but the label wil be added to alerts for both Alertmanagers.
Maybe alert_relabel_configs helps, but I don't know how to distinguish the two Alertmanagers in alert_relabel_configs.
Appreciate it if someone can help on this.
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3a9acc69-08d8-4655-a15d-30f5e8bdb6d0n%40googlegroups.com.
Brian Brazil
Morven Cao
May 13, 2021, 5:24:14 AM5/13/21
to Prometheus Users
Thanks Brian,
Let me try to explain.
I setup one prometheus with two Alertmanagers in my k8s cluster, one of the two Alertmanagers is in-cluster and configured by dynamic discovery,
while another Alertmanager is out of the cluster and it is statically configured via remote URL.
What I'm try to do is adding additional label, for example, cluster=<cluster-name> to alerts sent to remote Alertmanager, but I don't want to add the label
for the same alert sent to the in-cluster Alertmanager.
Is it possible?
Let me try to explain.
I setup one prometheus with two Alertmanagers in my k8s cluster, one of the two Alertmanagers is in-cluster and configured by dynamic discovery,
while another Alertmanager is out of the cluster and it is statically configured via remote URL.
What I'm try to do is adding additional label, for example, cluster=<cluster-name> to alerts sent to remote Alertmanager, but I don't want to add the label
for the same alert sent to the in-cluster Alertmanager.
Is it possible?
Brian Brazil
May 13, 2021, 6:32:49 AM5/13/21
to Morven Cao, Prometheus Users
No, that's not possible. You want all alertmanagers to get the same labels, otherwise you'll get duplicate notifications. I'd recommend setting the cluster label on all alerts, generally via external_labels.
Brian Brazil
Morven Cao
May 13, 2021, 6:37:41 AM5/13/21
to Prometheus Users
thanks Brian, for the information.
I will check if setting the cluster label on all alerts is acceptable to our team.
I will check if setting the cluster label on all alerts is acceptable to our team.
Reply all
Reply to author
Forward
0 new messages