Prometheus external labels for the thanos causes alertmanager to raise duplicate alerts

[nesa...@gmail.com]

Hi

We have two Prometheus servers with the same configuration which are connected to two alertmanager instances.

After setting up a Thanos instance, we added an external label to each Prometheus instance (replica:0 and replica:1) for the metric deduplication in the Thanos.

But now we are receiving every alert twice, one with the label 'replica:0' from the first Prometheus and another one with the label 'replica:1' from the second Prometheus server

Is there any way to fix this problem?

[Brian Brazil]

See how to use alert_relabel_configs here: https://www.robustperception.io/high-availability-prometheus-alerting-and-notification

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/46bb5c9a-66b4-46c0-a8a9-596d065b4274n%40googlegroups.com.

--

Brian Brazil

www.robustperception.io