Add additional label to a vector v with no labels

[Sayan Das]

Hi,
I have multiple prometheus instances sending alerts to one alertmanager, which deduplicates the alerts. To ensure alerting pipeline between each prometheus and alertmanager is working properly, I want to send a always active alert to the alertmanager. I am configuring a alert rule in each prometheus which would always trigger. I have rule:

      - alert: WatchDog

        annotations:

          message: 'This alert should always be active in alertmanager, to ensure the prometheus-alertmanager integration is functional. We should be notified immediately if this alert is not active'

        expr: vector(1)

        labels:

          severity: none 

The tricky part is how do we differentiate this alert in alertmanager, since all prometheuses send the same alert. Is there a way to add labels to a vector v before sending it to the alert manager. I saw prometheus has alert_relabel_configs, but how do we use it to add labels to a vector v has no labels ? Can I somehow use label_replace() or label_join() functions for this use case?

Regards,

Sayan

[Brian Brazil]

Each Prometheus should have distinct external_labels, which will be added to all alerts to indicate they're from.

--

Brian Brazil

www.robustperception.io

[Sayan Das]

Hi,
I have multiple prometheus instances sending alerts to one alertmanager, which deduplicates the alerts. To ensure alerting pipeline between each prometheus and alertmanager is working properly, I want to send a always active alert to the alertmanager. I am configuring a alert rule in each prometheus which would always trigger. I have rule:

      - alert: WatchDog

        annotations:

          message: 'This alert should always be active in alertmanager, to ensure the prometheus-alertmanager integration is functional. We should be notified immediately if this alert is not active'

        expr: vector(1)

        labels:

          severity: none 

The tricky part is how do we differentiate this alert in alertmanager, since all prometheuses send the same alert. Is there a way to add labels to a vector v before sending it to the alert manager. I saw prometheus has alert_relabel_configs, but how do we use it to add labels to a vector v has no labels ? Can I somehow use label_replace() or label_join() functions for this use case?

Regards,

Sayan

[Sayan Das]

Please ignore this. Brian has answered me.