Team Based Alerting for alert manager

74 views
Skip to first unread message

Anup Tripathi

unread,
Jan 25, 2021, 7:06:55 AM1/25/21
to Prometheus Users
Hi team,

i want to send alert A to Team (T1 and T2) then alert B to team (T1 and T3) then alert C to team (T1 and T4). So, I have to send all alert to team T1 and rest on the basis of requirement

Julius Volz

unread,
Jan 25, 2021, 9:44:47 AM1/25/21
to Anup Tripathi, Prometheus Users
Hi Anup,

You could use the "continue" flag in your Alertmanager config to first send everything to T1, then continue on to the rest of your routes. See https://www.robustperception.io/sending-alert-notifications-to-multiple-destinations.

Regards,
Julius

On Mon, Jan 25, 2021 at 1:06 PM Anup Tripathi <imanupt...@gmail.com> wrote:
Hi team,

i want to send alert A to Team (T1 and T2) then alert B to team (T1 and T3) then alert C to team (T1 and T4). So, I have to send all alert to team T1 and rest on the basis of requirement

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/21df88e7-38a4-4bb9-81a7-07035bbd7574n%40googlegroups.com.


--
Julius Volz
PromLabs - promlabs.com

Remya Ramachandran

unread,
Jan 26, 2021, 12:00:03 AM1/26/21
to Prometheus Users
Hi Julius, 

I have used 'continue' in my configuration, however it is matching only the first match_re entry in my case and not considering the next set of entries and just route it to my default destination.  Is there anything wrong with the below configuration ?

global:

  smtp_smarthost: 'localhost:25'

  smtp_from: 'critica...@xxx.com'

  smtp_require_tls: false

route:

  group_by: ['instance', 'alert']

  group_wait: 30s

  group_interval: 5m

  repeat_interval: 3h

  receiver: 'UX team'

  routes:

  - match_re:

       job: ^(TX Windows Servers|ECD Windows Set 1|ECD Windows Set 2)$

    receiver: 'TX team'

    repeat_interval: '5h'

    continue: true

  - match_re:

       job: ^(CX Servers1|CX Servers2)$

    receiver: 'Windows Team'

    repeat_interval: '5h'

receivers:

  - name: 'Windows Team'

    email_configs:

      - to: 'a...@xxx.com , d...@xxx.com'

        send_resolved: true

  - name: 'UX team'

    email_configs:

      - to: 'p...@xxx.com , x...@xxx.com'

        send_resolved: true

  - name: 'TX team'

    email_configs:

      - to: 'cb...@xxx.com , d...@xxx.com'

        send_resolved: true


Thanks, 
Remya 

Julius Volz

unread,
Jan 26, 2021, 6:45:44 AM1/26/21
to Remya Ramachandran, Prometheus Users
Hi Remya,

I don't see anything immediately wrong in that configuration (although the child route with "continue" doesn't have any label-matching overlap with the following route yet, as was your initial problem statement). I guess we'd have to see your actual alerts and their labels to debug further why the second route doesn't seem to match anything.

Regards,
Julius

Reply all
Reply to author
Forward
0 new messages