multiple match_re in a receiver

[Alan Miller]

I have the (generic?) use case where I want all my alert manager notifications to go to

the "default" receiver but for several specific alarms I want an additional notification sent to a 2nd receiver. Using AM version 0.26.0

This seems to work but I had to duplicate the SpecialUser since adding both match_re blocks under the same receiver results in a config error. I think this also means the emails to the SpecialUser 1 and 2 will be grouped separately which isn't what I'd want.

global:
  smtp_smarthost: 'smtp.mydomain.com:25'
  smtp_from: 'no-reply@mydomain.com'
  smtp_require_tls: false
  resolve_timeout: 5m
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'TeamNotifications'
  routes:
  - receiver: 'TeamNotifications'
    group_wait: 10s
    continue: true
  - receiver: 'SpecialUser'
    group_wait: 10s
    match_re:
      alertname: 'TargetDown'
      instance: 'host1|host2'
  - receiver: 'SpecialUser2'
    group_wait: 10s
    match_re:
      alertname: 'ProbeFailing'
      instance:  'https://site.mydomain.com/login'
    continue: true

receivers:
- name: 'Team-Notifications'
  email_configs:
  - to: '123456.my...@amer.teams.ms'
    send_resolved: true

- name: 'SpecialUser'
  email_configs:
  - to: 'Specia...@mydomain.com'
    send_resolved: true
    html: '{{ template "email.html" . }}'
- name: 'SpecialUser2'
  email_configs:
  - to: 'Specia...@mydomain.com'
    send_resolved: true
    html: '{{ template "email.html" . }}'

templates:
- '/etc/alertmanager/templates/default.tmpl'
- '/etc/alertmanager/templates/email.tmpl'

[Brian Candler]

To be pedantic those match_re blocks are within routing rules, not receivers. A routing rule doesn't have to have a "receiver" attribute at all: it can have nested routes instead.

I can't see why you have defined both SpecialUser and SpecialUser2, because Google Groups has mangled them. Are these two different destinations?  If you want two rules to send to the same receiver, then both rules can refer to the same receiver.

On the flip side, if you want one rule to send to multiple receivers, there are two ways to go about it:

(1) Make a single "receiver" with multiple "email_configs" (and/oor other types of destination)

- name: 'SpecialUsers'
  email_configs:
  - to: 'Speci...@mydomain.com'

  - to: 'Speci...@mydomain.com'

(2) Make a routing rule with multiple receivers, by having nested routes which always match:

  - match_re:

      alertname: 'TargetDown'
      instance: 'host1|host2'

    routes: [ {receiver: SpecialUser1, continue: true}, {receiver: SpecialUser2} ]

 When you say "adding both match_re blocks under the same receiver", I don't really understand what you mean. Are you trying to do an OR configuration (label A and label B *OR* label X and label Y)? If so then yes, you'll need two routing rules, but they can have the same receiver.

Incidentally, "match_re" is deprecated in favour of "matchers", which are more powerful PromQL-inspired label matchers.

[Alan Miller]

Sorry,  SpecialUser and SpecialUser2 both point to the same destination. That was just a way to get around not being able to define a set of matchers for 1 receiver.

Here's what I want.

 - ALL notifications go to my 'default' receiver. (host1,2,3,....host99 are down, sites A,B,C.... is down, etc.)

 - 3 specific notifications ALSO go to a 'secondary' receiver  (host2 is down or host3 is down or siteC is down)

Since both my "host down" and "site down" alarms each require 2 labels (alertname and instance) do I need two routing rules?

How would I do that using matchers.

[Brian Candler]

> That was just a way to get around not being able to define a set of matchers for 1 receiver.

What I'm saying is, routing rules are an ordered list of independent rules which are tested in turn, and each one can trigger the same or different receivers. For example:

  - receiver: 'SpecialUser'

    match_re:
      alertname: 'TargetDown'
      instance: 'host1|host2'

  - receiver: 'SpecialUser'

    match_re:
      alertname: 'ProbeFailing'
      instance:  'https://site.mydomain.com/login'

For each rule, the conditions are first tested. If all conditions pass (or there are no conditions), then the child "routes" (if any) are processed. If there are no child routes, or none of them match, then the "receiver" is used as the destination.

If the conditions don't all pass, or if continue: true is set, then processing continues with the next rule.

You may find it easier to read each rule if you put the matcher attribute(s) before the receiver attribute:

  - match_re:
      alertname: 'TargetDown'
      instance: 'host1|host2'

    receiver: 'SpecialUser'

  - match_re:
      ... etc

> Here's what I want.

> - ALL notifications go to my 'default' receiver. (host1,2,3,....host99 are down, sites A,B,C.... is down, etc.)

> - 3 specific notifications ALSO go to a 'secondary' receiver  (host2 is down or host3 is down or siteC is down)

There are a couple of ways to do that. Either you put the "ALL notifications" rule first (as you have done) with continue: true; or you put the "ALL notifications" rule at the end. That depends on whether you want to have the ability for individual rules to suppress the "all notifications" or not. If all rules have "continue: true" then it doesn't matter.

> Since both my "host down" and "site down" alarms each require 2 labels (alertname and instance) do I need two routing rules?

Yes.

> How would I do that using matchers

    - matchers:
        - 'alertname="TargetDown"'

        - 'instance=~"host1|host2"'

      receiver: 'SpecialUser'
    - matchers:
        - ... etc

[Alan Miller]

I tried this and according to amtool output it looks like it should reflect what I want.

  routes:
  - receiver: 'TeamNotifications'
    group_wait: 10s
    continue: true
  - receiver: 'SpecialUser'
    group_wait: 10s

    continue: true
    matchers:
      - alertname=~"TargetDown|ProbeFailing"
      - instance=~"host2|host3|siteC"

amtool test output

prometheus:/# amtool  --alertmanager.url=http://127.0.0.1:9093/ config routes show
Routing tree:
.
└── default-route  receiver: TeamNotifications
    ├── default-route  continue: true  receiver: TeamNotifications
    ├── {alertname=~"TargetDown|ProbeFailing",instance=~"host2|host3|siteC"}  continue: true  receiver: SpecialUser

prometheus:/# amtool  --alertmanager.url=http://127.0.0.1:9093/ config routes test --tree --verify.receivers=TeamNotifications,SpecialUser  alertname="ProbeFailing" instance="siteC"

Matching routes:
.
└── default-route
    ├── default-route  receiver: TeamNotifications
    └── {alertname=~"TargetDown|ProbeFailing",instance=~"host2|host3|siteC"}  receiver: SpecialUser


TeamNotifications,SpecialUser

[Brian Candler]

    matchers:
      - alertname=~"TargetDown|ProbeFailing"
      - instance=~"host2|host3|siteC"

That matches a slightly wider set of conditions: all alerts where the instance is host2 or host3 or siteC, and the alertname is TargetDown or ProbeFailing.

For example, this would match an alert with {alertname="ProbeFailing",instance="host2"} which your original set of rules would not have matched. Of course, if no such alert is ever generated, then it makes no difference in practice.

There's also an online web-based tool you can use for testing routing rules:

https://prometheus.io/webtools/alerting/routing-tree-editor/