Problem with multiple receivers in alertmanager

[PASS IT Consulting GmbH Research]

Hello

I have the following situation: I am running several shops  on a server.
I have three groups of persons (receivers)

•  The tech admins, which should receive all alarms
• The shop team  which should see all shop related alarm
• The shop owner who should only see the results for his shop

I tried to achieve this using the following route definition

# The root route on which each incoming alert enters.

route:

  # A default receiver

  receiver: tech-admins

  # All the above attributes are inherited by all child routes and can 

  # overwritten on each.

  

  # The child route trees.

  routes:

  # All  alerts

  # are dispatched to the technical admins

  - receiver: 'tech-admins'

    continue: true

    group_wait: 10s

    match_re:

      job: '.*'

    routes: 

 

    # All shop alerts

    # are dispatched to the shop team.

    - receiver: 'shop-admins'

      group_by: [product]

      continue: true

      group_wait: 10s

      match_re:

        product: 'Shop.*'

      routes:

    

    

      # Alerts for individual shops are send to th the shop admins     

      - receiver: 'ownerTest' 

        match:

          product: 'ShopTest'

  

When I now trigger an alarm for ShopTest te shop team and the shop owner receive email but not the tech-admins

I do not understand why this happens?

Is there a way to debug or test the alarms

P.S.  I did not made the rule for the techadmins the default route, since there a contionue true is forbidden.

[Brian Candler]

There is a tool you can use to debug your alert routing tree here:

https://prometheus.io/webtools/alerting/routing-tree-editor/

The parent receiver is only called if none of the child routes match.  "Continue" only passes onto the the next route at the same level.  See https://prometheus.io/docs/alerting/latest/configuration/#route

Hence you need to move your routes to the same level.  Side note: you don't need any match_re on the tech-admins, since it matches all alerts.

Try something like this:

# The root route on which each incoming alert enters.

route:

  # A default receiver (not used)

  receiver: tech-admins

  routes:

  # All  alerts

  # are dispatched to the technical admins

  - receiver: 'tech-admins'

    continue: true

    group_wait: 10s

 

  # All shop alerts

  # are dispatched to the shop team.

  - receiver: 'shop-admins'

    group_by: [product]

    continue: true

    group_wait: 10s

    match_re:

      product: 'Shop.*'

    

  # Alerts for individual shops are send to the shop admins     

  - receiver: 'ownerTest' 

    match:

      product: 'ShopTest'

The top-level receiver will never be used, since at least one child always matches.  You could replace it with a null receiver (i.e. a receiver with no targets) if you prefer.  However it's nice to have a safety net in case something breaks in the child rules.

[PASS IT Consulting GmbH Research]

Thanks,  this did it