PagerDuty Root fires all alerts

[Miko Tarriman]

Hi Guys,

My alertmanager works fine but not in a way I like,  all alerts are fired by the root pagerduty and it doesn't query with its child receivers. Any recommendation please!


route:

group_wait: 10s

group_by: [alertname]

group_interval: 30s

repeat_interval: 30m

receiver: "rootPageduty"

routes:

- receiver: "X"

group_wait: 10s

match_re:

alertname: "/^(?:test1)$/"

continue: true

- receiver: 'Y'

group_wait: 10s

match_re:

alertname: "/^(?:test2)$/"

continue: true

- receiver: "Z"

group_wait: 10s

match_re:

alertname: "/^(?:test3)$/"

continue: true

receivers:

- name: rootPageduty

pagerduty_configs:

- service_key: xxx

- name: X

pagerduty_configs:

- service_key: xxx

- name: Y

pagerduty_configs:

- service_key: xxx

- name: Z

pagerduty_configs:

- service_key: xxx

[George B]

You might want to try inputing the configuration file into the visualization tree and entering some label values to see where things break.  https://www.prometheus.io/webtools/alerting/routing-tree-editor/

I've found the following example very useful and what we use to for routes and child-routes: https://github.com/prometheus/alertmanager/blob/master/doc/examples/simple.yml.  In this structure you can create main routes to catch informational alerts without specifying a specific severity level label and then have warning/critical severity labels as child-routes.  Alerts that don't match that the warming/critical severity would automatically be routed to the top level route.