Config Check for AlertManager with Slack and PagerDuty

[Johnny Blaze]

Hi folks,

I am trying to setup AlertManager to send notifications to both Slack and PagerDuty. With my current configuration, alerts only go to Slack.

Can someone kindly review my alertmanager.yml and let me know what I'm missing?

global:
  resolve_timeout: 1m
  slack_api_url: 'https://hooks.slack.com/services/XXXXXX/XXXXX/XXXXXXXX'
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'
route:
  receiver: 'slack-notifications'
  routes:
    - match:
        severity: critical,warning
      receiver: 'slack-notifications'
      continue: true
    - match:
        severity: critical,warning
      receiver: 'pagerduty-notifications'
      continue: false
receivers:
  - name: 'slack-notifications'
    slack_configs:
    - channel: '#operation-alerts-test'
      send_resolved: true
      icon_url: https://avatars3.githubusercontent.com/u/3380462
      title: |-
        [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
        {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
          {{" "}}(
          {{- with .CommonLabels.Remove .GroupLabels.Names }}
            {{- range $index, $label := .SortedPairs -}}
              {{ if $index }}, {{ end }}
              {{- $label.Name }}="{{ $label.Value -}}"
            {{- end }}
          {{- end -}}
          )
        {{- end }}
      text: >-
        {{ range .Alerts -}}
        *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}

        *Description:* {{ .Annotations.description }}

        *Details:*
          {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
          {{ end }}
        {{ end }}
  - name: 'pagerduty-notifications'
    pagerduty_configs:
    - service_key: xxxxxxx
      send_resolved: true

Thank you and Happy Holidays

[Brian Candler]

    - match:
        severity: critical,warning

This matches only a label with name "severity" and value exactly equal to "critical,warning", e.g. it would match

myalert{severity="critical,warning"} 1

You need to use either the new matchers or the older and deprecated match_re to match multiple values of the same label.

However, you will make your life easier if you create a single receiver which sends to both slack and pagerduty:

receivers:

  - name: 'slack-and-pagerduty'

    slack_configs:

      - ...

    pagerduty_configs:
      - ...

This can end up being a bit messy, because if you also need other receivers for *only* slack or *only* pagerduty then you'll end up duplicating some of these configs.  But once you start doing selective routing, you'll find your actual routing rules get way more complicated if you have to duplicate those, with correct nested routes and "continue" statements to send one alert to multiple receivers.  (It would be nice if alertmanager allowed multiple receivers for a single rule, but unfortunately it doesn't)

One other thing I'll mention: your final "continue: false" does nothing. Firstly because it defaults to false, but secondly even "continue: true" on the last rule in a list of routes has no effect.  It only means "after successful match, continue onto the next routing rule", and if there are no more, it doesn't do anything.  The default "receiver" which exists outside all of the routing rules is only triggered if *none* of the rules match, regardless of whether continue is set.

HTH,

Brian.

[Brian Candler]

To clarify, here's how it would work with a single receiver which delivers to both slack and pagerduty:

route:
  receiver: 'slack-notifications'
  routes:
    - matchers:
        - 'severity =~ "critical|warning"'
      receiver: slack-and-pagerduty

Whereas if you want a single rule to deliver to multiple receivers, you need a nested set of routes (each of which has no matchers, so they always trigger):

route:
  receiver: 'slack-notifications'
  routes:
    - matchers:
        - 'severity =~ "critical|warning"'

      routes:

        - receiver: slack-notifications
          continue: true

        - receiver: pagerduty-notifications

You can turn this into a one-liner [untested]:

route:
  receiver: 'slack-notifications'
  routes:
    - matchers:
        - 'severity =~ "critical|warning"'

      routes: [ {receiver: slack-notifications, continue: true}, {receiver: pagerduty-notifications} ]

[Johnny Blaze]

Hi Brian,

I was able to get it to work based on your provided examples (haven't tried the one liner)

Now I just got to read up on how I should go about building more detailed routing rules :)

Lastly, I really appreciate you taking the time on getting me over the first initial bump.

Take Care,

JB

[Brian Candler]

Great.  There's an online alertmanager route tester you may find useful here:

https://prometheus.io/webtools/alerting/routing-tree-editor/

Unfortunately I don't think it has been updated to work with "matchers" yet (at least, these don't seem to work, whereas match_re does)

Also, if you want all notifications to go to slack, and certain ones also to go to pagerduty, then you can have a route with "continue: true" that sends everything to slack first:

route:
  receiver: slack-notifications    # this is never used, I think it can be omitted completely
  routes:

    - receiver: slack-notifications

      continue: true

    - matchers:
        - 'severity =~ "critical|warning"'

      receiver: pagerduty-notifications