Mapping alert labels/annotations to PagerDuty "details"

[tsim...@digitalocean.com]

Hello!

Within our organization, we have teams that intake alerts from a variety of different monitoring solutions via PagerDuty.

When looking at the alerts that come from Prometheus, we noticed the default templates write a pretty difficult to parse JSON body in the "details" field for a PagerDuty alert:

{

  "client": "AlertManager",

  "client_url": "http://alertmanager....",

  "description": "BlackboxTargetDown",

  "details": {

    "description": "<instance> is down\n",

    "firing": "Labels:\n - alertname = BlackboxTargetDown\n - instance = ...com/api/status\n - job = ....-blackbox\n - service = ...\n - severity = critical\n - team = eng-observability\nAnnotations:\n - description = <instance>/api/status is down\nSource: http://prometheus:9090/graph?g0.expr=probe_success+%3C+1&g0.tab=1\n",

    "num_firing": "1",

    "num_resolved": "0",

    "resolved": "",

    "severity": "critical",

    "status": "firing"

  },

  "event_type": "trigger",

  "incident_key": "...",

  "service_key": "..."

}

Our teams doing 24/7 coverage would ideally like to have the data in annotations/labels be easily parseable JSON (ie. just be fields in "details":{).

I understand it is possible to check for specific fields (like a runbook URL) by adding a section the receiver like:

details: {
  firing:       '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
  resolved:     '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
  num_firing:   '{{ .Alerts.Firing | len }}'
  num_resolved: '{{ .Alerts.Resolved | len }}'
  alertname:    '{{ .GroupLabels.alertname }}'
  runbookURL:   '{{ .CommonAnnotations.runbook }}'
}

or would this fail if there was no runbook?

But I think it'd be awesome to allow a label mapping, similar to how relabel_configs work, where all the labels and annotations are automatically added to this details section. Some of our alerts won't have a specific runbook url, but will have other annotations/labels that provide valuable context.

I think this would make PagerDuty alerts much more contextual and easier to parse/use. There's obviously some corner cases (what if someone has a severity/description/resolved label, is it overridden?), but it feels like

if this label mapping were an option that you had to set explicitly, you'd probably be ok with the label/annotations overriding the defaults.

Thanks for your time, I'd love to hear your thoughts :)

Tim Simmons

tim...@digitalocean.com
@timsim on freenode
github.com/timsimmons

[tsim...@digitalocean.com]

Any thoughts on this from anyone? I'd be happy to work up an Alertmanager PR to show what this might look like.