Subject/Title line templates for Slack and PagerDuty.
Maxim V
I spend quite some time to understand how to modify Title/Subject line for alerts firing from Alertmanager to PagerDuty or slack and still have luck in understanding.
Is there some thread in github with description for it, which I have not found so far? I’d like to put more meaningful information into “Description” field as it seems like it is being use by PD to show in the WEB UI as Subject of the alert.
Any help here would be highly appreciated.
-------------------------------------------------------------------------
Extra Details:
Alertmanager: As I can see Alertmanager fires messages through generic API call to PD in JSON like:
{ "client": "Alertmanager", "client_url": "URL HERE ***********************************", "contexts": [], "description": "[FIRING:1] flux-system (AutomationReconciliationIssue lsv-prod manager http-prom flux-system 10.233.82.36:8080 flux-system/flux-system Kustomization p0 critical devops monitoring-system kustomize-controller-7b4bfdd8b-n7s4c p0 monitoring-system/kps-kube-prometheus-stack-prometheus 2y critical False pagerduty-devops Ready)", "event_type": "trigger", "incident_key": "*******************************************", "service_key": "*****************************", "details": { "firing": "Labels: - alertname = AutomationReconciliationIssue - cluster = lsv-prod - container = manager - endpoint = http-prom - exported_namespace = flux-system - instance = 10.233.82.36:8080 - job = flux-system/flux-system - kind = Kustomization - label_default_priority = p0 - label_default_severity = critical - label_team = devops - name = monitoring-system - namespace = flux-system - pod = kustomize-controller-7b4bfdd8b-n7s4c - priority = p0 - prometheus = monitoring-system/kps-kube-prometheus-stack-prometheus - retention = 2y - severity = critical - status = False - target = pagerduty-devops - type = ReadyAnnotations: - description = Kustomization monitoring-system in flux-system is in False status - summary = Kustomization monitoring-system in flux-system is in False statusSource: <URL here ******************>", "num_firing": "1", "num_resolved": "0", "resolved": "" }}
Slack: The same issue is valid for Slack alerts.
I have done configuration for alerts on receivers definition as follow:
- name: slack-internal
slackConfigs:
- apiURL:
key: url
name: slack-url-internal
channel: '#pgmt-k8s-alerts-internal'
username: lsv-prod
sendResolved: true
text: |-
{{ range .Alerts.Firing }}
*{{ .Labels.severity | toUpper }}* - `{{ .Labels.alertname }}`: {{ .Annotations.summary }}{{ .Annotations.message }}
{{ .Annotations.description -}}
{{ if .Annotations.runbook_url }}{{ .Annotations.runbook_url }}{{ end }}
*Additional Details:*
{{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
{{ end }}
{{ end }}
And alerts looks like:
CRITICAL - AutomationReconciliationIssue: HelmRelease app-ai in app-prod is in False status HelmRelease app-ai in app-prod is in False status
Additional Details:
• alertname: AutomationReconciliationIssue
• cluster: lsv-prod
• container: manager
• endpoint: http-prom
• exported_namespace: app-prod
• instance: 10.233.111.140:8080
• job: flux-system/flux-system
• kind: HelmRelease
• label_default_priority: p0
• label_default_severity: critical
• label_team: si
• name: app-ai
• namespace: app-prod
• pod: helm-controller-7868945f7f-5slwc
• priority: p0
• prometheus: monitoring-system/kps-kube-prometheus-stack-prometheus
• retention: 2y
• severity: critical
• status: False
• target: slack-internal
• type: Ready
Where first line of the message is generated automatically and does not have inheritance from the receiver configuration (as I can see).
Brian Candler
{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}