Alertmanager configuration

[Karthike Ezhilarasan]

Hello,

I've recently configured prometheus and alertmanager with a slack webhook, everything works fine but the notifications I get in slack uses the same text for firing and resolved making it look a bit odd.

[FIRING:1] warning@my_server high_storage_load
warning@my_server: Host storage usage is 81%.

 

[RESOLVED] high_storage_load@my_server
Resolved@my_server: Host storage usage is 81%.

I would like to know if there is a simple way to use 2 different text fields one for firing messages and one for resolved messages.

Thanks for helping me.

EZHILARASAN Karthike

[Christian Hoffmann]

Hi,

Yes, you can use Alertmanager's notification templating feature to
format such messages differently, e.g. based on the Status field (firing
vs. resolved). You can add/change the static text or you could even
reference a different alert-specific annotation.

I suggest starting based on the built-in defaults:

https://prometheus.io/docs/alerting/configuration/#slack_config
https://github.com/prometheus/alertmanager/blob/master/template/default.tmpl
https://prometheus.io/docs/alerting/notification_examples/
https://prometheus.io/docs/alerting/notifications/

Kind regards,
Christian

[Karthike Ezhilarasan]

Thanks a lot, I'll look into templates details.

EZHILARASAN Karthike

[Da Sm]

Thanks for the template snip - 

Here's what I finally ended up going with - 

alertmanager.yml

  slack_configs:

  - api_url: 

    channel: '#xxxxxxxxxxxxxxxxxxxx'

    color: '{{ template "SLACK_MSG_COLOR" . }}'

    send_resolved: true

    title: '{{ template "SLACK_MSG_TITLE" . }}'

    text: '{{ template "SLACK_MSG_TEXT" . }}'

  pagerduty_configs:

  - routing_key: '{{ template "Global_PD_Service_Key" . }}'

    description: '{{ template "PAGERDUTY_DESCRIPTION" . }}'

    severity: '{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toLower }}{{ else }}critical{{ end }}'

    links: 

    - text: 'Prometheus'

      href: '{{ (index .Alerts 0).GeneratorURL }}'

    - text: 'Search Runbooks'

      href: '{{ template "RUNBOOK_SEARCH" . }}'

and the .tmpl

# Pagerduty description  
{{ define "PAGERDUTY_DESCRIPTION" }}
    {{if .CommonAnnotations.summary }}{{ .CommonAnnotations.summary }}
    {{ else }}{{ .GroupLabels.alertname }}
    {{ if .CommonLabels.instance }}{{ .CommonLabels.instance }}
    {{ end }}
    {{ end }}
{{ end }}


################  
# Runbook  
################
# Runbook Search
{{ define "RUNBOOK_SEARCH" }}https://dsmith73.github.io/101-docs/search/?q={{ .CommonLabels.alertname }}{{ end }}


################  
# Slack  
################  
# Slack Color
{{ define "SLACK_MSG_COLOR" }}{{ if eq .Status "firing" }}{{ if eq .CommonLabels.severity "critical" }}danger{{ else if eq .CommonLabels.severity "error" }}danger{{ else if eq .CommonLabels.severity "warning" }}warning{{ else }}#439FE0{{ end }}{{ else}}good{{ end }}{{ end }}


# Slack Text  
{{define "SLACK_MSG_TEXT" }}
      <!here> - {{ .CommonAnnotations.description }}
      `View:` :chart_with_upwards_trend:*<{{ (index .Alerts 0).GeneratorURL }}|Prometheus>* or :notebook:*<{{ template "RUNBOOK_SEARCH" . }}|Runbook>*


      *Details:*
      {{ range .CommonLabels.SortedPairs }}• *{{ .Name }}:* `{{ .Value }}`
    {{ end }}
{{ end }}


#Slack Summary
{{ define "SLACK_TITLE_SUMMARY" -}}
    {{- if .CommonAnnotations.summary -}}
        {{- .CommonAnnotations.summary -}}
    {{- else -}}
        {{- with index .Alerts 0 -}}
            {{- .Annotations.summary -}}
        {{- end -}}
    {{- end -}}
{{- end -}}


# Slack Title  
{{ define "SLACK_MSG_TITLE" }}
    {{ if eq .Status "resolved" }}
        {{- .Status | toUpper }} : {{ template "SLACK_TITLE_SUMMARY" . }}
    {{ else if eq .Status "firing" }}
        {{ .CommonLabels.severity | toUpper }} : {{ template "SLACK_TITLE_SUMMARY" . }}
    {{ end }}
{{ end }}

It's all a wire-frame, but I like using the product label and alert-name to execute a query style search for runbooks. I've been tossing around the idea of moving our runbooks to git, and the docksy-jeckyll website brings it all together IMO.

cheers!