Alertmanager - playing with the templating engine to create custom emails

Adrian Popa

unread,

Oct 21, 2021, 8:02:26 AM10/21/21

to Prometheus Users

Hello everyone!

I'm new here, but I've been playing with prometheus/alertmanager for about a year now and I'm still learning. I've set up most of the monitoring I want to do, but now I'm working on alerts.

I have some alert-centric questions I'd like to ask, in order to be able to learn more.

I found the email template online and I'm trying to understand how to use the templating engine to customize the messages I get. I have some Jinja2 experience, but not Go (yet). I'd like to know if there is a tool that can:

a) expose the internal state of an alert (expose the fields that are accessed by the template) so that I can have a look around and see which is which in terms of content, so that I know what I can use to display in my alerts.

b) allow me to validate/render the template with fake data/static data, so that I know that my changes work or not

c) allow me to test my alerts by triggering my alert rules one by one, so that I can validate the entire alerting system end-to-end. I know that I can change the thresholds, but sometimes triggering some alerts may be difficult. Maybe there is an easy way...

In general, I'd like to see how others are using alertmanager to get alerts and act on them (mostly from node_exporter data), and how they are making their alerts to be meaningful and not overwhelming.

Thanks a lot!,

Adrian

Adrian Popa

unread,

Nov 8, 2021, 4:33:22 AM11/8/21

to Prometheus Users

Kind bump?

How do you guys play with email templates and tune them until you like it? There must be an easier way than poking around every time and restarting alertmanager...

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9953ee98-6832-4224-8dd6-479f3d0f5039n%40googlegroups.com.

Julien Pivotto

unread,

Nov 8, 2021, 8:41:49 AM11/8/21

to Adrian Popa, Prometheus Users

There is such a functionality in amtool:

amtool template render --template.glob='/foo/bar/*.tmpl'
--template.text='{{ template "slack.default.markdown.v1" . }}'

> > <https://groups.google.com/d/msgid/prometheus-users/9953ee98-6832-4224-8dd6-479f3d0f5039n%40googlegroups.com?utm_medium=email&utm_source=footer>
> > .

> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAOKso16c4EJ2vsnS2p2NPvQXe9%3DVL1p3bkcxEVFizSMsbeX%3D0Q%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie

Adrian Popa

unread,

Nov 9, 2021, 7:10:44 AM11/9/21

to Adrian Popa, Prometheus Users

Thank you for the tip! I'll look into it!

Adrian Popa

unread,

Nov 9, 2021, 8:10:38 AM11/9/21

to Adrian Popa, Prometheus Users

Well, I'm back sooner than I had hoped.

I've upgraded my alertmanager (and prometheus) to their latest version to get the template option in amtool. Now, I'm trying to get it to work...

I am able to query for a specific alert:

```

/alertmanager $ amtool -o extended --alertmanager.url=http://127.0.0.1:9093 alert query alertname="high_temperature" instance="10.19.53.4:9100" chip="platform_coretemp_0" sensor="temp2"
Labels Annotations Starts At Ends At Generator URL State
alertname="high_temperature" chip="platform_coretemp_0" instance="10.19.53.4:9100" job="syslog_server" monitor="docker-host-alpha" platform="syslog" production="True" send_alerts="True" sensor="temp2" severity="warning" description="Host temperature is 100C. Reported by instance 10.19.53.4:9100 of job syslog_server." summary="Server temperature over 75C" 2021-11-09 12:45:26 UTC 2021-11-09 12:51:56 UTC http://6260171c95c4:9090/graph?g0.expr=node_hwmon_temp_celsius%7Bsend_alerts%3D%22True%22%7D+%3E+75&g0.tab=1 active

```

I can view it as json, which I suspect I'll need, in order to find out what fields I can use in the template:

```

/alertmanager $ amtool -o json --alertmanager.url=http://127.0.0.1:9093 alert query alertname="high_temperature" instance="10.19.53.4:9100" chip="platform_coretemp_0" sensor="temp2"
[{"annotations":{"description":"Host temperature is 100C. Reported by instance 10.19.53.4:9100 of job syslog_server.","summary":"Server temperature over 75C"},"endsAt":"2021-11-09T12:53:11.340Z","fingerprint":"4e8fa68805104681","receivers":[{"name":"raptor-ssms"}],"startsAt":"2021-11-09T12:45:26.340Z","status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},"updatedAt":"2021-11-09T12:49:11.371Z","generatorURL":"http://6260171c95c4:9090/graph?g0.expr=node_hwmon_temp_celsius%7Bsend_alerts%3D%22True%22%7D+%3E+75\u0026g0.tab=1","labels":{"alertname":"high_temperature","chip":"platform_coretemp_0","instance":"10.19.53.4:9100","job":"syslog_server","monitor":"docker-host-alpha","platform":"syslog","production":"True","send_alerts":"True","sensor":"temp2","severity":"warning"}}]

```

Now, simple strings work when rendering a template, and also things like "if":

```

/alertmanager $ amtool --alertmanager.url=http://127.0.0.1:9093 template render --template.glob=/etc/alertmanager/config/*.tmpl --template.text='This is a test'
This is a test

/alertmanager $ amtool --alertmanager.url=http://127.0.0.1:9093 template render --template.glob=/etc/alertmanager/config/*.tmpl --template.text='This is a test {{if gt 2 1}} True {{else}} False {{end}}'
This is a test True

```

But how am I supposed to pipe a specific (active) alert through the template rendering? I tried the example, but I'm not sure where I would get the template name from (it's not the file name, I tried that too):

```

/alertmanager $ amtool --alertmanager.url=http://127.0.0.1:9093 template render --template.glob=/etc/alertmanager/config/*.tmpl --template.text='{{ template "slack.default.markdown.v1" . }}'
amtool: error: template: :1:12: executing "" at <{{template "slack.default.markdown.v1" .}}>: template "slack.default.markdown.v1" not defined

/alertmanager $ amtool --alertmanager.url=http://127.0.0.1:9093 template render --template.glob=/etc/alertmanager/config/template-email-slack.tmpl --template.text='{{ template "template-email-slack" . }}'
amtool: error: template: :1:12: executing "" at <{{template "template-email-slack" .}}>: template "template-email-slack" not defined

```

Thanks for the pointers!

Adrian Popa

unread,

Jan 10, 2022, 5:17:33 AM1/10/22

to Adrian Popa, Prometheus Users

Hello everyone - sorry to bump my own thread, but I've figured it out (after going through a basic golang course) and playing around. Hope this information helps others as well (and maybe improves documentation too).

So - my original goal was to modify the original email template and have customized subject/content for email alerts. The same information below can be used to customize other types of messages (slack, etc). The original template I got from here: https://github.com/prometheus/alertmanager/blob/main/template/default.tmpl

Let's look a bit at the subject line:

{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}

In order to understand it, you'll need some background about:

* the templating engine: https://pkg.go.dev/text/template

* golang data structures: https://medium.com/@victorsteven/understanding-data-structures-in-golang-f55205afdcaa

* the Alertmanager datastructure that gets passed to the templating engine (something that I missed at first), explains what data gets sent, but without an obvious example: https://prometheus.io/docs/alerting/latest/notifications/

So, the question is - what variables are available to the templating engine to build the subject line. Where does it get .CommonLabels from, and what do those contain for my alerts?

I've managed to find a way to dump all the variables that get to the templating engine in some sort of Python's pretty print, or Perl's Data::Dumper way, by using Golang's fmt.Printf with the "%#v" option in order to print the data structure with names.

So, here's what I did - {{ define "email.default.html" }} defines the start of the HTML email that you receive on an alert. Inside the HTML code, after the <body> tag, I added a (optionally hidden) div that dumps the variables it got:

<div name="GLOBAL DEBUG" style="display: none;">

<br>
{{ printf "%#v" . }}
<br>
</div>

You can use the same trick inside for loops and it will dump the variables you see inside the loop. This way, we can see a full data structure for an alert:

&template.Data{
Receiver:"my-email-receiver",
Status:"firing",
Alerts:template.Alerts{
template.Alert{
Status:"firing",
Labels:template.KV{
"alertname":"monitor_service_down",
"instance":"10.10.10.10:9182",
"job":"myservername",
"monitor":"docker-host-alpha",
"platform":"MyPlatform",

"production":"True",
"send_alerts":"True",

"severity":"critical"
},
Annotations:template.KV{
"description":"[myservername] Service 10.10.10.10:9182 is down.",
"summary":"Monitor service non-operational"
},
StartsAt:time.Time{
wall:0x34553780,
ext:63777397532,
loc:(*time.Location)(nil)
},
EndsAt=:time.Time{
wall:0x0,
ext:0,
loc:(*time.Location)(nil)
},
GeneratorURL:"http://6260171c95c4:9090/graph?g0.expr=XXXXXXXXXXXXXX",
Fingerprint:"380272c1fa92c19a"
}
},
GroupLabels:template.KV{
"alertname":"monitor_service_down",
"instance":"10.10.10.10:9182",
"job":"myservername",
"monitor":"docker-host-alpha",
"platform":"MyPlatform",

"production":"True",
"send_alerts":"True",

"severity":"critical"
},
CommonLabels:template.KV{
"alertname":"monitor_service_down",
"instance":"10.10.10.10:9182",
"job":"myservername",
"monitor":"docker-host-alpha",
"platform":"MyPlatform",

"production":"True",
"send_alerts":"True",

"severity":"critical"
},
CommonAnnotations:template.KV{
"description":"[myservername] Service 10.10.10.10:9182 is down.",
"summary":"Monitor service non-operational"
},
ExternalURL:"http://ad4c4424ca70:9093"
}

In my case I wanted to reformat the subject line, to show the description and platform of my alert. I'm also using grouping in a way that doesn't do grouping, and sends individual alerts (this is the way I'm used to seeing alerts), so this may not apply to everyone. In order to access the description from CommonAnnotations, I did the following:

{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonAnnotations.description }} [platform {{ .CommonLabels.platform }}]{{ end }}

And I got the subject I was expecting.

Next, I edited the HTML code and printed what I felt was useful for my case, by looping through elements of the Alerts array and printing the fields that I needed.

Note that you should use error handling and check that a field exists before printing it, otherwise the email may not leave and would generate a log entry in alertmanager that it couldn't render the template!

Note for alertmanager devs! It may be useful to include a dump of sample alert data like above in https://prometheus.io/docs/alerting/latest/notifications/, to allow users to understand what is available and help them access the data. For a golang expert it would be fairly easy to use the dump above and load it into golang play and render the template (as a test) when developing it, without regenerating the alerts.