Separate mails from FIRING and RESOLVED

Sebastian Glock

unread,

May 14, 2021, 7:57:26 AM5/14/21

to Prometheus Users

Hi,

I have send resolved set to true and I found an interesting option to show the alert time in the alert:

duration: {{(.EndsAt.Sub .StartsAt).Truncate 1000000000}}

But when alerts fires i have something like this:

-2562047h47m16s

When resolved alert comes it looks gr8:

2m15s

How can i separate those two emails? Or maybe u have better query to show duration between StartsAt and EndsAt?

Is there way to separate templates?

My actual config:

- name: alert-emailer-cpu-1h

email_configs:

- to: 'sebasti...@ecom.software'

send_resolved: true

from:***

smarthost: ***

auth_username: ***

auth_password: ***

auth_secret:***

auth_identity:***

html: '{{ template "email" .}}'

headers:

subject: "[{{ .Status | toUpper }}][CUK] {{ .CommonLabels.severity }} {{ .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{ .CommonAnnotations.description }}"

and template:

<html>

<head>

table {

font-family: verdana,arial,sans-serif;

font-size:14px;

color:#333333;

border-width: 1px;

border-color: #999999;

border-collapse: collapse;

}

table th {

background-color:#ff6961;

border-width: 1px;

padding: 8px;

border-style: solid;

border-color: #F54C44;

}

table td {

border-width: 1px;

padding: 8px;

border-style: solid;

border-color: #F54C44;

text-align: center;

}

</style>

</head>

<body>

<thead>

<tr>

<th>Alert name</th>

<th>Instance</th>

<th>Summary</th>

<th>Description</th>

</tr>

</thead>

<tbody>

<tr>

<td><b>{{ .Labels.alertname }}</td></b>

<td>{{ .Labels.instance }}</td>

<td>{{ .Annotations.summary }}</td>

<td>{{ .Annotations.description }}</td>

<td>{{ .StartsAt.Format "2006-01-02 15:04:05" }}</td>

<td>{{(.EndsAt.Sub .StartsAt).Truncate 1000000000}}</td>

</tr>

</tbody>

</table>

</body>

</html>

Maybe there's way to see only duration between firing and resolved ? If i can separate alerts (maybe with template) resolved and firing it would be also gr8. But I don't know how :(

I'm not best in it so please be kind :D

Thanks for all replies!

Screenshot_108.jpg

Julien Pivotto

unread,

May 18, 2021, 7:17:30 PM5/18/21

to Sebastian Glock, Prometheus Users

On 14 May 13:57, Sebastian Glock wrote:
> Hi,
>
> I have send resolved set to true and I found an interesting option to show
> the alert time in the alert:
>
> duration: {{(.EndsAt.Sub .StartsAt).Truncate 1000000000}}

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAFn4LSqxfWGMvR8d4obDgOgZ3EFQ5L5nDwWvaYhx6tpqys_xYg%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie

Sebastian Glock

unread,

May 19, 2021, 6:20:03 AM5/19/21

to Sebastian Glock, Prometheus Users

Thanks for your reply

It seems to work! :D

But now how to set time between firing and resolved? I have something like this:

global:

resolve_timeout: 4m

route:

receiver: alert-emailer-cpu-30m

group_by: ['alertname', 'priority', 'instance']

group_wait: 10s

group_interval: 10s

repeat_interval: 30m

But i get mail with resolved after few seconds. How to change it?

And my second question - how to tell alertmanager that i want only to see status with "resolved" not with "firing".

html: '{{ template "email" .}}'

headers:

subject: "[CUK][{{ .Status | toUpper }}{{ if eq .Status 'resolved' }}]{{ .CommonLabels.severity }} {{ .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{ .CommonAnnotations.description }}"

I tried something like this but it's not working. I just want to see that word "RESOLVED" if alert is resolved. If still firing i don't want to see "FIRING".

Thanks for your help!

Reply all

Reply to author

Forward