Send resolved without slack

182 views
Skip to first unread message

Sebastian Glock

unread,
Mar 29, 2021, 3:31:53 AM3/29/21
to Prometheus Users
Hi, is there any way to configure send resolved without slack? I only found configuration with slack channel. I tried to add send resolved on true but it's not working. Is that even possible?

Stuart Clark

unread,
Mar 29, 2021, 8:05:13 AM3/29/21
to Sebastian Glock, Prometheus Users
On 29/03/2021 08:31, Sebastian Glock wrote:
> Hi, is there any way to configure send resolved without slack? I only
> found configuration with slack channel. I tried to add send resolved
> on true but it's not working. Is that even possible? --

Which method(s) are you trying to use?

--
Stuart Clark

Sebastian Glock

unread,
Mar 29, 2021, 9:20:53 AM3/29/21
to Stuart Clark, Prometheus Users
I tried to do something like this:

```
- name: alert-emailer-cpu-1h
  email_configs:
  - to: 'sebasti...@ecom.software'
    send_resolved: false
    from: '*****'
    smarthost:'*****'
    auth_username: '*****'
    auth_password: '*****'
    auth_secret: '*****'
    auth_identity:'*****'
    html: '{{ template "email" .}}'
    headers:
      subject: "[CUK]{{ .CommonLabels.severity }} {{ .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{ .CommonAnnotations.description }}"

- name: alert-emailer-cpu-1h-resolved
  email_configs:
  - to: 'sebasti...@ecom.software'
    send_resolved: true
    from: '*****'
    smarthost: '*****'
    auth_username: '*****'
    auth_password: '*****'
    auth_secret: '*****'
    auth_identity: '*****'
    html: '{{ template "email" .}}'
    headers:
      subject: "[CUK][RESOLVED]{{ .CommonLabels.severity }} {{ .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{ .CommonAnnotations.description }}"
```

1 alert without send resolved, second with send resolved: true. I thought i'll get an email "alert-emailer-cpu-1h-resolved" after 5 minutes when 1st alert "alert-emailer-cpu-1h" goes out. Is there any other way to solve that?

Thanks for your help!

Stuart Clark

unread,
Mar 30, 2021, 5:02:30 AM3/30/21
to Sebastian Glock, Prometheus Users
Could you post the whole alertmanager config?

--
Stuart Clark

Matthias Rampke

unread,
Mar 30, 2021, 6:24:06 AM3/30/21
to Stuart Clark, Sebastian Glock, Prometheus Users
I *think* you can have one email config, and toggle whether to show [RESOLVED] as part of the template. Looking at the default templates, it seems like the way to do that is to check for the length of .Alerts.Firing and .Alerts.Resolved: {{ if gt (len .Alerts.Firing) 0 }} etc.

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c6ee84ab-4b5b-9dde-db46-ae45f0dc8944%40Jahingo.com.

Sebastian Glock

unread,
Mar 31, 2021, 3:19:48 AM3/31/21
to Matthias Rampke, Stuart Clark, Prometheus Users
I have something like this:

global:
route:
  receiver: alert-emailer-cpu-30m

  group_by: ['alertname', 'priority', 'instance']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 30m
#  resolve_timeout: 10s
  routes:


#rebooted


  - receiver: alert-emailer-reboot-30m
    repeat_interval: 30m
    match:
      severity: "[Rebooted]"


#CPU

  - receiver: alert-emailer-cpu-30m
    repeat_interval: 1m
    match:
      severity: "[Disaster-CPU]"


  - receiver: alert-emailer-cpu-1h
    repeat_interval: 1m
    match:
      severity: "[High-CPU]"


  - receiver: alert-emailer-cpu-3h
    repeat_interval: 1m
    match:
      severity: "[Average-CPU]"


#resolved

  - receiver: alert-emailer-cpu-30m-resolved
    repeat_interval: 1m
    match:
      severity: "[Disaster-CPU]"


  - receiver: alert-emailer-cpu-1h-resolved
    repeat_interval: 1m
    match:
      severity: "[High-CPU]"


  - receiver: alert-emailer-cpu-3h-resolved
    repeat_interval: 1m
    match:
      severity: "[Average-CPU]"


#cpu

- name: alert-emailer-cpu-1h
  email_configs:
  - to: 'sebasti...@ecom.software'
    send_resolved: false
    from: '******'
    smarthost: '******'
    auth_username: '******'
    auth_password: '******'
    auth_secret: '******'
    auth_identity:'******'
    html: '{{ template "email" .}}'
    headers:
      subject: "[CUK]{{ .CommonLabels.severity }} {{ .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{ .CommonAnnotations.description }}"

- name: alert-emailer-cpu-1h-resolved
  email_configs:
  - to: 'sebasti...@ecom.software'
    send_resolved: true
    from: '******'
    smarthost: '******'
    auth_username: '******'
    auth_password: '******'
    auth_secret: '******'
    auth_identity:'******'
    html: '{{ template "email" .}}'
    headers:
      subject: "[CUK][RESOLVED]{{ .CommonLabels.severity }} {{ .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{ .CommonAnnotations.description }}"

- name: alert-emailer-cpu-30m
  email_configs:
  - to: 'sebasti...@ecom.software'
    send_resolved: false
    from: '******'
    smarthost: '******'
    auth_username: '******'
    auth_password: '******'
    auth_secret: '******'
    auth_identity:'******'
    html: '{{ template "email" .}}'
    headers:
      subject: "[CUK]{{ .CommonLabels.severity }} {{ .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{ .CommonAnnotations.description }}"

- name: alert-emailer-cpu-30m-resolved
  email_configs:
  - to: 'sebasti...@ecom.software'
    send_resolved: true
    from: '******'
    smarthost: '******'
    auth_username: '******'
    auth_password: '******'
    auth_secret: '******'
    auth_identity:'******'
    html: '{{ template "email" .}}'
    headers:
      subject: "[CUK][RESOLVED]{{ .CommonLabels.severity }} {{ .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{ .CommonAnnotations.description }}"


@Matthias Rampke so U think i should add .Alerts.Resolved: {{ if gt (len .Alerts.Firing) 0 }} to alert? Where prometheus hold's that alert is resolved? I thought in slack it will show that alert has been resolved, but in prometheus that alert is just green and show nothing.

Matthias Rampke

unread,
Mar 31, 2021, 4:31:43 PM3/31/21
to Sebastian Glock, Stuart Clark, Prometheus Users
To Prometheus, "resolved" is not a separate first class event.

As long as an alert is "firing" in Prometheus, it sends a constant stream of "still firing" messages to Alertmanager. If there are multiple alerts firing, or multiple label combinations for one alert, there's more notifications. AM takes these streams of "still bad", deduplicates, and groups them.

This leads to an alert group with 1 or more alerts in it. In configurable intervals, Alertmanager will send a notification to the configured receiver(s). This is when the templates are evaluated. While a group exists, individual alerts can be added and removed, this often triggers additional notifications. Finally, when the last alert in a group is resolved, another notification is fired.

It is important here that this notification uses the same receiver, and the same templates, as all the previous ones! The choice of Slack color is only done based on the state of each notification, you can see how it's done in the default value of the "color" setting: https://prometheus.io/docs/alerting/latest/configuration/#slack_config

You need only one receiver and set send_resolved: true on it. Then you can use templating to make these emails have different subjects depending on the contents of each notification.

/MR

Sebastian Glock

unread,
May 14, 2021, 2:45:54 AM5/14/21
to Matthias Rampke, Stuart Clark, Prometheus Users
Ok so this one should look like this:

  - alert: CPU load is more than 80%!
    expr: 100 - (1 -avg(irate(windows_cpu_time_total{mode="user"}[10m])) by (instance)) * 100 >= 40
    for: 10s
    labels:
      severity: "[High]"
    annotations:
      summary: "CPU load is more than 80%!"
      description: "{{ humanize $value }}%"


and alertmanager:

  - receiver: alert-emailer-cpu-1h-resolved
    repeat_interval: 1m
    match:
      severity: "[High-CPU]"


- name: alert-emailer-cpu-1h-resolved
  email_configs:
  - to: 'sebasti...@ecom.software'
    send_resolved: true
    from: ****
    smarthost: ****
    auth_username: '****
    auth_password: ****
    auth_secret: ****
    auth_identity: ***
    html: '{{ template "email" .}}'
    headers:
      subject: "[CUK][RESOLVED]{{ .CommonLabels.severity }} {{ .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{ .CommonAnnotations.description }}"

I'm right or something wrong? What if I want to configure send resolved for all alerts? I have many recivers based on severity in recivers and i want only to see mail with "[RESOLVED]" in subject. 

I think there's something wrong because it's not working :(

Sebastian Glock

unread,
May 14, 2021, 4:36:20 AM5/14/21
to Matthias Rampke, Stuart Clark, Prometheus Users
Ok I see problem, but how can i separate email subjects from alert and alert-resolved? Can I somehow insert 2 templates of email or 2 templates of subject into 1 alert? How alertmanager will know when to use 1st template and second?

Matthias Rampke

unread,
May 14, 2021, 4:39:13 AM5/14/21
to Sebastian Glock, Stuart Clark, Prometheus Users
Use conditions within the template. See the Go documentation for the basic language:


/MR
Reply all
Reply to author
Forward
0 new messages