Sent resolved or inactive status (alertmanager)

Dmitry

unread,

Jul 13, 2020, 7:27:34 AM7/13/20

to Prometheus Users

Hello!
I have standard rule for prometheus alertmanager:

  rules:
  - alert: Instance_down
    expr: up == 0
    for: 1m
    # Labels - additional labels to be attached to the alert
    labels:
      severity: 'critical'
    annotations:
      title: 'Instance {{ $labels.instance }} down'
      description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute.'

I receive firing alert messages on my email and slack when instance falls. But i don't receive any messages when instance work is restored, and alert status changes from firing.
Is there a way to sent a message when problem is resolved or alert comes from firing to inactive?

Christian Hoffmann

unread,

Jul 13, 2020, 9:24:48 AM7/13/20

to Dmitry, Prometheus Users

Hi,

Can you confirm that you've set the send_resolved: true option in the
relevant email_config and slack_config blocks?

Kind regards,
Christian

Dmitry

unread,

Jul 13, 2020, 12:00:12 PM7/13/20

to Prometheus Users

No, I did not set send_resolved: true. I am trying to add this line to my configuration file, but there is no result. But if I look at the alertmanager status page, I will see the line send_resolved: false in my configuration.

    global:
    slack_api_url: 'https://hooks.........'
  
  route:
    group_by: [alertname]
    # Send all notifications to me.
    receiver: notify-me
  
  receivers:
  - name: notify-me
    email_configs:
    - to: em...@gmail.com
      from: em...@gmail.com
      smarthost: smtp.gmail.com:587
      auth_username: "em...@gmail.com"
      auth_identity: "em...@gmail.com"
      auth_password: "pass"
    - to: em...@gmail.com
      from: em...@gmail.com
      smarthost: smtp.gmail.com:587
      auth_username: "em...@gmail.com"
      auth_identity: "em...@gmail.com"
      auth_password: "pass"
    slack_configs:
    - channel: '#some channel'
      text: 'https://prometheus/alerts' 

Brian Candler

unread,

Jul 13, 2020, 1:11:43 PM7/13/20

to Prometheus Users

It goes under each receiver separately, e.g.

email_configs:

- to: em...@gmail.com

from: em...@gmail.com

send_resolved: true

Dmitry

unread,

Jul 14, 2020, 5:17:59 AM7/14/20

to Prometheus Users

Thank you. It helped me.

Reply all

Reply to author

Forward