Alertmanager is sending the resolved notification everytime the current value of metric is changing

25 views
Skip to first unread message

Rahul Hada

unread,
Mar 10, 2020, 1:10:41 PM3/10/20
to Prometheus Users
I have alerts configured for various metrics, and has set send_resolved : true, now once alert is in active state, we are getting [Firing] emails as configured, but we are getting the [resolved] email even if there is a slight change in the values. For ex:- we get Firing alert when disk usage is 98% and the [resolved] emails comes in if the values goes to 97% or anything. Please suggest how to eradicate this issue. 

Brian Brazil

unread,
Mar 10, 2020, 1:30:30 PM3/10/20
to Rahul Hada, Prometheus Users
On Tue, 10 Mar 2020 at 17:10, Rahul Hada <rh....@gmail.com> wrote:
I have alerts configured for various metrics, and has set send_resolved : true, now once alert is in active state, we are getting [Firing] emails as configured, but we are getting the [resolved] email even if there is a slight change in the values. For ex:- we get Firing alert when disk usage is 98% and the [resolved] emails comes in if the values goes to 97% or anything. Please suggest how to eradicate this issue. 

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a9060291-9ea5-40a0-aec6-cea866d7a3e2%40googlegroups.com.


--

Rahul Hada

unread,
Mar 10, 2020, 1:41:32 PM3/10/20
to Prometheus Users
Thanks for the quick response Brian, I was checking same. We are not using Value as label, only using in annotations. Below is the alert config. Please have a look and suggest what changes can help us achieve better alert notifications. One more help i would request is, like below alert expression, we have alert rule for warning(which is if usage reaches more than 80%), we get both Critical & warning alert notification if value goes to more than 90%. Would really appreciate help. 

  - alert: OutOfDiskSpace-Crit
    expr: ((node_filesystem_size_bytes{fstype!~"nfs.*"} - node_filesystem_avail_bytes{fstype!~"nfs.*"}) / node_filesystem_size_bytes{fstype!~"nfs.*"} * 100 > 90) * on (instance) group_left(nodename) node_uname_info
    for: 5m
    labels:
      severity: "CRITICAL"
    annotations:
      summary: "High Disk Usage on *{{ $labels.instance }}* - *{{ $labels.nodename }}* on the filesystem *{{ $labels.mountpoint }}*"
      description: "Current Usage of {{ $labels.mountpoint }} = *{{ $value | humanize }}%*"
      identifier: "*Cluster:* `{{ $labels.cluster }}`, *node:* `{{ $labels.node }}`, *Mountpoint:* `{{ $labels.mountpoint }}` "


On Tuesday, March 10, 2020 at 11:00:30 PM UTC+5:30, Brian Brazil wrote:
On Tue, 10 Mar 2020 at 17:10, Rahul Hada <rh....@gmail.com> wrote:
I have alerts configured for various metrics, and has set send_resolved : true, now once alert is in active state, we are getting [Firing] emails as configured, but we are getting the [resolved] email even if there is a slight change in the values. For ex:- we get Firing alert when disk usage is 98% and the [resolved] emails comes in if the values goes to 97% or anything. Please suggest how to eradicate this issue. 

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

Brian Brazil

unread,
Mar 10, 2020, 2:08:05 PM3/10/20
to Rahul Hada, Prometheus Users
On Tue, 10 Mar 2020 at 17:41, Rahul Hada <rh....@gmail.com> wrote:
Thanks for the quick response Brian, I was checking same. We are not using Value as label, only using in annotations. Below is the alert config. Please have a look and suggest what changes can help us achieve better alert notifications. One more help i would request is, like below alert expression, we have alert rule for warning(which is if usage reaches more than 80%), we get both Critical & warning alert notification if value goes to more than 90%. Would really appreciate help. 


Brian
 

  - alert: OutOfDiskSpace-Crit
    expr: ((node_filesystem_size_bytes{fstype!~"nfs.*"} - node_filesystem_avail_bytes{fstype!~"nfs.*"}) / node_filesystem_size_bytes{fstype!~"nfs.*"} * 100 > 90) * on (instance) group_left(nodename) node_uname_info
    for: 5m
    labels:
      severity: "CRITICAL"
    annotations:
      summary: "High Disk Usage on *{{ $labels.instance }}* - *{{ $labels.nodename }}* on the filesystem *{{ $labels.mountpoint }}*"
      description: "Current Usage of {{ $labels.mountpoint }} = *{{ $value | humanize }}%*"
      identifier: "*Cluster:* `{{ $labels.cluster }}`, *node:* `{{ $labels.node }}`, *Mountpoint:* `{{ $labels.mountpoint }}` "


On Tuesday, March 10, 2020 at 11:00:30 PM UTC+5:30, Brian Brazil wrote:
On Tue, 10 Mar 2020 at 17:10, Rahul Hada <rh....@gmail.com> wrote:
I have alerts configured for various metrics, and has set send_resolved : true, now once alert is in active state, we are getting [Firing] emails as configured, but we are getting the [resolved] email even if there is a slight change in the values. For ex:- we get Firing alert when disk usage is 98% and the [resolved] emails comes in if the values goes to 97% or anything. Please suggest how to eradicate this issue. 

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a9060291-9ea5-40a0-aec6-cea866d7a3e2%40googlegroups.com.


--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7c692d96-f542-4c53-a5e2-61ee079d7588%40googlegroups.com.


--
Reply all
Reply to author
Forward
0 new messages