I have alerts configured for various metrics, and has set send_resolved : true, now once alert is in active state, we are getting [Firing] emails as configured, but we are getting the [resolved] email even if there is a slight change in the values. For ex:- we get Firing alert when disk usage is 98% and the [resolved] emails comes in if the values goes to 97% or anything. Please suggest how to eradicate this issue.
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a9060291-9ea5-40a0-aec6-cea866d7a3e2%40googlegroups.com.
On Tue, 10 Mar 2020 at 17:10, Rahul Hada <rh....@gmail.com> wrote:
I have alerts configured for various metrics, and has set send_resolved : true, now once alert is in active state, we are getting [Firing] emails as configured, but we are getting the [resolved] email even if there is a slight change in the values. For ex:- we get Firing alert when disk usage is 98% and the [resolved] emails comes in if the values goes to 97% or anything. Please suggest how to eradicate this issue.Sounds like it might be https://www.robustperception.io/dont-put-the-value-in-alert-labels
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a9060291-9ea5-40a0-aec6-cea866d7a3e2%40googlegroups.com.
Thanks for the quick response Brian, I was checking same. We are not using Value as label, only using in annotations. Below is the alert config. Please have a look and suggest what changes can help us achieve better alert notifications. One more help i would request is, like below alert expression, we have alert rule for warning(which is if usage reaches more than 80%), we get both Critical & warning alert notification if value goes to more than 90%. Would really appreciate help.
--- alert: OutOfDiskSpace-Critexpr: ((node_filesystem_size_bytes{fstype!~"nfs.*"} - node_filesystem_avail_bytes{fstype!~"nfs.*"}) / node_filesystem_size_bytes{fstype!~"nfs.*"} * 100 > 90) * on (instance) group_left(nodename) node_uname_infofor: 5mlabels:severity: "CRITICAL"annotations:summary: "High Disk Usage on *{{ $labels.instance }}* - *{{ $labels.nodename }}* on the filesystem *{{ $labels.mountpoint }}*"description: "Current Usage of {{ $labels.mountpoint }} = *{{ $value | humanize }}%*"identifier: "*Cluster:* `{{ $labels.cluster }}`, *node:* `{{ $labels.node }}`, *Mountpoint:* `{{ $labels.mountpoint }}` "
On Tuesday, March 10, 2020 at 11:00:30 PM UTC+5:30, Brian Brazil wrote:On Tue, 10 Mar 2020 at 17:10, Rahul Hada <rh....@gmail.com> wrote:I have alerts configured for various metrics, and has set send_resolved : true, now once alert is in active state, we are getting [Firing] emails as configured, but we are getting the [resolved] email even if there is a slight change in the values. For ex:- we get Firing alert when disk usage is 98% and the [resolved] emails comes in if the values goes to 97% or anything. Please suggest how to eradicate this issue.Sounds like it might be https://www.robustperception.io/dont-put-the-value-in-alert-labels--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a9060291-9ea5-40a0-aec6-cea866d7a3e2%40googlegroups.com.
--Brian Brazil
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7c692d96-f542-4c53-a5e2-61ee079d7588%40googlegroups.com.