Alert Manager alert rules spamming

[kishor...@gmail.com]

I have a quick question on alert manager alerting rules, i have 500 IoT devices connected to the server and those are being monitored using prometheus, and i configured the alert rules in such a way to get an alert when ever a device goes offline. every thing is fine till now but, when multiple devices are going offline at different time, alert manager is sending alerts of all the players which are currently offline rather than showing the alert of particular player on particular time.

ALERT IoT_online
IF IoT_online == 0
LABELS {severity="critical"}
ANNOTATIONS {description="This device is offline {{ $value }}.", summary="Instance {{ $labels.instance }} IoT device is offline"}

I wanted to configure alerts in such a way that alert manager should only send alert of particular device alert on particular time rather than sending the alert of all offline devices every-time when a device goes offline. any one can help me ?

How to group these alerts without spamming.? Any examples

[Nicholas Capo]

By default the Alertmanager will try to minimize the notifications it sends you.

If you really want it to send you one notification for each machine, take a look at the `group_by` [0] configuration option.
If you set it to group by `instance` it will send one notification per instance.

Nicholas

[0] https://prometheus.io/docs/alerting/configuration/#example

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/fb159250-3cea-4146-942f-a316841e9d2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nicholas Capo]

Also, this sort of question is probably better asked on the Prometheus-Users mailing list [0]

[0] https://prometheus.io/docs/alerting/configuration/#example

[Kishore Routu]

Hello Nicholas,

Thanks for your answer. I think i need to be more specific on my question all those 500 devices are connected to one instance. and i wrote a custom exporter which gives me the CPU, memory, Heap , network quality , Online and offline status of the device, so all those 500 are up and online.

In my scenario. device-1 goes offline at 12:00 AM - i got an alert saying this device-1 is off-line ( very great and good )

                         device -2 goes offline at 3:00 AM  - i got an alert saying device-2 is offline and also it shows device -1 is also offline ( i just wanted to see the device 2 alert) in this time not device -1
                        device -3 goes offline at 4:00 AM .  - i got an alert saying device -1 ,2 amd 3 are offline ( i want only alert of device-3)

How to tackle these kind of scenarios in real time production environment if i have 50000 devices connected to one instance.

Hope i'm clear now ?

Thanks

On Mon, Feb 5, 2018 at 1:49 PM, Nicholas Capo <nichol...@gmail.com> wrote:

Also, this sort of question is probably better asked on the Prometheus-Users mailing list [0]

[0] https://prometheus.io/docs/alerting/configuration/#example

On Mon, Feb 5, 2018 at 12:48 PM Nicholas Capo <nichol...@gmail.com> wrote:

By default the Alertmanager will try to minimize the notifications it sends you.

If you really want it to send you one notification for each machine, take a look at the `group_by` [0] configuration option.
If you set it to group by `instance` it will send one notification per instance.

Nicholas

[0] https://prometheus.io/docs/alerting/configuration/#example

On Mon, Feb 5, 2018 at 10:29 AM <kishor...@gmail.com> wrote:

I have a quick question on alert manager alerting rules, i have 500 IoT devices connected to the server and those are being monitored using prometheus, and i configured the alert rules in such a way to get an alert when ever a device goes offline. every thing is fine till now but, when multiple devices are going offline at different time, alert manager is sending alerts of all the players which are currently offline rather than showing the alert of particular player on particular time.

ALERT IoT_online
  IF IoT_online == 0
  LABELS {severity="critical"}
  ANNOTATIONS {description="This device is offline  {{ $value }}.", summary="Instance {{ $labels.instance }} IoT device is offline"}


I wanted to configure alerts in such a way that alert manager should only send alert of particular device alert on particular time rather than sending the alert of all offline devices every-time when a device goes offline. any one can help me ?

How to group these alerts without spamming.? Any examples

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

[Simon Pasquier]

On Mon, Feb 5, 2018 at 7:54 PM, Kishore Routu <kishor...@gmail.com> wrote:

Hello Nicholas,

Thanks for your answer. I think i need to be more specific on my question all those 500 devices are connected to one instance. and i wrote a custom exporter which gives me the CPU, memory, Heap , network quality , Online and offline status of the device, so all those 500 are up and online.

In my scenario. device-1 goes offline at 12:00 AM - i got an alert saying this device-1 is off-line ( very great and good )

                         device -2 goes offline at 3:00 AM  - i got an alert saying device-2 is offline and also it shows device -1 is also offline ( i just wanted to see the device 2 alert) in this time not device -1
                        device -3 goes offline at 4:00 AM .  - i got an alert saying device -1 ,2 amd 3 are offline ( i want only alert of device-3)

In that case, add whatever label represents the device to 'group_by'.

 

How to tackle these kind of scenarios in real time production environment if i have 50000 devices connected to one instance.

Hope i'm clear now ?

Thanks

On Mon, Feb 5, 2018 at 1:49 PM, Nicholas Capo <nichol...@gmail.com> wrote:

Also, this sort of question is probably better asked on the Prometheus-Users mailing list [0]

[0] https://prometheus.io/docs/alerting/configuration/#example

On Mon, Feb 5, 2018 at 12:48 PM Nicholas Capo <nichol...@gmail.com> wrote:

By default the Alertmanager will try to minimize the notifications it sends you.

If you really want it to send you one notification for each machine, take a look at the `group_by` [0] configuration option.
If you set it to group by `instance` it will send one notification per instance.

Nicholas

[0] https://prometheus.io/docs/alerting/configuration/#example

On Mon, Feb 5, 2018 at 10:29 AM <kishor...@gmail.com> wrote:

I have a quick question on alert manager alerting rules, i have 500 IoT devices connected to the server and those are being monitored using prometheus, and i configured the alert rules in such a way to get an alert when ever a device goes offline. every thing is fine till now but, when multiple devices are going offline at different time, alert manager is sending alerts of all the players which are currently offline rather than showing the alert of particular player on particular time.

ALERT IoT_online
  IF IoT_online == 0
  LABELS {severity="critical"}
  ANNOTATIONS {description="This device is offline  {{ $value }}.", summary="Instance {{ $labels.instance }} IoT device is offline"}


I wanted to configure alerts in such a way that alert manager should only send alert of particular device alert on particular time rather than sending the alert of all offline devices every-time when a device goes offline. any one can help me ?

How to group these alerts without spamming.? Any examples

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/fb159250-3cea-4146-942f-a316841e9d2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CANg4Exd3CYAQF1N_%3DzzdCcsoPigbj2Gtp_etjTh5DHBT6LAVQw%40mail.gmail.com.