Isolated Alerts Telegram / email

22 views
Skip to first unread message

Alexander Varejão

unread,
Jun 17, 2024, 1:29:27 PMJun 17
to Prometheus Users
Hi,

I am testing server monitoring with Prometheus and creating alerts with Alertmanager.

My current scenario involves two machines (I'll call them A and B) being monitored by Prometheus.

I have created alerts so that when one of the servers is down (or both), I get notified via Telegram and email.

In my test, when I shut down server A, I am correctly notified of its downtime. However, if I shut down server B, I receive an alert whose body contains information not only about the downtime of B but also of A.

This would be very problematic considering I have a large number of devices to monitor.

How can I ensure that each alert refers only to the specific event it corresponds to?

Thank you.

Chris Siebenmann

unread,
Jun 17, 2024, 1:41:59 PMJun 17
to Alexander Varejão, Prometheus Users, Chris Siebenmann
Alertmanager sends notifications about *groups* of alerts, not
individual alerts, although you can make a group any size you want, down
to a single alert. Grouping is based on the labels assigned to each
individual alert from Prometheus, and is set in the relevant 'route:'
block, using 'group_by'.

https://prometheus.io/docs/alerting/latest/configuration/#route-related-settings

If you want separate alert notifications about each separate host, you
need to group by a label that names the host, possibly among other
labels. If you really want one notification for each alert, there is an
option for this, covered in the configuration documentation.

- cks

Alexander Varejão

unread,
Jun 17, 2024, 2:52:05 PMJun 17
to Prometheus Users
Thank you Chris :)

It really seems work.

Now I can continue my tests

[ ] 's
Reply all
Reply to author
Forward
0 new messages