How to manage the alerting frequency via alertmanager?

492 views
Skip to first unread message

Loïc

unread,
Apr 26, 2022, 11:58:31 AM4/26/22
to Prometheus Users
Hi,

I want use an exporter (mtail or fluentd) to alerting when a pattern match into the kubernetes logs. 
Currently, i use alertmanager receivers that are not very advanced (mattermost, mail).

When my exporter matchs a pattern, a metric is updated to 1.  An alerting is send from Alertmanager. Then the metric is updated to 0 because no new recording. So alertmanager close the issue. 
If several logs match during the following minutes, an alert will be generated almost every time and the end-user will be flooded of messages.

Do you know how I can deal with this situation? For a rule, Is it possible not to alarm again for a period of time?

Thanks

Abhi Rathod

unread,
Apr 26, 2022, 1:31:02 PM4/26/22
to Prometheus Users
You can use group_interval.
You can specify the time you want to wait after initial notification is sent before firing new alerts. 

Loïc

unread,
Jun 22, 2022, 11:11:28 AM6/22/22
to Prometheus Users

Thanks for your reply.
group_interval is for any alarms sent to my email receiver.  Maybe i can create a specific receiver with matcher for my log alarm.

Also, for a email receiver do you know if's possible to limit the max number of alarms sent?

Thanks
Loïc
Reply all
Reply to author
Forward
0 new messages