How limit alerting after the first alert?

Skip to first unread message

Loic Fouray

Mar 29, 2022, 5:52:49 AM3/29/22
to mtail-users

I want use mtail to alerting end-users when a pattern match into the logs.
Mtail is used as exporter with prometheus/alertmanager. Currently, i use alertmanager receivers that are not very advanced (mattermost, mail).

When mtail matchs a pattern, a metric of type gauge is updated to 1.  An alerting is send from Alertmanager. Then the metric is updated to 0 because no new recording. So alertmanager close the issue. 
If several logs match during the following minutes, an alert will be generated almost every time and the end-user will be flooded of messages.

Do you know how I can deal with this situation? For example, is it possible with mtail to keep my metric with 1 as value during a defined period of time (1 hour for example) without take into account new record? 
Otherwise, have you already used a specific configuration of Prometheus/Alertmanager to take into account this use case ? 

Reply all
Reply to author
0 new messages