delay "Resolved notification" only for one rule or other solution?
11 views
Skip to first unread message
Daniel Trüssel
Mar 23, 2020, 5:34:05 AM3/23/20
to Prometheus Users
Hey
This is the alert based on blackbox exporter
'avg_over_time(probe_duration_seconds{env="Prod"}[15m]) > 1'
It is when the performance gets slow, we need to wake up and try to
solve the root cause.
When we restart web server, this alert temporary resolved, but later
fires again.
Is there a hack to send the resolved notification in delay? For example
only close alert if the situation is fine for 2 hours?
I not wish the alert to fire, close, fire, close,..
We see this behavior only with slow performance alerts, others which
check things like HTTP status are fine.
what possible solutions?
kind regards
Daniel
This is the alert based on blackbox exporter
'avg_over_time(probe_duration_seconds{env="Prod"}[15m]) > 1'
It is when the performance gets slow, we need to wake up and try to
solve the root cause.
When we restart web server, this alert temporary resolved, but later
fires again.
Is there a hack to send the resolved notification in delay? For example
only close alert if the situation is fine for 2 hours?
I not wish the alert to fire, close, fire, close,..
We see this behavior only with slow performance alerts, others which
check things like HTTP status are fine.
what possible solutions?
kind regards
Daniel
Brian Candler
Mar 23, 2020, 6:06:04 AM3/23/20
to Prometheus Users
It's a long-standing issue, see (closed) https://github.com/prometheus/alertmanager/issues/204
Brian Brazil's suggestions are here.
Reply all
Reply to author
Forward
0 new messages