Sending inhibited or muted Alertmanager alerts when the mute expires?

[Chris Siebenmann]

Alertmanager has supported inhibiting alert notifications for a long
time, and somewhat more recently it's added support for muting them
during specific time ranges (or only sending them in specific time
ranges). As far as I know, the behaviour (for both inhibited alerts and
muted alerts) is that the alert notification that would be sent is
entirely suppressed; Alertmanager basically 'sends' it into the void,
and afterwards will only re-send an (unchanged) alert notification at
your repeat_interval.

What we'd like for our purposes is some way to re-send all such
suppressed alert notifications if the alert is still active when an
inhibition or mute expires. So, for example, if we mute alert
notifications out of hours and some alert triggers then, at the start of
working hours we'd get an alert notification for that alert (or group of
them, etc). We don't want to do this by reducing our repeat_interval
(which is currently large).

(I am aware that we could build a dashboard of Prometheus alerts that
are currently active, but such a dashboard wouldn't contain our specific
and helpful alert message text and so on.)

Is there any clever way to do this within the current features of
Alertmanager (and/or Prometheus)?

(I've even looked at the Alertmanager API reference, but as far as I can
see there's no way to trigger the re-sending of an alert notification. I
suppose we could write a script that tried to add an artificial new
alert to every outstanding alert group that matched suitable criteria to
trigger an Alertmanager resend because the alert group has 'changed',
but that seems like a bad hack and potentially fragile.)

Thanks in advance.

- cks

[George Robinson]

Hi!

> What we'd like for our purposes is some way to re-send all such
suppressed alert notifications if the alert is still active when an
inhibition or mute expires. So, for example, if we mute alert
notifications out of hours and some alert triggers then, at the start of
working hours we'd get an alert notification for that alert (or group of
them, etc).

Alertmanager isn't able to re-send alerts when a mute time interval
becomes inactive, because time intervals are not checked until it's
time to send the notification again.

> We don't want to do this by reducing our repeat_interval
> (which is currently large).

However, I think you can approximate this by having a short
group_interval (for example, 5 minutes). Each time the group_interval
elapses, Alertmanager will check if there are new alerts in the group
for which a notification has not been sent. This will be true if an
alert fired over night, and is still firing in working hours.
However, if the alert resolved before the start of working hours, then
a notification might not be sent as the alert is resolved.

Best regards,

George