Inhibit resolved messages from inhibited alerts

[Michael Kogelman]

It seems that resolved messages are still thrown/received when an inhibited alert is resolved. Is there any way to squelch these as well? Or is this pretty much as intended.

Thanks!

[Chris Siebenmann]

> It seems that resolved messages are still thrown/received when an inhibited
> alert is resolved. Is there any way to squelch these as well? Or is this
> pretty much as intended.

Inhibitions normally stop notifications about resolved alerts. However,
I suspect that you may be running into a tricky issue of timing and how
inhibitions (and silences, and mutes) behave.

As far as I know, Alertmanager doesn't necessarily immediately send
notifications about resolved alerts; instead they sometimes or always
are sent only at the next group_interval tick. If the resolved alert is
not inhibited at this point, you will get a notification for its alert
group. In turn, an inhibition will only be in effect if the alert that
inhibits other alerts is still active at this point, which is some time
*after* the inhibited alert has resolved.

So if alert A inhibits alert B and they both resolve at the same time,
or A resolves shortly after alert B, then (resolved) alert B may not
still be inhibited when it comes time to send a notification about it,
because alert A is now resolved and not inhibiting anything any more. To
get around this, you need to keep alert A active for an extra period
that's longer than the group_interval for alert B.

(We often say that silences, inhibitions, and so on 'suppress'
notifications. As far as I can tell, this is not actually what happens;
instead, inhibitions make notifications not happen at all. This means
that inhibited things are considered to not yet have been part of a
successful alert group notification, and Alertmanager will continue to
consider them new and worthy of triggering a group_interval new alert
group notification.)

- cks