Silenced alerts and web.hook receiver

[Anthony Dakhin]

Hi,

I'm new to Prometheus and currently trying to implement a proxy between Alertmanager and our monitoring system. I've configured web.hook receiver based on sample Alertmanager config, so I'm able to receive POSTs when alert is firing or resolved - so far so good.

If I create silence before an alert starts firing, receiver does not get any information about this alert - that's OK. But if a silence is created after an alert started firing, Alertmanager does not inform (web.hook) receiver about this silence in any way. I expected to receive some sort of notification, so I could supress the corresponding alert in my monitoring system.

Is this behaviour intenional? If so, is there any possible way to work around this problem?

Thanks,

Anthony

[Christian Hoffmann]

Hi,

I think this behaviour is intentional and I cannot think of any way to
circumvent this, besides extending Alertmanager.

The only thing which comes to mind would be observing (polling)
Alertmanager via its APIs in order to find any silenced alerts. This
should be possible, but would probably not really be that elegant.

https://github.com/prometheus/alertmanager#api

Kind regards,
Christian

[Laurent Dumont]

That something that I investigated as well. So far, it seems a silence will mute an alarm at the root level on the AM side. There is no notifications that a silence was created either.

In your case, you could have the reverse where your monitoring system could create the silences locally and push them to AM.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/14562d52-6708-babd-d7d0-4b245f550f67%40hoffmann-christian.info.

[Anthony Dakhin]

It's an interesting idea, thank you!

вторник, 11 августа 2020 г., 8:48:14 UTC+3 пользователь Laurent Dumont написал:

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.