Best way to monitor Alerting itself?

[dcadwa...@gmail.com]

From the Prometheus Alerting docs:

> As always, if it is possible to alert on symptoms rather than causes, this helps to reduce noise. For example, a blackbox test that alerts are getting from PushGateway to Prometheus to Alertmanager to email is better than individual alerts on each.

Wondering if there are any best practices for how to implement something like this?

[Ben Kochie]

One option is to have an always firing alert send notifications to a service like https://deadmanssnitch.com/.

Or there was a lightning talk at PromCon that talked a bit about monitoring email server pings.  https://www.youtube.com/watch?v=AWUytyzwPyE

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/66a2a15f-f464-4662-9c99-4fb82351aa07%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Needles]

If the goal is to provide monitoring as a service (internally or externally), then use another instance of Prometheus. Nothing prioritizes work better for the group than living and breathing the customer experience.  Also it gives a place for beta testing 8-)  In absence of that, what is often used for other solutions is end to end heartbeating through the system with synthetic transactions and then monitoring the existence and performance of these.