Alert hysteresis

[Brian Candler]

I had an idea. Now that there is the "keep_firing_for" feature, it would be possible to use this for alert hysteresis if you had a second expression to cancel the alert.

For example:

expr: disk_space_used > 0.8
keep_firing_for: 1d
cancel_expr: disk_space_used < 0.7

Then your alert starts firing at 80%, and stops firing at 70% (or after 1 day if under 80%)

Anyway, I just thought the idea was worth a mention!

[Jan Fajerski]

This was discussed a while ago in the dev summit, see
https://docs.google.com/document/d/11LC3wJcVk00l8w5P3oLQ-m3Y37iom6INAMEu2ZAGIIE/edit?tab=t.0#heading=h.ir9z26welaum
It's also on the list to be discussed again:
https://docs.google.com/document/d/1uurQCi5iVufhYHGlBZ8mJMK_freDFKPG0iYBQqJ9fvA/edit?tab=t.0#heading=h.7jp762fwzkeo

Afaict this feature is waiting for someone to gather use cases and
outline the different approaches and their trade-offs. Once we have a
document on this, ideally in https://github.com/prometheus/proposals
the dev summit can form a concensus.

The first dev summit discussion has some more context on what has been
proposed before, especially when it comes to debugging and usability.
Those seem to be the tricky bits.
The contribution of a designproposal would be more then welcome.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/42b6f0d8-9852-4b23-ab46-e8f64a097bedn%40googlegroups.com.