Query to show the start of firing Alerts

[wcooley...@gmail.com]

I'd like to query prometheus to show when Alerts start and end. I want to either create a graph or table in grafana with these metrics.

I can use ALERTS{alertstate="firing"} == 0 to determine when alerts stop firing. changes(ALERTS{alertstate="firing"} [2m]) > 0 gives me a similar result.

However I haven't figured out a way to create a query to show the start of an alert.

I think I figured out that I can use count_over_time(ALERTS [1m]) < 4 in my environment to show both the start and end of alerts but I need to be able to distinguish between the start and end.

Any ideas on how to do this? 

William

[Brian Brazil]

This is not really possible. What is your end goal?

--

Brian Brazil

www.robustperception.io

[Brian Brazil]

On 12 March 2018 at 18:09, <wcooley...@gmail.com> wrote:

I'd like to build a dashboard in grafana that allows me to easily identify when firing alerts started and ended during some period of time in the past.

I've tried creating a stacked graph of ALERTS{alertstate="firing"} but this becomes unusable when displaying multiple non continuous alerts.
I'm guessing using elastic search might be better for recording these kinds of events.

That sounds more like event logging than metrics alright, you could use the webhook in the alertmanager to pass this information on to ES.

--

Brian Brazil

www.robustperception.io