alert if there is a change on gauge?

[Dan S]

So we have a gauge metric that is just zero or one.  Starts at zero, goes to one when event happens, rarely if ever goes back to zero.

We want to alert after event change 0->1 happens, but have the alert stop after X time, and not alert indefinitely.

Basically this returns the data of what triggered:

sum by (customer) (event_triggered) > 0

{customer="A"} 1

{customer="B"} 1

{customer="C"} 1

But we want triggered in X past time.  I realize rate doesn't work because it's not a counter... but ideally would want something like this:

sum by (customer) (rate(event_triggered[5m])) > 0

{customer="A"} .003 

i.e. just return the most recent one that triggered in the past 5 minutes and have it stop alerting once it returns to base (which is now 1)

Thanks

Dan

[Brian Candler]

If you would also expect an alert when it changes from 1 to 0, then changes() may be what you're looking for.  It counts the number of changes for each series in a range vector.

changes(blah[5m]) > 0

(or "> bool 0" if you want a 0/1 value which is amenable to summing)

> But we want triggered in X past time.

...

> i.e. just return the most recent one that triggered in the past 5 minutes

I don't understand either of those requirements.

You can use grouping in alertmanager so that alerts for customerA, customerB and customerC are grouped into the same alert.