Timeperiod in Alert definition

848 views
Skip to first unread message

Julien D.

unread,
Oct 4, 2016, 3:37:07 AM10/4/16
to Prometheus Developers
Hello,

currently if I want prometheus not to fire an alert during the night, I have to modify my query like this :
IF (time() % 86400 > bool 6.5*3600) * (time() % 86400 < bool 21*3600) * my_alert_condition

thats for between 6.30AM to 9PM.
Maybe there is a better query for that, but in any case I have to put this condition everywhere in my alert.rules configuration.
It's very unintuitive and far too complicated. It also has nothing to do with alerts at the prometheus level, as it should fire anytime there is a problem, wether it's night time or not.
To me it should be on the alertmanager's side, to decide if the alert should be sent or not depending on the time period.

I see two solutions :
- on prometheus side : add a TIMEPERIOD statement with a specific, human readable syntax lie 21:00 to 06:30
- on alertmanager side : add permanent silences based on tags : if an alert has a tag "period":"workhours" then silence if it's outside predefined workhours.

What do you think ?

Björn Rabenstein

unread,
Oct 4, 2016, 9:08:36 AM10/4/16
to Julien D., Prometheus Developers
On 4 October 2016 at 09:37, Julien D. <julien...@gmail.com> wrote:
> currently if I want prometheus not to fire an alert during the night, I have
> to modify my query like this :
> IF (time() % 86400 > bool 6.5*3600) * (time() % 86400 < bool 21*3600) *
> my_alert_condition
>
> thats for between 6.30AM to 9PM.
> Maybe there is a better query for that,

Since v1.1, there are functions for that. Your case would use
https://prometheus.io/docs/querying/functions/#hour()

v1.2 will also have a `minute()` function.

Note, however, that those function have no notion of time zones or
so-called "daylight saving time".

> but in any case I have to put this
> condition everywhere in my alert.rules configuration.

You could have a recording rule "nighttime" or something.

> It's very unintuitive and far too complicated. It also has nothing to do
> with alerts at the prometheus level, as it should fire anytime there is a
> problem, wether it's night time or not.

That's true for most use cases. If the timing is not business logic,
it should not live in your recording rule.

> To me it should be on the alertmanager's side, to decide if the alert should
> be sent or not depending on the time period.

The wisdom so far is that the alertmanager is not concerned with
schedules. The receiver of a notification should handle that. An
obvious example is Pagerduty, which is all about schedules (and has
knowledge of time zones of the user being paged etc.).

So:

- If it is business logic, put it into the alerting expression.
- If it is a scheduling issue, put it into PagerDuty.

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

Kris Berry

unread,
Oct 12, 2016, 4:25:56 PM10/12/16
to Prometheus Developers, julien...@gmail.com
Hi,

This sounds more like what I was looking for when I replied to a different post.  I should have looked further in the forum.  I tried to create a recording rule similar to the "nighttime" rule that was suggested but it is getting "no data" when I run a query on it.  If I look at the data on the right side it gives a result.  Am I mistaken in the syntax?

  workhours=day_of_week() < bool 6 or hour() >= bool 13 or hour() < bool 22

Once I have the recording rule working, would this be the proper syntax for the alert?

  IF workhours * my_alert_condition

Thanks!
Kris.

Brian Brazil

unread,
Oct 12, 2016, 4:35:51 PM10/12/16
to Kris Berry, Prometheus Developers, Julien Dehee
On 12 October 2016 at 21:25, Kris Berry <be...@mosaicatm.com> wrote:
Hi,

This sounds more like what I was looking for when I replied to a different post.  I should have looked further in the forum.  I tried to create a recording rule similar to the "nighttime" rule that was suggested but it is getting "no data" when I run a query on it.  If I look at the data on the right side it gives a result.  Am I mistaken in the syntax?

  workhours=day_of_week() < bool 6 or hour() >= bool 13 or hour() < bool 22

This will always return day_of_week() < bool 6. I think want * rather than "or" (I presume you want a logical and).
 
Once I have the recording rule working, would this be the proper syntax for the alert?

  IF workhours * my_alert_condition

my_alert_condition and by () workhours == 1

Brian
 

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d57c9d68-5959-43b9-9f13-426326ce9232%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Kris Berry

unread,
Oct 12, 2016, 4:47:56 PM10/12/16
to Prometheus Developers, be...@mosaicatm.com, julien...@gmail.com
Hi Brian,

Yeah, thanks, I had been a little indecisive with the rule.  It went from

    workhours=day_of_week() < bool 6 and hour() >= bool 13 and hour() < bool 22
to
    offhours=day_of_week() >= bool 6 or hour() < bool 13 or hour() > bool 21

and then when I changed it back I forgot to change the ors to ands.  I never restarted Prometheus after the first one, though, so that's what's running and giving me "no data" in the query console.   Thanks for catching that!  This is what my current Prometheus Rule display shows:

    workhours{} = day_of_week() < BOOL 6 and hour() >= BOOL 13 and hour() < BOOL 21

I appreciate the clarification of the syntax I will need for the IF statement!

Thanks,
Kris.
To post to this group, send email to prometheus...@googlegroups.com.



--

Brian Brazil

unread,
Oct 12, 2016, 5:38:14 PM10/12/16
to Kris Berry, Prometheus Developers, Julien Dehee
On 12 October 2016 at 21:47, Kris Berry <be...@mosaicatm.com> wrote:
Hi Brian,

Yeah, thanks, I had been a little indecisive with the rule.  It went from

    workhours=day_of_week() < bool 6 and hour() >= bool 13 and hour() < bool 22
to
    offhours=day_of_week() >= bool 6 or hour() < bool 13 or hour() > bool 21

and then when I changed it back I forgot to change the ors to ands.  I never restarted Prometheus after the first one, though, so that's what's running and giving me "no data" in the query console.   Thanks for catching that!  This is what my current Prometheus Rule display shows:

    workhours{} = day_of_week() < BOOL 6 and hour() >= BOOL 13 and hour() < BOOL 21

This also always returns day_of_week() < BOOL 6. Bool returns 0 or 1, and "and" ignores values. You want the * operator.

Brian
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Kris Berry

unread,
Oct 12, 2016, 5:58:18 PM10/12/16
to Prometheus Developers, be...@mosaicatm.com, julien...@gmail.com
Hey Brian,

This has been a great help!  That resolves the recording rule.

When I tried your recommendation on IF statement, though, promtool was not kind:

  IF rate(datafeed[10m]) == 0 and by () workhours == 1

  FAILED: parse error at line 4, char 36: no valid expression found

Any thoughts?

Thanks!
Kris.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.



--

Brian Brazil

unread,
Oct 12, 2016, 6:46:57 PM10/12/16
to Kris Berry, Prometheus Developers, Julien Dehee
On 12 October 2016 at 22:58, Kris Berry <be...@mosaicatm.com> wrote:
Hey Brian,

This has been a great help!  That resolves the recording rule.

When I tried your recommendation on IF statement, though, promtool was not kind:

  IF rate(datafeed[10m]) == 0 and by () workhours == 1

  FAILED: parse error at line 4, char 36: no valid expression found

Any thoughts?

It's on rather than by. I get confused with another language sometimes.

Brian
 

Thanks!
Kris.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Kris Berry

unread,
Oct 13, 2016, 9:01:26 AM10/13/16
to Prometheus Developers, be...@mosaicatm.com, julien...@gmail.com
I understand about the language thing!

When I enter "rate(datafeed[10m]) == 0 and on () workhours == 1" on the Prometheus query window it returns the proper value when the feed is not active during those hours.  When I add that into the rules file and restart Prometheus, though, it is removing "on ()" from the rules and it shows up as just "rate(datafeed[10m]) == 0 and workhours == 1".  That query is not returning "no data".  Is that a bug?  Or does it need to be expressed differently in the rules file?

Thanks again for all your help!
Kris.

Thanks!
Kris.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.



--

Brian Brazil

unread,
Oct 13, 2016, 12:59:13 PM10/13/16
to Kris Berry, Prometheus Developers, Julien Dehee
On 13 October 2016 at 14:01, Kris Berry <be...@mosaicatm.com> wrote:
I understand about the language thing!

When I enter "rate(datafeed[10m]) == 0 and on () workhours == 1" on the Prometheus query window it returns the proper value when the feed is not active during those hours.  When I add that into the rules file and restart Prometheus, though, it is removing "on ()" from the rules and it shows up as just "rate(datafeed[10m]) == 0 and workhours == 1".  That query is not returning "no data".  Is that a bug?  Or does it need to be expressed differently in the rules file?

That's a bug, can you put exactly that into an issue please?

Brian
 

Thanks!
Kris.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Kris Berry

unread,
Oct 13, 2016, 2:40:41 PM10/13/16
to Prometheus Developers, be...@mosaicatm.com, julien...@gmail.com
I created the issue.  Thanks so much for your help!

Thanks!
Kris.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.



--
Reply all
Reply to author
Forward
0 new messages