Firing an alert only between certain hours of the day?

Eoin Gillen

unread,

Mar 15, 2016, 5:36:01 PM3/15/16

to Prometheus Developers

Hi all,

I'm new to Prometheus so apologies if this is covered somewhere. Basically, I have a success timestamp metric with one dimension (the name of the task), so if I look at my metric in the console it looks like:

success_timestamp

success_timestamp{task="a"} 1458075306
success_timestamp{task="b"} 1458075309

I'm trying to set up an alert that fires if any timestamp in the metric is older than a certain time, but I only want the alert to fire between certain hours of the day. I thought I could accomplish that with something like:

IF (time() - success_timestamp > 300) AND ((time() % 86400000) / 3600000 > 10) AND ((time() % 86400000) / 3600000 < 18)

but I get an error message "comparisons between scalars must use BOOL modifier".

I had a look at the documentation on operators and tried a few different permutations using the bool cast and "==", but I haven't managed to get it working yet.
Is there some way to accomplish this?

Thanks,
Eoin.

Brian Brazil

unread,

Mar 15, 2016, 5:46:08 PM3/15/16

to Eoin Gillen, Prometheus Developers

You're looking for the vector() function. Also, time() returns a value in seconds.

Brian

Thanks,
Eoin.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

Eoin Gillen

unread,

Mar 16, 2016, 7:01:32 AM3/16/16

to Prometheus Developers, eoing...@gmail.com

On Tuesday, March 15, 2016 at 9:46:08 PM UTC, Brian Brazil wrote:

On 15 March 2016 at 21:36, Eoin Gillen <eoing...@gmail.com> wrote:
Hi all,

I'm new to Prometheus so apologies if this is covered somewhere. Basically, I have a success timestamp metric with one dimension (the name of the task), so if I look at my metric in the console it looks like:

success_timestamp

success_timestamp{task="a"} 1458075306
success_timestamp{task="b"} 1458075309

I'm trying to set up an alert that fires if any timestamp in the metric is older than a certain time, but I only want the alert to fire between certain hours of the day. I thought I could accomplish that with something like:

IF (time() - success_timestamp > 300) AND ((time() % 86400000) / 3600000 > 10) AND ((time() % 86400000) / 3600000 < 18)

but I get an error message "comparisons between scalars must use BOOL modifier".

I had a look at the documentation on operators and tried a few different permutations using the bool cast and "==", but I haven't managed to get it working yet.
Is there some way to accomplish this?

You're looking for the vector() function. Also, time() returns a value in seconds.

Brian

Hey, thanks for the response. Just had a go at it there and I think I've managed to get it working. The task names (i.e. the values of the "task" label) were something I couldn't know apriori in Prometheus, so I couldn't get AND between two vectors working (without getting rid of the task names), but I think I've got it working with:

IF (time() - success_timestamp > 300) * ((time() % 86400 / 3600 > bool 9) == bool (time() % 86400 / 3600 < bool 18))

which is currently returning me elements with some nonzero value if today's hour is between 9 and 18, and elements with 0 otherwise. Going to try it with an alert now...

Thanks,
Eoin.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Brian Brazil
www.robustperception.io

Brian Brazil

unread,

Mar 16, 2016, 7:12:26 AM3/16/16

to Eoin Gillen, Prometheus Developers

On 16 March 2016 at 11:01, Eoin Gillen <eoing...@gmail.com> wrote:

On Tuesday, March 15, 2016 at 9:46:08 PM UTC, Brian Brazil wrote:
On 15 March 2016 at 21:36, Eoin Gillen <eoing...@gmail.com> wrote:
Hi all,

I'm new to Prometheus so apologies if this is covered somewhere. Basically, I have a success timestamp metric with one dimension (the name of the task), so if I look at my metric in the console it looks like:

success_timestamp

success_timestamp{task="a"} 1458075306
success_timestamp{task="b"} 1458075309

I'm trying to set up an alert that fires if any timestamp in the metric is older than a certain time, but I only want the alert to fire between certain hours of the day. I thought I could accomplish that with something like:

IF (time() - success_timestamp > 300) AND ((time() % 86400000) / 3600000 > 10) AND ((time() % 86400000) / 3600000 < 18)

but I get an error message "comparisons between scalars must use BOOL modifier".

I had a look at the documentation on operators and tried a few different permutations using the bool cast and "==", but I haven't managed to get it working yet.
Is there some way to accomplish this?

You're looking for the vector() function. Also, time() returns a value in seconds.

Brian

Hey, thanks for the response. Just had a go at it there and I think I've managed to get it working. The task names (i.e. the values of the "task" label) were something I couldn't know apriori in Prometheus, so I couldn't get AND between two vectors working (without getting rid of the task names), but I think I've got it working with:

IF (time() - success_timestamp > 300) * ((time() % 86400 / 3600 > bool 9) == bool (time() % 86400 / 3600 < bool 18))

which is currently returning me elements with some nonzero value if today's hour is between 9 and 18, and elements with 0 otherwise. Going to try it with an alert now...

That'll just multiply it by 0 or 1 depending on if it's in the right timeframe.

(time() - success_timestamp > 300) AND ON (dummy) (vector(time() % 86400 / 3600) > 9 < 18)

should do the trick, but seems not to. This looks to be a bug in promql.

Brian

Thanks,
Eoin.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Brian Brazil
www.robustperception.io

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

Eoin Gillen

unread,

Mar 16, 2016, 1:38:15 PM3/16/16

to Prometheus Developers, eoing...@gmail.com

On Wednesday, March 16, 2016 at 11:12:26 AM UTC, Brian Brazil wrote:

On 16 March 2016 at 11:01, Eoin Gillen <eoing...@gmail.com> wrote:

On Tuesday, March 15, 2016 at 9:46:08 PM UTC, Brian Brazil wrote:
On 15 March 2016 at 21:36, Eoin Gillen <eoing...@gmail.com> wrote:
Hi all,

I'm new to Prometheus so apologies if this is covered somewhere. Basically, I have a success timestamp metric with one dimension (the name of the task), so if I look at my metric in the console it looks like:

success_timestamp

success_timestamp{task="a"} 1458075306
success_timestamp{task="b"} 1458075309

I'm trying to set up an alert that fires if any timestamp in the metric is older than a certain time, but I only want the alert to fire between certain hours of the day. I thought I could accomplish that with something like:

IF (time() - success_timestamp > 300) AND ((time() % 86400000) / 3600000 > 10) AND ((time() % 86400000) / 3600000 < 18)

but I get an error message "comparisons between scalars must use BOOL modifier".

I had a look at the documentation on operators and tried a few different permutations using the bool cast and "==", but I haven't managed to get it working yet.
Is there some way to accomplish this?

You're looking for the vector() function. Also, time() returns a value in seconds.

Brian

Hey, thanks for the response. Just had a go at it there and I think I've managed to get it working. The task names (i.e. the values of the "task" label) were something I couldn't know apriori in Prometheus, so I couldn't get AND between two vectors working (without getting rid of the task names), but I think I've got it working with:

IF (time() - success_timestamp > 300) * ((time() % 86400 / 3600 > bool 9) == bool (time() % 86400 / 3600 < bool 18))

which is currently returning me elements with some nonzero value if today's hour is between 9 and 18, and elements with 0 otherwise. Going to try it with an alert now...

That'll just multiply it by 0 or 1 depending on if it's in the right timeframe.

(time() - success_timestamp > 300) AND ON (dummy) (vector(time() % 86400 / 3600) > 9 < 18)

should do the trick, but seems not to. This looks to be a bug in promql.

Brian

Ah, yes I forgot to add a > 0 at the end. So the final IF statement is actually:

IF (time() - success_timestamp > 300) * ((time() % 86400 / 3600 > bool 9) == bool (time() % 86400 / 3600 < bool 18)) > 0

which leaves me with a vector of elements to start firing the alert(s) on. Just tested it there and it seems to work as I'd hoped. Thanks for the help!

Thanks,
Eoin.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Brian Brazil
www.robustperception.io

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Brian Brazil
www.robustperception.io

Brian Brazil

unread,

Mar 24, 2016, 8:53:13 AM3/24/16

to Eoin Gillen, Prometheus Developers

On 16 March 2016 at 11:12, Brian Brazil <brian....@robustperception.io> wrote:

On 16 March 2016 at 11:01, Eoin Gillen <eoing...@gmail.com> wrote:

On Tuesday, March 15, 2016 at 9:46:08 PM UTC, Brian Brazil wrote:
On 15 March 2016 at 21:36, Eoin Gillen <eoing...@gmail.com> wrote:
Hi all,

I'm new to Prometheus so apologies if this is covered somewhere. Basically, I have a success timestamp metric with one dimension (the name of the task), so if I look at my metric in the console it looks like:

success_timestamp

success_timestamp{task="a"} 1458075306
success_timestamp{task="b"} 1458075309

I'm trying to set up an alert that fires if any timestamp in the metric is older than a certain time, but I only want the alert to fire between certain hours of the day. I thought I could accomplish that with something like:

IF (time() - success_timestamp > 300) AND ((time() % 86400000) / 3600000 > 10) AND ((time() % 86400000) / 3600000 < 18)

but I get an error message "comparisons between scalars must use BOOL modifier".

I had a look at the documentation on operators and tried a few different permutations using the bool cast and "==", but I haven't managed to get it working yet.
Is there some way to accomplish this?

You're looking for the vector() function. Also, time() returns a value in seconds.

Brian

Hey, thanks for the response. Just had a go at it there and I think I've managed to get it working. The task names (i.e. the values of the "task" label) were something I couldn't know apriori in Prometheus, so I couldn't get AND between two vectors working (without getting rid of the task names), but I think I've got it working with:

IF (time() - success_timestamp > 300) * ((time() % 86400 / 3600 > bool 9) == bool (time() % 86400 / 3600 < bool 18))

which is currently returning me elements with some nonzero value if today's hour is between 9 and 18, and elements with 0 otherwise. Going to try it with an alert now...

That'll just multiply it by 0 or 1 depending on if it's in the right timeframe.

(time() - success_timestamp > 300) AND ON (dummy) (vector(time() % 86400 / 3600) > 9 < 18)

should do the trick, but seems not to. This looks to be a bug in promql.

This bug is fixed now, it'll be in the next release. https://github.com/prometheus/prometheus/issues/1489

Brian

Brian

Thanks,
Eoin.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Brian Brazil
www.robustperception.io

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Brian Brazil
www.robustperception.io

--

Brian Brazil

www.robustperception.io

Reply all

Reply to author

Forward