Max unspecified value for Promtool unit testing

[Tashfia Rahman]

Hi all,

I'm using Promtool for unit testing alert rules, and I'm having trouble understanding the behavior for unspecific input_series values.

For example, say I have an alert rule that fires for the expression x == 5. 

- alert: test_alert

expr: x == 5

labels:

severity: page

annotations:

description: "x is {{ $value }}"

summary: "x is {{ $value }}"

When I run this test with my input_series values '5 5 5' I'm only specifying input values for minutes 0, 1, and 2.

- interval: 1m

input_series:

- series: 'x'

values: '5 5 5'

alert_rule_test:

- eval_time: 7m

alertname: test_alert

exp_alerts:

- exp_labels:

severity: page

exp_annotations:

description: "x is 5"

summary: "x is 5"

As expected, the alert fires for eval_time 0m through 2m. However, the alert still fires when I make the eval_time anywhere from 3m through 7m, but then fails with 8m and later. I'm not sure why 3-7 work and then at 8 it stops working.

I've tried some other scenarios and can't find a pattern for when unspecified values do/don't work. I'm not sure if I am misinterpreting something here, so any help would be greatly appreciated.

Thanks in advance! 

[Tashfia Rahman]

Additionally, if you have multiple input series values, how is it reflected on the expression? Is the '5 5 5' applied to every minute, or does it mean that each minute 0 through 2 has a value of 5?

[Brian Candler]

The value at time T is defined as the most recent value received *prior* to time T, but prometheus only looks back 5 minutes.

[Tashfia Rahman]

Thanks for the response! To clarify, does time T refer to the eval_time or the interval time?

[Brian Candler]

The scrape time.

[Jhih Cheng Luo]

I am also curious about this behavior. Can anyone explain this case in more detail? I appreciate it!

Since the time series data 'x' only has the values between 0m to 2m, why the alert is still fired between 3m to 7m? Thank!

Tashfia Rahman於 2020年6月9日星期二 UTC+8下午9時57分19秒寫道：

[Stuart Clark]

On 2020-07-15 19:17, Jhih Cheng Luo wrote:
> I am also curious about this behavior. Can anyone explain this case in
> more detail? I appreciate it!
> Since the time series data 'x' only has the values between 0m to 2m,
> why the alert is still fired between 3m to 7m? Thank!
> Tashfia Rahman於 2020年6月9日星期二
> UTC+8下午9時57分19秒寫道：

When a query is made for a value of a timeseries at a particular point
it is very unlikely that there is an entry at that exact timestamp. By
default Prometheus will look back by up to 5 minutes to find a value.

> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/9a4e4b21-730b-4f10-ab83-2f856649218fo%40googlegroups.com
> [1].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/prometheus-users/9a4e4b21-730b-4f10-ab83-2f856649218fo%40googlegroups.com?utm_medium=email&utm_source=footer

--
Stuart Clark

[Jhih Cheng Luo]

Hi Stuart,

Thank yo so much for your quick response! I would like to go more details for it.

(1) Is there any official documentation has mentioned about this mechanism?
(2) The default look back interval is 5 minutes. Is it configurable for Prometheus and promtool during the unit testing for alert?

Honestly, I still couldn’t understand why Prometheus has this mechanism. Just for this test case, it didn’t make sense to keep firing an alert during the 3m to 7m, as the time series metrics has already gone.

Thank you so much!

[toyo...@gmail.com]

Just found some useful article might explain this behavior and share it with you. Really appreciate Stuart's explanation on this case!

• https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness
• https://www.robustperception.io/staleness-and-promql
• https://www.robustperception.io/alerting-on-gauges-in-prometheus-2-0 

toyo...@gmail.com 在 2020年7月16日 星期四上午2:53:00 [UTC+8] 的信中寫道：