PromQL - Check for specific value in the past

[fiala...@gmail.com]

Hi,

I have a metric bgp_state_info. Ok state is when metric has value = 3, other values (from 1 to 7) are considered as error.

I want to fire alert only for metrics that has value 3 at least only once. In other words I dont' want to fire alert for bgp that never worked.

Is it possible via promQL to do this? I have data retention 60 days and I'm aware of this limitations.

Thank you.

[Brian Candler]

You can use a subquery which will sample the data, something like this:

bgp_state_info != 3 and present_over_time((bgp_state_info == 3)[60d:1h])

You can reduce the sampling interval from 1h to reduce the risk of missing times when BGP was up, but then the query becomes increasingly expensive.

It would be nice if PromQL allowed you to do filtering and arithmetic expressions between range vectors and scalars, e.g. present_over_time(bgp_state_info[60d] == 3), but it doesn't.

Another approach is to use a recording rule, where you can combine the current value with a new value, e.g.

- record: bgp_seen

  expr: bgp_seen or bgp_state_info == 3

Temporarily set the expression to the subquery to prime it from historical data.  With a bit of tweaking you could make the value of this expression be the timestamp when bgp_state_info == 3 was first seen.

The alert then becomes:

bgp_state_info != 3 and bgp_seen