Easy way to parse out rule expression to extract value (remove comparator)

[Callum Jones]

Hi,

With alerting rules the Prometheus API (and templating language) has access to the "Value" of an alert which appears to be the first metric specified without the comparator.

I was wondering if this is something that is easily accessible in the Prometheus Go packages such that I could give it the expression string and it would strip out the comparison part and return just the metric query.

For example passing in:

> sum by (instance) (rate(node_disk_read_bytes_total[2m])) / 1024 / 1024 > 50
would return
> sum by (instance) (rate(node_disk_read_bytes_total[2m])) / 1024 / 1024

Thanks,
Callum

[Julius Volz]

Hi Callum,

In general yes, the PromQL Go library lets you parse an expression and then traverse its abstract syntax tree, using the types from the "ast" package in https://github.com/prometheus/prometheus/blob/main/promql/parser/ast.go. You'd first use https://pkg.go.dev/github.com/prometheus/prometheus/promql/parser#ParseExpr to do the parsing, and that gives you back the AST to look at.

For example, this is how PromLens shows the different sub-expressions of a query (for your expression: https://demo.promlens.com/?l=eF6PYANAlbQ).

However: Not every alerting query is of the shape "<something> <comp-op> <number>". Comparison / filter operators are normal binary expressions that may or may not occur at any part of a query (even deeply nested in a query tree, not at the end), and you will also have alerting queries which do not contain a filter operator at all. So you would have to apply certain assumptions about the structure of a query to be able to extract just the part you are interested in.

Regards,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a5b35b15-ebae-4030-8a9c-f67adb836eacn%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[laixintao]

Hi Callum

For what it's worth, I met the same issue, and I wrote a python lib that can split alert rules into multiple parts of expressions, based on grammar, so it can handles and/or/group_left/on, etc as well. The split_binary_op function takes a string expression and returns the parsed result in json format.

https://github.com/laixintao/promqlpy/

Also thanks Julius, your information helped a lot. ;D

[Brian Candler]

On Thursday, 3 June 2021 at 00:04:30 UTC+1 cal...@callumj.com wrote:

With alerting rules the Prometheus API (and templating language) has access to the "Value" of an alert which appears to be the first metric specified without the comparator.

That statement isn't very accurate.

Firstly, PromQL works with values which are a *vector* or zero or more values.  For example, the expression

foo

might give the following vector of values (at the current point in time, or the specific time which you are querying for):

[

foo{bar="a",baz="x"} 123
foo{bar="a",baz="y"} 456
foo{bar="b",baz="z"} 789

]

An expression like foo{bar="a"} is a filter: it will limit the vector to just those values with label bar="a"

[

foo{bar="a",baz="x"} 123
foo{bar="a",baz="y"} 456

]

Similarly, an expression like foo > 200 is a filter: it will limit the vector to just those with value over 200, which in this case would be

[

foo{bar="a",baz="y"} 456
foo{bar="b",baz="z"} 789

]

And the expression "foo > 1000" will give an empty vector.

An alerting rule is just an expression like this, and the alert triggers whenever the vector is *not empty* - and generates multiple alerts if the vector has multiple values.  For each alert, the "Value" is simply the value of the element of that vector.  Prometheus is not working out "the first metric specified without the comparator" or anything like that.  *Any* value generates an alert.

Given that foo > 200 does not return a "boolean", it's a filter which narrows down the result set, these filters can be combined, e.g. "foo > 200 < 500".

If you *do* want boolean values (which I find quite rare in practice), then there is another form "foo > bool 200", which will give you:

[

foo{bar="a",baz="x"} 0
foo{bar="a",baz="y"} 1
foo{bar="b",baz="z"} 1

]

This is completely useless for an alerting rule, because you will get three alerts firing continuously: one alert with value 0 and two alerts with value 1.  *Any* value in a vector will generate an alert, even 0.

All the above is considering the case where the LHS is an instant vector and the RHS is a scalar.  You can also have expressions where both LHS and RHS are vectors: in this case, the values in one vector will be matched to those in the other vector which have identical label sets.  For example:

node_filesystem_avail_bytes < node_filesystem_free_bytes

My apologies if you know all this already :-)