Why do logical / set operators only consider label sets, and not values?

[co...@freshtracks.io]

While trying to set up some fancy annotations in Grafana from a Prometheus query, I found that using, say, "or" between two comparison queries which use the "bool" modifier (https://prometheus.io/docs/querying/operators/)  causes me to lose all elements on the right side of the set operator.

As a simplified example, here's a query that (I think) should result in a flat line graph:

(sum(node_load1{instance="foo"}) > bool 3) or (sum(node_load1{instance="foo"}) < bool 3)

In other words, "node_load1" is either going to be above or below 3 - there are no other options.

As can be seen in the attached screenshot, this is not the behavior I see because all of the labels are stripped out when using the "bool" modifier. As stated in the docs linked above, this causes the union operator to retain all elements on the left side of the operator and drop all elements on the right side of the operator with label sets that match the left side.

Basically, Prometheus is behaving precisely as documented - but I'm curious as to why it was designed to only consider label sets, and ignore values. Since the label sets will always be empty when using the "bool" modifier on both sides of a set operator, it seems like it would be helpful to allow the right side's element values to act as the "tie-breaker" and allow the behavior I'd expect. There's a good chance I'm misunderstanding the reasons for this design, so I wanted to see if I could get some extra clarity.

Note: Oddly, this behavior doesn't happen when I remove the "bool" modifiers - I get a valid union of the two sets, near as I can tell. I would have thought it would behave the same as described above since the label sets would match, but I feel like I'm misunderstanding something.

[Brian Brazil]

On 19 July 2017 at 15:56, <co...@freshtracks.io> wrote:

While trying to set up some fancy annotations in Grafana from a Prometheus query, I found that using, say, "or" between two comparison queries which use the "bool" modifier (https://prometheus.io/docs/querying/operators/)  causes me to lose all elements on the right side of the set operator.

As a simplified example, here's a query that (I think) should result in a flat line graph:

(sum(node_load1{instance="foo"}) > bool 3) or (sum(node_load1{instance="foo"}) < bool 3)

In other words, "node_load1" is either going to be above or below 3 - there are no other options.

As can be seen in the attached screenshot, this is not the behavior I see because all of the labels are stripped out when using the "bool" modifier.

The labels are removed by the sum(), the bool modifier would only remove the __name__ label.

 

As stated in the docs linked above, this causes the union operator to retain all elements on the left side of the operator and drop all elements on the right side of the operator with label sets that match the left side.

Basically, Prometheus is behaving precisely as documented - but I'm curious as to why it was designed to only consider label sets, and ignore values. Since the label sets will always be empty when using the "bool" modifier on both sides of a set operator, it seems like it would be helpful to allow the right side's element values to act as the "tie-breaker" and allow the behavior I'd expect. There's a good chance I'm misunderstanding the reasons for this design, so I wanted to see if I could get some extra clarity. 

Note: Oddly, this behavior doesn't happen when I remove the "bool" modifiers - I get a valid union of the two sets, near as I can tell. I would have thought it would behave the same as described above since the label sets would match, but I feel like I'm misunderstanding something.

The use of the bool modifier is the issue here, as it always results in 0/1 rather than the default filtering behaviour of the > comparison operator. Using bool with or like this doesn't make sense, as it'll always return the left hand side.

--

Brian Brazil

www.robustperception.io

[co...@freshtracks.io]

Ah, ok, I didn't realize sum was removing the labels, that makes sense. Oops.

That said, I understand that it doesn't currently make sense to do what I'm doing, as the behavior of Prometheus is (intentionally, it seems) to only retain the left hand side when using bool. My confusion is around why it is this way - if I have two sets of binary values, and I "or" them together, I should get a new same-length set that contains all of the "one" values from the two input sets... yes?

I guess what I'm expecting it to do is something like:

0 1 0 1 0 0 0 1

OR

0 0 0 1 1 1 0 0

---------------

0 1 0 1 1 1 0 1

If this is not what's intended with the union operator, then I don't guess I understand why that's the case... :-)

Thanks!

[Brian Brazil]

On 19 July 2017 at 17:00, <co...@freshtracks.io> wrote:

Ah, ok, I didn't realize sum was removing the labels, that makes sense. Oops.

That said, I understand that it doesn't currently make sense to do what I'm doing, as the behavior of Prometheus is (intentionally, it seems) to only retain the left hand side when using bool. My confusion is around why it is this way - if I have two sets of binary values, and I "or" them together, I should get a new same-length set that contains all of the "one" values from the two input sets... yes?

I guess what I'm expecting it to do is something like:

0 1 0 1 0 0 0 1

OR

0 0 0 1 1 1 0 0

---------------

0 1 0 1 1 1 0 1

You want the max() aggregator. See https://www.robustperception.io/booleans-logic-and-math/ for more tricks with booleans.

The three logical operators and, or, and unless operate only on labels.

Brian

 

If this is not what's intended with the union operator, then I don't guess I understand why that's the case... :-)

Thanks!

On Wednesday, July 19, 2017 at 9:45:49 AM UTC-6, Brian Brazil wrote:

On 19 July 2017 at 15:56, <co...@freshtracks.io> wrote:

While trying to set up some fancy annotations in Grafana from a Prometheus query, I found that using, say, "or" between two comparison queries which use the "bool" modifier (https://prometheus.io/docs/querying/operators/)  causes me to lose all elements on the right side of the set operator.

As a simplified example, here's a query that (I think) should result in a flat line graph:

(sum(node_load1{instance="foo"}) > bool 3) or (sum(node_load1{instance="foo"}) < bool 3)

In other words, "node_load1" is either going to be above or below 3 - there are no other options.

As can be seen in the attached screenshot, this is not the behavior I see because all of the labels are stripped out when using the "bool" modifier.

The labels are removed by the sum(), the bool modifier would only remove the __name__ label.

 

As stated in the docs linked above, this causes the union operator to retain all elements on the left side of the operator and drop all elements on the right side of the operator with label sets that match the left side.

Basically, Prometheus is behaving precisely as documented - but I'm curious as to why it was designed to only consider label sets, and ignore values. Since the label sets will always be empty when using the "bool" modifier on both sides of a set operator, it seems like it would be helpful to allow the right side's element values to act as the "tie-breaker" and allow the behavior I'd expect. There's a good chance I'm misunderstanding the reasons for this design, so I wanted to see if I could get some extra clarity. 

Note: Oddly, this behavior doesn't happen when I remove the "bool" modifiers - I get a valid union of the two sets, near as I can tell. I would have thought it would behave the same as described above since the label sets would match, but I feel like I'm misunderstanding something.

The use of the bool modifier is the issue here, as it always results in 0/1 rather than the default filtering behaviour of the > comparison operator. Using bool with or like this doesn't make sense, as it'll always return the left hand side.

--

Brian Brazil

www.robustperception.io

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/260c37ad-e527-4b74-b41d-4d32b2164ad1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[co...@freshtracks.io]

Aha! That's the piece I was missing - didn't realize that max() could act as an "or" in the way I wanted. Given that, it makes sense why or / and / unless are restricted in scope to just label sets. 

Thanks a million!

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/260c37ad-e527-4b74-b41d-4d32b2164ad1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[co...@freshtracks.io]

For anyone else who might be looking for the working query:

(sum(node_load1{instance="foo"}) > bool 3) + (sum(node_load1{instance="foo"}) < bool 3) > bool 0

I used this form instead of max() because of the multiple variable bit, but I imagine it could be made to work either way.