How is Prometheus Remembering this instance in this query?

[Kristopher Kahn]

I have this query:

((node_memory_MemTotal{instance="hostname.example.com:9100"} - (node_memory_MemFree + node_memory_Buffers + node_memory_Cached)) / node_memory_MemTotal) * 100 

I'm surprised to find that every other metric after:

node_memory_MemTotal{instance="hostname.example.com:9100"}

"knows" to use that instance, such as node_memory_MemFree, and that I don't need to declare that instance every time thereafter so that the query grows into:

((node_memory_MemTotal{instance="hostname.example.com:9100"} - (node_memory_MemFree{instance="hostname.example.com:9100"} + node_memory_Buffers{instance="hostname.example.com:9100"} + node_memory_Cached{instance="hostname.example.com:9100"})) / node_memory_MemTotal{instance="hostname.example.com:9100"}) * 100

How does Prometheus know to keep and use that same returned instance for the entire query?

[Julius Volz]

This is an effect of the automatic join (vector matching) that Prometheus does around binary operators. By default, a binary operator will look for series with exactly identical label sets on the left and right side of the operator, and then make an identically labeled output series with that operation applied. For series that do *not* find an exact correspondence on the other side, they are simply dropped from the result. So if one of your sides is a subset of the other (as in your case), only that subset will find a label match and make it into the result. You will still have some cost though by first selecting those extra series that then get thrown away by the binary operator matching, so it may or may not be a good idea (efficiency-wise) to do the filtering everywhere (if otherwise the set of selected series would be very large).

I visualized it with a somewhat simpler scenario here, see the "Explain" tab on this query showing how two of the three series on the right side do not find a match on the left side:

https://demo.promlens.com/?l=bqxzm3klwDa

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/06b8d835-2c30-43dd-9467-16f6c032e28an%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Kristopher Kahn]

Really useful! I thought I was going mad trying to understand how this was working. Thank you very much for your thorough explanation!