can we combine different metrics

[BHARATH KUMAR]

Hello All,

Can we add two Prometheus queries as a single query?

For ex: I want to add CPU cores and memory as a single query. Is it possible?

I tried something like this:

((node_memory_MemTotal_bytes{instance=~"$ip"} - 0) /1024*1024) + ((count(node_cpu_seconds_total{mode='system',instance=~"$ip"}) by (instance)))

Thanks & regards,

Bharath Kumar.

[Brian Candler]

Yes.  In short:

1. you can directly combine two instance vectors, if the metrics have *exactly* the same set of labels

e.g.

    node_filesystem_avail_bytes / node_filesystem_size_bytes

2. if the label sets are not exactly the same, then you have to say which labels to match on, or which labels to exclude, so that the label sets on LHS and RHS correspond.

3. as long as each metric on the LHS matches exactly one metric on the RHS,  then this works fine

4. if there are N metrics on the LHS which match one metric on the RHS, or vice versa, then you can specify "group left" or "group right"

5. you can't have a situation where N metrics on the LHS match M metrics on the RHS (N>1 and M>1)

The full details are in the documentation:

https://prometheus.io/docs/prometheus/latest/querying/operators/

And there are more useful links about PromQL querying here:

https://prometheus.io/docs/prometheus/latest/querying/examples/
https://github.com/infinityworks/prometheus-example-queries
https://timber.io/blog/promql-for-humans/
https://www.weave.works/blog/promql-queries-for-the-rest-of-us/
https://www.slideshare.net/weaveworks/promql-deep-dive-the-prometheus-query-language
https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085
https://www.robustperception.io/blog
https://www.robustperception.io/extracting-raw-samples-from-prometheus
https://www.robustperception.io/common-query-patterns-in-promql
https://www.robustperception.io/booleans-logic-and-math
https://www.robustperception.io/composing-range-vector-functions-in-promql
https://www.robustperception.io/rate-then-sum-never-sum-then-rate
https://www.robustperception.io/using-group_left-to-calculate-label-proportions

As for your specific example: it seems to be pretty meaningless to add memory (in units of MB) to number of cores (dimensionless). But it can be done.  Note that:

node_memory_MemTotal_bytes/1024/1024  has one unique timeseries for each value of the 'instance' label

((count(node_cpu_seconds_total{mode='system'}) by (instance)))  also has one unique timeseries for each value of the 'instance' label

Hence you can match up timeseries on the LHS and RHS which have corresponding values of the "instance" label, ignoring all other labels.  This gives you the following solution:

node_memory_MemTotal_bytes/1024/1024 + on (instance) ((count(node_cpu_seconds_total{mode="system'") by (instance)))

Note: I have left out the additional label filter {instance=~"$ip"} here; the query will return a vector which contains this calculated value for *all* instances.  But you can add it back, if you only want a subset of instances.

In fact, you only need to put it on one side, either the LHS or the RHS, like this:

node_memory_MemTotal_bytes{instance=~"$ip"}/1024/1024 + on (instance) ((count(node_cpu_seconds_total{mode="system'") by (instance)))

This is because any values on the RHS which don't match the LHS will be ignored, because of the rule given before: the labels on the LHS and the RHS must match.