How to deal with multivalue labels

[audrius.b...@gmail.com]

So let's talk about a hypothetical scenario where I have a set of machines, and each machine runs one or more applications. Each machine can be owned by one or more teams (as apps from different teams could run on each machine).

How does one label the metrics for this type of situation in the context of node_exporter?

I have two workarounds but both are flawed:

1. Have multiple scrape configs, one for each app/team combination scraping the same target multiple times. This works okayish, yet for basic things as in how many CPU cores does this $instance have to work out average load the queries become complicated as I somehow have to deduplicate the fact that the instance is scraped multiple times (potentially tens of times) and this also explodes the amount of metrics stored. Also all out of the box dashboards don't work.
2. Concat the apps into a single comma separated label and teams into the same comma separated label, yet this breaks group by clauses, and grafana can't split these fields, so query templating doesn't work as you'd expect.

[Brian Brazil]

 https://www.robustperception.io/how-to-have-labels-for-machine-roles/

--

Brian Brazil

www.robustperception.io

[audrius.b...@gmail.com]

So if I understand this correctly, this ends up working as a join condition, as in give me metric X where it has metric Y with value A joining on value B?

[audrius.b...@gmail.com]

It seems filtering is easy by multiple other metrics, which is nice.

Is it possible to group by two other metrics?

So I have datacenter{datacenter=X} team{team=t1} team{team=t2}

And I'd like to get average cpu utilization (load/core count) grouped by team and datacenter. A machine can belong to multiple teams, so the same instance might have to be included in multiple averages.

It seems you cannot group left twice to annotate other metrics with labels from two other metrics.

[audrius.b...@gmail.com]

Also it seems if you have an All value that is .* to disable filtering by datacenter for example it breaks. Not sure why, as I thought all you care about is the unique (job, instance) pairs the submetric filter returns.

so

up * (job, instance) team{team=t1} * on (job, instance) datacenter{datacenter=X}

works yet if you replace team{team=t1} with team{team=~.*} it doesn't work, yet having a grafana option on not filtering by team to get upness status makes sense?

[Moshe Beeri]

~.* is mistake since it includes empty values as well, the right way is ~.+