many-to-many matching not allowed

[Rakesh Kollakandi]

Hi,

I have a query like below:

(kubelet_volume_stats_used_bytes * on (namespace,persistentvolumeclaim) group_left(storageclass, provisioner) (kube_persistentvolumeclaim_info * on (storageclass)  group_left(provisioner) kube_storageclass_info {provisioner=~"(.*rbd.csi.ceph.com)|(.*cephfs.csi.ceph.com)"})) / (kubelet_volume_stats_capacity_bytes * on (namespace,persistentvolumeclaim) group_left(storageclass, provisioner) (kube_persistentvolumeclaim_info * on (storageclass)  group_left(provisioner) kube_storageclass_info {provisioner=~"(.*rbd.csi.ceph.com)|(.*cephfs.csi.ceph.com)"})) > 0.75

this query failed and report an error like below, 

many-to-many matching not allowed: matching labels must be unique on one side

Has anyone dealt with this specific error and can you recommend something?

[Brian Candler]

You have lots of sub-expressions there. Try breaking them down into parts until you find where the error is: you can enter them into the PromQL web interface.

A * on (uuu) group_left(vvv) B

A * on (uuu) group_left(vvv) B * on (www) group_left(xxx) C

A * on (uuu) group_left(vvv) B * on (www) group_left(xxx) C * on (yyy) group_left(zzz) D

... etc

Then when you find the error, perform the LHS and RHS queries separately, and inspect the labels.

The error is telling you that you have multiple instances of the same combination of labels on both sides of one of those operators.  With an expression like

    A * on (foo,bar) group_left(...) B

then every value of B must have a distinct (unique) combination of labels foo and bar, if there is at least one match on the A side.  That is, one or more "A" must match exactly one "B", and the matching involves the on(...) labels.

The solution could involve adding extra labels to the on(...) clause, but it really depends on what metrics you have there.

[Brian Candler]

BTW, there is an online PromQL pretty-printer here: https://jiacai2050.github.io/promql-prettier/

This formats your expression as:

  (
      (
          kubelet_volume_stats_used_bytes
        * on (namespace, persistentvolumeclaim) group_left (storageclass, provisioner)
          (
              kube_persistentvolumeclaim_info
            * on (storageclass) group_left (provisioner)
              kube_storageclass_info{provisioner=~"(.*rbd.csi.ceph.com)|(.*cephfs.csi.ceph.com)"}
          )
      )
    /
      (
          kubelet_volume_stats_capacity_bytes
        * on (namespace, persistentvolumeclaim) group_left (storageclass, provisioner)
          (
              kube_persistentvolumeclaim_info
            * on (storageclass) group_left (provisioner)
              kube_storageclass_info{provisioner=~"(.*rbd.csi.ceph.com)|(.*cephfs.csi.ceph.com)"}
          )
      )
  )
>
  0.75

I think this makes it much easier to understand and break down - for example, you can remove the ( .. ) > 0.75 wrapper easily.

Aside: you can use this format inside your alerting rules, using the YAML multi-line syntax (|): just make sure the whole expression is indented below the starting line.

expr: |

    (
        (
            kubelet_volume_stats_used_bytes
          * on (namespace, persistentvolumeclaim) group_left (storageclass, provisioner)
            (
                kube_persistentvolumeclaim_info
              * on (storageclass) group_left (provisioner)
                kube_storageclass_info{provisioner=~"(.*rbd.csi.ceph.com)|(.*cephfs.csi.ceph.com)"}
            )
        )
      /
        (
            kubelet_volume_stats_capacity_bytes
          * on (namespace, persistentvolumeclaim) group_left (storageclass, provisioner)
            (
                kube_persistentvolumeclaim_info
              * on (storageclass) group_left (provisioner)
                kube_storageclass_info{provisioner=~"(.*rbd.csi.ceph.com)|(.*cephfs.csi.ceph.com)"}
            )
        )
    )
  >
    0.75

Anyway, I would proceed by testing this expression:

              kube_persistentvolumeclaim_info
            * on (storageclass) group_left (provisioner)
              kube_storageclass_info{provisioner=~"(.*rbd.csi.ceph.com)|(.*cephfs.csi.ceph.com)"}

Then this one:

          kubelet_volume_stats_used_bytes
        * on (namespace, persistentvolumeclaim) group_left (storageclass, provisioner)
          (
              kube_persistentvolumeclaim_info
            * on (storageclass) group_left (provisioner)
              kube_storageclass_info{provisioner=~"(.*rbd.csi.ceph.com)|(.*cephfs.csi.ceph.com)"}
          )

Then if those are OK then do the same for the lower half, and then finally combine them.

[Julius Volz]

Besides Brian's answers, it can greatly help to use PromLens (https://github.com/prometheus/promlens, see promlens.com for some screenshots/videos) to analyze complex expressions like this. Using PromLens, you can click on any part of the query to run it separately, and you can look at binary operators in an "Explain" tab that shows you the matching behavior. Currently it only shows you the matching when there is no error, but I'm working on a complete revamp of that binary operator visualizer to show matchings for all error cases as well. Eventually we also hope to put the same functionality into Prometheus itself, but the benefit of PromLens as a standalone tool is that you can run it against any PromQL-compatible system, not just Prometheus itself.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a42b0758-bc55-4f1b-9698-45186b978e65n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com