Prometheus-Grafana error - many-to-many matching not allowed

[Hemal Solgama]

Hello,

I'm trying to get the result from two different prometheus metrics. Basically my need is, I want to set an alert on container restart count but the list also includes cron jobs which I don't want. I only want to get container restart count for deployments, statefulsets and daemonsets. For which I built following query expression:

app:kube_cronjob_info:sum =

sum by (container, label_app) (sum(kube_cronjob_info{cronjob=~".*"}) by (cronjob, namespace) * on (namespace, cronjob) group_left(app) label_replace(kube_pod_container_status_restarts_total, "cronjob", "$1", "container", "(.*)"))

I just took a reference from here: https://groups.google.com/forum/#!topic/prometheus-users/NBbShXgn-6E

But this recording rule doesn't work in Prometheus (v2.21). It returns an error of "many-to-many matching not allowed: matching labels must be unique on one side"

Can anyone please help me to achieve this?

Thank you..

[Bjoern Rabenstein]

On 23.10.20 07:26, 'Hemal Solgama' via Prometheus Users wrote:
>
> app:kube_cronjob_info:sum =
> sum by (container, label_app) (sum(kube_cronjob_info{cronjob=~".*"}) by
> (cronjob, namespace) * on (namespace, cronjob) group_left(app) label_replace
> (kube_pod_container_status_restarts_total, "cronjob", "$1", "container", "(.*)
> "))

> [...]

> But this recording rule doesn't work in Prometheus (v2.21). It returns an error
> of "many-to-many matching not allowed: matching labels must be unique on one
> side"

The error message means that the right side of the multiplication has
more than one element matching at least one (namespace, cronjob) pair
that exists on the left side.

I haven't analyzed your query in depth, but `group_left` cannot be
helpful here because it would imply that you have multiple matches per
(namespace, cronjob) pair on the left side. But the left side is a
`sum(...) by (namespace, cronjob)` expression, so it's guaranteed to
have only one element for each (namespace, cronjob) pair.

You could just try `group_right` but it would be better to understand
the query better.

https://prometheus.io/docs/prometheus/latest/querying/operators/#many-to-one-and-one-to-many-vector-matches
is a dense but good read in the official docs.

To understand complicated queries like this, the proprietary service
https://promlens.com/ might be helpful.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in