Promql JOIN many-to-many matching

[Oltion Alimerko]

Hi all,

is it possible to use in promql the JOIN with many-to-many matching?I merged 2 metrics using group_left but it looks like the many-to-many matching is not supported because i get the error:

"Error executing query: found duplicate series for the match group {} on the left hand-side of the operation. Many-to-many matching not allowed: matching labels must be unique on one side"

On both sides of JOIN i have multitiple rows

Metric 1

kube_deployment_labels{label_chart=~".*sdc-5.17.*",project!=""}

Result of metric 1

kube_deployment_labels{deployment="sdc", instance="cfor-aks-dev", job="metrics-forwarder", label_app="sdc", label_app_kubernetes_io_managed_by="Helm", label_chart="sdc-5.17.2-HF01", label_heritage="Helm", label_release="sdc", namespace="dev-workloads", project="C4R"} 1 kube_deployment_labels{deployment="sdc", instance="sop-aks-dev", job="metrics-forwarder", label_app="sdc", label_app_kubernetes_io_managed_by="Helm", label_chart="sdc-5.17.1-b03", label_heritage="Helm", label_release="sdc", namespace="dev-workloads", project="SOP"} 1 kube_deployment_labels{deployment="sdc", instance="sop-aks-dev", job="metrics-forwarder", label_app="sdc", label_app_kubernetes_io_managed_by="Helm", label_chart="sdc-5.17.1-b03", label_heritage="Helm", label_release="sdc", namespace="test-workloads", project="SOP"} 1 kube_deployment_labels{deployment="sdc", instance="stu-aks-dev", job="metrics-forwarder", label_app="sdc", label_app_kubernetes_io_managed_by="Helm", label_chart="sdc-5.17.2-HF04-b01", label_heritage="Helm", label_release="sdc", namespace="dev-workloads", project="STU"}

Metric2

kube_node_info{container_runtime_version="containerd://1.4.4+azure", instance="cfor-aks-dev", job="metrics-forwarder", kernel_version="5.4.0-1051-azure", kubelet_version="v1.20.7", kubeproxy_version="v1.20.7", node="aks-default-13254112-vmss000001", os_image="Ubuntu 18.04.5 LTS", pod_cidr="10.244.2.0/24", project="C4R", provider_id="azure:///subscriptions/693c9868-a960-4590-b23d-7220a5a8ba04/resourceGroups/mc_rg_aks-c4r-aks-dev_c4r-aks-dev_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-default-13254112-vmss/virtualMachines/1"} 1 kube_node_info{container_runtime_version="containerd://1.4.4+azure", instance="cfor-aks-dev", job="metrics-forwarder", kernel_version="5.4.0-1051-azure", kubelet_version="v1.20.7", kubeproxy_version="v1.20.7", node="aks-default-13254112-vmss000004", os_image="Ubuntu 18.04.5 LTS", pod_cidr="10.244.3.0/24", project="C4R", provider_id="azure:///subscriptions/693c9868-a960-4590-b23d-7220a5a8ba04/resourceGroups/mc_rg_aks-c4r-aks-dev_c4r-aks-dev_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-default-13254112-vmss/virtualMachines/4"} 1 kube_node_info{container_runtime_version="containerd://1.4.4+azure", instance="stu-aks-dev", job="metrics-forwarder", kernel_version="5.4.0-1047-azure", kubelet_version="v1.20.7", kubeproxy_version="v1.20.7", node="aks-default-36930916-vmss000002", os_image="Ubuntu 18.04.5 LTS", pod_cidr="10.244.3.0/24", project="STU", provider_id="azure:///subscriptions/3fb69224-6feb-4c6c-9f55-0b233b82d4a2/resourceGroups/mc_rg_aks-stu-aks-dev_stu-aks-dev_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-default-36930916-vmss/virtualMachines/2"}

Now i want all fields from metric1 and the label “kubelet_version” from 2nd metric,let say i want to join kubelet_version to the 1st metric result.

I have wrote this metric:

kube_deployment_labels{label_chart=~".*sdc-5.17.*",project!=""} * on () group_right (kubelet_version)kube_node_info{}

Since both metrics give me more then 1 timeseries data it looks like that i want to achieve a many-to-many match.Would be this possible?

[Brian Candler]

You can't do a many-to-many join.  Even if you could it's unclear what the semantics would be.  (Would it be a cross-product, and what labels would the results have?)

Usually the solution is to summarise one side, or to add more fields to the on (...) clause, so that there is a one-to-many relationship.

After reformatting, I think the metrics you posted are these:

--------

kube_deployment_labels{deployment="sdc", instance="cfor-aks-dev", job="metrics-forwarder", label_app="sdc", label_app_kubernetes_io_managed_by="Helm", label_chart="sdc-5.17.2-HF01", label_heritage="Helm", label_release="sdc", namespace="dev-workloads", project="C4R"} 1

kube_deployment_labels{deployment="sdc", instance="sop-aks-dev", job="metrics-forwarder", label_app="sdc", label_app_kubernetes_io_managed_by="Helm", label_chart="sdc-5.17.1-b03", label_heritage="Helm", label_release="sdc", namespace="dev-workloads", project="SOP"} 1

kube_deployment_labels{deployment="sdc", instance="sop-aks-dev", job="metrics-forwarder", label_app="sdc", label_app_kubernetes_io_managed_by="Helm", label_chart="sdc-5.17.1-b03", label_heritage="Helm", label_release="sdc", namespace="test-workloads", project="SOP"} 1

kube_deployment_labels{deployment="sdc", instance="stu-aks-dev", job="metrics-forwarder", label_app="sdc", label_app_kubernetes_io_managed_by="Helm", label_chart="sdc-5.17.2-HF04-b01", label_heritage="Helm", label_release="sdc", namespace="dev-workloads", project="STU"} 1

--------

--------

kube_node_info{container_runtime_version="containerd://1.4.4+azure", instance="cfor-aks-dev", job="metrics-forwarder", kernel_version="5.4.0-1051-azure", kubelet_version="v1.20.7", kubeproxy_version="v1.20.7", node="aks-default-13254112-vmss000001", os_image="Ubuntu 18.04.5 LTS", pod_cidr="10.244.2.0/24", project="C4R", provider_id="azure:///subscriptions/693c9868-a960-4590-b23d-7220a5a8ba04/resourceGroups/mc_rg_aks-c4r-aks-dev_c4r-aks-dev_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-default-13254112-vmss/virtualMachines/1"} 1

kube_node_info{container_runtime_version="containerd://1.4.4+azure", instance="cfor-aks-dev", job="metrics-forwarder", kernel_version="5.4.0-1051-azure", kubelet_version="v1.20.7", kubeproxy_version="v1.20.7", node="aks-default-13254112-vmss000004", os_image="Ubuntu 18.04.5 LTS", pod_cidr="10.244.3.0/24", project="C4R", provider_id="azure:///subscriptions/693c9868-a960-4590-b23d-7220a5a8ba04/resourceGroups/mc_rg_aks-c4r-aks-dev_c4r-aks-dev_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-default-13254112-vmss/virtualMachines/4"} 1

kube_node_info{container_runtime_version="containerd://1.4.4+azure", instance="stu-aks-dev", job="metrics-forwarder", kernel_version="5.4.0-1047-azure", kubelet_version="v1.20.7", kubeproxy_version="v1.20.7", node="aks-default-36930916-vmss000002", os_image="Ubuntu 18.04.5 LTS", pod_cidr="10.244.3.0/24", project="STU", provider_id="azure:///subscriptions/3fb69224-6feb-4c6c-9f55-0b233b82d4a2/resourceGroups/mc_rg_aks-stu-aks-dev_stu-aks-dev_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-default-36930916-vmss/virtualMachines/2"} 1

--------

The question here is, is there *any* relationship between kube_deployment_labels and kube_node_info, and if so, what is it?  It looks like "instance" is a common label - does the instance refer to the cluster?  But then a cluster can have many deployments, and a cluster can also have many nodes.

In Kubernetes, a "deployment" is a cluster-level object, and the nodes are associated with the cluster.  But there's no relationship between a deployment and a node that I can think of.

Where it's more typical is to do joins between *pods* and *nodes*, because each pod is running on a node. There is an N:1 relationship between pod and node, and that lets you do a group_left or group_right join.

[Oltion Alimerko]

Hello,

the relationship between both metrics is that they have 2 fields in common,namespace and project.What i want to achieve is a JOIN of both metrics with these labels

1. label_chart (from kube_deployment_label)
2. namespace  (from kube_deployment_label)
3. project (from kube_deployment_label)
4. kernel_version  (from kube_node_info)
5. kubelet_version (from kube_node_info)
6. os_image  (from kube_node_info)

This result i want to visualize in grafana.But this combination leads to N:N matching because i have N rows on each side.The solution would be to add some filter at least on one side so that i can get 1:N matching but that its not what i wantend.The main idea was that for every project to list all enviroments(namespaces) and then join the k8s version.But since on other side i have more then 1 node its very hard how match all these labels

[Brian Candler]

(Grr: I gave a detailed reply yesterday and Google Groups deleted it.  I will try one more time)

Let's just think about kubelet_version.  The nodes of a cluster may have different kubelet versions.  The deployment is deployed to a cluster, and hence could be deployed to any or all nodes of that cluster.  Therefore, there is not necessarily a single value for "kubelet_version" that you can associate with a deployment.

What you *can* do is to group together the kubelet versions:

    count by (instance, kubelet_version) (kube_node_info)     # I'm assuming that "instance" is the cluster name

This will give a unique value for each (instance, kubelet_version) pair.  If all the nodes in a given cluster have identical kubelet versions, then you'll just get 1 metric per cluster.  At that point you can do a N:1 join:

    kube_deployment_labels * on (instance) group_left(kubelet_version) (count by (instance, kubelet_version) (kube_node_info))

That will annotate each deployment with the kubelet_version, as you wanted.  But if the cluster has inconsistent kubelet versions then it will fail, as expected, because it can't associate each cluster with a single kubelet version.

[Oltion Alimerko]

very helpful.

This is how it looks in alertmanager:

and this is how it looks in grafana:

Many thnx