Incorrect Node Labels in kube_pod_container_info

[Murali Krishna Kanagala]

Hi All,

While working on some alert rules I have noticed that the metric kube_pod_container_Info has wrong values under the node label. This label uses the below scrape/relabel config which looks right. Whereas the other metric from the same job kube_pod_info has the correct node label value. The node value in the kube_pod_container_info is giving the name of the node where Prometheus is running (only returns 1 node value for all the pods in the cluster).

  - source_labels: [__meta_kubernetes_pod_node_name]
    separator: ;
    regex: (.*)
    target_label: node
    replacement: $1
    action: replace

Another observation is that a bunch of metrics under this job kubernetes-service-endpoints (like kube_pod_container_status_restarts_total) are also reporting only one node value for all the pods on the cluster. 

This is deployed using helm chart version 19.3.1 with the default scrape config. 

Any suggestions/ recommendations?

--

Regards,
Murali Krishna Kanagala

[Julius Volz]

Hi Murali,

So I guess both of those metrics are coming from the same kube-state-metrics target? While I'm not sure why those two metrics would be behaving differently, I'm not sure you'd want put the "__meta_kubernetes_pod_node_name" service discovery meta label into the "node" target label, because this discovery meta label contains only the node name of the discovered kube-state-metrics target and not of the individual specific pods that the metric from kube-state-metrics is about. So I would actually expect both metrics to only show one single "node" value ever after your relabeling rule (or as many values as you have k-s-m instances).

One thing I did notice on https://github.com/kubernetes/kube-state-metrics/blob/main/docs/pod-metrics.md is that "kube_pod_info" already has a "node" label to begin with (coming from kube-state-metrics), while "kube_pod_container_info" does not. But your relabeling rule sets a "node" label on the target itself, which will conflict during the scrape with the "node" label on the "kube_pod_info" metric. That should cause the original "node" label on "kube_pod_info" to be moved to an "exported_node" label to resolve the label collision. That is, unless you are setting the "honor_labels: true" option in the scrape config (see https://github.com/prometheus/prometheus/blob/64842f137e1ae6e041e12a2707d99d6da4ba885b/docs/configuration/configuration.md?plain=1#L135-L155), which would 100% explain why you'd see the behavior you're describing, since that would keep the original label from kube-state-metrics and throw away the conflicting target label.

Anyway, for diagnosing discovery and relabeling issues in general, it's always a good idea to check out the /service-discovery page on your Prometheus server. There you can inspect the labels of any target both before and after relabeling ("Discovered Labels" column vs. "Target Labels" column). You can also use the relabeling simulator / explainer tool at https://relabeler.promlabs.com/ (shameless plug) to enter your relabeling rules and the target labels before relabeling (both in YAML format) to see what effect the rules have on the labels.

Regards,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAKimyZvTqw9jqW%3DoWFa8StAUV2UnBFOZCp6xE-YwaUz0ZmjF4g%40mail.gmail.com.

--

Julius Volz

PromLabs - promlabs.com

[Murali Krishna Kanagala]

Thanks for the clarification Julius! I will read through the links you have shared.