Instance value in PromQL output is different from the actual node where container is running

[Binu Nadarajan]

Hi all,

I am using Prometheus and cAdvisor to monitor containers running in 4 nodes Kubernetes cluster.

Let's say,

Master IP: 192.168.13.197

Worker-node 1: 192.168.13.198

Worker-node 2: 192.168.13.234

Worker-node 3: 192.168.13.235

I deployed cAdvisor as daemonset and scraping using below configuration in prometheus.yml

- job_name: cadvisor

  static_configs:

      - targets: ['192.168.13.197:31223', '192.168.13.198:31223', '192.168.13.234:31223', '192.168.13.235:31223']

I deployed a container busybox in Worker-node3 (192.168.13.235) and ensure that the container is running as expected.

kube[root@master-node ~]# kubectl get pods  --all-namespaces -o wide | grep busybox

monitoring       busybox                                      1/1     Running   0          29m   173.168.200.67   worker-node3   <none>           <none>

[root@master-node ~]#

When I query the statistics of this container only when I do below query 192.168.13.198:31223. But, I am expecting it should be available when I query 192.168.13.235:31223.

container_network_transmit_packets_total{container_label_annotation_hash_operator_tigera_io_cni_config="",container_label_annotation_hash_operator_tigera_io_node_cert="",container_label_annotation_hash_operator_tigera_io_typha_ca="",container_label_annotation_hash_operator_tigera_io_typha_cert="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubectl_kubernetes_io_last_applied_configuration="{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"labels\":{\"run\":\"busybox\"},\"name\":\"busybox\",\"namespace\":\"monitoring\"},\"spec\":{\"containers\":[{\"args\":[\"sleep\",\"1000000\"],\"image\":\"busybox\",\"name\":\"busybox\"}]}}\n",container_label_annotation_kubernetes_io_config_seen="2021-01-12T17:16:41.579490144+05:30",container_label_annotation_kubernetes_io_config_source="api",container_label_app="",container_label_component="",container_label_controller_revision_hash="",container_label_description="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="busybox",container_label_io_kubernetes_pod_namespace="monitoring",container_label_io_kubernetes_pod_uid="fb5da70b-f8db-4394-8725-4e28a4635ac3",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_run="busybox",container_label_summary="",container_label_vendor="",container_label_version="",id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb5da70b_f8db_4394_8725_4e28a4635ac3.slice/docker-2e56f2f1ffa9ffaf1f3078fa0f409ec08e6e81358ad44bf23e5eb3e1797f4fb4.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_busybox_monitoring_fb5da70b-f8db-4394-8725-4e28a4635ac3_0"} 8.973906e+06 1610452889584

container_network_transmit_packets_total{container_label_annotation_hash_operator_tigera_io_cni_config="",container_label_annotation_hash_operator_tigera_io_node_cert="",container_label_annotation_hash_operator_tigera_io_typha_ca="",container_label_annotation_hash_operator_tigera_io_typha_cert="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubectl_kubernetes_io_last_applied_configuration="{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"labels\":{\"run\":\"busybox\"},\"name\":\"busybox\",\"namespace\":\"monitoring\"},\"spec\":{\"containers\":[{\"args\":[\"sleep\",\"1000000\"],\"image\":\"busybox\",\"name\":\"busybox\"}]}}\n",container_label_annotation_kubernetes_io_config_seen="2021-01-12T17:16:41.579490144+05:30",container_label_annotation_kubernetes_io_config_source="api",container_label_app="",container_label_component="",container_label_controller_revision_hash="",container_label_description="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="busybox",container_label_io_kubernetes_pod_namespace="monitoring",container_label_io_kubernetes_pod_uid="fb5da70b-f8db-4394-8725-4e28a4635ac3",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_run="busybox",container_label_summary="",container_label_vendor="",container_label_version="",id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb5da70b_f8db_4394_8725_4e28a4635ac3.slice/docker-2e56f2f1ffa9ffaf1f3078fa0f409ec08e6e81358ad44bf23e5eb3e1797f4fb4.scope",image="k8s.gcr.io/pause:3.2",interface="tunl0",name="k8s_POD_busybox_monitoring_fb5da70b-f8db-4394-8725-4e28a4635ac3_0"} 0 1610452889584

Can anyone tell me is it working as expected and my expectation is wrong?

Please help me to understand how this works.

Let me know if any additional details needed.

Thanks in advance.

Best regards,

Binu