Pod Labels in Metrics

[neha...@gmail.com]

Hi Guys,

I am using prometheus-operator to monitor my cluster installed via kubeadm. I want me replicaset(pod) labels to appear in all related pod & container metrics. I installed latest prometheus-operator as mention on following git page:

https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus

My replicaset pod is labeled as follows:

ubuntu@ip-10-18-31-100:~$ kubectl describe pod test-container-0809-6797d846bc-2vgrr -n ns-7w2lnkp-rafay1

Name:           test-container-0809-6797d846bc-2vgrr

Namespace:      ns-7w2lnkp-rafay1

Node:           ip-10-18-1-10/10.18.1.10

Start Time:     Fri, 17 Aug 2018 23:34:37 +0000

Labels:         pod-template-hash=2353840267

                rep-app=test-container-0809

                rep-organization=7w2lnkp

                rep-partner=rx28oml

                rep-workload=jkedj4m

Annotations:    <none>

Status:         Running

IP:             10.244.4.239

Controlled By:  ReplicaSet/test-container-0809-6797d846bc

but when I pull metrics none of labels (like rep-app, rep-partner etc) appear:

container_cpu_load_average_10s{container_name="POD",endpoint="https-metrics",id="/kubepods/burstable/pod1b166fd",image="gcr.io/google_containers/pause-amd64:3.0",instance="10.18.1.10:10250",job="kubelet",name="k8s_POD_test-container-0809-6797d846bc-2vgrr_ns-7w2lnkp-rafay1_1b166fde-a276-11e8-b731-024022312e08_0",namespace="ns-7w2lnkp-rafay1",pod_name="test-container-0809-6797d846bc-2vgrr",service="kubelet"}

Do I need to enable this feature via config?

Thanks For help!

[neha...@gmail.com]

Look like joining metrics coming out of cadvisor with kube_pod_labels is only way. Am I correct?