up wrongly set to 0 on 2 pods

4 views
Skip to first unread message

klavs....@gmail.com

unread,
Sep 2, 2020, 4:28:48 AM9/2/20
to Prometheus Users
My prometheus is suddenly seeing 2 pods as up == 0 - but when I look at them - they have status Running 3/3 and Running 2/2 respectively - so they seem just fine.
How can I debug how they end up with having the up metric set to 0?

promql query: up == 0
returns:
up{app="dynamic-gateway-service",controller_revision_hash="rcd4359800f-dynamic-gateway-service-898c67dc8",instance="192.168.26.20:63512",job="kubernetes-pods",kubernetes_namespace="api-klavs-gwb-d1",kubernetes_pod_name="rcd4359800f-dynamic-gateway-service-0",release="rcd4359800f",statefulset_kubernetes_io_pod_name="rcd4359800f-dynamic-gateway-service-0"}  
up{app="dynamic-gateway-service",controller_revision_hash="r7fdf61c6a9-dynamic-gateway-service-8867b4f6c",instance="192.168.97.149:63512",job="kubernetes-pods",kubernetes_namespace="api-klavs-gwa-d1",kubernetes_pod_name="r7fdf61c6a9-dynamic-gateway-service-0",release="r7fdf61c6a9",statefulset_kubernetes_io_pod_name="r7fdf61c6a9-dynamic-gateway-service-0"} 

and kubectl get pods in klavs-api-gwa has this:
r7fdf61c6a9-dynamic-gateway-service-0            3/3       Running   0          21h

 
Reply all
Reply to author
Forward
0 new messages