Pod with Pending phase is in endpoints scraping targets (Prometheus 2.46.0)

[Vu Nguyen]

Hi,

With endpoints scraping role, the job should scrape POD endpoint that is up and running. That is what we are expected. 

I think by concept, K8S does not create an endpoint if Pod is in other phases like Pending, Failed, etc.

In our environments, Prometheus 2.46.0 on K8S v1.28.2, we currently have issues: 

1) POD is up and running from `kubectl get pod`, but from Prometheus discovery page, it shows:

__meta_kubernetes_pod_phase="Pending" 

__meta_kubernetes_pod_ready="false"  

2) The the endpoints job discover POD targets with pod phase=`Pending`.

Those issues disappear after we restart Prometheus pod.  

I am not sure if 1) that is K8S that does not trigger event after POD phase changes so Prometheus is not able to refresh its endpoints discovery or 2) it is a known problem of Prometheus? 

And do you think it is worth to add the following relabeling rule to endpoints job role?

  - source_labels: [ __meta_kubernetes_pod_phase ]

    regex: Pending|Succeeded|Failed|Completed
    action: drop

Thanks, Vu

[Vu Nguyen]

Hi,

Do you have a response to this thread? Has anyone ever encountered the issue?

Regards,

Vu

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c0f97ed7-1421-4c7c-a57d-2d301bb12418n%40googlegroups.com.

[Brian Candler]

Have you looked in the changelog for Prometheus? I found:

## 2.51.0 / 2024-03-18

* [BUGFIX] Kubernetes SD: Pod status changes were not discovered by Endpoints service discovery #13337

=> fixes #11305, which looks similar to your problem

## 2.50.0 / 2024-02-22

* [ENHANCEMENT] Kubernetes SD: Check preconditions earlier and avoid unnecessary checks or iterations in kube_sd. #13408

I'd say it's worth trying the latest release, 2.51.2.

[Vu Nguyen]

Thank you, Brian. #11305 looks exactly what we are encountering. 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0641d658-e295-418b-ae00-af6ce83e7ccbn%40googlegroups.com.