kuberenetes-pods VS. kubernetes-service-endpoints scrape jobs

[Eric]

Hello,

I'm curious if there is a difference between the kubernetes-pods job and the kubernetes-service-endpoints job. Are these both collecting metrics reported by the application?

I'm currently testing some application metrics like `http requests by path and method`. When using the service-endpoints job I have some concerns that since services will load balance between pods, the metrics being returned from each pod will be different on each scrape interval (or worse if client affinity is set the job will only ever scrape data from a single pod). So, is the response to this problem to simply just scrape the pods directly using the kubernetes-pods config, or will this config scrape some default kubernetes metrics?

Any extra information around these two configs is much appreciated.

Regards,

Eric

[Tom Wilkie]

Hi Eric

> I'm curious if there is a difference between the kubernetes-pods job and the kubernetes-service-endpoints job. Are these both collecting metrics reported by the application?

An important distinction is for Pods that don't have any corresponding Services - they won't be scraped by the endpoints job, but will be by the pods job.  If you ever find yourself adding a Service just to collect metrics, you might consider scraping pods instead.

> When using the service-endpoints job I have some concerns that since services will load balance between pods, 

With the endpoints type Prometheus will see all endpoints for a service, and as such will be directly scraping the pods - the scrapes will not be load balanced.  

That being said, I tend to use the pods job over endpoints as when a Pod is terminating the Pod is removed from the list of endpoints, causing Prometheus to stop scraping it.  Some of my Pods take a long time to terminate and its very important that I gather metrics from them during this time.

Thanks

Tom

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6de02c15-88d6-42ed-9234-415f2fb3cca9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[er...@restorationmedia.com]

Thanks, Tom! I appreciate the feedback. I agree scraping the pods directly seems to be optimal per your comparison of the two.

[manans...@gmail.com]

Hi,

I am newbie to Prometheus and wanted to explore Pod scrapping rather than service-endpoint scraping. If possible please share me some examples to refer.

Thanks,

Manan