How to expose a custom metrics from a pod/service in Kubernetes ?

[SS. OO.]

Hi!

I have a Kubernetes/Prometeus newbie question.

I have a Kubernetes application that I want to scale up via Horizontal Pod Autoscaler based on custom metrics reflecting the length of the app's work queue backlog in the app's internal database.

I can expose the metrics on a pod port in Prometheus format, so there is more than one scrape endpoint for fail-over purposes. However I do not want Prometheus to call the scrape endpoint on every pod's instance, only one instance at a time and failover to another instance only if necessary. All instance endpoints report the same data, and calling them in parallel would serve no useful purpose and only increase a workload on storage queries behind the app runtime.

Right now the application is packaged as a Kubernetes Deployment. Do I need to convert it for the above as a Service, or can Prometeus understand intrinsically or be instructed in the configuration that only one instance is to be called at a time?

Also, can someone please provide an example of the snippet I need to insert into 

https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml

so Prometheus picks up the endpoint? Assuming, for example, it is exposed on pod port 3333, path /metrics, service/pod named ABC or with label appname=LABC? 

Or do I simply tag the pod/service/deployment with some tag and Prometeus starts to collect on it? If yes, how do I pack the endpoint port+protocol+path into the tag?

Thanks for the advice.

[Murali Krishna Kanagala]

I think this page can help you to write the sd config for kubernetes pods.

https://github.com/nlighten/tomcat_exporter

For scraping only master pod of a given service it should have a special label so that Prometheus can discover it. An alternative approach would be a stateful set where every pod in a service can be given different labels.

Did you try reading about kube dd config to scrape service endpoints? If that is possible then you will be scraping the master pod if you scrape the service endpoint.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cabf6ba3-79c8-4f5e-865d-1fc51ba52c72%40googlegroups.com.

[Murali Krishna Kanagala]

Sorry for the wrong link

https://medium.com/@zhimin.wen/custom-prometheus-metrics-for-apps-running-in-kubernetes-498d69ada7aa