How to properly monitoring kubernetes service up/down status

[Xiao Han]

Hi,

  I'm using prometheus with kubernetes in our production environment, it's greate and helpful. But I met some problem about monitoring the k8s Service status using prometheus.

  Here is the basic information and my problem in detail.

  I run prometheus inside k8s as a Pod, and using kubernetes_sd to find all pods and services, it's all good.

  Generally our application is running in Pod, and we create one Kubernetes Service for each application with two Pods replica. We have an health check endpoint like `/health` and it returns 200 OK.

  The thing I want to achieve is: create a dashboard showing the Service is running correctly, which mean up or down.

  The way I use now is configure prometheus with kubernetes_sd, role='service', and for each service, the endpoint to pull is like `http://<service-name>:<service-port>/health`, and the prometheus

query I can use is like `up{job='kubernetes-service', kubernetes_service_name=<service-name>}`.

  The problem of this way is, it only works if the `health` endpoint returns EMPTY content or prometheus METRICS format data. But to let a /health endpoint to return metrics is not proper way, so now

we have to make it return empty content. Actually we want the '/health' endpoint to return some text like 'ok', but it will not work in prometheus.

  So I want to ask is my solution a proper way to monitoring kubernetes service status? If yes, does prometheus have a better way to support general health check endpoint like 'ok' response?

Thanks!

[Ben Kochie]

Please take this question to the prometheus-users list.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/5ef7a7cb-edda-4b89-93de-553430066149%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Xiao Han]

sure

On Thursday, February 23, 2017 at 6:18:39 PM UTC+9, Ben Kochie wrote:

Please take this question to the prometheus-users list.

On Thu, Feb 23, 2017 at 5:10 AM, Xiao Han <justl...@gmail.com> wrote:

Hi,

  I'm using prometheus with kubernetes in our production environment, it's greate and helpful. But I met some problem about monitoring the k8s Service status using prometheus.

  Here is the basic information and my problem in detail.

  I run prometheus inside k8s as a Pod, and using kubernetes_sd to find all pods and services, it's all good.

  Generally our application is running in Pod, and we create one Kubernetes Service for each application with two Pods replica. We have an health check endpoint like `/health` and it returns 200 OK.

  The thing I want to achieve is: create a dashboard showing the Service is running correctly, which mean up or down.

  The way I use now is configure prometheus with kubernetes_sd, role='service', and for each service, the endpoint to pull is like `http://<service-name>:<service-port>/health`, and the prometheus

query I can use is like `up{job='kubernetes-service', kubernetes_service_name=<service-name>}`.

  The problem of this way is, it only works if the `health` endpoint returns EMPTY content or prometheus METRICS format data. But to let a /health endpoint to return metrics is not proper way, so now

we have to make it return empty content. Actually we want the '/health' endpoint to return some text like 'ok', but it will not work in prometheus.

  So I want to ask is my solution a proper way to monitoring kubernetes service status? If yes, does prometheus have a better way to support general health check endpoint like 'ok' response?

Thanks!

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.