Howto have Prometheus *outside* Kubernetes scrape metrics from pods *inside* Kubernetes?

1,855 views
Skip to first unread message

Jan Algermissen

unread,
Sep 30, 2016, 3:44:31 PM9/30/16
to Prometheus Developers
Hi,

I am running a Prometheus to collect metrics from services running as pods in Kubernetes cluster(s). I want to keep Prometheus out of Kubernetes. Each pod exposes a /metrics scraping resource.

Can someone explain (or point me to) how I need to configure Kubernetes to be able to scrape the pods?

I do not want to include the Prometheus VM into SkyDNS.

So I imagine I need to configure somehow that Prometeus is using minion IPs and pod ports to access the /metrics resources.

Has anyone done this before?

( I have seen https://www.weave.works/prometheus-kubernetes-deploying/ but this leaves exactly my use case open as "slightly harder" :o)


Jan

Tom Wilkie

unread,
Sep 30, 2016, 3:53:29 PM9/30/16
to Jan Algermissen, Prometheus Developers
Where are you running your kubernetes cluster? What do you user for pod networking? 

If you use weave net, it's as easy as installing weave on your Prometheus host and running 'weave expose'.  I think you can achieve similar with flannel and vpc networking. 

Thanks

Tom
--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/b1235a62-31e7-4d96-bda1-d4e90984aa73%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Matthias Rampke

unread,
Oct 1, 2016, 11:45:24 AM10/1/16
to Tom Wilkie, Jan Algermissen, Prometheus Developers
As for the discovery itself, instead of `in_cluster: true` you need to configure basic auth, a client certificate/key pair or a bearer token as part of the `kubernetes_sd_config`. That's all.

For finding the port from which to scrape, build on the standard config[0]. The general gist is to use annotations on pods to tell Prometheus which port to scrape. Alternatively, if you have only few apps and want to keep all configuration on the Prometheus side, you can also use relabelling to simply filter by different namespaces and labels for each job. You'll have to duplicate the kubernetes_sd_config for every job.

/MR




On Fri, Sep 30, 2016 at 7:53 PM Tom Wilkie <t...@weave.works> wrote:
Where are you running your kubernetes cluster? What do you user for pod networking? 

If you use weave net, it's as easy as installing weave on your Prometheus host and running 'weave expose'.  I think you can achieve similar with flannel and vpc networking. 

Thanks

Tom


On Friday, 30 September 2016, Jan Algermissen <algermi...@gmail.com> wrote:
Hi,

I am running a Prometheus to collect metrics from services running as pods in Kubernetes cluster(s). I want to keep Prometheus out of Kubernetes. Each pod exposes a /metrics scraping resource.

Can someone explain (or point me to) how I need to configure Kubernetes to be able to scrape the pods?

I do not want to include the Prometheus VM into SkyDNS.

So I imagine I need to configure somehow that Prometeus is using minion IPs and pod ports to access the /metrics resources.

Has anyone done this before?

( I have seen https://www.weave.works/prometheus-kubernetes-deploying/ but this leaves exactly my use case open as "slightly harder" :o)


Jan

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAGoUHOMsVO_SDiX4%2Bsu3Ajpu%3Dr2LP%2B-e9yrDxZ_4xDMuhDzRtw%40mail.gmail.com.

Jan Algermissen

unread,
Oct 1, 2016, 5:51:06 PM10/1/16
to Prometheus Developers, algermi...@gmail.com
Thank you for the replies.

I can not make use of K8S magic because I cannot make Prometheus node part of the overlay network (I am monitoring more than one K8S cluster with one Prometheus)

However, I wanted to quickly share the solution I found:

I have set up a simple HTTP proxy per K8S cluster on a node that is in the given cluster's overlay network and can thus resolve the individual pod IPs. I can now make proxy requests to that proxy using the pod IPs.

Then I use Prometheus' file_sd_configs and generate the targets file dynamically from the K8S API as a list of pod IPs and exposed ports.

What is left is to configure the proxy in the file_sd_configs section.

Thanks all,

Jan





Ben Kochie

unread,
Oct 2, 2016, 9:07:21 AM10/2/16
to Jan Algermissen, Prometheus Developers
Another option is to run one Prometheus inside the k8s cluster, and use federation to extract the data.  We've been doing this with some success, however the large the data, the more lag you get.

Someday soon we'll have a Prometheus-as-a-proxy with the generic write API.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/16bb3861-0310-4f1f-89f9-bb5886420261%40googlegroups.com.

Brian Brazil

unread,
Oct 2, 2016, 9:14:21 AM10/2/16
to Ben Kochie, Jan Algermissen, Prometheus Developers
On 2 October 2016 at 14:07, Ben Kochie <sup...@gmail.com> wrote:
Another option is to run one Prometheus inside the k8s cluster, and use federation to extract the data.  We've been doing this with some success, however the large the data, the more lag you get.

Someday soon we'll have a Prometheus-as-a-proxy with the generic write API.

There's no planned features which in combination with the generic write API which would allow you to do this.

Brian
 

On Sat, Oct 1, 2016 at 11:51 PM, Jan Algermissen <algermi...@gmail.com> wrote:
Thank you for the replies.

I can not make use of K8S magic because I cannot make Prometheus node part of the overlay network (I am monitoring more than one K8S cluster with one Prometheus)

However, I wanted to quickly share the solution I found:

I have set up a simple HTTP proxy per K8S cluster on a node that is in the given cluster's overlay network and can thus resolve the individual pod IPs. I can now make proxy requests to that proxy using the pod IPs.

Then I use Prometheus' file_sd_configs and generate the targets file dynamically from the K8S API as a list of pod IPs and exposed ports.

What is left is to configure the proxy in the file_sd_configs section.

Thanks all,

Jan





--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

To post to this group, send email to prometheus-developers@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Reply all
Reply to author
Forward
0 new messages