Running Prometheus outside outside of k8s

[huihui...@gmail.com]

Hi,

I want to use Prometheus monitoring K8s cluster, but Prometheus are not running inside K8s, they are in different network environment.

In the condition, how can I config Prometheus to gather the metrics of K8s, or is there any doc for this? Thanks!

[Ionut Ilie]

Hi,

It is a not a good idea.
We inherit a similar one:
 - a proxy deployed in Kubernetes that should have allowed access to pods from outside of the k8s cluster
 - prometheus deployed outside of the cluster that should have scanned node-exporter, kube-state metrics and the pods

The proxy (tinyproxy) dropped a high number of requests which made detecting running pods impossible.
as we could not made it to work in an acceptable way, we ended up deploying a small Prometheus (1h retention)  inside Kubernetes ( and then used federate to get the metrics )
Basically made a small proxy out of Prometheus.

Now the problem is that the main Prometheus has a lot of metrics that we can only store for 15 days, most of them useless for us, in order to de stress our server we will make a more serious deployment in k8s cluster and for us only get what we need.

and if you also consider having only one server, from alerting perspective, also bad ideea :)


If you want i can put the k8s files on git.

[Danny Kulchinsky]

I have been pondering around this question for a while now, we are running a similar setup to what you describe below (in-cluster Prometheus with short retention scraped by two out-of-cluster Prometheus servers with week worth of retention) but in fact I feel we are over extending the use of federation to something it was not meant to cover, for now it's OK - but I fear things will get ugly when we scale.

We have several reasons for wanting to run Prometheus servers on dedicated nodes outside the K8s cluster and I have been looking into several way to achieve that.

1) Using some kind of a proxy - ruled out as unreliable (I see you have made the same conclusion)

2) This one is tricky and I'm hoping to have some time to PoC it soon, but will require the following:

• Extend the K8s overlay network (flannel in our case) to the out-of-cluster Prometheus servers - thus achieving access to Pod IPs
  
• Side-effect: we will "lose" two subnets from flannel's range (minor concern for us)
• Accessing the K8s API Servers from outside the cluster should be doable as well 
  
• We don't need access to "service-cluster-ip-range" since Prometheus needs the endpoints (Pod IPs) and doesn't actually access the Service ClusterIPs
  
• Access to K8s SD (i.e. DNS service), not sure yet how to approach this exactly, but should be able to figure something out
  

Any comments/feedback/suggestions ?

Danny

[Matthias Rampke]

We had the small-Prometheus-in-cluster setup in the past, but did hit the limits of federation when it took >10s to get all the current values.

Danny– your option 2 is spot on. For accessing the K8s API, use the address that the kubelet/kube-proxy are using. Extending the overlay network is the best way to get Prometheus connected to the pods. Don't worry about the K8s DNS, use the IPs from Kubernetes service discovery directly. Prometheus will have to authenticate against the Kubernetes API and kubelets (or tunnel through the API for the latter, there is an example config in the main repo).

If you are scraping metrics you don't want at all, use metric relabel configs to drop them on ingestion (the cAdvisor metrics can be quite volumous).

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/93c4557f-08ac-482e-ae92-93bbb914722b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Danny Kulchinsky]

Thanks for the feedback Matthias, we'll give it a try soon hopefully.

Danny

[Robin Bartholdson]

Bumping an old thread here — but I'm trying to figure out how I could do non-proxy based scraping of pods outside of a Kubernetes cluster.

Curious if anyone has a concrete configuration example of "option 2" mentioned in this thread.

• Extend the K8s overlay network (flannel in our case) to the out-of-cluster Prometheus servers - thus achieving access to Pod IPs
• Side-effect: we will "lose" two subnets from flannel's range (minor concern for us)
• Accessing the K8s API Servers from outside the cluster should be doable as well 
• We don't need access to "service-cluster-ip-range" since Prometheus needs the endpoints (Pod IPs) and doesn't actually access the Service ClusterIPs
• Access to K8s SD (i.e. DNS service), not sure yet how to approach this exactly, but should be able to figure something out

For accessing the K8s API, use the address that the kubelet/kube-proxy are using. Extending the overlay network is the best way to get Prometheus connected to the pods. Don't worry about the K8s DNS, use the IPs from Kubernetes service discovery directly. Prometheus will have to authenticate against the Kubernetes API and kubelets (or tunnel through the API for the latter, there is an example config in the main repo).

Thanks,

-Robin