How to scrape K8s pod metrics from prometheus running in VM instance

David Frank

unread,

Feb 11, 2021, 12:28:59 PM2/11/21

to Prometheus Users

Hey community, I am currently running Prometheus inside a K8s cluster in our production. However, we have frequent updates to K8s version and this keeps restarting our pods, causing outage for couple of hours (WAL loading).

Is it possible to run Prometheus on a VM instance outside the K8s cluster but have service discovery to scrape the metrics from the pods running inside K8s cluster?
I have had no success so far to get this to work.

Thanks in advance.

Stuart Clark

unread,

Feb 12, 2021, 12:52:18 PM2/12/21

to David Frank, Prometheus Users

Yes. As per the Kubernetes SD documentation at
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config
you can set the api_server parameter to point to the cluster's API
server (which would need to be accessible from your Prometheus VM). You
would also need to set the correct authentication parameters (basic auth
or bearer token). You'd be responsible for setting those up within the
cluster and updating Prometheus if the token changes. You might also
need to set tls_config section depending on what CA you are using (to
supply the CA certificate file).

--
Stuart Clark

Aliaksandr Valialkin

unread,

Feb 16, 2021, 6:45:57 PM2/16/21

to David Frank, Prometheus Users

Try substituting Prometheus with vmagent inside the K8S and pushing metrics to a remote storage outside the Kubernetes. See https://victoriametrics.github.io/vmagent.html#drop-in-replacement-for-prometheus

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0702263b-7219-480c-8346-ed0289345b8bn%40googlegroups.com.