How to get rid of context deadline exceeded without changing the scrape_timeout and scrape_interval

[suyog kulkarni]

Hello ,

I have prometheus deployed  as container and currently i am trying to scrape 100 static/dynamic  endpoints in single job and the corresponding scrape interval is 20s and timeout is 15s. with this interval and timeout all my endpoints are showing context-deadline-exceeded. While if i change the scrape timeout and scrape interval to 50s each then my 100 endpoints are up. my target endpoint is node-expoter with heavy list of metrics.
I dont want to change the scrape interval or scrape timeout. How do i get rid of context-deadline-exceeded without changing the scrape interval and scrape timeout.
Thanks in advance.

Regards,

Suyog Kulkarni

[Julien Pivotto]

Can we know more about your setup? Which version of prometheus do you use? Do you use TLS? Do you directly target the node exporters or is there a proxy in the middle?

Thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/374154c1-c27a-4ee1-ae24-a41922034db7n%40googlegroups.com.

[Matthias Rampke]

How long does it take when you request the metrics directly from node exporter ? Can you exclude one collector at a time to see which one is the slow one?

Generally, node exporter doesn't "do" much other than collect metrics from the kernel; if it is slow most likely something else is too.

"context timeout exceeded" is Go's words for "this took longer than the configured timeout". The only way to make it go away is to make whatever is happening take less time than the timeout, or extend the timeout to be more than it takes.

/MR

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAFJ6V0r_T5%2BuF%3DYf6Sh%2BF6ZqTT9khx5yXDCczrU9_FoOQsRdow%40mail.gmail.com.

[suyog kulkarni]

Current prometheus version is 2.20. There is proxy settings also. 
Here are my prometheus-config.

- job_name: spc-targets-node-exporter

  proxy_url: <Proxy-URL>

  scrape_interval: 15s

  scrape_timeout: 30s

  static_configs:

  - targets:

    - pce-9100:9100

    - pce-9101:9100

    - pce-9102:9100

    - pce-9103:9100

    - pce-9104:9100

         .

.

.

    - pce-N:9100

[suyog kulkarni]

The current node-exporter is setup as daemon-set in kubernetes cluster same where prometheus is deployed.

[Matthias Rampke]

Try removing the proxy, within a cluster Prometheus should be able to connect to the pods directly. Consider using Kubernetes SD to discover the daemonset pods dynamically.

/MR

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2e7978e9-d702-4820-9ddd-c6568b69c304n%40googlegroups.com.

[Julien Pivotto]

Indeed, it seems this seems an issue with the proxy rather than prometheus.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gZSbJ1crTQW%3DV%2B3omWF4ERkSjWZ2-%2BwLZ%2BLvWKQ8X9BEQ%40mail.gmail.com.