Prometheus context deadline exceeded with PushProx proxy

72 views

Skip to first unread message

Mykola Buhryk

unread,

Nov 15, 2024, 11:30:02 AM11/15/24

to Prometheus Users

Hello

Could someone help me resolve an issue in my configuration?

I have three Prometheus jobs that use the proxy_url parameter, pointing to a PushProx deployment located within the same AWS EKS cluster as Prometheus.

PushProx is also externally accessible via an NGINX ingress controller to collect metrics from hosts that are not directly reachable by Prometheus

I am currently facing an issue where Prometheus cannot scrape metrics from certain hosts, even though their metrics endpoints are accessible. The affected hosts appear to be random.

I have already tried increasing the scrape_interval and scrape_timeout settings in Prometheus, as well as the scrape.default-timeout parameter for PushProx. Unfortunately, these changes have not resolved the issue.

Also I checked pod resources like CPU and memory, they also not at the max level

Below are logs from the PushProx proxy pod

ts=2024-11-15T16:04:04.346Z caller=main.go:179 level=error msg="Error scraping:" err="Timeout reached for \"http://10.44.44.161:9000/metrics\": context canceled" url=http://10.44.44.161:9000/metrics

ts=2024-11-15T16:04:04.363Z caller=main.go:179 level=error msg="Error scraping:" err="Timeout reached for \"http://10.44.57.96:9000/metrics\": context canceled" url=http://10.44.57.96:9000/metrics

ts=2024-11-15T16:04:04.463Z caller=main.go:179 level=error msg="Error scraping:" err="Timeout reached for \"http://10.44.57.100:9000/metrics\": context canceled" url=http://10.44.57.100:9000/metrics

ts=2024-11-15T16:04:04.539Z caller=main.go:179 level=error msg="Error scraping:" err="Timeout reached for \"http://10.44.66.66:9000/metrics\": context canceled" url=http://10.44.66.66:9000/metrics

ts=2024-11-15T16:04:04.634Z caller=main.go:179 level=error msg="Error scraping:" err="Timeout reached for \"http://10.10.8.61:9000/metrics\": context canceled" url=http://10.10.8.61:9000/metrics

ts=2024-11-15T16:04:04.664Z caller=main.go:179 level=error msg="Error scraping:" err="Timeout reached for \"http://10.44.66.47:9000/metrics\": context canceled" url=http://10.44.66.47:9000/metrics

Reply all

Reply to author

Forward

0 new messages