unexpected EOF error using prometheus proxy

[Ian Banks]

Hi,

I am using the Prometheus Proxy (https://github.com/pambrose/prometheus-proxy) to push metrics to Prometheus. This works fine for some exporters, but metrics pushed to Prometheus which have been scraped from the node exporter are resulting in an 'unexpected EOF' error on the prometheus targets.

I have tested scraping the same node exporter metrics directly from Prometheus and this works. Looking at the metrics pages, both directly on the host the node exporter is running on and the one pushed form the Prometheus Proxy look the same.

Can anyone advise what might cause an 'unexpected EOF' error and how to troubleshoot further.

Thanks

[er...@nil-database.com]

Hi,

I stumbled on the same issue recently.

First, the Accept and Accept-Encoding headers used for the requests can change the behaviour of the exporters. Without specifying them with curl, I was not able to reproduce the issue.

It seems to be a problem with the gzip encoding. Nginx receives a shorter answer from the exporter (gziped), but is awaiting a longer answer (Content-Length?, wrong headers?) and triggers an error, thinking that the backend closed the connection too early. I had not enough time to understand all the details, and just tried to force HTTP1.1 on the backend with "proxy_http_version 1.1;". It solved the issue!

Hope it helps,

--

Éric

[er...@nil-database.com]

I forgot to say, I had the issue with Nginx. This is a Nginx solution.