Federated prometheus endpoint fails to return full response

[Kass Chäralampidøu]

I have set up federated prometheus, and scraping in one particular environment (with high load of metrics) is failing most of the times (not always though).

In the Targets UI I often see the 'Unexpected EOF' error for this target and when I curl the endpoint manually I get a 'transfer closed with outstanding read data remaining' error.

I increased the scrape_timeout but didn't make any difference. Does anyone have any idea how can I tackle this issue?

Thank you in advance!

[Bjoern Rabenstein]

I can only speculate here, but it might very well be that you are
trying to federate too many metrics. Federation isn't suitable to
mirror all metrics of a server (use remote-write instead). Federation
is meant to propagate a few aggerated metrics (generated with
recording rules) to a (usually very small) "higher order" Prometheus
server.


--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in