Error encoding and sending metrics family - Node Exporter.

[yagyans...@gmail.com]

Hi. I am running node_exporter version 0.18.1 on around 2500 servers. On some of the servers I am observing that node_exporter is throwing error intermittently for sending the metrics to Prometheus server.

Dec 22 17:10:29 hostname node_exporter[10807]: time="2020-12-22T17:10:29+05:30" level=error msg="error encoding and sending metric family: write tcp ServerIP:9100->PrometheusIP:40984: write: broken pipe\n" source="log.go:172"

Dec 22 17:10:29 hostname node_exporter[10807]: time="2020-12-22T17:10:29+05:30" level=error msg="error encoding and sending metric family: write tcp ServerIP:9100->PrometheusIP:40984: write: broken pipe\n" source="log.go:172"

Dec 22 17:10:29 hostname node_exporter[10807]: 2020/12/22 17:10:29 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:59)

Thanks in advance!

[Stuart Clark]

That suggests the Prometheus server disconnected before all the data was
returned. What do the Prometheus logs at that time say? Are there any
firewalls between the two that could have broken the connection?

[yagyans...@gmail.com]

No firewall between them. Infact, I checked on another server of the same cluster, it does not throw any error. My Prometheus server and target server both reside under the same Datacenter.
No error logs whatsoever for that time period on Prometheus side.