Prometheus- Standalone HTTP/1. transport connecti on broken: too many transfer encoding s: ["chunked" "chunked"]

[Rayala Rajashekar]

Hi All,

Can some one help me on below Prometheus  issue.

we are using Prometheus standalone V2.19.2 and planning to upgrade it to latest version V2.43.0.

We are getting below issue  with latest version, as per my observation it working fine until V.20.0 getting same issue from V2..21.0 till latest version.

Get "http://192.168.113.125:8080/metric

s": net/http: HTTP/1. transport connecti on broken: too many transfer encoding
s: ["chunked" "chunked"]

[Brian Candler]

Newer versions of Prometheus are compiled with newer versions of go:

https://stackoverflow.com/questions/72046302/go-how-to-solve-transport-connection-broken-too-many-transfer-encodings-ch

It looks like your exporter has a bug and is setting multiple Transfer-Encoding headers, and newer versions of go reject this.

You should be able to demonstrate with curl:

    curl -vfsS -o /dev/null http://192.168.113.125:8080/metrics    # or /metric; I wasn't sure if you had made a typo.

or with tcpdump:

    tcpdump -i eth0 -nn -s0 -A host 192.168.113.125 and tcp port 8080

(Run the tcpdump command on the prometheus server while prometheus is running and scraping. Change eth0 to your appropriate ethernet interface, or use "lo" if the prometheus server itself is 192.168.113.125)

Then take the problem up with the author of the exporter you are using.

[Brian Candler]

Some more references, thanks to google:

https://github.com/grafana/k6/issues/1780

https://github.com/golang/go/issues/44591   # there is an open change request which *may* permit this in future versions of go, if accepted

[Rayala Rajashekar]

Thanks for your time and input Brian,

getting below errors while executing commands.

Google\Cloud SDK›curl -vfsS -o /dev/null http: / /192.168.27.175:8080/metric
Trying 192.168.27.175:8080..
* connect to 192.168.27.175 port 8080 failed: Timed out
* Failed to connect to 192.168.27.175 port 8080 after 21055 ms: Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect to 192.168.27.175 port 8080 after 21055 ms: Couldn't connect to server
C: \Program Files (X86)\Google\Cloud SDK>tcpdump -i ethe -n -se -A host 192.168.271175:8080
"tepdump' is not recognized as an internal or external command, operable program or batch file.

[Brian Candler]

Well, firstly you're connecting with curl to a different IP address than the one you gave in your original message.

Secondly, it's "tcpdump" not "tepdump".  And that's a Linux command - you should run it on the same server or VM where Prometheus itself is running (or where the problematic exporter is running).  If you're running Prometheus under Windows then I can't help you, although I understand you should be able to install "wireshark" for packet capture.

You're just trying to capture the HTTP traffic between prometheus and the exporter, so you can show the duplicate transfer encoding header to the author of the exporter.