Can't read >32KBytes from http response body.

[Min Xie]

All,

I got puzzled by how to read a response size > 32KBytes from http resp.Body. All I get is 32768 bytes. With either io.Read functions or bufio.Reader.

buffer := make([]byte, 1024)
var total_bytes_read int
for {
  len, err := io.ReadAtLeast(res.Body, buffer, 1024)
  total_bytes_read += len
  log.Println(len, "bytes recevied, total received bytes: ", total_bytes_read)
  if err != nil {
    if err == io.EOF {
      log.Println("EOF: last chunk received")
    } else {
      log.Println(err)
    }
    break
  }
  time.Sleep(1 * time.Second)
}  
log.Println("Total bytes received:", total_bytes_read)
err = res.Body.Close()

What I see is: 

{"level":"info","msg":"1024 bytes recevied, total received bytes: 1024","time":"2022-06-19T19:47:48-05:00"} 

{"level":"info","msg":"1024 bytes recevied, total received bytes: 2048","time":"2022-06-19T19:47:49-05:00"} 

{"level":"info","msg":"1024 bytes recevied, total received bytes: 3072","time":"2022-06-19T19:47:50-05:00"}

...

{"level":"info","msg":"1024 bytes recevied, total received bytes: 30720","time":"2022-06-19T19:48:17-05:00"}

{"level":"info","msg":"1024 bytes recevied, total received bytes: 31744","time":"2022-06-19T19:48:18-05:00"} 

{"level":"info","msg":"1024 bytes recevied, total received bytes: 32768","time":"2022-06-19T19:48:19-05:00"}

Then it got stuck and waiting for return from the io.Read. Not sure if what I did is correct. Got the same observation when using more popular bufio.Read as well.

Thanks!

-Min

[Brian Candler]

1. What's the web server you're connecting to?  (Is it written in go, and if so, can you share it?)

2. What happens if you use io.Read instead of io.ReadAtLeast?

3. What happens if you remove the time.Sleep()? (Maybe the target webserver has a timeout which requires the transaction to be completed within 30 seconds?)

io.ReadAtLeast does exactly what it says: read *at least* that number of bytes.  If the sender has nothing more to send, and does not end the stream, then it will wait indefinitely for more data.  Note that io.ReadAtLeast never returns io.EOF, but it may return io.ErrUnexpectedEOF if it's unable to read the number of requested bytes.  For source code see here.

However, even if the sender does drop the connection, it does seem weird that you don't get an error.  (I don't suppose you are going through a badly-configured firewall with an aggressive TCP timeout and which doesn't send RST?)

[Min Xie]

We've managed to identify the issue and found a temporary solution. 

The issue only happens when the stream encoding is "gzip" and the uncompressed size is greater than 32KBytes. Side by side we've compared the server stream with plain text and gzip. The gzip uncompression is automatically engaged if server response stream encoding is detected. If the uncompressed data is truncated at 32768 bytes.

https://go.googlesource.com/go/+/refs/heads/dev.boringcrypto.go1.17/src/net/http/h2_bundle.go#8835. (same in 1.19).

The client could use either "Accept-Encoding: identity" header or disable the compression in the transport instance to ask for an uncompressed stream.

Look like a Golang net/http bug to me.

[Kevin Chowski]

Do you have a self-contained reproducer? E.g., a single Go program that hosts a server (either using the server functionality from package net/http, or hard-coded HTTP responses) and shows that using the client logic in net/http causes this problem? Preferably shared via play.golang.org link. If the problem is in the Go standard library, that is the best avenue for getting the root cause fixed, which usually is better than using workarounds in your application.

[Miha Vrhovnik]

I'm reading a lot of gzipped responses with Go client, so I'd say that this is probably problematic http server and not the client.