early close of http.Client's Response.Body

[Liam]

Scenario:

Sending a client request and only reading the whole Response.Body for certain .StatusCode values, and/or reading only the first N bytes.

Docs for Client.Do()

"If the Body is not both read to EOF and closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection..."

Docs for Client.Get/Post()

"Caller should close resp.Body when done reading from it."

Questions:

- does the read-to-EOF stipulation also apply to Client.Get/Post() ?

- why does Response.Body.Close() before io.EOF not release unread buffers or otherwise prepare it for persistence?

[Andy Balholm]

> On Nov 25, 2019, at 9:54 AM, Liam <networ...@gmail.com> wrote:
>
> - does the read-to-EOF stipulation also apply to Client.Get/Post() ?

Yes. Those methods are fairly simple wrappers around Do.

> - why does Response.Body.Close() before io.EOF not release unread buffers or otherwise prepare it for persistence?

The connection can’t be reused for another request before all the data from the first request is read. The server is sending those bytes over the wire, and they need to go somewhere before another response can be read. So there were two options for how to implement Close when the whole body hasn’t been read:

1. Copy the rest of the body to /dev/null (or equivalent), and reuse the connection like normal.
2. Close the connection, thus communicating to the server that we don’t need that data after all.

Option 1 would generally be preferable for short responses, and option 2 for long ones. For consistency, and because we don’t always know the response length in advance (e.g. if it is chunked), it always does option 2.

Andy

[Liam Breck]

So after read-to-EOF, Close() behaves differently, i.e. it does reset for next request?

In the event of early Close(), should I discard that Client instance? Or will it work for subsequent requests?

[Andy Balholm]

Read-to-EOF is one of several conditions for connection reuse:

- Was the response body fully read (to EOF)?

- Does the server support HTTP keep-alives?

- Is DisableKeepAlives set to false (the default) on the http.Transport that the client uses?

A connection will only be reused if all of these are met. Otherwise Close will close the connection.

There is no need to discard a Client after an early Close. A Client is not tied to a single connection.

Andy

[Kevin Powick]

You may find the following articles on http.Client useful.

https://medium.com/hackernoon/avoiding-memory-leak-in-golang-api-1843ef45fca8

http://devs.cloudimmunity.com/gotchas-and-common-mistakes-in-go-golang/index.html#close_http_resp_body

https://tleyden.github.io/blog/2016/11/21/tuning-the-go-http-client-library-for-load-testing/

--

Kevin Powick