question re: http.Client, re-used connections, and timeouts

[Davis Ford]

Let's suppose I have a client which I keep around, and often use to issue transactions.  I re-use it b/c the full connection is TLS and expensive, and I issue a lot of transactions, so I'd rather not spin up a new connection every time.  I need to set a reasonable timeout, as well.

var client = &http.Client{
  /* etc */
  Timeout: (60 * time.Second),
}


func Execute(req *http.Request) (interface{}, error) {
   resp, err := client.Do(req)
   if resp != nil && resp.Body != nil {
     defer resp.Body.Close()
   }
   if err != nil {
     return nil, err
   }
   body, err := ioutil.ReadAll(resp.Body)
   // deal with body
}

If a timeout occurs -- i.e. server fails to respond in 60 seconds, using Wireshark, I see that the connection is closed, b/c the client side issues a TCP RST.  Here's an example of this happening:

At No. 7, the client side issues FIN, ACK but this is before the server has responded with a 200 OK in step 8.  Nevertheless, in the code, client.Do will return an error of the sort "use of closed connection" and the response will be nil, meaning there is no chance to read the body...even though the server has sent the body -- it was too late, and happened after the timeout and thus after the FIN/ACK was issued by the client side.

So, here's my dilemma...on a single request / reply where we bootstrap a whole new connection like this each time, potentially losing that response (from step 8), is understandable.  The timeout occurred, and anything after that is toast.

But, if you're re-using the client for many such request / replies -- thus, there are many such posts interleaved on the same connection..if a timeout occurs, I believe there is the potential to lose the response from any outstanding request on that connection, and this is really bad.

Example timeline:

  1. Client.Do(request1)

  2. Client.Do(request2)

  3. Timeout for request1 occurs...client initiates connection close

  4. Server response for request 1 = "error: use of closed connection"

  5. Server response for request 2 = "error: use of closed connection"

This is a problem, and I'm wondering outside of spinning up a whole new connection for every request/response and disabling connection re-use altogether, how can I work around this issue.  Would Transport.MaxIdleConnsPerHost help here https://golang.org/pkg/net/http/#Transport ?  Would that allow multiple connections to the same host, so perhaps minimizing this effect?

Thanks in advance

[James Bardin]

On Wednesday, July 22, 2015 at 2:34:33 PM UTC-4, Davis Ford wrote:

So, here's my dilemma...on a single request / reply where we bootstrap a whole new connection like this each time, potentially losing that response (from step 8), is understandable.  The timeout occurred, and anything after that is toast.

But, if you're re-using the client for many such request / replies -- thus, there are many such posts interleaved on the same connection..if a timeout occurs, I believe there is the potential to lose the response from any outstanding request on that connection, and this is really bad.

This is incorrect. HTTP/1.1 is s serial protocol -- you can't interleave requests on a single connection. 

This is a problem, and I'm wondering outside of spinning up a whole new connection for every request/response and disabling connection re-use altogether, how can I work around this issue.  Would Transport.MaxIdleConnsPerHost help here https://golang.org/pkg/net/http/#Transport ?  Would that allow multiple connections to the same host, so perhaps minimizing this effect?

You get multiple connections to a host as needed. MaxIdleConnsPerHost determines how many are left open in the idle pool for re-use. If you're making a lot of requests to a few hosts, then increasing MaxIdleConnsPerHost will almost definitely help improve performance.

[Davis Ford]

Hi James,

HTTP is certainly a request/response, but it is my understanding the the underlying TCP connection will be re-used.  If a Timeout occurs on the Transport, it appears to me that the client will issue TCP RST and completely abort that TCP connection.  If that happens while the client has two in-flight requests on that TCP connection, then there is the risk that the HTTP responses will both be lost, is there not?

[James Bardin]

On Wed, Jul 22, 2015 at 10:31 PM, Davis Ford <davi...@gmail.com> wrote:

HTTP is certainly a request/response, but it is my understanding the the underlying TCP connection will be re-used.  If a Timeout occurs on the Transport, it appears to me that the client will issue TCP RST and completely abort that TCP connection.  If that happens while the client has two in-flight requests on that TCP connection, then there is the risk that the HTTP responses will both be lost, is there not?

No. Like you say, http/1.1 is a series of "request/response". Fundamentally, there's no way to have more than one outstanding request on a single tcp connection. Within http.Transport, a persistent connection isn't returned to the idle pool until it completes a successful RoundTrip, after which it's free for the next request. 

[Davis Ford]

Thanks for the quick / clear responses.  That makes sense.  I wasn't sure how the transport was utilizing pooled connections.  I did bump up Transport.MaxIdleConnsPerHost, and on a cursory level it does appear to help out my situation.