Connection reuse for HTTP client requests

[Brian G. Merrell]

Hi all,

I've written a small HTTP client that issues many (250,000) requests--50 maximum at a time--to a simple Go server[0] that listens for requests and writes back a small, simple response.  Here's the code: http://play.golang.org/p/4uKHGXOUxn.

I must be missing something, but my client doesn't behave like I would expect:  On both Linux and OS X, the client opens orders of magnitude more connections than I would expect.  I would expect somewhere around 50+ as the client is making, at most, that many requests at the same time.  I would understand a number a little higher than 50 as connections were closing and reopening.  On Linux, I'm seeing thousands and tens of thousands being opened (and almost all of them are in the TIME_WAIT state according to netstat--the ESTABLISHED connections stay in around the 50ish region that I would expect).  On OS X, I get a `127.0.0.1:8081: can't assign requested address` failure from http.Client.Do soon after starting the client.

My client is using DefaultTransport (as I am just using a http.Client{} client), which sounds OK according to the documentation: "DefaultTransport is the default implementation of Transport and is used by DefaultClient. It establishes network connections as needed and caches them for reuse by subsequent calls."

Am I running into https://code.google.com/p/go/issues/detail?id=6785 ?  If so, is there a suggested workaround?  My actual use case is a reverse proxy that multiplexes requests to multiple high-traffic servers (so that their respective responses can be compared) and then responds to the original request with one of those responses.

I've seen several[1a-c] questions about similar things, but the answers seem to fall into four categories 1) read the entire response, 2) close the response, 3) use or don't use keep-alive headers, and 4) set Transport.MaxIdleConnsPerHost.  Those suggestions don't appear to fix the issue I am seeing with my client, however.

Any information would be greatly appreciated.

Thanks,

Brian

[0] http://play.golang.org/p/68003LFqKY

[1a] http://stackoverflow.com/questions/17948827/reusing-http-connections-in-golang

[1b] https://groups.google.com/forum/#!topic/golang-nuts/l2rEBV6XhhI

[1c] http://stackoverflow.com/questions/19648217/getting-127-0-0-1-cant-assign-requested-address-http-client

[Kyle Wolfe]

Well, first I'd refactor a little bit. Your firing off 250k go routines and they aren't all being used (or supposed to be used) because of your buffered channel. Only fire up as many workers as your max concurrent calls.

See a quick example: http://play.golang.org/p/aBQlNWTlH1

[Brian G. Merrell]

Hi Kyle,

I am OK with being wasteful with goroutines for this test code.  My goal was illustrating the behavior I was seeing in the most simple, readable way I could.

Thanks,

Brian

[Caleb Spare]

Hi Brian, I can see the same behavior.

Make a custom http.Transport for your http.Client, and set
MaxIdleConnsPerHost to 50 (something >= the number of concurrent
connections you wish to make.) The default Transport will close many
connections (over the default limit of 2) during the brief time they
are idle after the response body is closed but before another
goroutine makes a request.

Cheers,
Caleb

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Brian G. Merrell]

Hi Caleb,

Thanks for the tip.  It seems to have helped things on the OS X side with my test scripts.  On Linux (which is where I'd be deploying), though, I am still seeing the connections with the TIME_WAIT state skyrocket into the tens of thousands.

The more I think about it, I must be running into issues https://code.google.com/p/go/issues/detail?id=6785 and https://code.google.com/p/go/issues/detail?id=8536.  Brad Fitzpatrick's suggestion (on issue 8536) is to "provide your own http.Transport.Dial func that has the policy you'd like. You'll have to do a little bit of bookkeeping, but you can do exactly what you need."  Of course he says it is easy; I'm going to start looking into it.

Thanks,

Brian

[Caleb Spare]

Interesting. I'm running on linux, and with this change I don't see
any new connections being established after the initial 50.

var client = &http.Client{
Transport: &http.Transport{
Dial: func(network, addr string) (net.Conn, error) {
log.Println("dial!")
return net.Dial(network, addr)
},
MaxIdleConnsPerHost: 50,
},
}

(confirmed by monitoring the tcp active opens as well.)

[Caleb Spare]

By the way, what version of Go are you using? If it's old enough, HTTP
keepalives might not be enabled by default. I seem to recall that
changed at some point.

[Brian G. Merrell]

Neat idea to add the "dial!" prints.

I'm using Go 1.3 on Ubuntu 12.04 (kernel version is 3.2.0-68-virtual) and the "dial!" messages get printed out like crazy (which is obviously why I'm seeing tens of thousands of TIME_WAIT connections in netstat).  On the OS X environment, I see the initial 50 "dial!" messages and then those connections are apparently getting reused (as long as the MaxIdleConnsPerHost is set).

Thanks,

Brian

[Caleb Spare]

Interesting. I'm not sure how much more help I can provide, given that
I can no longer reproduce it after that change.

Random idea: does it still happen if you set maxOutstanding to 5? 2?

-Caleb

[Brian G. Merrell]

Thanks Caleb.

I do still see the issue with maxOutstanding set to 5 or 2.  Just out of curiosity, what Linux distro and kernel are you using?

[Caleb Spare]

I'm using Go 1.3.1 with Ubuntu 14.04 and kernel 3.13.0-35-generic.

[Mikio Hara]

On Fri, Sep 26, 2014 at 6:19 AM, Brian G. Merrell <bgme...@gmail.com> wrote:

> test scripts. On Linux (which is where I'd be deploying), though, I am
> still seeing the connections with the TIME_WAIT state skyrocket into the
> tens of thousands.

try http://play.golang.org/p/tZT1HHy96K for dissecting your issue.

[Benjamin Measures]

On Thursday, 25 September 2014 20:16:22 UTC+1, Brian G. Merrell wrote:

I've written a small HTTP client that issues many (250,000) requests--50 maximum at a time [...]

On Linux, I'm seeing thousands and tens of thousands being opened (and almost all of them are in the TIME_WAIT state according to netstat--the ESTABLISHED connections stay in around the 50ish region that I would expect).

TIME_WAIT is the state where the local endpoint has closed the connection. You're getting tens of thousands of connections closing.

http://superuser.com/questions/173535/what-are-close-wait-and-time-wait-states

 

 On OS X, I get a `127.0.0.1:8081: can't assign requested address` failure from http.Client.Do soon after starting the client.

Eventually, you'll run out of ephemeral ports.

http://unix.stackexchange.com/questions/65475/ephemeral-port-what-is-it-and-what-does-it-do

My client is using DefaultTransport [...]. It establishes network connections as needed and caches them for reuse by subsequent calls."

The default value of Transport's MaxIdleConnsPerHost is DefaultMaxIdleConnsPerHost = 2 . If the number of idle connections in the pool exceeds MaxIdleConnsPerHost, it is closed.

Am I running into https://code.google.com/p/go/issues/detail?id=6785 ?

The issue centers around the variance in the number of connections going idle.

In your case, you've got 50 goroutines all started at roughly the same time, and so 50 connections at roughly the same time. Suppose each request takes exactly 10ms. You'll end up with 50 connections going idle at roughly the same time. Clearly, 50>2 so you'll get some -- rather high -- proportion of closed connections. Repeat a few thousand times.

There are two ways of solving:

1. Increase MaxIdleConnsPerHost to accomodate idle variance; or

2. Alter your code to minimise idle variance (global synchronisation).

I view option 2 as a proper fix but 1 is easier and oft suggested.