HTTP Client - Lots of Connections in TIME_WAIT

[sebastian...@mercadolibre.com]

I have a problem where a simple program performing concurrent requests to a Node.JS webserver starts to pile up LOTS of connections in TIME_WAIT and just keeps increasing.
This seems to only happen when I share the same Client between goroutines, having separate clients works just right, though Clients are supposed to be safe to use concurrently.
The code is really simple and can be found here https://gist.github.com/sschepens/062c7dbdf29272eea4af
One has to replace URL to some valid url of course.

Does anyone know if i'm doing something wrong or if this is a bug?

I'm running go 1.5.2 on Ubuntu Precise 12.04, kernel: Linux e-0000e768 3.8.0-44-generic #66~precise1-Ubuntu SMP Tue Jul 15 04:01:04 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Thanks,
Cheers,
Sebastian

[Dave Cheney]

Defer only runs on function exit. Your function does not exit so the defers will not run.

[sebastian...@mercadolibre.com]

Thanks for realizing that, but the code I published is an abstract, this happens even though I explicitly close the Body.

[James Bardin]

TIME_WAIT is the expected state after a successful active close of a TCP connection. Per the TCP protocol, its supposed to stay in that state for 2MSL.

If you're making a lot of TCP connections, it's expected you will see a lot of TIME_WAIT. 

[sebastian...@mercadolibre.com]

James,
I know that, but the amount of TIME_WAIT connections just keeps getting higher, this means go is not reusing connections to this service, the expected behaviour would be to have X amount of established connections and use those, every once in a while one would get closed and another created, but not all the time.

[Dave Cheney]

Thank you for providing your code sample. Can you provide the real code that is causing this issue ?

[James Bardin]

On Wednesday, December 30, 2015 at 9:54:06 AM UTC-5, sebastian...@mercadolibre.com wrote:

James,
I know that, but the amount of TIME_WAIT connections just keeps getting higher, this means go is not reusing connections to this service, the expected behaviour would be to have X amount of established connections and use those, every once in a while one would get closed and another created, but not all the time.

We can't say what's going on without an example to reproduce the problem.

If you're making more simultaneous requests to a host than http.DefaultMaxIdleConnsPerHost, the extra connections will be closed after each request. Increasing Transport.MaxIdleConnsPerHost will allow you to keep more connections alive.

[sebastian...@mercadolibre.com]

Setting MaxIdleConnsPerHost doesn't help with the problem.
It's hard to reproduce, i've been testing and it seems to be something with our LoadBalancer, requests to a single node don't generate connections in TIME_WAIT, but  requests to our LoadBalancer generates LOTS of them.
The thing is, TIME_WAIT means Go is closing connections, not the server, then why would Go close connections to the LoadBalancer but not to a single node, that's what i'm trying to figure out now, I don't know in which cases Go decides to close connections or how to know exactly what condition is being hit.

[Dave Cheney]

Can you get out wireshark and check the request that the LB is setting. It might be using http/1.0 or explicitly sending Connection: close

[sebastian...@mercadolibre.com]

I'm gonna check that, but theoretically LB is set in TCP mode which should not interfere with HTTP.

[Jesse McNelis]

On Thu, Dec 31, 2015 at 2:16 AM, <sebastian...@mercadolibre.com> wrote:
> Setting MaxIdleConnsPerHost doesn't help with the problem.
> It's hard to reproduce, i've been testing and it seems to be something with
> our LoadBalancer, requests to a single node don't generate connections in
> TIME_WAIT, but requests to our LoadBalancer generates LOTS of them.
> The thing is, TIME_WAIT means Go is closing connections, not the server,
> then why would Go close connections to the LoadBalancer but not to a single
> node, that's what i'm trying to figure out now, I don't know in which cases
> Go decides to close connections or how to know exactly what condition is
> being hit.
>

Are you reading the full response body before calling Close()?
If you don't read the body the http.Client is forced to close the
connection because it can't reuse it.

[Sebastián Schepens]

The code example i uploaded triggers the behaviour and it does read the body and discard it.
It also doesn't explain why it only happens when requesting the LoadBalancer.

[James Bardin]

On Wed, Dec 30, 2015 at 11:13 AM, 'Sebastián Schepens' via golang-nuts
<golan...@googlegroups.com> wrote:
> The code example i uploaded triggers the behaviour and it does read the body
> and discard it.
> It also doesn't explain why it only happens when requesting the
> LoadBalancer.

The gist you provided will definitely trigger the behavior, because
you haven't changed MaxIdleConnsPerHost.

You say you've tried raising that setting, but not what you set it to,
and this example theoretically needs 10 to be stable, but probably
needs many more than that.

Also, because you're requesting the same host in concurrent tight
loops, you might be triggering a pathological behavior in the
transport where connections can't be recycled fast enough so new
connections are continually created and closed. (There is an open
issue still for a hard upper limit on connections/host:
https://github.com/golang/go/issues/6785)

Try setting MaxIdleConnsPerHost very high and see what happens.

[sebastian...@mercadolibre.com]

James thank you MaxIdleConnsPerHost actually was the cause of the issue, setting it to 10 fixes the issue, though I think that the property is not named properly or does not work as one would expect.
I have never had such a problem with other languages or http clients.
Normally, a connection is considered Idle after a certain period of time of inactivity (maybe a whole keepalive period) and then they are closed as per a max idle connection settings.
In Go's case using a default client for concurrent connections ends up eating all ephemeral ports because only 2 connections are actually reused, the rest are closed instantly, though they are not idle at all.

Anyway,
Thanks for the help.
Cheers