dial tcp 127.0.0.1:3333: can't assign requested address - when calling http.Get too quickly..

[arb...@gmail.com]

Hello everyone,

I am trying to write a benchmark for Gin HTTP server. I started with very simple Ping - Pong example, so responses are very quick. Unfortunately my benchmark is not working. I am getting plenty of "can't assign requested address errors". I found similar topic on this user group, where the problem was related with concurrency and eating up all available system files. I rewrote my benchmark to be synchronous and now I have a for loop in where I am sending request, waiting for the response, close the response.

It is very simple - http://play.golang.org/p/B1wlziqRP5

This code runs fine for couple of seconds, then for about a minute it is throwing plenty of errors. I am trying to understand what happens there. What I think I see here, is for some reasons I have stale connections, and I am running out of available files to create more seconds. I think, on Mac, kernel default timeout is set to 60 seconds, so after this time all old connections are garbage collected, then my program is able to send some more requests, for couple of seconds and it gets blocked again. And btw when I add small sleep to the server handler (1ms), benchmark tool works fine.

Unfortunately, I can't confirm this theory. When I check number of open files (using lsof) or ports (netstat) it looks like the number grows by about 10-20, to the time it gets blocked with errors, then immediately it goes down to the initial level. Also I am calling resp.Body.Close() for every requests so the connections should be cleaned up immediately. 

I also noticed similar problems when I was trying to benchmark http servers using go-wrk library. With original wrk library I had no problems, but go-wrk was leaking clients connections (and this time I was able to see it using netstat). 

Is it problem with go http client tool? I am running out of ideas and would appreciate a lot if someone helps me to understand it.

Cheers

Adam

[James Bardin]

OSx seems to keep lo ports in an unlisted TIME_WAIT state for the MSL (15sec IIRC).

What that basically means, is that when you use up your allotted ephemeral ports in less than 15s, you'll get these errors until the kernel releases more ports. You'll need to make sure the connections are re-used to make calls this quickly.

[Aliaksandr Valialkin]

Try setting MaxIdleConnsPerHost to 10000:

c := &http.Client{
    Transport: &http.Transport{
        MaxIdleConnsPerHost: 10000,
    }
}

Then the created http client will keep up to 10000 open TCP connections to the server if server supports http keep-alive. If not, then take a look at ulimit -n and other tuning knobs in the OS like:

echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

echo "9000 65500" > /proc/sys/net/ipv4/ip_local_port_range

echo 500000 > /proc/sys/fs/file-max

[Adam Dratwiński]

 Hi James, thanks for your answer! Does it also mean, every MSL seconds kernel is allotting a new fixed pool of ephemeral ports that service can use, and before new ports are allotted it is impossible to reassign already used port? If yes, is this pool is per process or it is globally available? 

I always thought, once I close HTTP client connection, the used port is getting available immediately.

I would like to understand this in details, will be also very happy if you can redirect me to some readings on this topic.

Thanks!

On Thu, Jun 11, 2015 at 4:44 PM, Adam Dratwiński <arb...@gmail.com> wrote:

Hi James, thanks for your answer! Does it also mean, every MSL seconds kernel is allotting a new fixed pool of ephemeral ports that service can use, and before new ports are allotted it is impossible to reassign already used port? If yes, is this pool is per process or it is globally available? 

I always thought, once I close HTTP client connection, the used port is getting available immediately.

I would like to understand this in details, will be also very happy if you can redirect me to some readings on this topic.

Thanks!

--

pooozdrawiam :-)

--

pooozdrawiam :-)

[arb...@gmail.com]

On Linux I am able to observe it better and using netstat shows me all ports used for connections. What I can see is once a HTTP client creates a connection, it stays in TIME_WAIT state for 60 seconds. After I use all ~50k ports I started to see these errors. 

Is there any way to tell kernel to close this connection after shorter time then 60 seconds? These are my linux network tunning parameters:

net.core.somaxconn = 65535

net.core.netdev_max_backlog = 65536

net.core.rmem_max = 131071

net.core.wmem_max = 131071

net.ipv4.ip_local_port_range = 9101 65535

net.ipv4.tcp_fin_timeout = 3

net.ipv4.tcp_max_syn_backlog = 262144

net.ipv4.tcp_max_tw_buckets = 2097152

net.ipv4.tcp_no_metrics_save = 1

net.ipv4.tcp_low_latency = 1

net.ipv4.tcp_rmem = 4095 65535 131071

net.ipv4.tcp_sack = 1

net.ipv4.tcp_syncookies = 0

net.ipv4.tcp_timestamps = 0

net.ipv4.tcp_tw_recycle = 0

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_wmem = 4095 65535 131071

net.netfilter.nf_conntrack_max = 4194304

net.netfilter.nf_conntrack_tcp_timeout_established = 600

net.netfilter.nf_conntrack_tcp_timeout_time_wait = 5

I thought net.netfilter.nf_conntrack_tcp_timeout_time_wait should do this job.

[James Bardin]

On Thu, Jun 11, 2015 at 10:45 AM, Adam Dratwiński <arb...@gmail.com> wrote:

Hi James, thanks for your answer! Does it also mean, every MSL seconds kernel is allotting a new fixed pool of ephemeral ports that service can use, and before new ports are allotted it is impossible to reassign already used port?

Not exactly, It just means that when a connection goes through an active close, it goes into a wait queue, and is released some time later. This works just like TIME_WAIT for non loopback connections. 

 

If yes, is this pool is per process or it is globally available? 

Ports are per ip address.

 

I always thought, once I close HTTP client connection, the used port is getting available immediately.

Nope. When you actively close a connection to a remote host, it usually goes into TIME_WAIT.

I would like to understand this in details, will be also very happy if you can redirect me to some readings on this topic.

This is how TCP works, so any reading on that subject should help (though the loopback behavior we're discussing is osx specific)

[James Bardin]

On Thu, Jun 11, 2015 at 11:18 AM, <arb...@gmail.com> wrote:

Is there any way to tell kernel to close this connection after shorter time then 60 seconds? These are my linux network tunning parameters:

net.core.somaxconn = 65535

net.core.netdev_max_backlog = 65536

net.core.rmem_max = 131071

net.core.wmem_max = 131071

net.ipv4.ip_local_port_range = 9101 65535

net.ipv4.tcp_fin_timeout = 3

net.ipv4.tcp_max_syn_backlog = 262144

net.ipv4.tcp_max_tw_buckets = 2097152

net.ipv4.tcp_no_metrics_save = 1

net.ipv4.tcp_low_latency = 1

net.ipv4.tcp_rmem = 4095 65535 131071

net.ipv4.tcp_sack = 1

net.ipv4.tcp_syncookies = 0

net.ipv4.tcp_timestamps = 0

net.ipv4.tcp_tw_recycle = 0

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_wmem = 4095 65535 131071

net.netfilter.nf_conntrack_max = 4194304

net.netfilter.nf_conntrack_tcp_timeout_established = 600

net.netfilter.nf_conntrack_tcp_timeout_time_wait = 5

I thought net.netfilter.nf_conntrack_tcp_timeout_time_wait should do this job.

nf_conntrack_tcp_timeout_time_wait if for the iptables conntrack module.

You want net.ipv4.tcp_tw_reuse = 1, but that's already set, so I'm not sure offhand.

We're getting pretty far off topic for Go, but you really just need to make sure you're reusing the ports correctly. If your server or client is closing the connection every time, and the connections are this short lived, you're basically just testing how fast your system can make TCP connections, not your application code. 

[adam.dr...@gmail.com]

Thanks James for your answers, you helped me a lot! Seems like tw_reuse attribute wasn't working because tcp_timestamps was disabled, with this enabled everything works like expected. 

Cheers