"java.net.BindException: Cannot assign requested address" under heavy load with async posts - RESOLVED

[Ryan Barker]

Thought I'd post this in case anyone else runs into this. If you are using very heavy async calls, you can run out of threads. There is an internal threadpool used by jersey which needs to be configured in a non-obvious way.

This setting could very well be considered a bug in jersey client itself, but a change to async http client to set a more reasonable default in the builder would be appreciated.

Code:
DefaultAhcConfig config = new DefaultAhcConfig();
            config.getAsyncHttpClientConfigBuilder().setConnectionTimeoutInMs(connectTimeout)
                            .setRequestTimeoutInMs(readTimeout).setMaximumConnectionsPerHost(maxConnections)
                            .setMaximumConnectionsPerHost(maxConnections).setCompressionEnabled(true)
                            .setIdleConnectionInPoolTimeoutInMs(connectTimeout * 20);
            // Do not remove this line
            // If you do not set this, the async calls will essentially be unlimitted in thread pool size
            // until you run into OS level permissions (ulimit file handles and num processes)
            // then you blow up in spectacular ways!
            config.getProperties().put(ClientConfig.PROPERTY_THREADPOOL_SIZE, maxConnections);



Stack trace we were getting:

java.util.concurrent.ExecutionException: com.sun.jersey.api.client.ClientHandlerException: java.util.concurrent.ExecutionException: java.net.ConnectException: Cannot assign requested address to http://myurl.here.com
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
...
Caused by: com.sun.jersey.api.client.ClientHandlerException: java.util.concurrent.ExecutionException: java.net.ConnectException: Cannot assign requested address to http://myurl.here.com
    at org.sonatype.spice.jersey.client.ahc.AhcClientHandler.handle(AhcClientHandler.java:130)
    at com.sun.jersey.api.client.Client.handle(Client.java:648)
    at com.sun.jersey.api.client.AsyncWebResource$5.call(AsyncWebResource.java:797)
    at com.sun.jersey.api.client.AsyncWebResource$5.call(AsyncWebResource.java:795)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    ... 1 more
Caused by: java.util.concurrent.ExecutionException: java.net.ConnectException: Cannot assign requested address to http://myurl.here.com
    at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:297)
    at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:104)
    at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:399)
    at org.jboss.netty.channel.DefaultChannelFuture.addListener(DefaultChannelFuture.java:145)
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.doConnect(NettyAsyncHttpProvider.java:1041)
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.execute(NettyAsyncHttpProvider.java:858)
    at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:525)
    at org.sonatype.spice.jersey.client.ahc.AhcClientHandler.handle(AhcClientHandler.java:116)
    ... 11 more
Caused by: java.net.ConnectException: Cannot assign requested address to http://myurl.here.com
    at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:100)
    ... 17 more
Caused by: java.net.BindException: Cannot assign requested address
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:139)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:102)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:55)
    at org.jboss.netty.handler.codec.http.HttpClientCodec.handleDownstream(HttpClientCodec.java:97)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:108)
    at org.jboss.netty.channel.Channels.connect(Channels.java:642)
    at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:204)
    at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:230)
    at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.doConnect(NettyAsyncHttpProvider.java:999)
    ... 14 more