Async HTTP Client dropping requests

[Mani]

Hi,

I am trying to use Async HTTP Client with connection pool enabled.
I am using it to trigger lot of HTTP calls asynchronously using
multiple threads.

I am creating the HTTP Client as below:

Builder builder = new AsyncHttpClientConfig.Builder();
builder.setMaximumConnectionsPerHost(maxRequestPerRoute);
builder.setMaximumConnectionsTotal(httpConnections);
builder.setRequestTimeoutInMs(timeout*1000);
builder.setCompressionEnabled(true);
builder.setAllowPoolingConnection(true);
builder.build();
AsyncHttpClient client = new AsyncHttpClient(builder.build());

I am invoking the HTTP request as below:
httpClient.executeRequest(request,new LongPollResponseHandler(user));

I am noticing that some of the requests that were invoked are not
reaching the server.
What could be the reason for this?
Will there be problems in using Async HTTP Client in a Multi threaded
environment?
Thanks a lot for the help in advance.

[Jeanfrancois Arcand]

Salut,

the only way I can help would be you send me a unit test reproducing the
issue. I've never experienced such issue...Are you sure your
AsyncHandler#onThrowable isn't invoked with some exception (usually it's
connect exception produced because the OS is busy sending requests).

A+

-- Jeanfrancois

[Mani]

Hi Jean,

Thank you for your response.
We figured out the problem with above mentioned issue.
It finally turned out to be server really dropping the requests rather
than the client.
We resolved that issue now.

But have a new issue now altogether.
We keep creating connections (up to 200K we target to), but at around
30k connections we are seeing the following exception:
java.net.ConnectException: Cannot assign requested address
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:
333)
at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:
860)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:
801)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:
726)
at
sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:
904)
at com.ning.http.client.providers.jdk.JDKAsyncHttpProvider
$AsyncHttpUrlConnection.configure(JDKAsyncHttpProvider.java:506)
at com.ning.http.client.providers.jdk.JDKAsyncHttpProvider
$AsyncHttpUrlConnection.call(JDKAsyncHttpProvider.java:223)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

For both Client & Server OS, we configured to allow up to 200K
connections.
But still we see the above excepion.
Any thoughts as to how to resolve this issue?

Thank you.

Thanks & Regards,
M.K.N.Manikandar.

On Feb 15, 2:20 am, Jeanfrancois Arcand <jfarcand....@gmail.com>
wrote:

[Jeanfrancois Arcand]

Salut,

> For both Client& Server OS, we configured to allow up to 200K

> connections.
> But still we see the above excepion.
> Any thoughts as to how to resolve this issue?

How many file descriptor have you configured on your OS? You usually get
that exception when the OS run out of file descriptor.

A+

-- Jeanfrancois

>
> Thank you.
>
> Thanks& Regards,

[Jeffery Griffith]

Should also check netstat for time_wait connections also. If the
connection rate is too fast, you may have ephemeral port exhaustion.

> --
> You received this message because you are subscribed to the Google Groups
> "asynchttpclient" group.
> To post to this group, send email to asyncht...@googlegroups.com.
> To unsubscribe from this group, send email to
> asynchttpclie...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/asynchttpclient?hl=en.
>

[Jeffery Griffith]

Mani, maybe it's just me, but I'm not sure exactly what you mean here
by 'connections'. Connections aren't really exposed through the client
- you either execute requests that have connections that get closed,
or you have connections that get pooled after. By 200k connections, do
you really mean you are running a load test with 200k requests that
are always closed?

[Jeffery Griffith]

Ooo, I see now. You are keeping the connection open with your long
poll handler. It wouldn't surprise me at all that you'd run out of
ephemeral ports (64k at the max).

On Tue, Feb 21, 2012 at 4:51 PM, Jeffery Griffith

[Manikandar madduri]

Hi Jeffery,

 

Thanks a lot for all your help. I really appreciate!!

 

Coming to the issue we are seeing, I dont think we are going out of ephemeral ports.

Using netstat, when I monitor total (both ESTABLISHED & TIME_WAIT included) connections, it went up to 30K maximum.

So I am sure we can go much higher. But for some reason we are getting this exception very early.

Also at OS Level I have set the file descriptors to 300000 both on client & server.

So I am expecting it to go even higher than 64K.

 

One thing that I noticed is, as many connections are open, so many threads are also running.

Is it the expected behavior or can we tune Async Http Client better?

[Jeanfrancois Arcand]

Salut,

On 12-02-22 9:05 AM, Manikandar madduri wrote:

Hi Jeffery,

 

Thanks a lot for all your help. I really appreciate!!

 

Coming to the issue we are seeing, I dont think we are going out of ephemeral ports.

Using netstat, when I monitor total (both ESTABLISHED & TIME_WAIT included) connections, it went up to 30K maximum.

So I am sure we can go much higher. But for some reason we are getting this exception very early.

Also at OS Level I have set the file descriptors to 300000 both on client & server.

So I am expecting it to go even higher than 64K.

 

One thing that I noticed is, as many connections are open, so many threads are also running.

Is it the expected behavior or can we tune Async Http Client better?

Yes, you can configure both thread pool used. First you can configure the Netty internal one (be carefull)

https://github.com/sonatype/async-http-client/blob/master/src/main/java/com/ning/http/client/providers/netty/NettyAsyncHttpProviderConfig.java#L53

Netty by default using a CachedThreadPool. You might want to play with that ThreadPool.

You can set the application thread pool as well from the AsyncHttpClientConfig

https://github.com/sonatype/async-http-client/blob/master/src/main/java/com/ning/http/client/AsyncHttpClientConfig.java#L679

On thing you may want to try as well is to set the async connect to false and see if that help

https://github.com/sonatype/async-http-client/blob/master/src/main/java/com/ning/http/client/providers/netty/NettyAsyncHttpProviderConfig.java#L43

A+

-- Jeanfrancois

[Jeffery Griffith]

Hi Mani,

Can you describe the lifecycle of each request? I see you have a
request timeout, but didn't see its value in the example, however
based on the name of your handler LongPollResponseHandler, I take it
your goal is to establish >200k ESTABLISHED connections
*concurrently*?

Also, I don't see your OS mentioned. Are you using linux? The
available port range can be found with:
% cat /proc/sys/net/ipv4/ip_local_port_range
It could be that you have this set to around 32k.

Jean-Francois may correct me, but the last time I saw the connect
code, the client does an ephemeral bind (does not bind to any
particular IP or port) and therefore I don't see how one could ever
reach more than 64k without exhausting the port range.

--jg

On Wed, Feb 22, 2012 at 9:16 AM, Jeanfrancois Arcand

[Manikandar madduri]

Hi Jeffery,

 

Thanks a lot for all your guidance!!

It really helped us.

You are right that the port range was set to 32K.

We increased it to 64K and tried and we see it scaling further.

 

But as I understand 64K is the hard limit.

Is there a way to over come this hard limit and scale further on a single machine?

[Tatu Saloranta]

On Thu, Feb 23, 2012 at 2:56 AM, Manikandar madduri
<m.k.n.ma...@gmail.com> wrote:
> Hi Jeffery,
>

> Thanks a lot for all your guidance!!
> It really helped us.
> You are right that the port range was set to 32K.
> We increased it to 64K and tried and we see it scaling further.
>
> But as I understand 64K is the hard limit.
> Is there a way to over come this hard limit and scale further on a single
> machine?

IP port numbers are 16-bit, so it is very much a hard limit. not just
by lib, Java or OS.
But it only limits connections to specific IP+port combination. Also,
do you really need that many connections? HTTP 1.1 reuses connections
(unless explicitly disabled), and typically only couple should be
concurrently used even for large number of requests.

-+ Tatu +-

[Jeffery Griffith]

My pleasure, Mani.

Tatu, I think the difference here is that Mani wants to do long polls
on all many connections. Are the destinations all to different IPs,
Mani?

I've gone round and round this problem with my (netty-based) TCP
client also which needs massive scaling for outgoing connections. The
problem is that, even with multiple IPs, linux flattens its ephemeral
port space into a single table irrespective of the IP address and you
very quickly run out of ports if you allow ephemeral binding. The only
solution I have found for this is to configure my NIC for multiple IPs
and do the IP/port binding myself outside of the defined ephemeral
range. Adding this into the ahc client (or any other) might be quite a
challenge.

--jg

[Tatu Saloranta]

On Thu, Feb 23, 2012 at 10:59 AM, Jeffery Griffith
<jeffery....@gmail.com> wrote:
> My pleasure, Mani.
>
> Tatu, I think the difference here is that Mani wants to do long polls
> on all many connections. Are the destinations all to different IPs,

Ah yes, true. Not quite the same -- although, also one reason for
considering alternatives other than long-polling (or in general,
long-living connections)

-+ Tatu +-