setMaxConnectionsPerHost issues

[Eugene Prokopiev]

Hi,

I have some issue with setMaxConnectionsPerHost. My code looks like:

public class HttpClientApp {
   
    AsyncHttpClient client = new AsyncHttpClient(
        new NettyAsyncHttpProvider(new AsyncHttpClientConfig.Builder().setMaxConnectionsPerHost(1).build()));
   
    Request request = new RequestBuilder().setUrl("http://10.7.1.13:3000").build();
   
    public HttpClientApp() throws Exception {
        List<Future<Response>> futures = new ArrayList<Future<Response>>();
        for (int i=0;i<5;i++) {
            futures.add(client.executeRequest(request));
        }
        for (Future<Response> future : futures)
            System.out.println(future.get().getStatusText());
        System.out.println("END");
    }
   
    public static void main(String[] args) throws Exception {
        new HttpClientApp();       
    }

}

On running it I see:

Exception in thread "main" java.util.concurrent.ExecutionException: java.io.IOException: Too many connections -1
        at com.ning.http.client.providers.netty.future.NettyResponseFuture.abort(NettyResponseFuture.java:228)
        at com.ning.http.client.providers.netty.request.NettyRequestSender.abort(NettyRequestSender.java:420)
        at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:294)
        at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequestWithCertainForceConnect(NettyRequestSender.java:142)
        at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequest(NettyRequestSender.java:117)
        at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.execute(NettyAsyncHttpProvider.java:87)
        at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:486)
        at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:508)
        at ru.itx.proxyapp.HttpClientApp.<init>(HttpClientApp.java:24)
        at ru.itx.proxyapp.HttpClientApp.main(HttpClientApp.java:32)
Caused by: java.io.IOException: Too many connections -1

I see the same problem with any number in setMaxConnectionsPerHost() except 0. Why can it be?

[Stéphane Landelle]

Indeed , there was an issue with the Exception message: https://github.com/AsyncHttpClient/async-http-client/issues/799

Thanks for reporting!

--
You received this message because you are subscribed to the Google Groups "asynchttpclient" group.
To unsubscribe from this group and stop receiving emails from it, send an email to asynchttpclie...@googlegroups.com.
To post to this group, send email to asyncht...@googlegroups.com.
Visit this group at http://groups.google.com/group/asynchttpclient.
For more options, visit https://groups.google.com/d/optout.

[Eugene Prokopiev]

I see the same problem with grizzly implementation:

Exception in thread "main" java.util.concurrent.ExecutionException: java.io.IOException: Max connections exceeded
        at org.glassfish.grizzly.impl.SafeFutureImpl$Sync.innerGet(SafeFutureImpl.java:349)
        at org.glassfish.grizzly.impl.SafeFutureImpl.get(SafeFutureImpl.java:255)
        at com.ning.http.client.providers.grizzly.GrizzlyResponseFuture.get(GrizzlyResponseFuture.java:165)
        at ru.itx.proxyapp.HttpClientApp.<init>(HttpClientApp.java:28)
        at ru.itx.proxyapp.HttpClientApp.main(HttpClientApp.java:33)
Caused by: java.io.IOException: Max connections exceeded
        at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$ConnectionManager.doAsyncTrackedConnection(GrizzlyAsyncHttpProvider.java:2420)
        at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.execute(GrizzlyAsyncHttpProvider.java:250)

        at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:486)
        at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:508)

        at ru.itx.proxyapp.HttpClientApp.<init>(HttpClientApp.java:25)
        ... 1 more

I see this error only if using setMaxConnections or setMaxConnectionsPerHost. Is this a bug or misunderstanding? Need I use this methods to limit max concurent requests to remote resource and prevent failed responses or this methods has another goal?

[Stéphane Landelle]

I think you're misunderstanding this feature: requests don't get queued if the max is exceeded.

--

[Eugene Prokopiev]

2015-01-12 21:52 GMT+03:00 Stéphane Landelle <slan...@excilys.com>:

> I think you're misunderstanding this feature: requests don't get queued if
> the max is exceeded.

Thanks! But how can I queue requests? Is it impossible with AsyncHttpClient?

[Stéphane Landelle]

No, it's not implemented. Contribution welcome!

[Gordon Sun]

I try to catch the exception but cannot do it because it is a 

java.util.concurrent.ExecutionException

How about implement a check that can return if the client has reached the max connection? At least I can check that before blindly call the client to issue the request. In my case, I can block the main thread and wait a little bit. Only if I know that the max has been reached.

[Stéphane Landelle]

2015-01-13 4:51 GMT+01:00 Gordon Sun <suns...@gmail.com>:

I try to catch the exception but cannot do it because it is a 

java.util.concurrent.ExecutionException

It's an ExecutionException because of Future's signature.

 

How about implement a check that can return if the client has reached the max connection? At least I can check that before blindly call the client to issue the request.

Everything happens asynchronously. There's no guarantee that the limit has been reached after such a check and before actually sending the request.

In my case, I can block the main thread and wait a little bit. Only if I know that the max has been reached.

Blocking is not the point of such library.

[Gordon Sun]

Good points. I guess my use case requires me to manually slow things down.

I have about 10000 urls that I need to hit. Do them one by one is too slow so I'm trying this library.

But if I blindly loop through all of them and call the client, my machine run out of sockets.

If I set a limit then I run out of the limit.

Without the ability to catch the exception or know if it reached the limit, I had to guess how fast my machine can handle it and pause for a second for every 50 requests.

Is there a better solution?

[Stéphane Landelle]

Just store your requests in a queue, launch as many concurrent requests from the main thread as you want, listen to the response (either register a Listener on the ListenableFuture or write your own AsyncHandler) and fetch a new request from the pool. Wait in main thread for queue starvation (or use a CountdownLatch).

--

[Gordon Sun]

I'm kinda lost. If I launch as many concurrent requests as I want from the main thread, my machine ran out of socket or "Too many open files"