Connection leak on connection timeout

[Ryan Barker]

We have 2 systems, using synchronous posts, returning a ClientResponse object.

Under very high load, the target system begins to timeout followed by running out of connections. If we remove the connection limit, we will run out of file handles. If we raise the file limit super far, it has a very slow increase with the file handles being somewhat cleaned up by the garbage collector however we easily hit the higher 8K and crash under sustained load.

From eclipse memory analyzer, it appears the issue is that the netty connections are not being cleaned up, specifically org.jboss.netty.channel.socket.nio.NioClientSocketChannel
From jvisualvm, the vast majority of the references to the FileDescriptor go only to SocketChannelImpl which are linked to only from the Finalizer.

Heapdump can unfortunately not be provided directly however I can provide if required.


1st stack trace
Caused by: com.sun.jersey.api.client.ClientHandlerException: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: No response received after 5000
    at org.sonatype.spice.jersey.client.ahc.AhcClientHandler.handle(AhcClientHandler.java:130)
    at com.sun.jersey.api.client.Client.handle(Client.java:648)
    at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680)
    at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
    at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:558)
    at com.eharmony.protorest.RestClientImpl.post(RestClientImpl.java:278)
    ... 38 more
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: No response received after 5000
    at com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyResponseFuture.java:223)
    at com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyResponseFuture.java:187)
    at org.sonatype.spice.jersey.client.ahc.AhcClientHandler.handle(AhcClientHandler.java:116)
    ... 73 more
Caused by: java.util.concurrent.TimeoutException: No response received after 5000
    at com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyResponseFuture.java:215)
    ... 75 more

2nd stack trace
Caused by: com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: Too many connections 300
    at org.sonatype.spice.jersey.client.ahc.AhcClientHandler.handle(AhcClientHandler.java:130)
    at com.sun.jersey.api.client.Client.handle(Client.java:648)
    at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680)
    at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
    at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:558)
    at com.eharmony.protorest.RestClientImpl.post(RestClientImpl.java:278)
    ... 38 more
Caused by: java.io.IOException: Too many connections 300
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.doConnect(NettyAsyncHttpProvider.java:957)
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.execute(NettyAsyncHttpProvider.java:858)
    at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:525)
    at org.sonatype.spice.jersey.client.ahc.AhcClientHandler.handle(AhcClientHandler.java:116)



source:

    @Override
    public <R, T> R post(final String url, final T data, final Class<R> c) {
        final WebResource webResource               = restfulClient.resource(url);
        final WebResource.Builder requestBuilder    = webResource.getRequestBuilder();

        final Class<? extends Object> dataClass = data.getClass();
        final ClientResponse response =requestBuilder.type(MediaType.APPLICATION_JSON_TYPE).accept(MediaType.APPLICATION_JSON_TYPE).entity(data).post(ClientResponse.class);
        try {
            return getEntity(c, response);
        } finally {
            closeResponse(response);
        }
    }
    private <T> T getEntity(final Class<T> c, final ClientResponse response) {
        final int status = response.getStatus();
        log.debug("Response status: " + status);
        switch (status) {
        case 200:
            return response.getEntity(c);
        case 404:
            return null;
        default:
            log.error("Unexpected invalid response received! Status: " + status);
            return null;
        }
    }
    private static void closeResponse(final ClientResponse response) {
        if (response != null) {
            try {
                response.close();
            } catch (final ClientHandlerException che) {
                log.error("Could not close connection", che);
            }
        }
    }

Configuration:
//connectTimeout is 5000, readTimeout is 5000, threadPoolSize is 30, maxConnections is 300
// original code had no max total connections and noExecutorService and ran out of file descriptors

    public static final class AHCFactory {
        public AHCFactory() {
        }

        public Realm buildRealm(final String username, final String password, final boolean usePreemptiveAuth, final AuthScheme authScheme) {
            final Realm realm = new Realm.RealmBuilder()
            .setPrincipal(username)
            .setPassword(password)
            .setUsePreemptiveAuth(usePreemptiveAuth)
            .setScheme(authScheme)
            .build();

            return realm;
        }

        public Client create(final int maxConnections, final int connectTimeout, final int readTimeout,
                final String username, final String password,
                final boolean usePreemptiveAuth, final AuthScheme authScheme, final int threadPoolSize) {

            final Realm realm = this.buildRealm(username, password, usePreemptiveAuth, authScheme);

            final DefaultAhcConfig config = registerProviders();
            config.getAsyncHttpClientConfigBuilder()
            //setTimeouts
            .setConnectionTimeoutInMs(connectTimeout)
            .setRequestTimeoutInMs(readTimeout)
            .setIdleConnectionInPoolTimeoutInMs(connectTimeout * 2)
            // set maxConnections
            .setMaximumConnectionsPerHost(maxConnections)
            .setMaximumConnectionsTotal(maxConnections)
            // this executor service is used for callbacks by netty itself, make one with a maximum number of threads
            .setExecutorService( new ThreadPoolExecutor(0, threadPoolSize,
                            connectTimeout * 2, TimeUnit.MILLISECONDS,
                            new LinkedBlockingQueue<Runnable>()))
            .setCompressionEnabled(true)
            .setRealm(realm);

            // Do not remove this line
            // If you do not set this, the async calls will essentially be unlimitted in thread pool size
            // until you run into OS level permissions (ulimit file handles and num processes)
            // then you blow up in spectacular ways!
            config.getProperties().put(ClientConfig.PROPERTY_THREADPOOL_SIZE, threadPoolSize);

            final AhcHttpClient retval = AhcHttpClient.create(config);

            log.info("Setting HTTP Client connect timeout = " + connectTimeout);
            retval.setConnectTimeout(connectTimeout);
            log.info("Setting HTTP Client read timeout = " + readTimeout);
            retval.setReadTimeout(readTimeout);
            return retval;
        }

[Ryan Barker]

Sorry to be clear, the target system begins to timeout. the client system begins having timeout exceptions followed by running out of connections and/or file handles depending on configuration.

[Stéphane Landelle]

Hi,

Which version of AHC and Netty do you use?

2013/10/16 Ryan Barker <cela...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "asynchttpclient" group.
To unsubscribe from this group and stop receiving emails from it, send an email to asynchttpclie...@googlegroups.com.
To post to this group, send email to asyncht...@googlegroups.com.
Visit this group at http://groups.google.com/group/asynchttpclient.
For more options, visit https://groups.google.com/groups/opt_out.

[Ryan Barker]

I am using

 <dependency>
             <groupId>org.jfarcand</groupId>
             <artifactId>jersey-ahc-client</artifactId>
             <version>1.0.4</version>
         </dependency>

Which underneath is using AHC 1.7.6 and Netty 3.4.4 final

[Stéphane Landelle]

Quite a few things happened since then.

Could you upgrade AHC to 1.7.20 and Netty to 3.7.0, please?

2013/10/16 Ryan Barker <cela...@gmail.com>

[Ryan Barker]

That fixed it. Perhaps its time to make an update to https://github.com/AsyncHttpClient/jersey-ahc-client ;)

[Stéphane Landelle]

Great!

Regarding jersey-ahc-client, that's a side project where only Jean-François Arcand develops.

2013/10/16 Ryan Barker <cela...@gmail.com>