How do you timeout incoming connections from RestEasy in WildFly?

[Bill Crowell]

Good evening.

I am stuck using WildFly 24.0.1.Final, but I can also reproduce this issue with WildFly 38.0.1.Final.  This issue happens on Windows Server, but I have seen it on Rocky Linux as well.

Here is what is happening.  I have an Apache JMeter test script which hits a simple REST endpoint that sleeps between 1 and 10 seconds, and then spits out a success message.  What I have noticed as this test runs and ramps up to 5000 threads/connections is that the connections on the WildFly server are not automatically cleaned up and go into a CLOSE_WAIT state.  I have tried setting no-request-timeout, request-parse-timeout, read-timeout, and write-timeout to 10000ms in the https-listener in the Undertow subsystem.  I have also tried setting the ssl-session-timeout as well on the elytron subsystem.  This pile up of connections is causing WIldFly to hang.  I have even tried setting max-connections to 100.

This REST service uses WildFly's RestEasy 3.15.1.Final on WildFly 24.0.1 and uses RestEasy 6.2.14.Final on WildFly 38.0.1.Final.  There was a fix in WildFly 32.0.0.Final which uses RestEasy 6.2.8.Final: https://issues.redhat.com/browse/RESTEASY-3470

This fix calls evictExpiredConnections() and sets evictIdleConnections to 60 seconds.  This does not seem to be working as the connections persist on the server for up to 60 seconds on Linux (via the net.ipv4.tcp_fin_timeout through sysctl) and 240 seconds on Windows Server (via the registry's HKLM\SYSTEM\CurrentControlSet\Services\TCPIP\Parameters\TcpTimedWaitDelay DWORD setting).  

What am I forgetting so these connections do not pile up?  Is the eviction thread just not working?  I am at a loss.

Regards,

William Crowell

[Bill Crowell]

One correction on the testing I did.  CLOSE_WAIT connections do go away, but a lot of connections stay as ESTABLISHED even after the JVM is brought down.  

[James Perkins]

Are you closing the client? If it's possible, do you have a reproducer?

[Bill Crowell]

James,

Thank you for your reply.  I checked the jmeter.properties file, and I was not setting httpclient4.idletimeout.  I set that to 10000ms, and then edited my JMeter script to use a HTTPSampler.connect_timeout and HTTPSampler.response_timeout to both 10000.  I then set the HTTPSampler.implementation to HttpClient4 which I think the RestEasy client uses.  I did get better results this time on my test runs.   

I am assuming this fix would be helpful to evict connections that was introduced in RestEasy 6.2.8.Final?  https://issues.redhat.com/browse/RESTEASY-3470

It is not backported, but I could backport to RestEasy 3.15.1.Final because it is just a few lines of code.

As an alternative method of testing I created a test client where I am spinning up 100's of java.lang.Runnable threads, but the test client is not able to ramp up connections gradually like JMeter.  I set each thread with a readTimeout/connectTimeout of 10000 each that closes the response and client when it is done with it.  This is what it looks like:

import jakarta.ws.rs.client.*;
import jakarta.ws.rs.core.Response;
import org.jboss.resteasy.client.jaxrs.ResteasyWebTarget;
import java.util.concurrent.TimeUnit;

public class WebServiceClientRunnableTaskUnsecure implements Runnable {
    private final String URL;
    private long readTimeout = 0;
    private long connectTimeout = 0;

    public WebServiceClientRunnableTaskUnsecure(String targetUrl, long readTimeout, long connectTimeout) {
        this.URL = targetUrl;
        this.readTimeout = readTimeout;
        this.connectTimeout = connectTimeout;
    }

    @Override
    public void run() {
        Response response = null;
        Client client = null;
        try {
           ClientBuilder clientBuilder = ClientBuilder.newBuilder()
                    .readTimeout(readTimeout, TimeUnit.MILLISECONDS)
                    .connectTimeout(connectTimeout, TimeUnit.MILLISECONDS);
            client = clientBuilder.build();
            ResteasyWebTarget target = (ResteasyWebTarget) client.target(URL);
            Invocation.Builder request = target.request();
            response = request.get();
            int status = response.getStatus();
        } catch (Exception e) {
            Thread.currentThread().interrupt();
        } finally {
            if (response != null) {
                response.close();
            }
            if (client != null) {
                client.close();
            }
        }
    }
}

--
You received this message because you are subscribed to the Google Groups "WildFly" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wildfly+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/wildfly/a4e6f435-d344-4973-8957-803cd6dce3bbn%40googlegroups.com.

[James Perkins]

Great! Thank you for the information and test!

The 3.15.x branch is effectively dead. We've not been adding any fixes to that branch. That said, you can definitely fork it and add that fix if you need it.

[Bill Crowell]

James, that fix would only help on the client, correct? It would not help on the rest eaves, easy server side I am guessing. 

To view this discussion visit https://groups.google.com/d/msgid/wildfly/8c0005a8-6c85-44d1-ac6c-0d07c98ed708n%40googlegroups.com.

[James Perkins]

Correct. If you built it and replaced the libraries in WildFly it would fix the clients running inside WildFly too, but you'd need to replace the modules. That is easier to do in newer versions of WildFly that you can provision, but still doable in the older versions. It's just more of a manual process.

[Bill Crowell]

James, much appreciated.  Thank you.

To view this discussion visit https://groups.google.com/d/msgid/wildfly/623f4559-7297-4c7b-9364-d2a1e69e7830n%40googlegroups.com.