Undertow does not respond to HTTP requests after some time

[Steven]

Hi, I'm writing here in search for help.

My application runs on wildfly 18 and after some time the wildfly server stops responding to the client requests.

 
The structure is:
- WildFly18 cluster with 2 wildfly instance
- Apache web server as load balancer to route client requests to the application servers

-  The application is written using java ee8.


Observation:
1) The Apache log displayed 504 error from the node2 when other client requests were routed to node2 when the issue happened.
2) SocketTimeoutException was observed as below in server log file when one ejb component issued a web service request to the clustering, and no further error found.
Unable to send to url[http://localhost:8180/xxxx/WebServiceManager]: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.axis2.transport.http.impl.httpclient4.RequestImpl.execute(RequestImpl.java:210)
3) node1 worked well so far
4) node1 has the same configuration as the node2


the serve configuration:
       <subsystem xmlns="urn:jboss:domain:io:3.0">
            <worker name="default"/>
            <buffer-pool name="default"/>
        </subsystem>
        <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
        <subsystem xmlns="urn:jboss:domain:jca:5.0">
            <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
            <bean-validation enabled="true"/>
            <default-workmanager>
                <short-running-threads>
                    <core-threads count="50"/>
                    <queue-length count="50"/>
                    <max-threads count="50"/>
                    <keepalive-time time="10" unit="seconds"/>
                </short-running-threads>
                <long-running-threads>
                    <core-threads count="50"/>
                    <queue-length count="50"/>
                    <max-threads count="50"/>
                    <keepalive-time time="10" unit="seconds"/>
                </long-running-threads>
            </default-workmanager>
            <cached-connection-manager/>
        </subsystem>
<subsystem xmlns="urn:jboss:domain:undertow:10.0" default-server="default-server" default-virtual-host="default-host" default-servlet-container="default" default-security-domain="other" statistics-enabled="${wildfly.undertow.statistics-enabled:${wildfly.statistics-enabled:false}}">
            <buffer-cache name="default"/>
            <server name="default-server">
                <ajp-listener name="ajp" socket-binding="ajp" max-parameters="5000" allow-equals-in-cookie-value="true"/>
                <http-listener name="default" socket-binding="http" max-post-size="104857600" max-parameters="20000" allow-equals-in-cookie-value="true" redirect-socket="https" http2-enable-push="false"/>
                <https-listener name="https" socket-binding="https" max-post-size="104857600" max-parameters="20000" allow-equals-in-cookie-value="true" security-realm="ApplicationRealm" http2-enable-push="false"/>
                <host name="default-host" alias="localhost">
                    <location name="/" handler="welcome-content"/>
                    <http-invoker security-realm="ApplicationRealm"/>
                </host>
            </server>
            <servlet-container name="default" default-session-timeout="30">
                <jsp-config development="true"/>
                <websockets/>
            </servlet-container>
            <handlers>
                <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
            </handlers>
        </subsystem>



I really don't get why the node2 did not respond to requests. and unsure if it is due to wildfly itself.


All advice welcome.

Thank you in advance.

[John Saccoccio]

High CPU load and SslConduit calls in stack trace on the node?  There's an Undertow/SSL bug introduced in minor JDK version updates, see https://access.redhat.com/solutions/6997164

and https://groups.google.com/g/wildfly/c/Qo8JQquIHZw