FATAL [org.jgroups.protocols.TCP] node01: thread pool is full (max=100, active=200); thread dump (dumped once, until thread_dump is reset)

[VJ REDDY]

Hi All,

I recently upgraded our servers to WildFly-26.1.2 with Java 1.8 and suddenly we started seeing this errors. 

Anyone has any idea what causing this error and please help me to resolve this issue

FYI: We have 3 servers in cluster

Here is the Error we are getting:

2023-10-20 04:38:41,426 [,,] FATAL [org.jgroups.protocols.TCP] prdqtradevm01: thread pool is full (max=100, active=200); thread dump (dumped once, until thread_dump is reset):
"thread-9063,ejb,node01" #22239 prio=0 tid=0x56df nid=NA waiting
   java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x3a86f1a0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at org.jgroups.protocols.FRAG3$FragEntry.set(FRAG3.java:335)
at org.jgroups.protocols.FRAG3.unfragment(FRAG3.java:296)
at org.jgroups.protocols.FRAG3.up(FRAG3.java:157)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:355)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1051)
at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:774)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:755)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:408)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:595)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132)
at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:281)
at org.jgroups.protocols.Discovery.up(Discovery.java:300)
at org.jgroups.protocols.Discovery.up(Discovery.java:300)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1404)
at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
at org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70)
at java.lang.Thread.run(Thread.java:750)


   Locked ownable synchronizers:
- <0x46c786b8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"thread-8971,ejb,node01" #22147 prio=0 tid=0x5683 nid=NA timed_waiting
   java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x400e520e> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
at org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70)
at java.lang.Thread.run(Thread.java:750)

"Transaction Reaper Worker 2" #18043 prio=0 tid=0x467b nid=NA waiting
   java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <0x717742f9> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForWork(TransactionReaper.java:360)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)


"Keep-Alive-SocketCleaner" #13509 prio=0 tid=0x34c5 nid=NA waiting
   java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <0x435bf0fb> (a sun.net.www.http.KeepAliveStreamCleaner)
at sun.net.www.http.KeepAliveStreamCleaner.run(KeepAliveStreamCleaner.java:101)
at java.lang.Thread.run(Thread.java:750)


"Java2D Disposer" #3539 prio=0 tid=0xdd3 nid=NA waiting
   java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <0x71d3d4cd> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:150)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:171)
at sun.java2d.Disposer.run(Disposer.java:148)
at java.lang.Thread.run(Thread.java:750)


"FD_SOCK conn-handler-41,ejb, node01" #282 prio=0 tid=0x11a nid=NA runnable
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:224)
at org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1078)
at org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
at org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70)
at java.lang.Thread.run(Thread.java:750)


"Connection.Receiver [10.26.129.11:54423 - 10.26.129.13:7600]-40,ejb, node01" #281 prio=0 tid=0x119 nid=NA runnable
   java.lang.Thread.State: RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
at org.jgroups.util.Util.dumpThreads(Util.java:2201)
at org.jgroups.protocols.TP.submitToThreadPool(TP.java:1543)
at org.jgroups.protocols.TP.submitToThreadPool(TP.java:1531)
at org.jgroups.util.SubmitToThreadPool.process(SubmitToThreadPool.java:34)
at org.jgroups.util.MaxOneThreadPerSender.process(MaxOneThreadPerSender.java:63)
at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1508)
at org.jgroups.protocols.TP.receive(TP.java:1479)
at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:182)
at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:306)
at org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
at org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70)
at java.lang.Thread.run(Thread.java:750)

Thanks,

VJ

[VJ REDDY]

Hi All,

No one experienced with this issue before :(

I am really having major issue.... and I am counting on you guys... please help me to find a soluition

Thanks,

VJ

[Paul Ferraro]

Can you paste your JGroups protocol stack?

[vijen reddy]

Hi Paul,

Here it is:

        <subsystem xmlns="urn:jboss:domain:jgroups:8.0">
            <channels default="ee">
                <channel name="ee" stack="tcp" cluster="ejb"/>
            </channels>
            <stacks>
                <stack name="udp">
                    <transport type="UDP" socket-binding="jgroups-udp"/>
                    <protocol type="PING"/>
                    <protocol type="MERGE3"/>
                    <socket-protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
                    <protocol type="FD_ALL"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK2"/>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="pbcast.GMS"/>
                    <protocol type="UFC"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG3"/>
                </stack>
                <stack name="tcp">
                    <transport type="TCP" socket-binding="jgroups-tcp"/>
                    <protocol type="org.jgroups.protocols.TCPPING">
                        <property name="initial_hosts">10.101.18.41[7600],10.101.18.42[7600]</property>
                        <property name="port_range">0</property>
                    </protocol>
                    <socket-protocol type="MPING" socket-binding="jgroups-mping"/>
                    <protocol type="MERGE3"/>
                    <socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                    <protocol type="FD_ALL"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK2"/>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="pbcast.GMS"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG3"/>
                </stack>
            </stacks>
        </subsystem>

Thanks,

VJ

--
You received this message because you are subscribed to the Google Groups "WildFly" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wildfly+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/d0ab3ee0-05f3-4aab-918b-9206bb379ec0n%40googlegroups.com.

[vijen reddy]

Hi Paul,

Did you get a chance to look at this issue?

Thanks,

VJ

[Paul Ferraro]

Your protocol stack is missing <protocol type="UFC"/> (normally sandwiched between pbcast.GMS and MFC), which would otherwise mitigate the risk of unicast senders overwhelming receivers.