WF 39.0.1 huge CPU utilization caused by PerDestinationBundler

[Viktoriia Teliuk]

Hi!
We are switching from 37.0.1 to 39.0.1 Wildfly and experiencing a huge CPU utilization on the cluster with two nodes.

In WF 37 TransferQueueBundler was In WAITING time, but in WF 39 PerDestinationBundler is in RUNNABLE state and "eats" processor time

According to this advice https://infinispan.org/docs/16.0.x/titles/changes/changes.html#virtual_threads_enabled_by_default_jdk_21 set Dorg.infinispan.threads.virtual=false 

but it didn't help(

Here are the stacktraces with the highest CPU usage from the first and second nodes

pd-bundler-3,null,null
java...@21.0.10/java.util.concurrent.ConcurrentHashMap$Traverser.advance(ConcurrentHashMap.java:3383)
java...@21.0.10/java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:3483)
org.j...@5.5.2.Final//org.jgroups.protocols.PerDestinationBundler.run(PerDestinationBundler.java:135)
org.j...@5.5.2.Final//org.jgroups.util.Runner.run(Runner.java:87)
org.wildfly.clu...@9.0.2.Final//org.wildfly.clustering.context.ContextualExecutor$1.execute(ContextualExecutor.java:96)
org.wildfly.clu...@9.0.2.Final//org.wildfly.clustering.context.Contextualizer$2$1.run(Contextualizer.java:133)
java...@21.0.10/java.lang.Thread.runWith(Thread.java:1596)
java...@21.0.10/java.lang.Thread.run(Thread.java:1583)

2)

java...@21.0.10/java.lang.Thread.run(Thread.java:1583)pd-bundler-3,null,null
org.j...@5.5.2.Final//org.jgroups.protocols.PerDestinationBundler.run(PerDestinationBundler.java:138)
org.j...@5.5.2.Final//org.jgroups.util.Runner.run(Runner.java:87)
org.wildfly.clu...@9.0.2.Final//org.wildfly.clustering.context.ContextualExecutor$1.execute(ContextualExecutor.java:96)
org.wildfly.clu...@9.0.2.Final//org.wildfly.clustering.context.Contextualizer$2$1.run(Contextualizer.java:133)
java...@21.0.10/java.lang.Thread.runWith(Thread.java:1596)
java...@21.0.10/java.lang.Thread.run(Thread.java:1583)

Appreciate any help or ideas 🙏

[Paul Ferraro]

Have you tried reverting the bundler?

e.g.

/subsystem=jgroups/stack=stack-name/transport=protocol-name:map-put(name=properties, key=bundler_type, value=transfer-queue)

[Viktoriia Teliuk]

Thank you for the quick answer 🙏

Setting no-bundler helps.

With transfer-queue bundler_type, deploy on second node fails with errors

ISPN000475: Error processing response for request 23 from eng1

java.lang.AssertionError at org.infin...@16.0.5//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1573) at org.infin...@16.0.5//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1470) at org.infin...@16.0.5//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$0(JGroupsTransport.java:1654) at java.base/java.lang.Iterable.forEach(Iterable.java:75) at org.infin...@16.0.5//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1646) at org.j...@5.5.2.Final//org.jgroups.JChannel.up(JChannel.java:694) at org.j...@5.5.2.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:164) at org.j...@5.5.2.Final//org.jgroups.stack.Protocol.up(Protocol.java:392) at org.j...@5.5.2.Final//org.jgroups.protocols.FORK.up(FORK.java:177) at org.j...@5.5.2.Final//org.jgroups.protocols.FRAG2.up(FRAG2.java:159) at org.j...@5.5.2.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:813) at org.j...@5.5.2.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254) at org.j...@5.5.2.Final//org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1194) at org.j...@5.5.2.Final//org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:958) at org.j...@5.5.2.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:587) at org.j...@5.5.2.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:684) at org.j...@5.5.2.Final//org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:118) at org.j...@5.5.2.Final//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:195) at org.j...@5.5.2.Final//org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:205) at org.j...@5.5.2.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:285) at org.j...@5.5.2.Final//org.jgroups.protocols.Discovery.up(Discovery.java:324) at org.j...@5.5.2.Final//org.jgroups.protocols.TP.passBatchUp(TP.java:1164) at org.j...@5.5.2.Final//org.jgroups.util.SubmitToThreadPool$BatchHandler.passBatchUp(SubmitToThreadPool.java:147) at org.j...@5.5.2.Final//org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:143) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at org.wildfly.clu...@9.0.2.Final//org.wildfly.clustering.context.ContextualExecutor$1.execute(ContextualExecutor.java:96) at org.wildfly.clu...@9.0.2.Final//org.wildfly.clustering.context.Contextualizer$2$1.run(Contextualizer.java:133) at java.base/java.lang.Thread.run(Thread.java:1583)

ISPN000136: Error executing command PutKeyValueCommand on Cache 'default-server', writing keys [eng1]

org.infinispan.remoting.RemoteException: ISPN000217: Received exception from eng2, see cause for remote stack trace
at org.infin...@16.0.5//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:26)
at org.infin...@16.0.5//org.infinispan.remoting.transport.ValidSingleResponseCollector.withException(ValidSingleResponseCollector.java:37)
at org.infin...@16.0.5//org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:21)
at org.infin...@16.0.5//org.infinispan.remoting.transport.impl.SingleTargetRequest.addResponse(SingleTargetRequest.java:69)
at org.infin...@16.0.5//org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:44)
at org.infin...@16.0.5//org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:25)
at org.infin...@16.0.5//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:50)
at org.infin...@16.0.5//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1574)
at org.infin...@16.0.5//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1470)
at org.infin...@16.0.5//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$0(JGroupsTransport.java:1654)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.infin...@16.0.5//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1646)
at org.j...@5.5.2.Final//org.jgroups.JChannel.up(JChannel.java:694)
at org.j...@5.5.2.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:164)
at org.j...@5.5.2.Final//org.jgroups.stack.Protocol.up(Protocol.java:392)
at org.j...@5.5.2.Final//org.jgroups.protocols.FORK.up(FORK.java:177)
at org.j...@5.5.2.Final//org.jgroups.protocols.FRAG2.up(FRAG2.java:159)
at org.j...@5.5.2.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:813)
at org.j...@5.5.2.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254)
at org.j...@5.5.2.Final//org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1194)
at org.j...@5.5.2.Final//org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:958)
at org.j...@5.5.2.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:587)
at org.j...@5.5.2.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:731)
at org.j...@5.5.2.Final//org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:118)
at org.j...@5.5.2.Final//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:195)
at org.j...@5.5.2.Final//org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:205)
at org.j...@5.5.2.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
at org.j...@5.5.2.Final//org.jgroups.protocols.Discovery.up(Discovery.java:324)
at org.j...@5.5.2.Final//org.jgroups.protocols.TP.passBatchUp(TP.java:1164)
at org.j...@5.5.2.Final//org.jgroups.util.SubmitToThreadPool$BatchHandler.passBatchUp(SubmitToThreadPool.java:147)
at org.j...@5.5.2.Final//org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:143)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at org.wildfly.clu...@9.0.2.Final//org.wildfly.clustering.context.ContextualExecutor$1.execute(ContextualExecutor.java:96)
at org.wildfly.clu...@9.0.2.Final//org.wildfly.clustering.context.Contextualizer$2$1.run(Contextualizer.java:133)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.infinispan.commons.CacheException
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at org.infin...@16.0.5//org.infinispan.marshall.protostream.impl.MarshallableThrowable.getInstance(MarshallableThrowable.java:119)
at org.infin...@16.0.5//org.infinispan.marshall.protostream.impl.MarshallableThrowable.lambda$recreateGenericThrowable$2(MarshallableThrowable.java:95)
at org.infin...@16.0.5//org.infinispan.marshall.protostream.impl.MarshallableThrowable.create(MarshallableThrowable.java:111)
at org.infin...@16.0.5//org.infinispan.marshall.protostream.impl.MarshallableThrowable.recreateGenericThrowable(MarshallableThrowable.java:95)
at org.infin...@16.0.5//org.infinispan.marshall.protostream.impl.MarshallableThrowable.get(MarshallableThrowable.java:78)
at org.infin...@16.0.5//org.infinispan.remoting.responses.ExceptionResponse.getException(ExceptionResponse.java:31)
... 34 more
Caused by: java.lang.AssertionError
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at org.infin...@16.0.5//org.infinispan.marshall.protostream.impl.MarshallableThrowable.getInstance(MarshallableThrowable.java:119)
at org.infin...@16.0.5//org.infinispan.marshall.protostream.impl.MarshallableThrowable.lambda$recreateGenericThrowable$0(MarshallableThrowable.java:91)
at org.infin...@16.0.5//org.infinispan.marshall.protostream.impl.MarshallableThrowable.create(MarshallableThrowable.java:111)
at org.infin...@16.0.5//org.infinispan.marshall.protostream.impl.MarshallableThrowable.recreateGenericThrowable(MarshallableThrowable.java:91)
at org.infin...@16.0.5//org.infinispan.marshall.protostream.impl.MarshallableThrowable.get(MarshallableThrowable.java:78)
at org.infin...@16.0.5//org.infinispan.marshall.protostream.impl.MarshallableThrowable.recreateGenericThrowable(MarshallableThrowable.java:85)
... 36 more 

[Paul Ferraro]

That's odd - is there a more descriptive stack trace on the other cluster member?

[Viktoriia Teliuk]

"transfer-queue" bundler works! I missed changing module from  "org.wildfly.clustering.server" to "org.wildfly.clustering.singleton.server"   Sorry

Tried one more time the default bundler and default -Dorg.infinispan.threads.virtual, still huge CPU Utilization

Thanks for creating https://redhat.atlassian.net/browse/WFLY-21671

[Viktoriia Teliuk]

Hi Paul Ferraro,

We noticed, that with switching to bundler_type="transfer-queue" you also replaced NAKACK4/UNICAST4 with NAKACK2/UNICAST3 in https://github.com/wildfly/wildfly/pull/19808/changes
Should we do the same in standalone for WF 39.0.1.Final? Thank you 🙏