WildFly26 "org.infinispan.container.entries.NullCacheEntry cannot be cast to org.infinispan.container.entries.MVCCEntry" that occurs during session rebalancing.
23 views
Skip to first unread message
onetop.han
Nov 21, 2024, 3:54:16 AM11/21/24
to WildFly
dear wildfly team
The following error occurred during a session clustering test in WildFly 26.
11:59:38,422 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (thread-12,lms,server12) ISPN000136: Error executing command PutMapCommand on Cache 'session-sample.war', writing keys [SessionAccessMetaDataKey(FUJcdLYSn31n0NdZ4zQc0_ZOKFbmQr8EveY9dzKB), SessionCreationMetaDataKey(FUJcdLYSn31n0NdZ4zQc0_ZOKFbmQr8EveY9dzKB)]: java.lang.ClassCastException: org.infinispan.container.entries.NullCacheEntry cannot be cast to org.infinispan.container.entries.MVCCEntry
at org.infinispan.interceptors.impl.CallInterceptor.lookupMvccEntry(CallInterceptor.java:540)
at org.infinispan.interceptors.impl.CallInterceptor.visitPutMapCommand(CallInterceptor.java:504)
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:74)
at org.infinispan.interceptors.impl.CallInterceptor.visitCommand(CallInterceptor.java:178)
at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:61)
at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:55)
at org.infinispan.interceptors.DDAsyncInterceptor.visitPutMapCommand(DDAsyncInterceptor.java:91)
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:74)
at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
at org.infinispan.interceptors.BaseAsyncInterceptor.lambda$new$0(BaseAsyncInterceptor.java:27)
at org.infinispan.interceptors.InvocationSuccessFunction.apply(InvocationSuccessFunction.java:25)
at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:124)
at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:87)
at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:33)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:67)
at org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:105)
at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:51)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1496)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1398)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:146)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1586)
at org.jgroups.JChannel.up(JChannel.java:780)
at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:135)
at org.jgroups.stack.Protocol.up(Protocol.java:309)
at org.jgroups.protocols.FORK.up(FORK.java:150)
at org.jgroups.protocols.RSVP.up(RSVP.java:164)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:174)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:347)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:347)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1051)
at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:774)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:755)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:408)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:595)
at org.jgroups.protocols.BARRIER.up(BARRIER.java:171)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132)
at org.jgroups.protocols.FD.up(FD.java:227)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:281)
at org.jgroups.protocols.Discovery.up(Discovery.java:300)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1404)
at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
at org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70)
at java.lang.Thread.run(Thread.java:750)
11:59:38,423 WARN [org.infinispan.CLUSTER] (thread-12,lms,server12) ISPN000071: Caught exception when handling command SingleRpcCommand{cacheName='session-sample.war', command=PutMapCommand{map={SessionAccessMetaDataKey(FUJcdLYSn31n0NdZ4zQc0_ZOKFbmQr8EveY9dzKB)=SimpleSessionAccessMetaData{since-creation=PT0Slast-access=PT0S}, SessionCreationMetaDataKey(FUJcdLYSn31n0NdZ4zQc0_ZOKFbmQr8EveY9dzKB)=SimpleSessionCreationMetaData{created=2024-11-08T02:59:38.181Z, max-inactive-interval=PT0S}}, flags=[IGNORE_RETURN_VALUES], metadata=EmbeddedExpirableMetadata{version=null, lifespan=-1, maxIdle=-1}, internalMetadata={}, isForwarded=true}}: java.lang.ClassCastException: org.infinispan.container.entries.NullCacheEntry cannot be cast to org.infinispan.container.entries.MVCCEntry
at org.infinispan.interceptors.impl.CallInterceptor.lookupMvccEntry(CallInterceptor.java:540)
at org.infinispan.interceptors.impl.CallInterceptor.visitPutMapCommand(CallInterceptor.java:504)
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:74)
at org.infinispan.interceptors.impl.CallInterceptor.visitCommand(CallInterceptor.java:178)
at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:61)
at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:55)
at org.infinispan.interceptors.DDAsyncInterceptor.visitPutMapCommand(DDAsyncInterceptor.java:91)
at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:74)
at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
at org.infinispan.interceptors.BaseAsyncInterceptor.lambda$new$0(BaseAsyncInterceptor.java:27)
at org.infinispan.interceptors.InvocationSuccessFunction.apply(InvocationSuccessFunction.java:25)
at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:124)
at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:87)
at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:33)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:67)
at org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:105)
at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:51)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1496)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1398)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:146)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1586)
at org.jgroups.JChannel.up(JChannel.java:780)
at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:135)
at org.jgroups.stack.Protocol.up(Protocol.java:309)
at org.jgroups.protocols.FORK.up(FORK.java:150)
at org.jgroups.protocols.RSVP.up(RSVP.java:164)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:174)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:347)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:347)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1051)
at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:774)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:755)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:408)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:595)
at org.jgroups.protocols.BARRIER.up(BARRIER.java:171)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132)
at org.jgroups.protocols.FD.up(FD.java:227)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:281)
at org.jgroups.protocols.Discovery.up(Discovery.java:300)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1404)
at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
at org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70)
at java.lang.Thread.run(Thread.java:750)
(session-sample.war is just a sample page with a <distributable/> tag.)
Here's how to cause this error:
1. Connect Apache to two Wildfly 26. (http proxy, mod_balancer)
2. Set up clustering of 2 Wildfly units with tcpping.
3. Start only one Wildfly.
4. I maintain about 500 tps (maximum load on my test server) for 30 seconds through Jmeter.
5. Start the remaining wildfly to cause clustering rebalancing.
6. Error occurs during rebalancing.
The settings in my standalone-ha.xml file are as follows:
<cache-container name="web" default-cache="dist" modules="org.wildfly.clustering.web.infinispan">
<transport lock-timeout="60000"/>
<replicated-cache name="sso">
<locking isolation="READ_COMMITTED"/>
<transaction mode="NONE"/>
<expiration interval="0"/>
</replicated-cache>
<replicated-cache name="routing">
<expiration interval="0"/>
</replicated-cache>
<replicated-cache name="repl" remote-timeout="60000">
<locking isolation="READ_COMMITTED"/>
<transaction locking="OPTIMISTIC" mode="NONE"/>
<state-transfer timeout="0"/>
</replicated-cache>
<distributed-cache name="dist">
<locking isolation="READ_COMMITTED"/>
<transaction locking="OPTIMISTIC" mode="NONE"/>
<expiration interval="0"/>
<file-store/>
<state-transfer timeout="0"/>
</distributed-cache>
</cache-container>
<transport lock-timeout="60000"/>
<replicated-cache name="sso">
<locking isolation="READ_COMMITTED"/>
<transaction mode="NONE"/>
<expiration interval="0"/>
</replicated-cache>
<replicated-cache name="routing">
<expiration interval="0"/>
</replicated-cache>
<replicated-cache name="repl" remote-timeout="60000">
<locking isolation="READ_COMMITTED"/>
<transaction locking="OPTIMISTIC" mode="NONE"/>
<state-transfer timeout="0"/>
</replicated-cache>
<distributed-cache name="dist">
<locking isolation="READ_COMMITTED"/>
<transaction locking="OPTIMISTIC" mode="NONE"/>
<expiration interval="0"/>
<file-store/>
<state-transfer timeout="0"/>
</distributed-cache>
</cache-container>
.........
<stack name="tcpping">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<protocol type="org.jgroups.protocols.TCPPING">
<property name="initial_hosts">172.21.134.57[8600],172.21.134.57[8700]</property>
<property name="port_range">0</property>
</protocol>
<protocol type="MERGE3"/>
<socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="BARRIER"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
<protocol type="RSVP"/>
</stack>
<transport type="TCP" socket-binding="jgroups-tcp"/>
<protocol type="org.jgroups.protocols.TCPPING">
<property name="initial_hosts">172.21.134.57[8600],172.21.134.57[8700]</property>
<property name="port_range">0</property>
</protocol>
<protocol type="MERGE3"/>
<socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="BARRIER"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
<protocol type="RSVP"/>
</stack>
........
This error only occurs in heavy load situations.
What I confirmed is that it doesn't happen in infinispan 12 version.
As a result of the same test on jboss 7.4 and wildfly 23, no errors occurred.
Is there a cause for this error and a way to prevent it?
As a result of the same test on jboss 7.4 and wildfly 23, no errors occurred.
Is there a cause for this error and a way to prevent it?
onetop.han
Nov 21, 2024, 3:56:10 AM11/21/24
to WildFly
Re-upload stacktrace.
2024년 11월 21일 목요일 오후 5시 54분 16초 UTC+9에 onetop.han님이 작성:
Reply all
Reply to author
Forward
0 new messages