Groups keyboard shortcuts have been updated
Dismiss
See shortcuts
Unexpected result during load test
158 views
Skip to first unread message
Cem Dayanik
Jan 21, 2025, 6:07:03 AMJan 21
to narayana-users
Hi,
We are working on REST-AT solution, while everything seems to work in simple manuel testing.
We have encountered some problems while doing some load testing.
We have tested the scenario with Jboss EAP 7.4.0 (Narayana-5.11.3), WildFly 26 (5.13.1) and Wildfly 35(Narayana-7.1.0)
Before investivating the issue further, we want to make sure we are not doing anything wrong with test scenario and the api usage.
We have a few quick questions;
1)TxSupport class seems deprecated, is there an alternative?
2)Is SRA annonation absolutely necessary? Or using @Transactional (jakarta) is fine on participants' end?
As far as I understand this should be fine from the docs but not sure since there are some problems.
Trace logs shows some potentials problems; (participant end, tracing only done with EAP version)
suggesting that some threadlocals are not cleared. (tracelog-1)
And also it suggests that once a thread is "corrupted", it does not recover and causes others tasks failed in the manner of transaction integrity.
When we trace this particular thread, there are some other exceptions, honestly it is hard to guess which one is the root cause where some internals/threadlocals are not cleared.
So before diving too deep, we wanted to make sure we are not doing something wrong on our part.
tracelog-1:
2025-01-15 14:49:37,766 TRACE [org.jboss.narayana.rest.integration.ParticipantResource] (default task-150) PUT request on ParticipantResource. ParticipantId: 0:ffff0ad79245:60fc5db7:67879a84:43530, content: txstatus=TransactionCommitted
2025-01-15 14:49:37,784 TRACE [org.jboss.narayana.rest.bridge.inbound.InboundBridgeParticipant] (default task-150) InboundBridgeParticipant.commit: xid=< 131081, 29, 64, 0000000000-1-110-41-1106996-493-73103-121-102-12404534449, 0000000000000000000000000000000000000000000000000000000000000000 >
2025-01-15 14:49:37,784 TRACE [org.jboss.narayana.rest.bridge.inbound.InboundBridgeManager] (default task-150) InboundBridgeManager.getInboundBridge: xid=< 131081, 29, 64, 0000000000-1-110-41-1106996-493-73103-121-102-12404534449, 0000000000000000000000000000000000000000000000000000000000000000 >
2025-01-15 14:49:37,784 TRACE [org.jboss.narayana.rest.bridge.inbound.InboundBridge] (default task-150) InboundBridge.start <InboundBridge: xid=< 131081, 29, 64, 0000000000-1-110-41-1106996-493-73103-121-102-12404534449, 0000000000000000000000000000000000000000000000000000000000000000 >, enlistmentUrl=http://localhost:8080/rest-at-coordinator/tx/transaction-manager/0_ffff0ad79245_65dfa452_67879a97_16735>
2025-01-15 14:49:37,784 TRACE [com.arjuna.ats.jta] (default task-150) TransactionImple.getStatus: javax.transaction.Status.STATUS_UNKNOWN
2025-01-15 14:49:37,784 TRACE [com.arjuna.ats.jta] (default task-150) TransactionImpleManager.resume
2025-01-15 14:49:37,784 TRACE [com.arjuna.ats.jta] (default task-150) TransactionImple.getStatus: javax.transaction.Status.STATUS_MARKED_ROLLBACK
2025-01-15 14:49:37,784 WARN [org.jboss.jbossts.star.logging] (default task-150) AT027003: Failed to start the bridge. 'BaseTransaction.checkTransactionState - ARJUNA016051: thread is already associated with a transaction!': java.lang.IllegalStateException: BaseTransaction.checkTransactionState - ARJUNA016051: thread is already associated with a transaction!
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.checkTransactionState(BaseTransaction.java:266)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionManagerImple.resume(TransactionManagerImple.java:96)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.resume(BaseTransactionManagerDelegate.java:127)
at org.jboss.n...@5.11.3.Final-redhat-00001//org.jboss.narayana.rest.bridge.inbound.InboundBridge.start(InboundBridge.java:106)
at org.jboss.n...@5.11.3.Final-redhat-00001//org.jboss.narayana.rest.bridge.inbound.InboundBridgeParticipant.startBridge(InboundBridgeParticipant.java:137)
at org.jboss.n...@5.11.3.Final-redhat-00001//org.jboss.narayana.rest.bridge.inbound.InboundBridgeParticipant.commit(InboundBridgeParticipant.java:85)
at org.jboss.n...@5.11.3.Final-redhat-00001//org.jboss.narayana.rest.integration.ParticipantResource.commit(ParticipantResource.java:213)
at org.jboss.n...@5.11.3.Final-redhat-00001//org.jboss.narayana.rest.integration.ParticipantResource.terminate(ParticipantResource.java:119)
at jdk.internal.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:138)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:546)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:435)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:396)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:398)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:365)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:338)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:440)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:245)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:61)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at javax.se...@2.0.0.Final-redhat-00001//javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:841)
at org.jbos...@2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jbos...@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jbos...@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jbos...@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at org.jbo...@3.8.4.Final-redhat-00001//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
at java.base/java.lang.Thread.run(Thread.java:833)
We are working on REST-AT solution, while everything seems to work in simple manuel testing.
We have encountered some problems while doing some load testing.
We have tested the scenario with Jboss EAP 7.4.0 (Narayana-5.11.3), WildFly 26 (5.13.1) and Wildfly 35(Narayana-7.1.0)
Before investivating the issue further, we want to make sure we are not doing anything wrong with test scenario and the api usage.
We have a few quick questions;
1)TxSupport class seems deprecated, is there an alternative?
2)Is SRA annonation absolutely necessary? Or using @Transactional (jakarta) is fine on participants' end?
As far as I understand this should be fine from the docs but not sure since there are some problems.
Trace logs shows some potentials problems; (participant end, tracing only done with EAP version)
suggesting that some threadlocals are not cleared. (tracelog-1)
And also it suggests that once a thread is "corrupted", it does not recover and causes others tasks failed in the manner of transaction integrity.
When we trace this particular thread, there are some other exceptions, honestly it is hard to guess which one is the root cause where some internals/threadlocals are not cleared.
So before diving too deep, we wanted to make sure we are not doing something wrong on our part.
tracelog-1:
2025-01-15 14:49:37,766 TRACE [org.jboss.narayana.rest.integration.ParticipantResource] (default task-150) PUT request on ParticipantResource. ParticipantId: 0:ffff0ad79245:60fc5db7:67879a84:43530, content: txstatus=TransactionCommitted
2025-01-15 14:49:37,784 TRACE [org.jboss.narayana.rest.bridge.inbound.InboundBridgeParticipant] (default task-150) InboundBridgeParticipant.commit: xid=< 131081, 29, 64, 0000000000-1-110-41-1106996-493-73103-121-102-12404534449, 0000000000000000000000000000000000000000000000000000000000000000 >
2025-01-15 14:49:37,784 TRACE [org.jboss.narayana.rest.bridge.inbound.InboundBridgeManager] (default task-150) InboundBridgeManager.getInboundBridge: xid=< 131081, 29, 64, 0000000000-1-110-41-1106996-493-73103-121-102-12404534449, 0000000000000000000000000000000000000000000000000000000000000000 >
2025-01-15 14:49:37,784 TRACE [org.jboss.narayana.rest.bridge.inbound.InboundBridge] (default task-150) InboundBridge.start <InboundBridge: xid=< 131081, 29, 64, 0000000000-1-110-41-1106996-493-73103-121-102-12404534449, 0000000000000000000000000000000000000000000000000000000000000000 >, enlistmentUrl=http://localhost:8080/rest-at-coordinator/tx/transaction-manager/0_ffff0ad79245_65dfa452_67879a97_16735>
2025-01-15 14:49:37,784 TRACE [com.arjuna.ats.jta] (default task-150) TransactionImple.getStatus: javax.transaction.Status.STATUS_UNKNOWN
2025-01-15 14:49:37,784 TRACE [com.arjuna.ats.jta] (default task-150) TransactionImpleManager.resume
2025-01-15 14:49:37,784 TRACE [com.arjuna.ats.jta] (default task-150) TransactionImple.getStatus: javax.transaction.Status.STATUS_MARKED_ROLLBACK
2025-01-15 14:49:37,784 WARN [org.jboss.jbossts.star.logging] (default task-150) AT027003: Failed to start the bridge. 'BaseTransaction.checkTransactionState - ARJUNA016051: thread is already associated with a transaction!': java.lang.IllegalStateException: BaseTransaction.checkTransactionState - ARJUNA016051: thread is already associated with a transaction!
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.checkTransactionState(BaseTransaction.java:266)
at org.jboss.jts//com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionManagerImple.resume(TransactionManagerImple.java:96)
at org.jboss.jts.integration//com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.resume(BaseTransactionManagerDelegate.java:127)
at org.jboss.n...@5.11.3.Final-redhat-00001//org.jboss.narayana.rest.bridge.inbound.InboundBridge.start(InboundBridge.java:106)
at org.jboss.n...@5.11.3.Final-redhat-00001//org.jboss.narayana.rest.bridge.inbound.InboundBridgeParticipant.startBridge(InboundBridgeParticipant.java:137)
at org.jboss.n...@5.11.3.Final-redhat-00001//org.jboss.narayana.rest.bridge.inbound.InboundBridgeParticipant.commit(InboundBridgeParticipant.java:85)
at org.jboss.n...@5.11.3.Final-redhat-00001//org.jboss.narayana.rest.integration.ParticipantResource.commit(ParticipantResource.java:213)
at org.jboss.n...@5.11.3.Final-redhat-00001//org.jboss.narayana.rest.integration.ParticipantResource.terminate(ParticipantResource.java:119)
at jdk.internal.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:138)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:546)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:435)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:396)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:398)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:365)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:338)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:440)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:245)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:61)
at org.jboss.restea...@3.15.1.Final-redhat-00001//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at javax.se...@2.0.0.Final-redhat-00001//javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
at io.undert...@2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
at io.under...@2.2.5.Final-redhat-00001//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:841)
at org.jbos...@2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jbos...@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jbos...@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jbos...@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at org.jbo...@3.8.4.Final-redhat-00001//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
at java.base/java.lang.Thread.run(Thread.java:833)
Michael Musgrove
Jan 21, 2025, 2:59:30 PMJan 21
to Cem Dayanik, narayana-users
On Tue, Jan 21, 2025 at 11:10 AM 'Cem Dayanik' via narayana-users <narayan...@googlegroups.com> wrote:
Hi,
We are working on REST-AT solution, while everything seems to work in simple manuel testing.
We have encountered some problems while doing some load testing.
We have tested the scenario with Jboss EAP 7.4.0 (Narayana-5.11.3), WildFly 26 (5.13.1) and Wildfly 35(Narayana-7.1.0)
Hi, so I assume that you are seeing a similar issue to the reported stack trace on each of these versions when running under load?
Before investivating the issue further, we want to make sure we are not doing anything wrong with test scenario and the api usage.
We have a few quick questions;
1)TxSupport class seems deprecated, is there an alternative?
The REST-AT specification only specifies how to participate in the protocol using HTTP REST calls and it does not define programming APIs. TxSupport is marked as deprecated because it is for internal use, for example we use it extensively in the testsuite - that said we expect it to work and would be interested in any bug reports. The alternative is any library that supports HTTP, for example if your service is written in Java and it uses a JAX-RS library then as long as you follow the spec (https://github.com/jbosstm/documentation/blob/main/rts/docs/REST-Atomic_v2_draft_8_comments_sept_4.pdf) then you can participate in the protocol.
2)Is SRA annonation absolutely necessary? Or using @Transactional (jakarta) is fine on participants' end?
As far as I understand this should be fine from the docs but not sure since there are some problems.
The SRA annotations are experimental and are not required, just program your services using the REST-based protocol defined in the specification.
It's fine to use @Transactional on participants' end - in detail the specification defines how to perform Atomic transactions using HTTP and as such it can be used by services that support HTTP. There is a direct one to mapping between the REST-AT the JTA transaction models and to improve ease of use we support an "inbound bridge" which can bridge between the two models - the bridge is inbound in the sense that if you start a REST-AT transaction and then call a JAX-RS service annotated with @Transactional, and the bridge is enabled, then a JTA transaction will be associated with the thread used to run the method and the usual JTA behaviour can be relied upon by the service method. There is no outbound bridge which means that if you start a JTA transaction and then start a REST-AT transaction then the two "transactions" are independent of each other.
Trace logs shows some potentials problems; (participant end, tracing only done with EAP version)
suggesting that some threadlocals are not cleared. (tracelog-1)
There is definitely an issue here, it looks like the thread used to handle the JAX-RS call (the resteasy thread) still has a transaction associated with it. I'm not sure which bit of system code manages these threads, the problem may or may not be with REST-AT and I would not like to commit to an opinion without further investigation, maybe someone else can suggest some explanation, so we'd need a reproducer to make progress, perhaps one of our own tests can be adapted to run under load, not sure.
Thanks for asking the questions, I expect that we will be able to make progress.
Michael Musgrove
Jan 21, 2025, 3:03:40 PMJan 21
to Cem Dayanik, narayana-users
A link to the model and spec is https://www.narayana.io//docs/project/index.html#_overview_3
Cem Dayanik
Jan 22, 2025, 12:25:48 PMJan 22
to narayana-users
Hi Michael,
Thanks for quick and concise answers.
We have investigated the issue.
Any exception on participant end, seems to cause the problem.
LocalTransaction is not cleared, but it is in ABORTED state.
In the next request, same trx -with same hashcode- just sitting there.
This causes any request to same thread fails.
Since 2PC protocols are executed in same worker pool, they are also affected.
This participant was the second one in trx, I assume 2PC protocols are executed in order.
Commit request gets the same exception, so the transaction integrity breaks. (this one is confirmed by trace logs, not in debug, see earlier attached logs pls, well.. they were for 5.11.3, this debugging session is done with 7.1.0 )
I havent really debugged the non-rest version but I just assume same implemention -classes- are used except maybe InboundBridgeFilter etc.
My best guess would there is no exactly an interceptor for top level trx here, it is just a tricky insert done by InboundBridgeFilter, but it should be really taken care of in this case.
Apart from this issue, I have seen this participant was doing a rest call to itself.
It really didnt make sense to me why rest/http is prefered over a method call in this case. (RESTRecord.topLevelAbort)
But I kinda overlooked it :) since it probably didnt have anything to do with this case.
I hope this helps.
Thanks for quick and concise answers.
We have investigated the issue.
Any exception on participant end, seems to cause the problem.
LocalTransaction is not cleared, but it is in ABORTED state.
In the next request, same trx -with same hashcode- just sitting there.
This causes any request to same thread fails.
Since 2PC protocols are executed in same worker pool, they are also affected.
This participant was the second one in trx, I assume 2PC protocols are executed in order.
Commit request gets the same exception, so the transaction integrity breaks. (this one is confirmed by trace logs, not in debug, see earlier attached logs pls, well.. they were for 5.11.3, this debugging session is done with 7.1.0 )
I havent really debugged the non-rest version but I just assume same implemention -classes- are used except maybe InboundBridgeFilter etc.
My best guess would there is no exactly an interceptor for top level trx here, it is just a tricky insert done by InboundBridgeFilter, but it should be really taken care of in this case.
Apart from this issue, I have seen this participant was doing a rest call to itself.
It really didnt make sense to me why rest/http is prefered over a method call in this case. (RESTRecord.topLevelAbort)
But I kinda overlooked it :) since it probably didnt have anything to do with this case.
I hope this helps.
Michael Musgrove
Jan 23, 2025, 8:22:10 AMJan 23
to narayana-users
Would it be possible for you share a small project that shows exactly how you are generating these errors?
Cem Dayanik
Jan 28, 2025, 7:33:58 AMJan 28
to narayana-users
Hi Michael,
Sorry, I am a little late, it is never easy to get some code out of a bank;)
I removed the irrevelant stuff and changed a few bits in order to make the setup easier.
Then it got interested, the issue was not there anymore.
Participants need to run on their own instances (JBoss/Wildfly), if you redirect them to same instance (i did this earlier for easy setup- although the protocol is still rest, the issue does not happen.
Apparently they join the same transaction as an optimization.
Paritipants run on 8070 and 8090. (ALL ports are shifted except smtp, project is deployed to all instances)
Main request is sent to 8080.
I added a "single" case, and an artificial exception for the second participant -to replicate SQLTimeoutException in our case-
You dont need to replicate those all sorts of errors, since they are probably just a side effect.
Please see the screenshot for the second participant. After the exception thrown -artifical one, always thrown-,
there is nothing on call stack left after this point, which can clear the threal local and its still there.
P.S. In multi thread version, rest client stalls once a while, this wasnt the case in javax version.
I think pooling has changed -probably-, I did saw ForkJoinPool in the call stack where the stall happens, tried with custom builder with an explicit pool, nothing has changed (hanging and ForkJoinPool was still there), so reverted it back;
AFAIK there is a known problem about ForkJoinPool but no any proper investigation on our end about this issue.
Sorry, I am a little late, it is never easy to get some code out of a bank;)
I removed the irrevelant stuff and changed a few bits in order to make the setup easier.
Then it got interested, the issue was not there anymore.
Participants need to run on their own instances (JBoss/Wildfly), if you redirect them to same instance (i did this earlier for easy setup- although the protocol is still rest, the issue does not happen.
Apparently they join the same transaction as an optimization.
Paritipants run on 8070 and 8090. (ALL ports are shifted except smtp, project is deployed to all instances)
Main request is sent to 8080.
I added a "single" case, and an artificial exception for the second participant -to replicate SQLTimeoutException in our case-
You dont need to replicate those all sorts of errors, since they are probably just a side effect.
Please see the screenshot for the second participant. After the exception thrown -artifical one, always thrown-,
there is nothing on call stack left after this point, which can clear the threal local and its still there.
P.S. In multi thread version, rest client stalls once a while, this wasnt the case in javax version.
I think pooling has changed -probably-, I did saw ForkJoinPool in the call stack where the stall happens, tried with custom builder with an explicit pool, nothing has changed (hanging and ForkJoinPool was still there), so reverted it back;
AFAIK there is a known problem about ForkJoinPool but no any proper investigation on our end about this issue.
Michael Musgrove
Jan 29, 2025, 4:36:01 AMJan 29
to narayana-users
Thanks for that and very much appreciated, it makes it clear what your use/test case is.
Please give me a few days to get it working on our infrastructure and I will report back.
Please give me a few days to get it working on our infrastructure and I will report back.
Michael Musgrove
Feb 4, 2025, 9:53:10 AMFeb 4
to narayana-users
Hello again, I haven't forgotten about this, do you have an example of the debit and credit participants used by TaskResource:
private static final String PARTICIPANT_DEBIT_URL_CORE = "http://localhost:8070/NarayanaWildfly/" + DEBIT_PATH_SEGMENT;
private static final String PARTICIPANT_CREDIT_URL_LOG = "http://localhost:8090/NarayanaWildfly/" + CREDIT_PATH_SEGMENT;
private static final String PARTICIPANT_DEBIT_URL_CORE = "http://localhost:8070/NarayanaWildfly/" + DEBIT_PATH_SEGMENT;
private static final String PARTICIPANT_CREDIT_URL_LOG = "http://localhost:8090/NarayanaWildfly/" + CREDIT_PATH_SEGMENT;
Plus anything else I might need to run your reproducer - skimming the code I think I just need the two participants that you are using?
Thanks
Cem Dayanik
Feb 4, 2025, 1:26:05 PMFeb 4
to narayana-users
Hi Michael,
Sorry I wasnt clear enough. It is the same class: TaskResource.
Same application deployed on three instances, participants different datasources.
Main one running on 8080. Actual main request should be sent to this one.
("single" case enough to see the problem, multiple was the one we have discovered the issue though)
@POST
@Path(DEBIT_PATH_SEGMENT + "/{accountId}/{amount}")
@Transactional
public String decreaseAmount(@Context HttpServletRequest httpServletRequest,
@PathParam("accountId") String accountId, @PathParam("amount") String amount) {
debit(accountId, amount);
return "SUCCESS";
}
@POST
@Path(CREDIT_PATH_SEGMENT + "/{accountId}/{amount}")
@Transactional
public String credit(@Context HttpServletRequest httpServletRequest,
@PathParam("accountId") String accountId, @PathParam("amount") String amount) {
credit(accountId, amount);
return "SUCCESS";
}
@Path(DEBIT_PATH_SEGMENT + "/{accountId}/{amount}")
@Transactional
public String decreaseAmount(@Context HttpServletRequest httpServletRequest,
@PathParam("accountId") String accountId, @PathParam("amount") String amount) {
debit(accountId, amount);
return "SUCCESS";
}
@POST
@Path(CREDIT_PATH_SEGMENT + "/{accountId}/{amount}")
@Transactional
public String credit(@Context HttpServletRequest httpServletRequest,
@PathParam("accountId") String accountId, @PathParam("amount") String amount) {
credit(accountId, amount);
return "SUCCESS";
}
Cem Dayanik
Feb 4, 2025, 1:29:06 PMFeb 4
to narayana-users
Oh, I see the confusion, my bad sorry :)
@PathParam("accountId") String accountId, @PathParam("amount") String amount) {
debit(accountId, amount);
return "SUCCESS";
}
@POST
@Path(CREDIT_PATH_SEGMENT + "/{accountId}/{amount}")
@Transactional
public String credit(@Context HttpServletRequest httpServletRequest,
@PathParam("accountId") String accountId, @PathParam("amount") String amount) {
credit(accountId, amount);
return "SUCCESS";
}
I havent checked it after the refactoring.
@POST
@Path(DEBIT_PATH_SEGMENT + "/{accountId}/{amount}")
@Transactional
@Path(DEBIT_PATH_SEGMENT + "/{accountId}/{amount}")
@Transactional
public String debit(@Context HttpServletRequest httpServletRequest,
@PathParam("accountId") String accountId, @PathParam("amount") String amount) {
debit(accountId, amount);
return "SUCCESS";
}
@POST
@Path(CREDIT_PATH_SEGMENT + "/{accountId}/{amount}")
@Transactional
public String credit(@Context HttpServletRequest httpServletRequest,
@PathParam("accountId") String accountId, @PathParam("amount") String amount) {
credit(accountId, amount);
return "SUCCESS";
}
Michael Musgrove
Feb 5, 2025, 7:07:36 AMFeb 5
to narayana-users
Thanks,
May I also have your standalone.xml for the datasource definitions, the lookups (in methods getDataSourceCore/Log) fail with javax.naming.NameNotFoundException: jdbc/bsaCoreDS -- service jboss.naming.context.java.jdbc.bsaCoreDS etc.
May I also have your standalone.xml for the datasource definitions, the lookups (in methods getDataSourceCore/Log) fail with javax.naming.NameNotFoundException: jdbc/bsaCoreDS -- service jboss.naming.context.java.jdbc.bsaCoreDS etc.
Reply all
Reply to author
Forward
0 new messages