Cross-posting from the Tx Newsgroup: XA Transaction spanning WLS 6.1 and 8.1 and involving MDB causing Timeout during commit processing

[Eric Ma]

I have a JMS Topic on WLS 6.1 SP4 on Solaris 8. The Topic is configured to retry
message delivery at 6 minutes interval for 5 times before an undelivered message
ends up in an error queue. Also, it uses a file based store to hold messages.
The JTA timeout setting for the WLS domain is 120 sec. A ConnectionFactory is
created that supports XA transaction, with a timeout of also 120 sec.

I have a MDB that is a durable subscriber to the above JMS topic. The specifics
are:

WLS 8.1 SP1 on WinNT 4
JTA of the WLS domain has timeout setting of 120 sec
Timeout for this MDB is set at 120 sec in weblogic-ejb-jar.xml
MDB used the XA-enabled connectionfactory in WLS 6.1 server
TX attribute of the MDB is Required and the TX is CMT (actually BMT behaves exactly
the same)

My problem is:

the onMessage method on my MDB exited normally, and the message seems to be acknowledged
(no redelivery was ever attempted). However, on the WLS 6.1 console, for the durable
subscriber (in this case my MDB), I saw the Messages Pending Count kept going
up. Also, on my WLS 8.1 command console, I got the following stack trace:

<Aug 23, 2003 9:08:49 PM EDT> <Error> <EJB> <BEA-010026> <Exception occurred during
commit of transaction Xid=BEA1-00002 727E3F0E9626320(26904898),Status=Unknown,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds
since begin=120,seconds left= 0,XAServerResourceInfo[JMS_JMSStore]=(ServerResourceInfo[JMS_JMSStore]=(state=new,assigned=none),xar=null)
,SCInfo[WLS81DOMAIN+wls81Server]=(state=active),SCInfo[WLS61DOMAIN+wls61Server]=(state=active),properties=({}),Owner
TransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=wls81Server+WLS81Host:7001+WLS81DOMAIN+t3+,
XAResource s={},NonXAResources={})],CoordinatorURL=WLS61DOMAIN+WLS61HOST:7001+WLS61DOMAIN+t3+):
javax.transaction.SystemException: Timeout during commit processing

Start server side stack trace: javax.transaction.SystemException: Timeout during
commit processing at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:243)
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:189)
at weblogic.transaction.internal.CoordinatorImpl.commit(CoordinatorImpl.java:68)
at weblogic.transaction.internal.CoordinatorImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:305) at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:274)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:22)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
End server side stack trace

at weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:108)
at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:138) at weblogic.transaction.internal.Coordinator2_WLStub.commit(Unknown
Source) at weblogic.transaction.internal.TransactionImpl.commit(TransactionImpl.java:308)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:233)
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:221)
at weblogic.ejb20.internal.MDListener.execute(MDListener.java:412) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170) Caused by: javax.transaction.SystemException:
Timeout during commit processing

Start server side stack trace: javax.transaction.SystemException: Timeout during
commit processing at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:243)
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:189)
at weblogic.transaction.internal.CoordinatorImpl.commit(CoordinatorImpl.java:68)
at weblogic.transaction.internal.CoordinatorImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:305) at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:274)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:22)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
End server side stack trace

The stack trace seems to suggested that the XA TX coordinator on the WLS 6.1 domain
tried to enlist the file-based JMS message store operation in the TX. Is this
guess correct? Why did it try to do that? Also, if the XA TX failed and the TX
was rolled back, why was the message acknowledged and not redelivered?

Any help is greatly appreciated.

Eric Ma

[Tom Barnes]

Hi Eric,

I recommend contacting customer support. Note that
SP4 in particular has a known problem with non-transactional
messages not getting acknowledged (pending count keeps going up) -
so make sure that the 6.1 MDB is transactional (the JMS FAQ
contains instrumentation code you can put in your MDB to test for this).

Tom, BEA

[Eric Ma]

Tom:

Thanks for your reply. I will open a case with BEA Support.

In the meantime, I was playing around a little bit trying to get around the Tx
problem. In my WLS 8.1 SP1 domain I created a JMS bridge, with the WLS 6.1 SP4
Topic as the source and another WLS 8.1 SP1 Topic as the target destination.
My WLS 8.1 SP1 MDB now subscribes to this new Topic instead of the WLS 6.1 SP4
Topic. When I published a message to the source destination, I got the following
in my WLS 8.1 SP1 command console:

<Aug 25, 2003 4:50:59 PM EDT> <Error> <MessagingBridge> <BEA-200015> <An error
occurred in bridge "MyJM
SBridge" during the transfer of messages (weblogic.transaction.RollbackException:
SubCoordinator 'WLS81ServerName+WLS81ServerIPAddreess:Port+WLS81DomainName+t3+'
not available

Start server side stack trace:

javax.transaction.SystemException: SubCoordinator 'WLS81ServerName+WLS81ServerIPAddreess:Port+WLS81DomainName+t3+'
not available
at weblogic.transaction.internal.TransactionImpl.abort(TransactionImpl.java:933)
at weblogic.transaction.internal.ServerSCInfo.startPrepare(ServerSCInfo.java:200)
at weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTransactionImpl.java:1604)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:217)

at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:189)
at weblogic.transaction.internal.CoordinatorImpl.commit(CoordinatorImpl.java:68)
at weblogic.transaction.internal.CoordinatorImpl_WLSkel.invoke(Unknown
Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:305)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:274)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:22)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)

--------------- nested within: ------------------
weblogic.transaction.RollbackException: SubCoordinator 'WLS81ServerName+WLS81ServerIPAddreess:Port+WLS81DomainName+t3+'
not available - with nest
ed exception:
[javax.transaction.SystemException: SubCoordinator 'napServer+165.250.128.76:2101+NAP+t3+'
not available]
at weblogic.transaction.internal.TransactionImpl.throwRollbackException(TransactionImpl.java:1490)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:265)

at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:189)
at weblogic.transaction.internal.CoordinatorImpl.commit(CoordinatorImpl.java:68)
at weblogic.transaction.internal.CoordinatorImpl_WLSkel.invoke(Unknown
Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:305)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:274)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:22)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
End server side stack trace

- with nested exception:
[javax.transaction.SystemException: SubCoordinator 'WLS81ServerName+WLS81ServerIPAddreess:Port+WLS81DomainName+t3+'
not available

Start server side stack trace:

javax.transaction.SystemException: SubCoordinator 'WLS81ServerName+WLS81ServerIPAddreess:Port+WLS81DomainName+t3+'
not available
at weblogic.transaction.internal.TransactionImpl.abort(TransactionImpl.java:933)
at weblogic.transaction.internal.ServerSCInfo.startPrepare(ServerSCInfo.java:200)
at weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTransactionImpl.java:1604)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:217)

at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:189)
at weblogic.transaction.internal.CoordinatorImpl.commit(CoordinatorImpl.java:68)
at weblogic.transaction.internal.CoordinatorImpl_WLSkel.invoke(Unknown
Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:305)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:274)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:22)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
End server side stack trace

What does this mean?

Eric

[Tom Barnes]

Hi Eric,

I do not know what the error message means - All I can suggest
is to post the tx stack-trace to the tx newsgroup, to check
your logs for other messages, and to take a look at the bridge
FAQ at http://e-docs.bea.com/wls/docs70/faq/msgbridge.html.

I wonder if the bridge and MDB transaction problems are related?
Seems likely.

Tom

[Eric Ma]

Tom:

Another quick/stupid question. With my JMS Message Bridge setup, I checked the
JTA status in WebLogic Console and found there are multiple rolled-back Tx's with
the file store as the Tx resource. Is this correct? How can a file I/O operation
be part of a XA transaction? Who is the Resource Manager, the underlying OS?

Eric

Tom Barnes <ple...@replyinnewsgroup.com> wrote:
>Hi Eric,
>

[Tom Barnes]

Hi Eric,

Each WebLogic JMS server is its own RM (Tx resource).
Transactions are supported equally reliably for both
the file store and the JDBC store. The tx resource is
given the same name as the store to help guarantee
that the tx resource is uniquely named.

In fact, arguably, the file store can be
more reliable, as it doesn't depend on a remote database
server being up or on a network connection.

I suggest you read the JMS Performance Guide or the JMS FAQ
which both contain sections comparing file stores to JDBC
stores.

Tom

[Tom Barnes]

Hi Eric,

There is a known issue that may be related
to 6.1SP4 to 8.1 JTA interop. It is fixed
in 6.1SP5. (Customer support should be
able to track down the particular case, if not,
give them my name.)

Tom, BEA

[Eric Ma]

Hi, Tom:

Thanks for the info. I also came across this while creating a support case earlier
today. I will try to upgrade to WLS 6.1 SP5 and let the group now about it.

Eric

[Eric Ma]

Yes, problem solved. Thanks as always for your help Tom.

Eric