MDB in WebSphere cluster , Enlist error

[bodnar13]

Hi, I'd like to know how to run/install properly an MDB in application server cluster. I have two nodes. (WebSphere 5.1 /MQ ) It looks like everything is OK when the system is used, but after a long idle period I always got a Timeout and later an Enlist exception on both nodes at the same time. Later the listener restarts and the message is processed, but the open input count (MQ) is increased. Any help appreciated, Thanks in advance.

[adw...@pulpjava.com]

Well, I'm not sure if this will totally hit your need, but it might
confirm what you are doing.

I get alot of people asking me for help setting up messaging, and
Message Driven Beans with WebSphere and IRAD. I put this multimedia
tutorial, along with some MDB code, to demonstrate how to set up a
service bus, message queue/topic, and ConnectionFactory using IRAD and
WebSphere. Here it is:

http://www.technicalfacilitation.com/examscam/tf/get.php?link=14creatingJMSMDBs

This might help a bit.

Good luck!

-Cameron McKenzie www.pulpjava.com
Author of What is WebSphere?

Free WebSphere Tutorials and Code: www.mcnz.com
Free Mock Certification Exams: www.examscam.com
Free SCJA Java Certification Exam Simulators: www.scjp.com

www.technicalfacilitation.com www.cameronmckenzie.com

[David Currie]

Can you post the exception?

Regards,
David

[bodnar13]

Hi, Thank you for your help.
The exceptions:
[10/31/06 12:17:11:764 GMT] 59b75634 TimeoutManage I WTRN0006W: Transaction 57415344:000000000000002b00000001dbe65e5f991fe681ace8a0d98c452af9a4491c8541564f4e434152445f4153[] has timed out after 120 seconds.
[10/31/06 12:21:13:354 GMT] 31251637 XATransaction E J2CA0027E: An exception occurred while invoking start on an XA Resource Adapter from dataSource JMS$CBCCard_QCF_RQST_PYMNT_CZ$JMSManagedConnection@2126681641, within transaction ID {XID: formatId(57415344), gtrid_length(43), bqual_length(28), data(000000000000002b00000001dbe65e5f991fe681ace8a0d98c452af9a4491c8541564f4e434152445f
at com.ibm.mq.MQXAResource.start(MQXAResource.java:443)
at com.ibm.ejs.jms.JMSManagedSession$JMSXAResource.start(JMSManagedSession.java:1234)
at com.ibm.ejs.j2c.XATransactionWrapper.start(XATransactionWrapper.java:1288)
at com.ibm.ws.Transaction.JTA.JTAResourceBase.start(JTAResourceBase.java:164)
at com.ibm.ws.Transaction.JTA.RegisteredResources.startRes(RegisteredResources.java:391)
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:2001)
at com.ibm.ws.Transaction.JTA.TranManagerSet.enlist(TranManagerSet.java:491)
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:613)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:754)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:990)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:833)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:511)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:494)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:58)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:375)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:936)
.
[10/31/06 12:21:13:642 GMT] 31251637 RegisteredRes E WTRN0078E: An attempt by the transaction manager to call start on a transactional resource has resulted in an error. The error code was XAER_RMERR. The exception stack trace follows: javax.transaction.xa.XAException: XA operation failed, see errorCode
at com.ibm.mq.MQXAResource.start(MQXAResource.java:443)
at com.ibm.ejs.jms.JMSManagedSession$JMSXAResource.start(JMSManagedSession.java:1234)
at com.ibm.ejs.j2c.XATransactionWrapper.start(XATransactionWrapper.java:1288)
at com.ibm.ws.Transaction.JTA.JTAResourceBase.start(JTAResourceBase.java:164)
at com.ibm.ws.Transaction.JTA.RegisteredResources.startRes(RegisteredResources.java:391)
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:2001)
at com.ibm.ws.Transaction.JTA.TranManagerSet.enlist(TranManagerSet.java:491)
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:613)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:754)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:990)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:833)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:511)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:494)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:58)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:375)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:936)

[10/31/06 12:21:14:330 GMT] 31251637 XATransaction E J2CA0030E: Method enlist caught javax.transaction.SystemException: Failed to start the transaction association.
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:2015)
at com.ibm.ws.Transaction.JTA.TranManagerSet.enlist(TranManagerSet.java:491)
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:613)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:754)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:990)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:833)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:511)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:494)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:58)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:375)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:936)
Caused by: javax.transaction.xa.XAException: XA operation failed, see errorCode
at com.ibm.mq.MQXAResource.start(MQXAResource.java:443)
at com.ibm.ejs.jms.JMSManagedSession$JMSXAResource.start(JMSManagedSession.java:1234)
at com.ibm.ejs.j2c.XATransactionWrapper.start(XATransactionWrapper.java:1288)
at com.ibm.ws.Transaction.JTA.JTAResourceBase.start(JTAResourceBase.java:164)
at com.ibm.ws.Transaction.JTA.RegisteredResources.startRes(RegisteredResources.java:391)
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:2001)
... 18 more
while trying to enlist resources from datasource JMS$CBCCard_QCF_RQST_PYMNT_CZ$JMSManagedConnection@2126681641 with the Transaction Manager for the current transaction, and threw a ResourceException.
[10/31/06 12:21:14:865 GMT] 31251637 ConnectionEve A J2CA0056I: The Connection Manager received a fatal connection error from the Resource Adaptor for resource JMS$CBCCard_QCF_RQST_PYMNT_CZ$JMSManagedConnection@2126681641. The exception which was received is javax.jms.JMSException: Enlist failed
[10/31/06 12:21:14:869 GMT] 31251637 ConnectionEve A J2CA0056I: The Connection Manager received a fatal connection error from the Resource Adaptor for resource jms/CBCCard_QCF_RQST_PYMNT_CZ. The exception which was received is javax.jms.JMSException: Enlist failed
[10/31/06 12:21:14:873 GMT] 31251637 JMSExceptionL E WMSG0018E: Error on JMSConnection for MDB PaymentListenerCZ , JMSDestination jms/CBCCard_DST_RQST_PYMNT_CZ : javax.jms.JMSException: Enlist failed
at com.ibm.ejs.jms.JMSCMUtils.mapToJMSException(JMSCMUtils.java:107)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:851)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:511)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:494)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:58)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:375)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:936)
---- Begin backtrace for Nested Throwables
javax.resource.ResourceException: enlist: caught Exception
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:714)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:754)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:990)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:833)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:511)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:494)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:58)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:375)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:936)

[David Currie]

Hmm - not sure what's causing the XA_RMERR but I'd try and address the
transaction timeout first anyway. Is there anything in the application
that is blocking and could be causing the transaction to time out?

David

[bodnar13]

Unfortunately I don't know how to determine which component throws the Timeout Exception.
I changed the server transaction timeout from the default 120 to a 500 sec.It did not help.
I think the timeout also relates to the MQ connection which was closed by a firewall. I tried to configure the connection pool in WebSphere so that the connection would be discarded by the Server not the Firewall, but did not help. I realized that the MQ server still shows old /closed connection on the status window. I don't have any idea what to do next, maybe an MQ server side diagnostics, etc. I spent more then a week on this issue....

[David Currie]

The connection pooling won't help as the message listener service holds
on to a connection from the pool. You need to resolve the issue with the
firewall closing the connection.

David

[gregg_...@statestreet.com]

Hi,

This sounds like a bug -- there may be a good reason for a firewall, and if MQ isn't doing the right keep alives, then its reasonable for it to be disconnected. Even if it is doing keep alives, the firewall policy might disconnect it or the connection might be lost simply due to a failure of some type -- regardless of the reason, if the connection gets closed, I'd expect WebSphere to throw away the connection.

[chauhan_vin1]

Has somebody got solution to this ?
Any IBMer ?? , who can solve this I am tired of debuggin this now , IBM PMR suggested me to disable the MQ disconnect setting on the MQ Server connection channel but it hasnt helped still.

[taolu]

See this:
http://publib.boulder.ibm.com/infocenter/tivihelp/v3r1/index.jsp?topic=/com.ibm.itcamrtt.doc/itcamftt_installcluster20.htm

I got the same issue, solved by above approach.

Cheers,
Lann

[chauhan_vin1]

Lann,

Its not something to do with Websphere unclean shutdown , which could cause hung connection between the app and database, its with the MDB and MQ .

Please let me know if you have other solution to it

Thanks,
Vineet