Hi guys,
Please help!!
We are running an XA-enabled Spring 3.2.5 based application on Wildfly 8.2, that uses Tibco EMS 8 for inbound and outbound JMS interactions.
Multiple EMS servers are involved.
JMS connection factories have been configured in standalone-full.xml as in jms-connection-config.xml (attached).
The configuration defines two sets of EMS connection factories - one set for our ESB EMS and one for our application's EMS.
Each set defines an XA topic connection factory for outgoing interactons (publishing) and an XA Queue connection factory for incoming interactions (receiving).
We use Spring DefualtMessageListenerContainer instances for XA-enabled message consumption.
The Spring listeners are defined as follows:
<bean id="abstractEventListenerContainer" abstract="true"
class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="transactionManager" ref="transactionManager" />
<property name="receiveTimeout" value="10000" />
<property name="sessionTransacted" value="true" />
</bean>
...
<bean id="fileNotificationContainer" parent="abstractEventListenerContainer">
<property name="destination" ref="bbgFileNotificationQueue" />
<property name="messageListener" ref="fileNotificationMessageListener" />
<property name="connectionFactory" ref="bbgConnectionFactory" />
<property name="maxConcurrentConsumers" value="10" />
</bean>
<bean id="bbgConnectionFactory" class="org.springframework.jms.connection.TransactionAwareConnectionFactoryProxy">
<property name="targetConnectionFactory">
<bean class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:/TibcoJmsXAbbg" />
</bean>
</property>
</bean>
...
<bean id="transactionManager" class="org.springframework.transaction.jta.JtaTransactionManager"/>
When the listeners are initialised, the following output (including stacktrace) is output in the logs:
2015-05-04 20:21:35,866 DEBUG [org.jboss.resource.adapter.jms.JmsManagedConnection] (fileNotificationContainer-1) using connection factory: XAQueueConnectionFactory[URL=tcp://localhost:7222;clientID=null;Properties={com.tibco.tibjms.multicast.enabled=true}]
2015-05-04 20:21:35,866 DEBUG [org.jboss.resource.adapter.jms.JmsManagedConnection] (fileNotificationContainer-1) using username/password: admin/-- not shown --
2015-05-04 20:21:35,871 DEBUG [org.jboss.resource.adapter.jms.JmsManagedConnection] (fileNotificationContainer-1) created XAConnection: XAQueueConnection[ClientId=null, Connected=tcp://localhost:7222, URL=tcp://localhost:7222]
2015-05-04 20:21:35,873 TRACE [org.jboss.resource.adapter.jms.JmsManagedConnection] (fileNotificationContainer-1) created connection: XAQueueConnection[ClientId=null, Connected=tcp://localhost:7222, URL=tcp://localhost:7222]
2015-05-04 20:21:35,873 TRACE [org.jboss.resource.adapter.jms.JmsManagedConnection] (fileNotificationContainer-1) Using a non-XA Connection. It will not be able to participate in a Global UOW
2015-05-04 20:21:35,875 DEBUG [org.jboss.resource.adapter.jms.JmsManagedConnection] (fileNotificationContainer-1) xaSession=null, Session=QueueSession[transacted=false, tx_started=false, ackMode=NO_ACKNOWLEDGE, listener=null]
2015-05-04 20:21:35,875 DEBUG [org.jboss.resource.adapter.jms.JmsManagedConnection] (fileNotificationContainer-1) transacted=false, ack=1
2015-05-04 20:21:35,875 TRACE [org.jboss.resource.adapter.jms.JmsManagedConnection] (fileNotificationContainer-1) created new managed connection: org.jboss.resource.adapter.jms.JmsManagedConnection@6632e8fd
2015-05-04 20:21:35,876 TRACE [org.jboss.resource.adapter.jms.JmsManagedConnection] (fileNotificationContainer-1) ConnectionEvent listener added: org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@77f0be6d[state=NORMAL managed connection=org.jboss.resource.adapter.jms.JmsManagedConnection@6632e8fd connection handles=0 lastUse=1430763695876 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolByCri@593f9b9d mcp=SemaphoreArrayListManagedConnectionPool@7b405103[pool=TibcoJmsXAbbg] xaResource=XAResourceWrapperImpl@5e66239f[xaResource=null pad=false overrideRmValue=null productName=JBoss Generic JMS JCA Resource Adapter productVersion=0.1 jndiName=java:/TibcoJmsXAbbg] txSync=null]
2015-05-04 20:21:35,884 ERROR [org.jboss.resource.adapter.jms.JmsSessionFactoryImpl] (requestListenerContainer-5) could not create session: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@7afd296e[state=NORMAL managed connection=org.jboss.resource.adapter.jms.JmsManagedConnection@47e878cb connection handles=0 lastUse=1430763695884 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolByCri@593f9b9d mcp=SemaphoreArrayListManagedConnectionPool@7b405103[pool=TibcoJmsXAbbg] xaResource=XAResourceWrapperImpl@2ab33d7[xaResource=null pad=false overrideRmValue=null productName=JBoss Generic JMS JCA Resource Adapter productVersion=0.1 jndiName=java:/TibcoJmsXAbbg] txSync=null]
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:780)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:518)
at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:362)
at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createSession(JmsSessionFactoryImpl.java:341)
at org.springframework.jms.connection.ConnectionFactoryUtils$1.createSession(ConnectionFactoryUtils.java:149) [spring-jms-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:300) [spring-jms-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.jms.connection.ConnectionFactoryUtils.getTransactionalSession(ConnectionFactoryUtils.java:138) [spring-jms-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.jms.connection.TransactionAwareConnectionFactoryProxy$TransactionAwareConnectionInvocationHandler.invoke(TransactionAwareConnectionFactoryProxy.java:231) [spring jms-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at com.sun.proxy.$Proxy116.createSession(Unknown Source)
at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196) [spring-jms-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.access$200(AbstractPollingMessageListenerContainer.java:77) [spring-jms-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer$MessageListenerContainerResourceFactory.createSession(AbstractPollingMessageListenerContainer.java:531) [spring-jms-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:300) [spring-jms-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:288) [spring-jms-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243) [spring-jms-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1103) [spring-jms-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1095) [spring-jms-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:992) [spring-jms-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on entering meta-aware object
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:551)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:775)
... 18 more
Caused by: javax.transaction.SystemException: IJ000356: Failed to enlist: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 0:ffff7f000101:26676f67:5547b8a4:10 status: ActionStatus.ABORT_ONLY >
at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.checkEnlisted(TxConnectionListener.java:848)
at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:383)
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:544)
... 19 more
2015-05-04 20:21:35,942 ERROR [stderr] (fileNotificationContainer-1) java.lang.NullPointerException
2015-05-04 20:21:35,943 ERROR [stderr] (fileNotificationContainer-1) at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.setTransactionTimeout(XAResourceWrapperImpl.java:204)
2015-05-04 20:21:35,943 ERROR [stderr] (fileNotificationContainer-1) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:612)
2015-05-04 20:21:35,943 ERROR [stderr] (fileNotificationContainer-1) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:398)
2015-05-04 20:21:35,943 ERROR [stderr] (fileNotificationContainer-1) at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.enlist(TxConnectionListener.java:883)
2015-05-04 20:21:35,944 ERROR [stderr] (fileNotificationContainer-1) at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:365)
2015-05-04 20:21:35,944 ERROR [stderr] (fileNotificationContainer-1) at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:544)
2015-05-04 20:21:35,944 ERROR [stderr] (fileNotificationContainer-1) at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:775)
2015-05-04 20:21:35,944 ERROR [stderr] (fileNotificationContainer-1) at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:518)
2015-05-04 20:21:35,945 ERROR [stderr] (fileNotificationContainer-1) at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:362)
2015-05-04 20:21:35,945 ERROR [stderr] (fileNotificationContainer-1) at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createSession(JmsSessionFactoryImpl.java:341)
2015-05-04 20:21:35,970 ERROR [stderr] (fileNotificationContainer-1) at org.springframework.jms.connection.ConnectionFactoryUtils$1.createSession(ConnectionFactoryUtils.java:149)
2015-05-04 20:21:35,970 ERROR [stderr] (fileNotificationContainer-1) at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:300)
2015-05-04 20:21:35,970 ERROR [stderr] (fileNotificationContainer-1) at org.springframework.jms.connection.ConnectionFactoryUtils.getTransactionalSession(ConnectionFactoryUtils.java:138)
2015-05-04 20:21:35,970 ERROR [stderr] (fileNotificationContainer-1) at org.springframework.jms.connection.TransactionAwareConnectionFactoryProxy$TransactionAwareConnectionInvocationHandler.invoke(TransactionAwareConnectionFactoryProxy.java:231)
2015-05-04 20:21:35,971 ERROR [stderr] (fileNotificationContainer-1) at com.sun.proxy.$Proxy116.createSession(Unknown Source)
2015-05-04 20:21:35,971 ERROR [stderr] (fileNotificationContainer-1) at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
2015-05-04 20:21:35,971 ERROR [stderr] (fileNotificationContainer-1) at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.access$200(AbstractPollingMessageListenerContainer.java:77)
2015-05-04 20:21:35,971 ERROR [stderr] (fileNotificationContainer-1) at org.springframework.jms.listener.AbstractPollingMessageListenerContainer$MessageListenerContainerResourceFactory.createSession(AbstractPollingMessageListenerContainer.java:531)
2015-05-04 20:21:35,971 ERROR [stderr] (fileNotificationContainer-1) at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:300)
2015-05-04 20:21:35,972 ERROR [stderr] (fileNotificationContainer-1) at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:288)
2015-05-04 20:21:35,972 ERROR [stderr] (fileNotificationContainer-1) at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
2015-05-04 20:21:35,972 ERROR [stderr] (fileNotificationContainer-1) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1103)
2015-05-04 20:21:35,972 ERROR [stderr] (fileNotificationContainer-1) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1095)
2015-05-04 20:21:35,972 ERROR [stderr] (fileNotificationContainer-1) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:992)
2015-05-04 20:21:35,973 ERROR [stderr] (fileNotificationContainer-1) at java.lang.Thread.run(Thread.java:745)
Looking at the RA's code, the decisions regarding XA sessions and resources vs not is made in JmsManagedConnection.setUp() :
/**
* Setup the connection.
*
* @throws ResourceException
*/
private void setup() throws ResourceException {
boolean trace = log.isTraceEnabled();
try {
Context context = JmsActivation.convertStringToContext(mcf.getJndiParameters());
Object factory;
boolean transacted = info.isTransacted();
int ack = transacted ? 0 : info.getAcknowledgeMode();
String connectionFactory = mcf.getConnectionFactory();
if (connectionFactory == null) {
throw new IllegalStateException("No configured 'connectionFactory'.");
}
ClassLoader oldTCCL = SecurityActions.getThreadContextClassLoader();
try {
SecurityActions.setThreadContextClassLoader(JmsManagedConnection.class.getClassLoader());
factory = context.lookup(connectionFactory);
} finally {
SecurityActions.setThreadContextClassLoader(oldTCCL);
}
con = createConnection(factory, user, pwd);
if (info.getClientID() != null && !info.getClientID().equals(con.getClientID())) {
con.setClientID(info.getClientID());
}
con.setExceptionListener(this);
if (trace) {
log.trace("created connection: " + con);
}
if (con instanceof XAConnection && transacted) {
if (mcf.getProperties().getType() == JmsConnectionFactory.QUEUE) {
xaSession = ((XAQueueConnection) con).createXAQueueSession();
session = ((XAQueueSession)xaSession).getQueueSession();
} else if (mcf.getProperties().getType() == JmsConnectionFactory.TOPIC) {
xaSession = ((XATopicConnection) con).createXATopicSession();
session = ((XATopicSession)xaSession).getTopicSession();
} else {
xaSession = ((XAConnection) con).createXASession();
session = xaSession.getSession();
}
xaTransacted = true;
} else {
if (mcf.getProperties().getType() == JmsConnectionFactory.QUEUE) {
session = ((QueueConnection)con).createQueueSession(transacted, ack);
} else if (mcf.getProperties().getType() == JmsConnectionFactory.TOPIC) {
session = ((TopicConnection)con).createTopicSession(transacted, ack);
} else {
session = con.createSession(transacted, ack);
}
if (trace) {
log.trace("Using a non-XA Connection. " +
"It will not be able to participate in a Global UOW");
}
}
log.debug("xaSession=" + xaSession + ", Session=" + session);
log.debug("transacted=" + transacted + ", ack=" + ack);
} catch (NamingException e) {
throw new ResourceException("Unable to setup connection", e);
} catch (JMSException e) {
throw new ResourceException("Unable to setup connection", e);
}
}
It seems that the RA is getting its cue from Spring - a stack trace taken in setup() above reveals that the transacted flag is passed directly from Spring, as configured in the TransactionAwareConnectionFactoryProxy bean above (the default is false) :
java.lang.Exception: setup!!
at org.jboss.resource.adapter.jms.JmsManagedConnection.setup(JmsManagedConnection.java:673)
at org.jboss.resource.adapter.jms.JmsManagedConnection.<init>(JmsManagedConnection.java:185)
at org.jboss.resource.adapter.jms.JmsManagedConnectionFactory.createManagedConnection(JmsManagedConnectionFactory.java:110)
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:1166)
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:446)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getTransactionNewConnection(AbstractPool.java:541)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:442)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:379)
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:421)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:515)
at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:364)
at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createSession(JmsSessionFactoryImpl.java:343)
at org.springframework.jms.connection.ConnectionFactoryUtils$1.createSession(ConnectionFactoryUtils.java:149)
at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:300)
at org.springframework.jms.connection.ConnectionFactoryUtils.getTransactionalSession(ConnectionFactoryUtils.java:138)
at org.springframework.jms.connection.TransactionAwareConnectionFactoryProxy$TransactionAwareConnectionInvocationHandler.invoke(TransactionAwareConnectionFactoryProxy.java:231)
at com.sun.proxy.$Proxy116.createSession(Unknown Source)
at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.access$200(AbstractPollingMessageListenerContainer.java:77)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer$MessageListenerContainerResourceFactory.createSession(AbstractPollingMessageListenerContainer.java:531)
at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:300)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:288)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1103)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1095)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:992)
With regards to local transactions vs XA, the Spring docs state that in a container, the presence of distributed transactions and an XA-capable transaction manager override all of the Spring configuration wrt transactionality.
The below taken from
setSessionTransacted
public void setSessionTransacted(boolean sessionTransacted)
Description copied from class: JmsAccessor
Set the transaction mode that is used when creating a JMS Session. Default is "false".
Note that within a JTA transaction, the parameters passed to
create(Queue/Topic)Session(boolean transacted, int acknowledgeMode)
method are not taken into account. Depending on the J2EE transaction
context, the container makes its
own decisions on these values. Analogously, these parameters are not taken into account within a locally managed transaction either, since the accessor operates on an existing JMS Session in this case.
Setting this flag to "true" will use a short local JMS transaction when running outside of a managed transaction, and a synchronized local JMS transaction in case of a managed transaction (other than an XA transaction) being present. The latter has the effect of a
Overrides:
setSessionTransacted in class JmsAccessor
See Also:
Connection.createSession(boolean, int)
Looking at the above stacktrace, and then at the Spring docs above, it almost looks like either Spring, or the RA is expecting something to intervene wrt the transacted flag etc., and that something is not happening, leading to the incorrect treatment of an XA connection.
Since I'm not clear on whether the issue is with the RA, Spring 3.x or with our configuration, it seems this forum is the best place to start looking for answers.
I was able to get around the error by forcing transacted to true for XA connections, as below.
We are obviously hesitant to go with that as a solution.
Is there something wrong/missing with our Wildfly/Spring configuration that causes this?
Has anyone seen this type of behaviour when using the RA with a Spring 3.x application against Tibco EMS?
Your urgent assistance will be greatly appreciated.
Thanks,
Jan
/**