ConnectorXAResource handleResourceException

[riccardo cazzola]

Hi,
we have an application installed on Payara 164.
There is a JDBC resource configured to access a MariaDB 10.1 database on the same host.
The resource is used not only by web part (JSF with PrimFaces), but also by some Java Beans.
Those beans work on different threads, all accessing and using the resource.
Sometimes, among others, we receive the following error

gen 18, 2018 10:00:01 AM com.sun.enterprise.resource.ConnectorXAResource handleResourceException
GRAVE: poolmgr.system_exception
com.sun.appserv.connectors.internal.api.PoolingException: java.lang.NullPointerException
at com.sun.enterprise.resource.ConnectorXAResource.getResourceHandle(ConnectorXAResource.java:255)
at com.sun.enterprise.resource.ConnectorXAResource.end(ConnectorXAResource.java:159)
at com.sun.enterprise.transaction.JavaEETransactionManagerSimplified.delistResource(JavaEETransactionManagerSimplified.java:551)
at com.sun.enterprise.resource.rm.SystemResourceManagerImpl.delistResource(SystemResourceManagerImpl.java:145)
at com.sun.enterprise.resource.pool.PoolManagerImpl.resourceClosed(PoolManagerImpl.java:379)
at com.sun.enterprise.resource.listener.LocalTxConnectionEventListener.connectionClosed(LocalTxConnectionEventListener.java:77)
at com.sun.gjc.spi.ManagedConnectionImpl.connectionClosed(ManagedConnectionImpl.java:840)
at com.sun.gjc.spi.base.ConnectionHolder.close(ConnectionHolder.java:214)
at com.sun.gjc.spi.jdbc40.ConnectionHolder40.close(ConnectionHolder40.java:604)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.closeDatasourceConnection(DatabaseAccessor.java:495)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.closeConnection(DatasourceAccessor.java:520)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.closeConnection(DatabaseAccessor.java:520)
at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.closeJTSConnection(DatasourceAccessor.java:187)
at org.eclipse.persistence.sessions.server.ClientSession.releaseJTSConnection(ClientSession.java:168)
at org.eclipse.persistence.transaction.AbstractSynchronizationListener.beforeCompletion(AbstractSynchronizationListener.java:175)
at org.eclipse.persistence.transaction.JTASynchronizationListener.beforeCompletion(JTASynchronizationListener.java:68)
at com.sun.enterprise.transaction.JavaEETransactionImpl.commit(JavaEETransactionImpl.java:452)
at com.sun.enterprise.transaction.JavaEETransactionManagerSimplified.commit(JavaEETransactionManagerSimplified.java:884)
at com.sun.ejb.containers.EJBContainerTransactionManager.completeNewTx(EJBContainerTransactionManager.java:721)
at com.sun.ejb.containers.EJBContainerTransactionManager.postInvokeTx(EJBContainerTransactionManager.java:505)
at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4542)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2021)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1992)
at com.sun.ejb.containers.BaseContainer.callEJBTimeout(BaseContainer.java:4029)
at com.sun.ejb.containers.EJBTimerService.deliverTimeout(EJBTimerService.java:1224)
at com.sun.ejb.containers.EJBTimerService.access$000(EJBTimerService.java:92)
at com.sun.ejb.containers.EJBTimerService$TaskExpiredWork.run(EJBTimerService.java:1954)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException


We tried to change the pool size and to enable the leak monitor 
(Connection Leak Timeout and Connection Leak Reclaim) with no luck
We are using the mysql driver and eclipselink, but we tested also the 
mariaDB driver with the same result

P.S.
A similar post in stackoverflow

[Steve Millidge (Payara)]

When you say these beans are using the resource on different threads how are the threads created and how are the beans getting the connection from the datasource?

 

It looks like the exception is from this line of code https://github.com/payara/Payara/blob/master/appserver/connectors/connectors-runtime/src/main/java/com/sun/enterprise/resource/ConnectorXAResource.java#L255

 

Which would imply the ResourceHandle is null or the ResourceState is null which implies the connection is not enlisted into the transaction.

 

Steve

--
You received this message because you are subscribed to the Google Groups "Payara Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to payara-forum...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/44ff3c3d-5a00-4423-9080-8ee3c1a0fbb9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[riccardo cazzola]

We have the jdbc connection created in payara

In domain.xml

   

<jdbc-connection-pool pool-resize-quantity="4" connection-leak-reclaim="true" validation-table-name="DUAL" res-type="javax.sql.DataSource" max-pool-size="64" max-wait-time-in-millis="10000" connection-leak-timeout-in-seconds="300" steady-pool-size="16" datasource-classname="com.mysql.jdbc.jdbc2.optional.MysqlDataSource" idle-timeout-in-seconds="60" name="mysql_xxxx_rootPool" is-connection-validation-required="true" wrap-jdbc-objects="false" transaction-isolation-level="repeatable-read">
      <property name="URL" value="jdbc:mysql://localhost:3306/dmlm?zeroDateTimeBehavior=convertToNull"></property>
      <property name="driverClass" value="com.mysql.jdbc.Driver"></property>
      <property name="Password" value="xxxxxxxxx"></property>
      <property name="portNumber" value="3306"></property>
      <property name="databaseName" value="xxxx"></property>
      <property name="User" value="user"></property>
      <property name="serverName" value="localhost"></property>
    </jdbc-connection-pool>
    <jdbc-resource pool-name="mysql_xxxx_rootPool" description="xxxx db resource" jndi-name="jdbc/xxxx"></jdbc-resource>

the persistence.xml

<?xml version="1.0" encoding="UTF-8"?>
<persistence version="2.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">
  <persistence-unit name="AppPU" transaction-type="JTA">
    <provider>org.eclipse.persistence.jpa.PersistenceProvider</provider>
    <jta-data-source>jdbc/xxxx</jta-data-source>
    <exclude-unlisted-classes>false</exclude-unlisted-classes>
    <shared-cache-mode>ENABLE_SELECTIVE</shared-cache-mode>
    <validation-mode>NONE</validation-mode>
    <properties>
      <property name="eclipselink.logging.logger" value="JavaLogger"/>
      <property name="eclipselink.logging.level" value="INFO"/>
      <property name="eclipselink.logging.parameters" value="false"/>      
      <property name="eclipselink.logging.exceptions" value="true"/>
      <property name="eclipselink.ddl-generation" value="drop-and-create-tables"/>
      <property name="eclipselink.create-ddl-jdbc-file-name" value="createDDL.jdbc"/>
      <property name="eclipselink.drop-ddl-jdbc-file-name" value="dropDDL.jdbc"/>
      <property name="eclipselink.ddl-generation.output-mode" value="sql-script"/>
    </properties>
  </persistence-unit>
</persistence>

One example of EJB that use the connection

@Stateless(name = "MailHelper", mappedName = "ejb/MailHelper")
@Interceptors(GenericInterceptor.class)
public class MailHelper
{

    @PersistenceContext(name = "AppPU")
    private EntityManager em;
.....

What appears strange to me is that is normally working, and it fails ten times a day, on hundreds of tries

[riccardo cazzola]

I forgot the thread

In a EJB the method called is annotated with the following code

@Stateless(name = "EmailSender")
public class EmailSender
{

    @Asynchronous
    @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
    public void sendMessages(Mailbox mailbox, List<Long> processingMailbox, Integer pidId)
    {
...

at org.eclipse.persistence.internal<span st

[Michael Ranaldo]

Hi Riccardo,

Do you have any logs thrown by the error?

[riccardo cazzola]

The one already posted, no more clues

[Ondro Mihályi]

Hi,

Can you try with version 171.1 or later? There are some fixes in ConnectorXAResource that might fix your problem too. See this commit: https://github.com/payara/Payara/commit/56047ebb6b29f1760da8c9125d1a98e5ff2c9191#diff-85bb4b5e0be4074e47a0e9c2d934a72c

Ondro

[riccardo cazzola]

Hi all,
we finally installed a new machine and replicate the server with version 174.
Again we have an exception on ConnectorXAResource, but on the previous step (getResourceHandle(ConnectorXAResource.java:255) )
I still have no idea why the resource became unuseful

Jan 31, 2018 11:06:40 PM com.sun.enterprise.resource.ConnectorXAResource getResourceHandle
SEVERE: poolmgr.system_exception
java.lang.NullPointerException
        at com.sun.enterprise.resource.ConnectorXAResource.getResourceHandle(ConnectorXAResource.java:255)
        at com.sun.enterprise.resource.ConnectorXAResource.getResourceHandleAndBeginTxIfNeeded(ConnectorXAResource.java:232)
        at com.sun.enterprise.resource.ConnectorXAResource.end(ConnectorXAResource.java:160)
        at com.sun.enterprise.transaction.JavaEETransactionManagerSimplified.delistResource(JavaEETransactionManagerSimplified.java:545)
        at com.sun.enterprise.resource.rm.SystemResourceManagerImpl.delistResource(SystemResourceManagerImpl.java:145)
        at com.sun.enterprise.resource.pool.PoolManagerImpl.resourceClosed(PoolManagerImpl.java:378)
        at com.sun.enterprise.resource.listener.LocalTxConnectionEventListener.connectionClosed(LocalTxConnectionEventListener.java:77)
        at com.sun.gjc.spi.ManagedConnectionImpl.connectionClosed(ManagedConnectionImpl.java:860)
        at com.sun.gjc.spi.base.ConnectionHolder.close(ConnectionHolder.java:214)
        at com.sun.gjc.spi.jdbc40.ConnectionHolder40.close(ConnectionHolder40.java:606)

        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.closeDatasourceConnection(DatabaseAccessor.java:495)
        at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.closeConnection(DatasourceAccessor.java:520)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.closeConnection(DatabaseAccessor.java:520)
        at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.closeJTSConnection(DatasourceAccessor.java:187)
        at org.eclipse.persistence.sessions.server.ClientSession.releaseJTSConnection(ClientSession.java:168)
        at org.eclipse.persistence.transaction.AbstractSynchronizationListener.beforeCompletion(AbstractSynchronizationListener.java:175)
        at org.eclipse.persistence.transaction.JTASynchronizationListener.beforeCompletion(JTASynchronizationListener.java:68)
        at com.sun.enterprise.transaction.JavaEETransactionImpl.commit(JavaEETransactionImpl.java:452)

        at com.sun.enterprise.transaction.JavaEETransactionManagerSimplified.commit(JavaEETransactionManagerSimplified.java:878)

        at com.sun.ejb.containers.EJBContainerTransactionManager.completeNewTx(EJBContainerTransactionManager.java:721)
        at com.sun.ejb.containers.EJBContainerTransactionManager.postInvokeTx(EJBContainerTransactionManager.java:505)

        at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4616)
        at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2089)
        at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)
        at com.sun.ejb.containers.BaseContainer.callEJBTimeout(BaseContainer.java:4103)
        at com.sun.ejb.containers.EJBTimerService.deliverTimeout(EJBTimerService.java:1228)
        at com.sun.ejb.containers.EJBTimerService.access$000(EJBTimerService.java:93)
        at com.sun.ejb.containers.EJBTimerService$TaskExpiredWork.run(EJBTimerService.java:1963)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

[Ondro Mihályi]

Hi,

Very recently we've added some fixes related to distributed transaction handling. They solve concurrency issues and it's very probable that also your exception happens because of those concurrency issues.

These fixes will be part of Payara Server 4.181 (it won't take too long before it's released). See https://github.com/payara/Payara/pull/2330 and

Even with Payara Server 4.174, most of the issues can be safely avoided with setting the system property "org.glassfish.jms.skip-resource-registration-in-transaction" to true.

Ondro

[Steve Millidge]

This looks like straight JPA code so the JMS system property won't have an effect unless in the code JMS 2.0 JMS Contexts are being used.

[riccardo cazzola]

I can confirm that the xaresource error keep came out even after setting the suggested properties