Engine stuck in PersistenceException

[Akshay Garg]

Hi,

 

I have been facing with this issue quite often where camunda starts throwing PersistenceException and then remains stuck in that case. Stack trace is copied below. I am running camunda on local box with tomcat and using REST API to create instance and do other things.

It works fine initially when tomcat is just started, but if I do not send any REST request for sometime say 10-15 mins and then send any request, I get PersistenceException.

 

And once exception starts, system remains stuck in that state. I have to restart the tomcat to make work.

 

I am using Camunda with Tomcat and MSSQL. I have copied content of bpm-platform.xml below stack trace.

 

I want to know the reason of such exception and how to avoid it. Any help is appreciated.

 

Stack Trace

Caused by: org.apache.ibatis.exceptions.PersistenceException:
### Error querying database.  Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit.  Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false.  Cause: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
### The error may exist in org/camunda/bpm/engine/impl/mapping/entity/Execution.xml
### The error may involve org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.selectProcessInstanceByQueryCriteria
### The error occurred while executing a query
### Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit.  Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false.  Cause: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
 at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
 at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:111)
 at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:102)
 at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.selectList(DbSqlSession.java:87)
 at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectListWithRawParameter(DbEntityManager.java:170)
 at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:162)
 at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:154)
 at org.camunda.bpm.engine.impl.persistence.entity.ExecutionManager.findProcessInstanceByQueryCriteria(ExecutionManager.java:129)
 at org.camunda.bpm.engine.impl.ProcessInstanceQueryImpl.executeList(ProcessInstanceQueryImpl.java:203)
 at org.camunda.bpm.engine.impl.AbstractQuery.evaluateExpressionsAndExecuteList(AbstractQuery.java:186)
 at org.camunda.bpm.engine.impl.AbstractQuery.execute(AbstractQuery.java:165)
 at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
 at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:95)
 ... 39 more
Caused by: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit.  Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false.  Cause: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
 at org.apache.ibatis.transaction.jdbc.JdbcTransaction.setDesiredAutoCommit(JdbcTransaction.java:105)
 at org.apache.ibatis.transaction.jdbc.JdbcTransaction.openConnection(JdbcTransaction.java:138)
 at org.apache.ibatis.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:61)
 at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:279)
 at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:72)
 at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:59)
 at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:267)
 at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:137)
 at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:96)
 at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:77)
 at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:108)
 ... 50 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
 at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:190)
 at com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:388)
 at com.microsoft.sqlserver.jdbc.SQLServerConnection.getAutoCommit(SQLServerConnection.java:1909)
 at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:497)
 at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:126)
 at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109)
 at org.apache.tomcat.jdbc.pool.DisposableConnectionFacade.invoke(DisposableConnectionFacade.java:80)
 at com.sun.proxy.$Proxy4.getAutoCommit(Unknown Source)
 at org.apache.ibatis.transaction.jdbc.JdbcTransaction.setDesiredAutoCommit(JdbcTransaction.java:96)
 ... 60 more

 

 

 

 

 

 

bpm-platform.xml

 

<?xml version="1.0" encoding="UTF-8"?>

<bpm-platform xmlns="http://www.camunda.org/schema/1.0/BpmPlatform" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://www.camunda.org/schema/1.0/BpmPlatform http://www.camunda.org/schema/1.0/BpmPlatform ">

<job-executor>

<job-acquisition name="default" />

</job-executor>

<process-engine name="default">

<job-acquisition>default</job-acquisition>

<configuration>org.camunda.bpm.engine.impl.cfg.StandaloneProcessEngineConfiguration</configuration>

<datasource>java:jdbc/ProcessEngine</datasource>

<properties>

<property name="history">audit</property>

<property name="databaseSchemaUpdate">true</property>

<property name="authorizationEnabled">true</property>

<property name="jobExecutorDeploymentAware">true</property>

</properties>

<plugins>

<!-- plugin enabling Process Application event listener support -->

<plugin>

<class>org.camunda.bpm.application.impl.event.ProcessApplicationEventListenerPlugin</class>

</plugin>

<!-- plugin enabling integration of camunda Spin -->

<plugin>

<class>org.camunda.spin.plugin.impl.SpinProcessEnginePlugin</class>

</plugin>

<!-- plugin enabling connect support -->

<plugin>

<class>org.camunda.connect.plugin.impl.ConnectProcessEnginePlugin</class>

</plugin>     

</plugins>

　

</process-engine>

</bpm-platform>

 

[Paolo Cabras]

Hi everyone,

I'm facing a similar problem (no REST api, java ones instead).

Sometime while trying to to interact with the Process Engine I get this exception:

00:00:17,180 SEVERE [org.camunda.bpm.engine.impl.interceptor.CommandContext] (job-executor-tp-threads - 1) Error while closing command context: org.apache.ibatis.exceptions.PersistenceException:

### Error querying database.  Cause: java.sql.SQLException: Closed Connection

### The error may exist in org/camunda/bpm/engine/impl/mapping/entity/Job.xml

### The error may involve org.camunda.bpm.engine.impl.persistence.entity.JobEntity.selectNextJobsToExecute

### The error occurred while executing a query

### SQL: select * from ( select a.*, ROWNUM rnum from (     select       RES.*      from ACT_RU_JOB RES      where (RES.RETRIES_ > 0)       and (RES.DUEDATE_ is null or RES.DUEDATE_ <= ?)       and (RES.LOCK_OWNER_ is null or RES.LOCK_EXP_TIME_ < ?)       and (RES.SUSPENSION_STATE_ = 1 or RES.SUSPENSION_STATE_ is null)                      ) a where ROWNUM < ?) where rnum  >= ?

### Cause: java.sql.SQLException: Closed Connection

                at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)

                at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:111)

                at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:102)

                at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.selectList(DbSqlSession.java:86) [camunda-engine-7.3.0.jar:7.3.0]

                at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectListWithRawParameter(DbEntityManager.java:164) [camunda-engine-7.3.0.jar:7.3.0]

                at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:156) [camunda-engine-7.3.0.jar:7.3.0]

                at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:152) [camunda-engine-7.3.0.jar:7.3.0]

                at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:141) [camunda-engine-7.3.0.jar:7.3.0]

                at org.camunda.bpm.engine.impl.persistence.entity.JobManager.findNextJobsToExecute(JobManager.java:165) [camunda-engine-7.3.0.jar:7.3.0]

                at org.camunda.bpm.engine.impl.cmd.AcquireJobsCmd.execute(AcquireJobsCmd.java:55) [camunda-engine-7.3.0.jar:7.3.0]

                at org.camunda.bpm.engine.impl.cmd.AcquireJobsCmd.execute(AcquireJobsCmd.java:38) [camunda-engine-7.3.0.jar:7.3.0]

                at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24) [camunda-engine-7.3.0.jar:7.3.0]

                at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:97) [camunda-engine-7.3.0.jar:7.3.0]

                at org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:59) [camunda-engine-7.3.0.jar:7.3.0]

                at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32) [camunda-engine-7.3.0.jar:7.3.0]

                at org.camunda.bpm.engine.impl.jobexecutor.SequentialJobAcquisitionRunnable.run(SequentialJobAcquisitionRunnable.java:64) [camunda-engine-7.3.0.jar:7.3.0]

                at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]

                at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:806) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]

                at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]

                at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:826) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]

                at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]

                at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]

Caused by: java.sql.SQLException: Closed Connection

                at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)

                at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)

                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:199)

                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:263)

                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:271)

                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:445)

                at oracle.jdbc.driver.PhysicalConnection.prepareStatement(PhysicalConnection.java:3127)

                at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.doPrepareStatement(BaseWrapperManagedConnection.java:738)

                at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.prepareStatement(BaseWrapperManagedConnection.java:724)

                at org.jboss.jca.adapters.jdbc.WrappedConnection.prepareStatement(WrappedConnection.java:405)

                at org.apache.ibatis.executor.statement.PreparedStatementHandler.instantiateStatement(PreparedStatementHandler.java:75)

                at org.apache.ibatis.executor.statement.BaseStatementHandler.prepare(BaseStatementHandler.java:85)

                at org.apache.ibatis.executor.statement.RoutingStatementHandler.prepare(RoutingStatementHandler.java:57)

                at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:73)

                at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:59)

                at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:267)

                at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:137)

                at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:96)

                at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:77)

                at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:108)

                ... 20 more

In my case the server is running in domain mode and the issue propagates itself in all running instances forcing then a reboot.

I also verified DB connection and it has always been available...

any idea?

[Sebastian Menski]

Hi,

maybe a solution would be to use a connection pool which has a keep-alive option for connections. I think the problem is that inactive connections are closed

by the server and this leads to this problems.

Cheers,

Sebastian

[Frank Langelage]

Hi,

can't speak for tomcat (problem description of the OP), but regarding JBoss AS / Wildfly the application server provides a connection pooling which can be configured within the datasource configuration.

There you also can configure validation of connections, see e.g. https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.4/html/Administration_and_Configuration_Guide/sect-Database_Connection_Validation.html, to avoid that your application gets a no longer open connection.

Regards, Frank

[Paolo Cabras]

Hi,

sorry for answering late but, after applying you suggestion, I was monitoring the server.

Apparently it worked!!!

So thanks a lot,

Cheers,

Paolo

Il giorno mercoledì 17 febbraio 2016 16:24:00 UTC+1, Akshay Garg ha scritto: