Saga persistence deadlock

[sheak]

We are sometimes experiences deadlocks when sagas are persisted as
shown in the stack trace below. All our sagas are running in async
mode. Any thoughts on what the issue might be?

2011-06-20 17:33:21,413 [taskExecutor-7] ERROR
org.hibernate.event.def.AbstractFlushingEventListener Could not
synchronize database state with session
org.hibernate.exception.LockAcquisitionException: could not update:
org.axonframework.saga.repository.jpa.SagaEntry#26aab18f-3554-4bc8-
a9f3-5d801d45ca69
at
org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:
105)
at
org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:
66)
at
org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:
2453)
at
org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:
2335)
at
org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:
2635)
at
org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:
115)
at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:
263)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:
168)
at
org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:
321)
at
org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:
50)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
at
org.hibernate.ejb.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:
304)
at sun.reflect.GeneratedMethodAccessor222.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.orm.jpa.SharedEntityManagerCreator
$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:
240)
at $Proxy135.flush(Unknown Source)
at
org.axonframework.saga.repository.jpa.JpaSagaRepository.updateSaga(JpaSagaRepository.java:
106)
at
org.axonframework.saga.repository.AbstractSagaRepository.commit(AbstractSagaRepository.java:
112)
at
org.axonframework.saga.AbstractSagaManager.commit(AbstractSagaManager.java:
136)
at
org.axonframework.saga.AbstractSagaManager.invokeSagaHandler(AbstractSagaManager.java:
117)
at org.axonframework.saga.AbstractSagaManager.access
$100(AbstractSagaManager.java:40)
at org.axonframework.saga.AbstractSagaManager
$SagaInvocationTask.run(AbstractSagaManager.java:210)
at org.axonframework.saga.AsynchronousSagaExecutor
$Task.execute(AsynchronousSagaExecutor.java:94)
at
org.axonframework.saga.AsynchronousSagaExecutor.doHandle(AsynchronousSagaExecutor.java:
61)
at
org.axonframework.saga.AsynchronousSagaExecutor.doHandle(AsynchronousSagaExecutor.java:
32)
at org.axonframework.eventhandling.AsynchronousExecutionWrapper
$1.doHandle(AsynchronousExecutionWrapper.java:149)
at
org.axonframework.eventhandling.EventProcessingScheduler.handleEventBatch(EventProcessingScheduler.java:
317)
at
org.axonframework.eventhandling.EventProcessingScheduler.processOrRetryBatch(EventProcessingScheduler.java:
239)
at
org.axonframework.eventhandling.EventProcessingScheduler.run(EventProcessingScheduler.java:
200)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException:
Transaction (Process ID 372) was deadlocked on lock resources with
another process and has been chosen as the deadlock victim. Rerun the
transaction.
at
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:
196)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:
1454)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:
388)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement
$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:338)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4026)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:
1416)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:
185)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:
160)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:
306)
at
org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:
46)
at
org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:
2431)

[Allard Buijze]

Hi Shea,

Seems to be an issue that has happened before with MS SQL:http://www.coderanch.com/t/415119/ORM/java/Deadlock-problems-Hibernate-Spring-MS.

Meanwhile, I will create a small test setup to find out if I can reproduce this problem.

Cheers,

Allard

[sheak]

Though I should post a follow up to this with some details on how we
got around this problem:

1) Make sure you put the appropriate indexes on the SagaEntry and
AssociationValueEntry table to speed up queries (look at the
JpaSagaRepository for the queries used)
2) If you get a deadlock from sql server when a saga executes consider
a retry. Axon support reties via the SpringTransactionManager however
it does not handle the
org.hibernate.exception.LockAcquisitionException only some base jdbc
recoverable expections. Unless there is a way around this I think
this class should change to handle the hibernate exception.
2) Limit the number of threads available to execute sagas. We have
limited it to use 1 thread for now to remove contention for the
SagaEntry table.

[Allard Buijze]

Hi all,

soon, I will review the Saga logic in Axon Framework. I've received a lot of feedback which I will use to improve it. Thanks for that!

In the meantime, the 1.1 version contains some fixes already.

Cheers,

Allard