Deadlocks / What may a database restart do regarding indexes?

[Michael Azerhad]

Hi,

I know my question is rather "abstract". 

But I've just come across the following issue: 

I have a running Neo4j 2.0.4.

I updated my source code (my application) on Heroku and restarted the dynos.

When dynos restarted, I was about to test my new features, but...a series of Deadlock exceptions occured regarding indexes: 

2014-08-03T16:22:18.693880+00:00 app[web.2]: at org.springframework.data.neo4j.config.Neo4jConfiguration$$EnhancerByCGLIB$$1a31c2a5.neo4jTemplate(<generated>)

2014-08-03T16:22:18.693982+00:00 app[web.2]: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

2014-08-03T16:22:18.694071+00:00 app[web.2]: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

2014-08-03T16:22:18.694159+00:00 app[web.2]: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2014-08-03T16:22:18.694246+00:00 app[web.2]: at java.lang.reflect.Method.invoke(Method.java:606)

2014-08-03T16:22:18.694336+00:00 app[web.2]: at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:166)

2014-08-03T16:22:18.694417+00:00 app[web.2]: ... 48 more

2014-08-03T16:22:18.695021+00:00 app[web.2]: Caused by: org.neo4j.rest.graphdb.RestResultException: Don't panic.

2014-08-03T16:22:18.695024+00:00 app[web.2]: 

2014-08-03T16:22:18.695026+00:00 app[web.2]: A deadlock scenario has been detected and avoided. This means that two or more transactions, which were holding locks, were wanting to await locks held by one another, which would have resulted in a deadlock between these transactions. This exception was thrown instead of ending up in that deadlock.

2014-08-03T16:22:18.695027+00:00 app[web.2]: 

2014-08-03T16:22:18.695028+00:00 app[web.2]: See the deadlock section in the Neo4j manual for how to avoid this: http://docs.neo4j.org/chunked/stable/transactions-deadlocks.html

2014-08-03T16:22:18.695029+00:00 app[web.2]: 

2014-08-03T16:22:18.695031+00:00 app[web.2]: Details: 'Transaction(36, owner:"qtp787473890-25")[STATUS_ACTIVE,Resources=1] can't wait on resource RWLock[SchemaLock] since => Transaction(36, owner:"qtp787473890-25")[STATUS_ACTIVE,Resources=1] <-[:HELD_BY]- RWLock[SchemaLock] <-[:WAITING_FOR]- Transaction(35, owner:"qtp787473890-33")[STATUS_ACTIVE,Resources=1] <-[:HELD_BY]- RWLock[SchemaLock]'. at

2014-08-03T16:22:18.695032+00:00 app[web.2]: DeadlockDetectedException

2014-08-03T16:22:18.695033+00:00 app[web.2]:    org.neo4j.kernel.impl.transaction.RagManager.checkWaitOnRecursive(RagManager.java:211)

2014-08-03T16:22:18.695034+00:00 app[web.2]:    org.neo4j.kernel.impl.transaction.RagManager.checkWaitOnRecursive(RagManager.java:238)

2014-08-03T16:22:18.695036+00:00 app[web.2]:    org.neo4j.kernel.impl.transaction.RagManager.checkWaitOn(RagManager.java:178)

2014-08-03T16:22:18.695037+00:00 app[web.2]:    org.neo4j.kernel.impl.transaction.RWLock.deadlockGuardedWait(RWLock.java:553)

2014-08-03T16:22:18.695055+00:00 app[web.2]:    org.neo4j.kernel.impl.transaction.RWLock.acquireWriteLock(RWLock.java:321)

2014-08-03T16:22:18.695056+00:00 app[web.2]:    org.neo4j.kernel.impl.transaction.LockManagerImpl.getWriteLock(LockManagerImpl.java:71)

2014-08-03T16:22:18.695058+00:00 app[web.2]:    org.neo4j.kernel.impl.api.LockHolderImpl.acquireSchemaWriteLock(LockHolderImpl.java:121)

2014-08-03T16:22:18.695059+00:00 app[web.2]:    org.neo4j.kernel.impl.api.LockingStatementOperations.indexCreate(LockingStatementOperations.java:102)

2014-08-03T16:22:18.695063+00:00 app[web.2]:    org.neo4j.kernel.impl.api.OperationsFacade.indexCreate(OperationsFacade.java:512)

2014-08-03T16:22:18.695064+00:00 app[web.2]:    org.neo4j.cypher.internal.spi.v2_0.TransactionBoundQueryContext.addIndexRule(TransactionBoundQueryContext.scala:238)

2014-08-03T16:22:18.695065+00:00 app[web.2]:    org.neo4j.cypher.internal.compiler.v2_0.spi.DelegatingQueryContext.addIndexRule(DelegatingQueryContext.scala:68)

2014-08-03T16:22:18.695067+00:00 app[web.2]:    org.neo4j.cypher.internal.compiler.v2_0.spi.ExceptionTranslatingQueryContext.org$neo4j$cypher$internal$compiler$v2_0$spi$ExceptionTranslatingQueryContext$$super$addIndexRule(ExceptionTranslatingQueryContext.scala:83) 

Unusable. 

I even tried to let only one dyno running...to avoid potential Deadlocks between cluster nodes.

So I just thought about restarting Neo4j itself, and....the whole works. 

How could a simple restart of  dynos impact the underlying Neo4j instance? 

It's just a question for my curiosity, since it's not the first time it happens.

Many thanks :)

Michael

[Michael Hunger]

A deadlock is if several operations access resources in a manner that a waits for b and b waits for a.

Perhaps this is during the initial index creation on starting your SDN instance if you do that in parallel with multiple instances.

Michael

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Azerhad]

Yes, probably but the point is I don't manage to figure out why a simple restart of the database prevent this issue to occur afterwards. 

It's like saying the application keeps lock on some indexes/nodes ( causing deadlocks) and release them at database restart ...

You received this message because you are subscribed to a topic in the Google Groups "Neo4j" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/neo4j/Zat9b0Ql7ZQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to neo4j+un...@googlegroups.com.