Concurrent reading and deleting on relationships leads to NotFoundException. How to ignore?

[chris]

I have a reading and a deleting cypher query that run in different processes at the same time which leads to a NotFoundException. The exception happens in the reading query.

- the read query is like this:

START partentNode=1 MATCH parentNode-[r1:HAS]->childNode WHERE r1.id = {x} return childNode

- while my deleting query is like this

START partentNode=1 MATCH parentNode-[r1:HAS]->childNode WHERE r1.id = {y} delete r1

My guess is that the reading query gathers nodes that fit the MATCH clause. Now the deleting query deletes r1's which happen to be in the just gathered set of nodes of the reading query.

When the reading query now starts to evaluate nodes to the WHERE clause the exception occurs because the node to evaluate is not existing anymore.

Neo4j even seems to detect this (cool :D) but, instead of ignoring the error and continuing the evaluation on the rest of the set, it is stopping and therefore does not return a result.

Is there a way to tell neo4j to go on instead of stopping and throwing an exception (catching the exception only does not lead to a result)?

Trace:

[ERROR] [09/21/2012 15:14:00.194] [...] The property 'instanceId' does not exist on Relationship[408]
org.neo4j.cypher.EntityNotFoundException: The property 'instanceId' does not exist on Relationship[408]
        at org.neo4j.cypher.internal.commands.Property.compute(Expression.scala:221)
        at org.neo4j.cypher.internal.commands.Expression$$anonfun$apply$1.apply(Expression.scala:30)
        at scala.collection.MapLike$class.getOrElse(MapLike.scala:122)
        at scala.collection.JavaConversions$JMapWrapper.getOrElse(JavaConversions.scala:792)
        at org.neo4j.cypher.internal.commands.Expression.apply(Expression.scala:30)
        at org.neo4j.cypher.internal.commands.Equals.isMatch(ComparablePredicate.scala:51)
        at org.neo4j.cypher.internal.pipes.matching.SimplePatternMatcherBuilder$$anonfun$getMatches$1$$anonfun$apply$3.apply(SimplePatternMatcherBuilder.scala:93)
        at org.neo4j.cypher.internal.pipes.matching.SimplePatternMatcherBuilder$$anonfun$getMatches$1$$anonfun$apply$3.apply(SimplePatternMatcherBuilder.scala:93)
        at scala.collection.LinearSeqOptimized$class.forall(LinearSeqOptimized.scala:69)
        at scala.collection.immutable.List.forall(List.scala:76)
        at org.neo4j.cypher.internal.pipes.matching.SimplePatternMatcherBuilder$$anonfun$getMatches$1.apply(SimplePatternMatcherBuilder.scala:93)
        at org.neo4j.cypher.internal.pipes.matching.SimplePatternMatcherBuilder$$anonfun$getMatches$1.apply(SimplePatternMatcherBuilder.scala:85)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
        at scala.collection.Iterator$class.foreach(Iterator.scala:772)
        at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
        at scala.collection.JavaConversions$JIterableWrapper.foreach(JavaConversions.scala:587)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
        at scala.collection.JavaConversions$JIterableWrapper.map(JavaConversions.scala:587)
        at org.neo4j.cypher.internal.pipes.matching.SimplePatternMatcherBuilder.getMatches(SimplePatternMatcherBuilder.scala:85)
        at org.neo4j.cypher.internal.pipes.matching.SimplePatternMatcherBuilder.getMatches(SimplePatternMatcherBuilder.scala:30)
        at org.neo4j.cypher.internal.pipes.matching.MatchingContext.getMatches(MatchingContext.scala:53)
        at org.neo4j.cypher.internal.pipes.MatchPipe$$anonfun$createResults$1.apply(MatchPipe.scala:32)
        at org.neo4j.cypher.internal.pipes.MatchPipe$$anonfun$createResults$1.apply(MatchPipe.scala:31)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:239)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:239)
        at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
        at scala.collection.immutable.List.foreach(List.scala:76)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:239)
        at scala.collection.immutable.List.flatMap(List.scala:76)
        at org.neo4j.cypher.internal.pipes.MatchPipe.createResults(MatchPipe.scala:31)
        at org.neo4j.cypher.internal.pipes.FilterPipe.createResults(FilterPipe.scala:28)
        at org.neo4j.cypher.internal.pipes.ColumnFilterPipe.createResults(ColumnFilterPipe.scala:39)
        at org.neo4j.cypher.internal.executionplan.ExecutionPlanImpl$$anonfun$4$$anonfun$apply$2.apply(ExecutionPlanImpl.scala:95)
        at org.neo4j.cypher.internal.executionplan.ExecutionPlanImpl$$anonfun$4$$anonfun$apply$2.apply(ExecutionPlanImpl.scala:95)
        at org.neo4j.cypher.PipeExecutionResult.immutableResult(PipeExecutionResult.scala:37)
        at org.neo4j.cypher.PipeExecutionResult.iterator(PipeExecutionResult.scala:153)
        at org.neo4j.cypher.PipeExecutionResult.hasNext(PipeExecutionResult.scala:155)
        at scala.collection.Iterator$$anon$19.hasNext(Iterator.scala:400)
        at scala.collection.JavaConversions$IteratorWrapper.hasNext(JavaConversions.scala:562)
        at org.neo4j.helpers.collection.IteratorWrapper.hasNext(IteratorWrapper.java:42)
        at org.neo4j.helpers.collection.IteratorUtil.addToCollection(IteratorUtil.java:322)
        at org.neo4j.helpers.collection.IteratorUtil.addToCollection(IteratorUtil.java:341)
        at org.springframework.data.neo4j.repository.query.GraphRepositoryQuery.dispatchQuery(GraphRepositoryQuery.java:92)
        at org.springframework.data.neo4j.repository.query.GraphRepositoryQuery.execute(GraphRepositoryQuery.java:70)
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:313)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:155)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at $Proxy51.findFlowNodeInstancesAtCurrentLevelByState(Unknown Source)

...

Caused by: org.neo4j.graphdb.NotFoundException: Relationship[408] not found. This can be because someone else deleted this entity while we were trying to read properties from it, or because of concurrent modification of other properties on this entity. The problem should be temporary.
        at org.neo4j.kernel.impl.core.Primitive.ensureFullProperties(Primitive.java:589)
        at org.neo4j.kernel.impl.core.Primitive.ensureFullProperties(Primitive.java:567)
        at org.neo4j.kernel.impl.core.Primitive.getProperty(Primitive.java:153)
        at org.neo4j.kernel.impl.core.RelationshipImpl.getProperty(RelationshipImpl.java:32)
        at org.neo4j.kernel.impl.core.RelationshipProxy.getProperty(RelationshipProxy.java:114)
        at org.neo4j.cypher.internal.commands.Property.compute(Expression.scala:219)
        ... 70 more
Caused by: org.neo4j.kernel.impl.nioneo.store.InvalidRecordException: PropertyRecord[682] not in use
        at org.neo4j.kernel.impl.nioneo.store.PropertyStore.getRecord(PropertyStore.java:441)
        at org.neo4j.kernel.impl.nioneo.store.PropertyStore.getLightRecord(PropertyStore.java:300)
        at org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getPropertyRecordChain(ReadTransaction.java:185)
        at org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.loadProperties(ReadTransaction.java:215)
        at org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.relLoadProperties(ReadTransaction.java:233)
        at org.neo4j.kernel.impl.persistence.PersistenceManager.loadRelProperties(PersistenceManager.java:119)
        at org.neo4j.kernel.impl.core.NodeManager.loadProperties(NodeManager.java:688)
        at org.neo4j.kernel.impl.core.RelationshipImpl.loadProperties(RelationshipImpl.java:92)
        at org.neo4j.kernel.impl.core.Primitive.ensureFullProperties(Primitive.java:584)
        ... 75 more

[chris]

Searching for a quick solution I stumbled over the lines of code that lead to this behavior: 

https://github.com/neo4j/community/pull/789

In the test comments of TestOperationsOnDeletedRelationships.java it correctly states: 

// Should it really do this? Wouldn't it be better if we could recover from a a relationship suddenly // missing in the chain? Perhaps that is really hard to do though.

So my vote is for "yeah, this would be better". ;)

Will this be fixed any time soon, or can you hint me a way of quickly hacking the behavior I need?

[Peter Neubauer]

Hi Christopher,

I've discussed this with our team, and unfortunately this is a
side-effect of the way Neo4j optimises for read performance - it
supports "read-committed" within a single cypher statement, which can
result in phantom reads.

In this specific case, the cypher engine has identified nodes to be
traversed as part of the query, but then those nodes are not available
when it attempts to traverse the graph during execution. So you see
this exception.

We don't particularly like this either, and believe the exception is
obtuse. So we will be looking at how to improve this, both in terms of
the exception message itself and also to try and support "repeatable
reads" at minimum within a single cypher statement. I don't have a
timeframe on this, but I'm hoping we'll see that in a future release
soon.

For now, there is no concern in ignoring those exceptions and
re-trying the query.

I have opened https://github.com/neo4j/community/issues/879 to track this.

Thanks for reporting!

Cheers,

/peter neubauer

Neo4j 1.8.RC1 "Vindeln Vy" -
http://blog.neo4j.org/2012/09/neo4j-18rc1-really-careful-ftw.html

G: neubauer.peter
S: peter.neubauer
P: +46 704 106975
L: http://www.linkedin.com/in/neubauer
T: @peterneubauer

Wanna learn something new? Come to @graphconnect.

> --
>
>

[chris]

Hi Peter,

thanks for your reply. I will until then just retry the queries that return this particular exception in my client code.

This seems to work and the speed hit seems acceptable to me at the moment. 

Kind regards,

Chris

[Debajyoti Roy]

Hi Peter,

We are noticing this on community 1.9.3  ...not sure if there is a more elegant way to address this issue.
I could not get further information about the resolution as the link is broken: https://github.com/neo4j/community/issues/879

Thanks,
Roy

[Jonathan Crosmer]

Updated link: https://github.com/neo4j/neo4j/issues/37
Appears to be unresolved at the moment.