deletion of relationships got stuck

111 views
Skip to first unread message

Ray Chengr

unread,
Aug 28, 2016, 6:02:30 PM8/28/16
to Neo4j
On our Neo4j 3.0.3, I tried to delete relationships and nodes from the browser by:

match (n:node_type_A {day: 17016}) detach delete(n)

I got this error:

Relationship[482224340,used=false,source=-1,target=-1,type=-1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,
secondaryUnitId=-1, sFirst, tFirst] not in use

match (n:node_type_A {day: 17016})-[r]->(m) with n, r limit 1000 delete n, r return count(n) as deletedRelsCount

also gave me exactly the same error message. 482224340 may be the ID of a relationship. I guess there is an inconsistency of this particular relationship within Neo4j.

Is there a way (or tool) to delete this "abnormal-shaped" relationship so that I can delete the rest of relationships?
Thanks!

Michael Hunger

unread,
Aug 28, 2016, 6:11:39 PM8/28/16
to ne...@googlegroups.com
Can you run a consistency check on your store?


Thx

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ray Chengr

unread,
Aug 30, 2016, 4:49:09 AM8/30/16
to Neo4j
Hi Michael,

This was the result of consistency check while Neo4j 3.0.3 was stopped:

# java -cp 'lib/*:system/lib/*' org.neo4j.consistency.ConsistencyCheckTool /data/databases/graph.db
Exception in thread "main" java.lang.IllegalArgumentException: Unknown entry type 34 for version 0. At position LogPosition{logVersion=3203, byteOffset=16} and entry version V2_0
    at org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion.entryParser(LogEntryVersion.java:141)
    at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:84)
    at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:53)
    at org.neo4j.kernel.recovery.LatestCheckPointFinder.find(LatestCheckPointFinder.java:77)
    at org.neo4j.kernel.recovery.PositionToRecoverFrom.apply(PositionToRecoverFrom.java:53)
    at org.neo4j.kernel.impl.recovery.RecoveryRequiredChecker.isRecoveryRequiredAt(RecoveryRequiredChecker.java:69)
    at org.neo4j.consistency.ConsistencyCheckTool.checkDbState(ConsistencyCheckTool.java:115)
    at org.neo4j.consistency.ConsistencyCheckTool.run(ConsistencyCheckTool.java:92)
    at org.neo4j.consistency.ConsistencyCheckTool.runConsistencyCheckTool(ConsistencyCheckTool.java:69)
    at org.neo4j.consistency.ConsistencyCheckTool.main(ConsistencyCheckTool.java:56)

Also, neo4j.log in our Neo4j 3.0.3 Community Edition produces these messages continuously while we are inserting relationships (not sure about inserting nodes due to the amount of log lines):

2016-08-30 08:26:36.179+0000 ERROR Failed to commit transaction. Relationship[499972766,used=false,source=-1,target=-1,type=-1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,secondaryUnitId=-1, sFirst, tFirst] not in use
org.neo4j.kernel.impl.store.InvalidRecordException: Relationship[499972766,used=false,source=-1,target=-1,type=-1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,secondaryUnitId=-1, sFirst, tFirst] not in use
    at org.neo4j.kernel.impl.store.record.RecordLoad.verify(RecordLoad.java:63)
    at org.neo4j.kernel.impl.store.CommonAbstractStore.verifyAfterReading(CommonAbstractStore.java:1263)
    at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1047)
    at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1025)
    at org.neo4j.kernel.impl.transaction.state.Loaders$3.load(Loaders.java:200)
    at org.neo4j.kernel.impl.transaction.state.Loaders$3.load(Loaders.java:190)
    at org.neo4j.kernel.impl.transaction.state.RecordChanges.getOrLoad(RecordChanges.java:72)
    at org.neo4j.kernel.impl.transaction.state.RelationshipCreator.connect(RelationshipCreator.java:203)
    at org.neo4j.kernel.impl.transaction.state.RelationshipCreator.connectRelationshipToDenseNode(RelationshipCreator.java:165)
    at org.neo4j.kernel.impl.transaction.state.RelationshipCreator.connectRelationship(RelationshipCreator.java:143)
    at org.neo4j.kernel.impl.transaction.state.RelationshipCreator.relationshipCreate(RelationshipCreator.java:67)
    at org.neo4j.kernel.impl.transaction.state.TransactionRecordState.relCreate(TransactionRecordState.java:282)
    at org.neo4j.kernel.impl.storageengine.impl.recordstorage.TransactionToRecordStateVisitor.visitCreatedRelationship(TransactionToRecordStateVisitor.java:96)
    at org.neo4j.storageengine.api.txstate.TxStateVisitor$Delegator.visitCreatedRelationship(TxStateVisitor.java:227)
    at org.neo4j.kernel.api.txstate.TransactionCountingStateVisitor.visitCreatedRelationship(TransactionCountingStateVisitor.java:108)
    at org.neo4j.kernel.impl.api.state.TxState$15.visitAddedRelationship(TxState.java:376)
    at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter$1.visit(RelationshipChangeVisitorAdapter.java:125)
    at org.neo4j.kernel.impl.api.state.RelationshipStateImpl.accept(RelationshipStateImpl.java:52)
    at org.neo4j.kernel.impl.api.state.TxState.relationshipVisit(TxState.java:1539)
    at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter$1.visit(RelationshipChangeVisitorAdapter.java:115)
    at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter.visitAddedRelationship(RelationshipChangeVisitorAdapter.java:66)
    at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter.visitAdded(RelationshipChangeVisitorAdapter.java:90)
    at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter.visitAdded(RelationshipChangeVisitorAdapter.java:49)
    at org.neo4j.kernel.impl.util.diffsets.SuperDiffSets.accept(SuperDiffSets.java:66)
    at org.neo4j.kernel.impl.util.diffsets.RelationshipDiffSets.accept(RelationshipDiffSets.java:43)
    at org.neo4j.kernel.impl.api.state.TxState.accept(TxState.java:265)
    at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.createCommands(RecordStorageEngine.java:312)
    at org.neo4j.kernel.impl.api.KernelTransactionImplementation.commit(KernelTransactionImplementation.java:437)
    at org.neo4j.kernel.impl.api.KernelTransactionImplementation.close(KernelTransactionImplementation.java:386)
    at org.neo4j.server.rest.transactional.TransitionalTxManagementKernelTransaction.commit(TransitionalTxManagementKernelTransaction.java:92)
    at org.neo4j.server.rest.transactional.TransactionHandle.closeContextAndCollectErrors(TransactionHandle.java:243)
    at org.neo4j.server.rest.transactional.TransactionHandle.commit(TransactionHandle.java:151)
    at org.neo4j.server.rest.web.TransactionalService.lambda$executeStatementsAndCommit$28(TransactionalService.java:202)
    at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
    at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
    at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
    at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
    at org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(CollectUserAgentFilter.java:69)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:497)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)

Is there a way to fix our Neo4j?

Thanks,
Ray

Michael Hunger

unread,
Aug 30, 2016, 6:12:33 AM8/30/16
to ne...@googlegroups.com
Can you tell me more about your use of Neo4j and the history of this store? Were there crashes, out of disk situations or hard kills? Or any other unusual incidents?
Perhaps you can even share a compressed copy with us / our engineering so that we can do some forensics?

To recover/repair, try my store utils to transform the store to a fixed copy, while skipping and logging the invalid records.


Michael

To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+unsubscribe@googlegroups.com.

Ray Chengr

unread,
Aug 30, 2016, 6:35:22 AM8/30/16
to Neo4j

Ray Chengr

unread,
Aug 30, 2016, 6:39:43 AM8/30/16
to Neo4j
Hi Michael,
Sorry, I copy-pasted wrong text in my last reply two minutes ago. Please ignore that one.

We do insertion at rate about 250 relationships&nodes per second(!) on a single server. The disk usage is as follows: (422GB, it's too big to send)

[root@sjc graph.db]# du -s *                 /* this is in Kbytes */
4    index
8    neostore
4    neostore.counts.db.a
4    neostore.counts.db.b
4    neostore.id
8    neostore.labeltokenstore.db
4    neostore.labeltokenstore.db.id
8    neostore.labeltokenstore.db.names
4    neostore.labeltokenstore.db.names.id
185148    neostore.nodestore.db
4    neostore.nodestore.db.id
8    neostore.nodestore.db.labels
4    neostore.nodestore.db.labels.id
89701060    neostore.propertystore.db
332845360    neostore.propertystore.db.arrays
4    neostore.propertystore.db.arrays.id
4    neostore.propertystore.db.id
8    neostore.propertystore.db.index
4    neostore.propertystore.db.index.id
8    neostore.propertystore.db.index.keys
4    neostore.propertystore.db.index.keys.id
1336196    neostore.propertystore.db.strings
4    neostore.propertystore.db.strings.id
29636    neostore.relationshipgroupstore.db
4    neostore.relationshipgroupstore.db.id
16814260    neostore.relationshipstore.db
88    neostore.relationshipstore.db.id
8    neostore.relationshiptypestore.db
4    neostore.relationshiptypestore.db.id
8    neostore.relationshiptypestore.db.names
4    neostore.relationshiptypestore.db.names.id
8    neostore.schemastore.db
4    neostore.schemastore.db.id
214076    neostore.transaction.db.3203
1302104    schema
0    store_lock
[root@ddna-sjc graph.db]# du -s .         /* also in Kbytes */
442428080    .
[root@ddna-sjc graph.db]# du -s -h .
422G    .


I started insertion and got these same messages from neo4j.log continuously, even with the same relationship ID (?) 500676928.

2016-08-30 10:19:02.252+0000 ERROR Failed to commit transaction. Relationship[500676928,used=false,source=-1,target=-1,type=-1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,secondaryUnitId=-1, sFirst, tFirst] not in use
org.neo4j.kernel.impl.store.InvalidRecordException: Relationship[500676928,used=false,source=-1,target=-1,type=-1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,secondaryUnitId=-1, sFirst, tFirst] not in use

Michael Hunger

unread,
Aug 30, 2016, 6:42:54 AM8/30/16
to ne...@googlegroups.com, Mattias Persson
Anything more about the history? Upgrades from previous versions?

Since when are you getting these errors?

Only now or also before?

Which version is it?

Michael

To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+unsubscribe@googlegroups.com.

Ray Chengr

unread,
Aug 30, 2016, 2:25:01 PM8/30/16
to Neo4j, mat...@neotechnology.com
I noticed this kind of serious errors occurred two days ago when I tried to use Cypher to delete relationships. We also brought up and down Neo4j 3.0.3 a few times last Thursday.

We didn't upgrade any data. We started fresh on Neo4j 3.0.3.

Is there a few such as the old Unix fsck to repair DB?
Luckily we are not in production yet.

Ray
Reply all
Reply to author
Forward
0 new messages