exception shutting down db after 200M property writes

[John Fry]

Hi All,

can someone help me decipher this exception below.

Some background:

• I have 16 threads running batches that create properties on relationships
• There are about 200M relationships
• I am batching about 100K property creations per transaction before i close a transaction
• all threads and transaction complete without error (the properties appear in the .db when I manually check in the neo4j shell)
• the error occurs when calling db.shutdown(); 

Thanks John.

Exception in thread "main" org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.NeoStoreDataSource@7c1110d4' failed to stop. Please see attached cause exception.

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.stop(LifeSupport.java:560)

at org.neo4j.kernel.lifecycle.LifeSupport.stopInstances(LifeSupport.java:161)

at org.neo4j.kernel.lifecycle.LifeSupport.stop(LifeSupport.java:143)

at org.neo4j.kernel.impl.transaction.state.DataSourceManager.stop(DataSourceManager.java:132)

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.stop(LifeSupport.java:552)

at org.neo4j.kernel.lifecycle.LifeSupport.stopInstances(LifeSupport.java:161)

at org.neo4j.kernel.lifecycle.LifeSupport.stop(LifeSupport.java:143)

at org.neo4j.kernel.lifecycle.LifeSupport.shutdown(LifeSupport.java:184)

at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.shutdown(GraphDatabaseFacade.java:285)

at PMIResultsToDB.threadedWrToDB(Unknown Source)

at PMIResultsToDB.main(Unknown Source)

Suppressed: org.neo4j.kernel.lifecycle.LifecycleException: Failed to transition component 'org.neo4j.kernel.impl.pagecache.PageCacheLifecycle@3fe0bec3' from STOPPED to SHUTTING_DOWN. Please see attached cause exception

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.shutdown(LifeSupport.java:591)

at org.neo4j.kernel.lifecycle.LifeSupport.shutdown(LifeSupport.java:199)

... 3 more

Caused by: java.lang.IllegalStateException: Cannot close the PageCache while files are still mapped:

neostore.counts.db.a (1 mapping)

neostore (1 mapping)

neostore.relationshipgroupstore.db (1 mapping)

neostore.schemastore.db (1 mapping)

neostore.labeltokenstore.db (1 mapping)

neostore.labeltokenstore.db.names (1 mapping)

neostore.relationshiptypestore.db (1 mapping)

neostore.relationshiptypestore.db.names (1 mapping)

neostore.relationshipstore.db (1 mapping)

neostore.propertystore.db (1 mapping)

neostore.propertystore.db.arrays (1 mapping)

neostore.propertystore.db.strings (1 mapping)

neostore.propertystore.db.index (1 mapping)

neostore.propertystore.db.index.keys (1 mapping)

neostore.nodestore.db (1 mapping)

neostore.nodestore.db.labels (1 mapping)

at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.close(MuninnPageCache.java:519)

at org.neo4j.kernel.impl.pagecache.PageCacheLifecycle.shutdown(PageCacheLifecycle.java:42)

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.shutdown(LifeSupport.java:583)

... 4 more

Caused by: java.lang.IllegalStateException: Weird, inUse was read in as true but the record is Property[134469435,used=false,prev=577553732818370560,next=1657324675757244416,node=288230376151711744]

at org.neo4j.kernel.impl.transaction.command.PhysicalLogCommandReaderV1_9$PhysicalNeoCommandReader.readPropertyRecord(PhysicalLogCommandReaderV1_9.java:500)

at org.neo4j.kernel.impl.transaction.command.PhysicalLogCommandReaderV1_9$PhysicalNeoCommandReader.visitPropertyCommand(PhysicalLogCommandReaderV1_9.java:199)

at org.neo4j.kernel.impl.transaction.command.Command$PropertyCommand.handle(Command.java:288)

at org.neo4j.kernel.impl.transaction.command.PhysicalLogCommandReaderV1_9.read(PhysicalLogCommandReaderV1_9.java:123)

at org.neo4j.kernel.impl.transaction.log.entry.LogEntryParsersV1_9$4.parse(LogEntryParsersV1_9.java:129)

at org.neo4j.kernel.impl.transaction.log.entry.LogEntryParsersV1_9$4.parse(LogEntryParsersV1_9.java:122)

at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:107)

at org.neo4j.kernel.NeoStoreDataSource$10.getTimestampForVersion(NeoStoreDataSource.java:863)

at org.neo4j.kernel.impl.transaction.log.PhysicalLogFileInformation.getFirstStartRecordTimestamp(PhysicalLogFileInformation.java:92)

at org.neo4j.kernel.impl.transaction.log.pruning.TransactionTimespanThreshold.reached(TransactionTimespanThreshold.java:54)

at org.neo4j.kernel.impl.transaction.log.pruning.ThresholdBasedPruneStrategy.prune(ThresholdBasedPruneStrategy.java:67)

at org.neo4j.kernel.impl.transaction.log.pruning.LogPruningImpl.pruneLogs(LogPruningImpl.java:54)

at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:188)

at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.forceCheckPoint(CheckPointerImpl.java:87)

at org.neo4j.kernel.NeoStoreDataSource.stop(NeoStoreDataSource.java:1213)

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.stop(LifeSupport.java:552)

... 10 more

[John Fry]

Quick addition - this is on a 120GB machine and the app runs steady state consuming not more than 70GB of mem.

[John Fry]

Turning off the transaction logs fixes this. 

Is this a known issue in ver 2.0.3?

[Michael Hunger]

Veesion 2.0.3 is very old and out of support 

Any chance that you could update to a recent version like 3.0.4?

Also 2.0.3 doesn't have a page cache, so sth is off. 

I think this issue was fixed recently. 

Caused by: java.lang.IllegalStateException: Cannot close the PageCache while files are still mapped:

neostore.counts.db.a (1 mapping)

neostore (1 mapping)

neostore.relationshipgroupstore.db (1 mapping)

neostore.schemastore.db (1 mapping)

neostore

Von meinem iPhone gesendet

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.