Neo4j cant't start after wrong shutdown: PageCache error

[Péterson Sampaio Procópio Júnior]

Hello all,

after what was probably a wrong shutdown, I am failing to start Neo4j server. The error occurred in the 2.2.0 windows version of Neo4j. I upgraded to 2.2.2, but it still remains.
The relevant log message follows below. I think the error is related, as the message says, to the cache mechanism.

2015-06-08 19:48:05.251+0000 ERROR [o.n.s.d.LifecycleManagingDatabase]: Failed to start database.
org.neo4j.kernel.lifecycle.LifecycleException: Failed to transition component 'org.neo4j.kernel.impl.pagecache.PageCacheLifecycle@5f8dbdab' from STOPPED to SHUTTING_DOWN. Please see attached cause exception
...
Caused by: java.lang.IllegalStateException: Cannot close the PageCache while files are still mapped:
    neostore.propertystore.db.index (1 mapping)
    neostore.propertystore.db.index.keys (1 mapping)
    neostore.labeltokenstore.db (1 mapping)
    neostore.labeltokenstore.db.names (1 mapping)
    neostore.relationshiptypestore.db (1 mapping)
    neostore.relationshiptypestore.db.names (1 mapping)
    at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.close(MuninnPageCache.java:483) ~[neo4j-io-2.2.2.jar:2.2.2]
    at org.neo4j.kernel.impl.pagecache.PageCacheLifecycle.shutdown(PageCacheLifecycle.java:42) ~[neo4j-kernel-2.2.2.jar:2.2.2]
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.shutdown(LifeSupport.java:555) [neo4j-kernel-2.2.2.jar:2.2.2]
    ... 11 common frames omitted

The most similar situation I found, and it's solution, are described in this stackoverflow answer:

I can think of two causes here:

1. another java process is accessing some of the files, unless something other helps consider a kill -9 <pid>
2. double check the file permissions in your graph.db folder.

However, this seems to not be the same case I am running into.

Could anyone help me with this?

I attached the relavant pieces of my messages.log.

Thank you,

Péterson

[Fyndee search]

Hi Peterson, 

I ran into the same issue as you did i.e. "Caused by: java.lang.IllegalStateException: Cannot close the PageCache while files are still mapped"

I asked a question here in the group but no one answered yet. In case you manage to find a solution for your problem, can you please let me know?

These kind of crashes and lack of support from neo4j staff makes me think, I should walk away from Neo4j and revisit it at a later time.  

Thanks in advance!

VJ

[Péterson Sampaio Procópio Júnior]

Thank you,

Péterson

[Mattias Persson]

I think that the exception seen here is hiding the real exception. Basically something goes wrong during startup and so neo4j tries to shut down and back out before throwing that exception out to the user. Shutting down somehow results in this exception and so that gets thrown instead of the real exception. I'll have a look at trying to reproduce with dummy exceptions at different points in startup.

[Fyndee search]

Thanks a lot Mattias! I look forward to hearing from you once you manage to reproduce this issue. 

@ Péterson : Did you encounter this problem in windows environment like I did?

[Péterson Sampaio Procópio Júnior]

Hello Mattias and Fyndee,

thank you a lot for the feedback.

After a while, I stopped trying to figure out what the issue was and simply rebuilt my database. I know it is not the best solution.

If I encounter the error again I will post in this thread.

Best regards,

Péterson

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rik Van Bruggen]

It seems like the most common cause for this is incorrect file permissions on the store files. Have you checked these?

https://neo4j.com/developer/kb/how-to-fix-cannot-close-the-pagecache-while-files-are-still-mapped/

Rik

On Tuesday, June 9, 2015 at 1:02:26 AM UTC+2, Péterson Sampaio Procópio Júnior wrote:

[asurya...@kiwiup.com]

I'm getting a similar error while starting database. Please LMK if anyone has a solution. My logs:

2018-06-19 06:28:34.954+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.kernel.impl.pagecache.PageCacheLifecycle@45cff11c' failed to transition from stopped to shutting_down. Please see the attached cause exception "Cannot close the PageCache while files are still mapped:
	/Users/aniket/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-c8f9681c-f456-4ca6-ab41-423f5e91b0b1/installation-3.3.5/data/databases/graph.db/neostore.nodestore.db.labels (1 mapping)". Starting Neo4j failed: Component 'org.neo4j.kernel.impl.pagecache.PageCacheLifecycle@45cff11c' failed to transition from stopped to shutting_down. Please see the attached cause exception "Cannot close the PageCache while files are still mapped:
	/Users/aniket/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-c8f9681c-f456-4ca6-ab41-423f5e91b0b1/installation-3.3.5/data/databases/graph.db/neostore.nodestore.db.labels (1 mapping)".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.kernel.impl.pagecache.PageCacheLifecycle@45cff11c' failed to transition from stopped to shutting_down. Please see the attached cause exception "Cannot close the PageCache while files are still mapped:
	/Users/aniket/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-c8f9681c-f456-4ca6-ab41-423f5e91b0b1/installation-3.3.5/data/databases/graph.db/neostore.nodestore.db.labels (1 mapping)".
	at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
	at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:220)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:111)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:79)
	at com.neo4j.server.enterprise.CommercialEntryPoint.main(CommercialEntryPoint.java:22)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.pagecache.PageCacheLifecycle@45cff11c' failed to transition from stopped to shutting_down. Please see the attached cause exception "Cannot close the PageCache while files are still mapped:
	/Users/aniket/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-c8f9681c-f456-4ca6-ab41-423f5e91b0b1/installation-3.3.5/data/databases/graph.db/neostore.nodestore.db.labels (1 mapping)".
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.shutdown(LifeSupport.java:519)
	at org.neo4j.kernel.lifecycle.LifeSupport.shutdown(LifeSupport.java:206)
	at org.neo4j.kernel.impl.factory.ClassicCoreSPI.shutdown(ClassicCoreSPI.java:175)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.shutdown(GraphDatabaseFacade.java:387)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:220)
	at org.neo4j.kernel.enterprise.EnterpriseGraphDatabase.<init>(EnterpriseGraphDatabase.java:36)
	at org.neo4j.server.enterprise.OpenEnterpriseNeoServer.lambda$static$1(OpenEnterpriseNeoServer.java:75)
	at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:88)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
	at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:212)
	... 3 more
	Suppressed: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory, /Users/aniket/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-c8f9681c-f456-4ca6-ab41-423f5e91b0b1/installation-3.3.5/data/databases/graph.db
		at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:211)
		... 9 more
	Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.NeoStoreDataSource@7be58f16' was successfully initialized, but failed to start. Please see the attached cause exception "Externally locked: /Users/aniket/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-c8f9681c-f456-4ca6-ab41-423f5e91b0b1/installation-3.3.5/data/databases/graph.db/neostore.nodestore.db".
		at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:466)
		at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
		at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:100)
		at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)
		at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
		at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:207)
		... 9 more
	Caused by: org.neo4j.kernel.impl.store.UnderlyingStorageException: Unable to open store file: /Users/aniket/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-c8f9681c-f456-4ca6-ab41-423f5e91b0b1/installation-3.3.5/data/databases/graph.db/neostore.nodestore.db
		at org.neo4j.kernel.impl.store.CommonAbstractStore.checkAndLoadStorage(CommonAbstractStore.java:210)
		at org.neo4j.kernel.impl.store.CommonAbstractStore.initialise(CommonAbstractStore.java:134)
		at org.neo4j.kernel.impl.store.NeoStores.initialize(NeoStores.java:266)
		at org.neo4j.kernel.impl.store.NeoStores.createNodeStore(NeoStores.java:543)
		at org.neo4j.kernel.impl.store.StoreType$2.open(StoreType.java:47)
		at org.neo4j.kernel.impl.store.StoreType$2.open(StoreType.java:42)
		at org.neo4j.kernel.impl.store.NeoStores.openStore(NeoStores.java:259)
		at org.neo4j.kernel.impl.store.NeoStores.getOrCreateStore(NeoStores.java:304)
		at org.neo4j.kernel.impl.store.NeoStores.<init>(NeoStores.java:148)
		at org.neo4j.kernel.impl.store.StoreFactory.openNeoStores(StoreFactory.java:160)
		at org.neo4j.kernel.impl.store.StoreFactory.openAllNeoStores(StoreFactory.java:124)
		at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.<init>(RecordStorageEngine.java:200)
		at org.neo4j.kernel.NeoStoreDataSource.buildStorageEngine(NeoStoreDataSource.java:587)
		at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:449)
		at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)
		... 14 more
	Caused by: org.neo4j.io.pagecache.impl.FileLockException: Externally locked: /Users/aniket/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-c8f9681c-f456-4ca6-ab41-423f5e91b0b1/installation-3.3.5/data/databases/graph.db/neostore.nodestore.db
		at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.acquireLock(SingleFilePageSwapper.java:208)
		at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.<init>(SingleFilePageSwapper.java:159)
		at org.neo4j.io.pagecache.impl.SingleFilePageSwapperFactory.createPageSwapper(SingleFilePageSwapperFactory.java:71)
		at org.neo4j.io.pagecache.impl.muninn.MuninnPagedFile.<init>(MuninnPagedFile.java:147)
		at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.map(MuninnPageCache.java:341)
		at org.neo4j.kernel.impl.store.CommonAbstractStore.checkAndLoadStorage(CommonAbstractStore.java:183)
		... 28 more
Caused by: java.lang.IllegalStateException: Cannot close the PageCache while files are still mapped:
	/Users/aniket/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-c8f9681c-f456-4ca6-ab41-423f5e91b0b1/installation-3.3.5/data/databases/graph.db/neostore.nodestore.db.labels (1 mapping)
	at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.close(MuninnPageCache.java:580)
	at org.neo4j.kernel.impl.pagecache.PageCacheLifecycle.shutdown(PageCacheLifecycle.java:42)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.shutdown(LifeSupport.java:511)
	... 13 more

[Mattias Finné]

`org.neo4j.io.pagecache.impl.FileLockException: Externally locked: /Users/aniket/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-c8f9681c-f456-4ca6-ab41-423f5e91b0b1/installation-3.3.5/data/databases/graph.db/neostore`

means that `neostore` file seems to be used by another process. That's strange because typically a higher level store lock should prevent the db from trying to start and start mapping its files before this. Are there any processes alive that may use that file somehow?

--
You received this message because you are subscribed to a topic in the Google Groups "Neo4j" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/neo4j/Xx420_ggHRo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to neo4j+unsubscribe@googlegroups.com.

[Aniket Suryavanshi]

@Mattias Yes ... 2 processes (java) were using 2 ports where Neo4j usually runs. These processes also might've been using the `neostore` file. Killing the processes worked for me.

To unsubscribe from this group and all its topics, send an email to neo4j+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

- Aniket