Neo4j database ALWAYS shuts down incorrectly if start/stop as a service from a list of windows services

[Denys Hryvastov]

Hi -

I use neo4j as a windows service. It was installed using something like "neo4j.bat install". After that "neo4j database" appeared in the list of windows services and I start/stop neo4j server from there.

But what I have noticed: when I stop neo4j from the list of windows services - it ALWAYS stops incorrectly. When I start neo4j server using another way - run neo4j.bat file, then new java window is opened and is hanging all the time neo4j server is running. After that if I stop neo4j server by pressing Ctrl+C when this Java window is active, then neo4j server stops correctly (next time when I start neo4j, there is no message that neo4j server was incorrectly shut down).

The question is: why does neo4j server always shut down incorrectly if I use neo4j as a service?
It causes a real issue, because I cannot upgrade my database from 1.9.5 to 2.0. It says me that upgrade was not performed because the database was incorrectly shut down previous time...

Could you please tell me how to workaround this, or what I do wrong?

I tried starting neo4j from a list of services and by running neo4j.bat (did comparison of behavior) with 1.9.5, 2.0 and 2.1.2 versions.

Thanks,
Denys

[Denys Hryvastov]

Here is a stack trace that I get when I try to do upgrade from 1.9.5 to 2.0:

2014-06-17 09:48:27.319+0000 INFO  [API] Setting startup timeout to: 120000ms based on -1
Detected incorrectly shut down database, performing recovery..
2014-06-17 09:48:28.108+0000 DEBUG [API]
org.neo4j.server.ServerStartupException: Starting Neo4j Server failed: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, D:\Neo4j\neo4j-enterprise-2.0.0\data\graph.db
        at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:209) ~[neo4j-server-2.0.0.jar:2.0.0]
        at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:87) [neo4j-server-2.0.0.jar:2.0.0]
        at org.neo4j.server.Bootstrapper.main(Bootstrapper.java:50) [neo4j-server-2.0.0.jar:2.0.0]
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, D:\Neo4j\neo4j-enterprise-2.0.0\data\graph.db
        at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:333) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:63) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:92) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:198) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.kernel.impl.recovery.StoreRecoverer.recover(StoreRecoverer.java:115) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.server.preflight.PerformRecoveryIfNecessary.run(PerformRecoveryIfNecessary.java:59) ~[neo4j-server-2.0.0.jar:2.0.0]
        at org.neo4j.server.preflight.PreFlightTasks.run(PreFlightTasks.java:70) ~[neo4j-server-2.0.0.jar:2.0.0]
        at org.neo4j.server.AbstractNeoServer.runPreflightTasks(AbstractNeoServer.java:319) ~[neo4j-server-2.0.0.jar:2.0.0]
        at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:144) ~[neo4j-server-2.0.0.jar:2.0.0]
        ... 2 common frames omitted
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.transaction.XaDataSourceManager@2b1eb67d' was successfully initialized, but failed to start.
Please see attached cause exception.
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:504) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:310) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        ... 10 common frames omitted
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource@1bf5df6a' was successfully initialized, but failed to start. P
lease see attached cause exception.
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:504) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.kernel.impl.transaction.XaDataSourceManager.start(XaDataSourceManager.java:164) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:498) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        ... 12 common frames omitted
Caused by: org.neo4j.kernel.impl.storemigration.StoreUpgrader$UpgradingStoreVersionNotFoundException: 'neostore' does not contain a store version, please ensure that the original datab
ase was shut down in a clean state.
        at org.neo4j.kernel.impl.storemigration.UpgradableDatabase.checkUpgradeable(UpgradableDatabase.java:85) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.kernel.impl.storemigration.StoreUpgrader.attemptUpgrade(StoreUpgrader.java:72) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.kernel.impl.nioneo.store.StoreFactory.tryToUpgradeStores(StoreFactory.java:143) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.kernel.impl.nioneo.store.StoreFactory.newNeoStore(StoreFactory.java:123) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.start(NeoStoreXaDataSource.java:323) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:498) ~[neo4j-kernel-2.0.0.jar:2.0.0]
        ... 15 common frames omitted
2014-06-17 09:48:28.109+0000 DEBUG [API] Failed to start Neo Server on port [7474]

[Jim Salmons]

Hi Denys,

I think you're experience is a variation of mine as related here. Seems to be a long-standing issue that hasn't bitten too many but is problematic for the 2.1 database migration. There are some tips/insight on the link. The Neo folks are aware of the issue so I expect we'll see a fix at some point, maybe soon.

Have you tried the "Ole Out and In" -- dump it from your 'corrupt' DB and load it in an empty 2.1? I had a number of tiny to small research and self-learning DBs that it was so much easier to go out and back in on a fresh 2.1 store.

That tip about timeouts might help you, too. And the Neo4j Mojo that I am not qualified to comment on and I suspect is a big factor has to do with leveraging 2.0+ version indexing and constraints, etc. There might be some tweaks to the schema that you can make before migrating that, along with the increased timeout value, might give the migration process the room to work its one-time-only procedure.

It's a painful problem but, as you can imagine, hits a relative small segment of the broader Neo4j community. BTW, if you run Neo4j as a Windows Service, have you tried my mini Control Panel? :-) http://jim-salmons.github.io/neo4jcp/

I've subscribed to this thread and will let you know if I learn anything more, etc.

--Jim--

www.FactMiners.org and www.SoftalkApple.com

[Denys Hryvastov]

Thank you, Jim!

Your post was really very useful for me, it has given me some more thoughts and information.
I will continue with my experiments and will post here the results that I will get.

-Denys

[Denys Hryvastov]

Ok,

After having an email thread with neo4j support team (thanks them for quick and full responses for all my questions) I have decided to use neo4jshell.bat as a workaround for closing the database properly.
Also, it is important to note that neo4j 1.9.5 cannot be directly upgraded to 2.1.2. 1.9.5 must be upgraded to 2.0.3 first, then to 2.1.2.

Thanks,
Denys

[Bill Scheidel]

I ran into this same problem trying to upgrade from 2.0.1 to 2.1.2.  

[Michael Hunger]

Hey,

the problem is related on how windows services are hard-killed after a too short timeout.

It's planned to be worked on resolving this issue, but I can give no date / release on when it will be fixed.

So for now, if you need a cleanly shut down database (e.g. for an upgrade), please use the shell to get a cleanly shutdown db, like this:

Neo4jShell.bat -path \path\to\graph.db -c quit 

If you installed the neo4j installer for windows, either download the zip-distribution which contains the shell or have a look at Mark's blog post:

http://www.markhneedham.com/blog/2013/11/29/neo4j-the-case-of-windows-neo4j-desktop-and-the-invisible-text/

Cheers,

Michael

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.