Unknown Entry?

[Sean Bowman]

We had a problem running out of disk space on the partition Neo4j was running.  When we cleaned out some space and restarted, this error kept coming up:

Dec 14, 2012 7:40:30 PM org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery

INFO: Non clean shutdown detected on log [/var/lib/singularity/neo4j/nioneo_logical.log.2]. Recovery started ...

Exception in thread "main" java.lang.RuntimeException: Unexpected Exception

at org.neo4j.helpers.Exceptions.launderedException(Exceptions.java:78)

at org.neo4j.helpers.Exceptions.launderedException(Exceptions.java:48)

at org.neo4j.helpers.Exceptions.launderedException(Exceptions.java:43)

at org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.<init>(NeoStoreXaDataSource.java:282)

at org.neo4j.kernel.InternalAbstractGraphDatabase.create(InternalAbstractGraphDatabase.java:418)

at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:223)

at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:79)

at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:70)

at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:55)

...

Caused by: java.io.IOException: Unknown entry[82]

at org.neo4j.kernel.impl.transaction.xaframework.LogIoUtils.readLogEntry(LogIoUtils.java:120)

at org.neo4j.kernel.impl.transaction.xaframework.LogIoUtils.readEntry(LogIoUtils.java:91)

at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.readEntry(XaLogicalLog.java:859)

at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.doInternalRecovery(XaLogicalLog.java:791)

at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.open(XaLogicalLog.java:205)

at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.open(XaLogicalLog.java:167)

at org.neo4j.kernel.impl.transaction.xaframework.XaContainer.openLogicalLog(XaContainer.java:64)

at org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.<init>(NeoStoreXaDataSource.java:244)

... 24 more

Are we just hosed, and have to rebuild the database from scratch?  My guess is that the log file tried to write out some information, got half-way through, and ran out of space.  Thus the log file is in an incomplete state and confusing Neo4j.  If this is the case, then I'd say this is an extremely serious bug.

[Sean Bowman]

Sorry, forgot to add we're running 1.8 embedded with Java 7 on Ubuntu 12.04 64-bit (Amazon EC2 instance).

[Peter Neubauer]

Sean,
do you have that store privately so we could take a look at it, or at
least the logs?

/peter

Cheers,

/peter neubauer

G: neubauer.peter
S: peter.neubauer
P: +46 704 106975
L: http://www.linkedin.com/in/neubauer
T: @peterneubauer

Neo4j 1.8 GA - http://www.dzone.com/links/neo4j_18_release_fluent_graph_literacy.html

> --
>
>

[Mattias Persson]

I believe this is a problem which was fixed recently and is provably available in the latest 1.8 maintenance release and 1.9 milestone

--

[Jacob Hansson]

+1, this is was an issue discovered in 1.8, and fixed in 1.8.1. Really sorry for the pain caused by it :( 

Your options are essentially to restore from a backup (or rebuild database if you have that option), or attempt restoring the database up to the last correct commit entry in the logical log. The latter option is available either by bringing in Neo4j consultants, getting a support contract, or attempting to use internal tools yourself to restore it. 

/jake

[Radu Kopetz]

I am having the same problem with 1.9 release. After running out of space, I can't restart Neo4j. The message says exactly the same thing:

SEVERE:

java.lang.RuntimeException: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.transaction.XaDataSourceManager@2a43dfc2' was successfully initialized, but failed to start. Please see attached cause exception.

        at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:281)

        at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:106)

        at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:88)

        at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:207)

        at org.neo4j.kernel.impl.recovery.StoreRecoverer.recover(StoreRecoverer.java:114)

        at org.neo4j.server.preflight.PerformRecoveryIfNecessary.run(PerformRecoveryIfNecessary.java:59)

        at org.neo4j.server.preflight.PreFlightTasks.run(PreFlightTasks.java:70)

        at org.neo4j.server.AbstractNeoServer.runPreflightTasks(AbstractNeoServer.java:279)

        at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:159)

        at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:86)

        at org.neo4j.server.Bootstrapper.main(Bootstrapper.java:49)

Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.transaction.XaDataSourceManager@2a43dfc2' was successfully initialized, but failed to start. Please see attached cause exception.

        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:497)

        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:104)

        at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:259)

        ... 10 more

Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource@67dc1e2' was successfully initialized, but failed to start. Please see attached cause exception.

        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:497)

        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:104)

        at org.neo4j.kernel.impl.transaction.XaDataSourceManager.start(XaDataSourceManager.java:128)

        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:491)

        ... 12 more

Caused by: java.lang.RuntimeException: Unexpected Exception

        at org.neo4j.helpers.Exceptions.launderedException(Exceptions.java:78)

        at org.neo4j.helpers.Exceptions.launderedException(Exceptions.java:48)

        at org.neo4j.helpers.Exceptions.launderedException(Exceptions.java:43)

        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.start(NeoStoreXaDataSource.java:289)

        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:491)

        ... 15 more

Caused by: java.io.IOException: Unknown entry[79]

        at org.neo4j.kernel.impl.transaction.xaframework.LogIoUtils.readLogEntry(LogIoUtils.java:120)

        at org.neo4j.kernel.impl.transaction.xaframework.LogIoUtils.readEntry(LogIoUtils.java:91)

        at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.readEntry(XaLogicalLog.java:833)

        at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.doInternalRecovery(XaLogicalLog.java:769)

        at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.open(XaLogicalLog.java:206)

        at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.open(XaLogicalLog.java:168)

        at org.neo4j.kernel.impl.transaction.xaframework.XaContainer.openLogicalLog(XaContainer.java:64)

        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.start(NeoStoreXaDataSource.java:251)

        ... 16 more

Is there any way of restarting the database ? I don't care if I loose some of the last transactions, I just want to be able to recover to some point in the past so that I don't have to rebuild everything from scratch.

Thanks,

Radu

[Radu Kopetz]

I managed to solve the problem by moving away the file "nioneo_logical.log.active". Is there any tool I can use to see what is in this log and what was the data that was lost ?

Thanks,

Radu

--
You received this message because you are subscribed to a topic in the Google Groups "Neo4j" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/neo4j/IQtrZBJD9r0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Michael Hunger]

That file only points to the active log (1 or 2)

The logfile contains the transactions that have been

- finished

- only committed

- started

The first two are applied to the store during recovery

There are some internal tools for dumping logfile contents. Nothing public I'm afraid.

If you send us your db (dropbox)  we can investigate and help you figure out what's missing

Michael

Sent from mobile device

You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.

[Radu Kopetz]

Thank you for the clarifications Michael and for your help proposal. It's nothing critical, only test data for the moment, so no need to loose time with it. I was asking more for the future if this problem comes back to me later.

Regards,

Radu