Problems Initializing Neo Graph in AWS

[Andrew Guldman]

We are seeing intermittent failures during the initialization of a neo4j graph running in Gremlin Server. We create 7 different graphs that are very similar, and one or two of them intermittently fail. Different ones fail at different times. I haven't been able to figure out a root source or a pattern yet.

It looks like the root exception is:

java.lang.RuntimeException: Unable to read neo store header information, read: 15B, expected 54B

Ring any bells? Got any advice?

The server is running in AWS using a t2.medium ec2 instance with an attached EBS volume. I have not  observed this problem when running locally on my Mac Book Pro.

The gremlin-server-xps-http.yaml and neo4j-sandbox-ag.properties configuration files are attached. The Gremlin Server log is shown below.

Thanks in advance for your help!

Cheers,

Andrew

         \,,,/

         (o o)

-----oOOo-(3)-oOOo-----

12:02:54.290 [INFO] GremlinServer - Configuring Gremlin Server from /usr/local/fluid/product/xps-graphdb.fluidretail.net/vendor/tinkerpop3/greml

in-server/target/gremlin-server-3.0.0.M6-standalone/conf/gremlin-server-xps-http.yaml

12:02:54.338 [INFO] MetricManager - Configured Metrics Slf4jReporter configured with interval=180000ms and loggerName=com.tinkerpop.gremlin.serv

er.Settings$Slf4jReporterMetrics

12:02:54.968 [WARN] Graphs - Graph [g_sandbox_ag] configured at [conf/neo4j-sandbox-ag.properties] could not be instantiated and will not be ava

ilable in Gremlin Server.  GraphFactory message: GraphFactory could not instantiate this Graph implementation [com.tinkerpop.gremlin.neo4j.struc

ture.Neo4jGraph]

java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.tinkerpop.gremlin.neo4j.structure.Neo4jGraph]

        at com.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:61)

        at com.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:80)

        at com.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:96)

        at com.tinkerpop.gremlin.server.Graphs.lambda$new$10(Graphs.java:29)

        at com.tinkerpop.gremlin.server.Graphs$$Lambda$8/1308244637.accept(Unknown Source)

        at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:663)

        at com.tinkerpop.gremlin.server.Graphs.<init>(Graphs.java:27)

        at com.tinkerpop.gremlin.server.GremlinServer.initializeGremlinExecutor(GremlinServer.java:121)

        at com.tinkerpop.gremlin.server.GremlinServer.run(GremlinServer.java:74)

        at com.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:176)

Caused by: java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:483)

        at com.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:57)

        ... 9 more

Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, /mnt/gremlin/sandbox-ag/neo4j

        at com.tinkerpop.gremlin.neo4j.structure.Neo4jGraph.<init>(Neo4jGraph.java:133)

        at com.tinkerpop.gremlin.neo4j.structure.Neo4jGraph.open(Neo4jGraph.java:148)

        ... 14 more

Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, /mnt/gremlin/sandbox-ag/neo4j

        at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:366)

        at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:59)

        at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:91)

        at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:181)

        at com.tinkerpop.gremlin.neo4j.structure.Neo4jGraph.<init>(Neo4jGraph.java:111)

        ... 15 more

Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.transaction.XaDataSourceManager@10feca44' was success

fully initialized, but failed to start. Please see attached cause exception.

        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:513)

        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115)

        at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:343)

        ... 19 more

Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource@24fcf36f' was successfully initialized, but failed to start. Please see attached cause exception.

        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:513)

        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115)

        at org.neo4j.kernel.impl.transaction.XaDataSourceManager.start(XaDataSourceManager.java:164)

        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507)

        ... 21 more

Caused by: java.lang.RuntimeException: Unable to read neo store header information, read: 15B, expected 54B

        at org.neo4j.kernel.impl.nioneo.store.NeoStoreUtil.<init>(NeoStoreUtil.java:64)

        at org.neo4j.kernel.impl.storemigration.StoreMigrator.needsMigration(StoreMigrator.java:130)

        at org.neo4j.kernel.impl.storemigration.StoreUpgrader.getParticipantsEagerToMigrate(StoreUpgrader.java:268)

        at org.neo4j.kernel.impl.storemigration.StoreUpgrader.migrateIfNeeded(StoreUpgrader.java:143)

        at org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.start(NeoStoreXaDataSource.java:344)

        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507)

        ... 24 more

[Mike Hobbs]

I looked into the source code for NeoStoreUtil, but as it's written, it's hard to tell whether the file is actually truncated or if FileChannel.read() simply didn't read all of the the bytes requested. The NeoStoreUtil constructor calls FileChannel.read() (via a StoreFileChannel) and then throws an exception if the read operation doesn't read in all of the bytes requested. However, the FileChannel.read() API explicitly denies making any guarantee that it will read all of the data. It may be possible that the read operation is returning a shortened result if the kernel would have to block the process to get more bytes from the EBS volume.

[Stephen Mallette]

I can't say I've every seen that error, but it looks like something specific to Neo4j.  You might consider trying to re-create it without Gremlin Server/TP3 wrapped around it and post a question to the Neo4j mailing list if you don't find resolution.  You might also consider your configuration for each of your Neo4j databases.  As you are running multiple embedded databases on a single machine there may be some competition for resources unless that is controlled.  I guess I would expect that scenario to manifest as a out of memory exception or something similar, but perhaps it is related.

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/6fbbe6df-3ac7-4aec-ab4c-dd9d73d83ffa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.