Graph databases, Neo4j Multi threading

[Reihane Boghrati]

Hi,

I'm now working to implement a new BG client for Neo4j. It works well with one thread, but when I increase the number of threads to more than 1, it throws these exception: 

java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, C:\Users\Reihane\Documents\Neo4j\default.graphdb

at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:333)

at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:63)

at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:92)

at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:198)

at org.neo4j.graphdb.factory.GraphDatabaseFactory.newEmbeddedDatabase(GraphDatabaseFactory.java:69)

at Neo4j.Neo4jClient.init(Neo4jClient.java:110)

at edu.usc.bg.base.DBWrapper.init(DBWrapper.java:62)

at edu.usc.bg.base.Client.executeDoLoad(Client.java:1013)

at edu.usc.bg.base.Client.runBG(Client.java:892)

at edu.usc.bg.BGMainClass.main(BGMainClass.java:41)

Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.StoreLockerLifecycleAdapter@2e8da6a9' was successfully initialized, but failed to start. Please see attached cause exception.

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:504)

at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115)

at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:310)

... 9 more

Caused by: org.neo4j.kernel.StoreLockException: Unable to obtain lock on store lock file: C:\Users\Reihane\Documents\Neo4j\default.graphdb\store_lock. Please ensure no other process is using this database, and that the directory is writable (required even for read-only access)

at org.neo4j.kernel.StoreLocker.checkLock(StoreLocker.java:87)

at org.neo4j.kernel.StoreLockerLifecycleAdapter.start(StoreLockerLifecycleAdapter.java:44)

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:498)

... 11 more

Caused by: java.io.IOException: Couldn't lock lock file C:\Users\Reihane\Documents\Neo4j\default.graphdb\lock because another process already holds the lock.

at org.neo4j.kernel.impl.nioneo.store.FileLock.getLockFileBasedFileLock(FileLock.java:115)

at org.neo4j.kernel.impl.nioneo.store.FileLock.getOsSpecificFileLock(FileLock.java:63)

at org.neo4j.kernel.DefaultFileSystemAbstraction.tryLock(DefaultFileSystemAbstraction.java:89)

at org.neo4j.kernel.StoreLocker.checkLock(StoreLocker.java:74)

... 13 more

It seams that with more than one thread, multiple threads try to lock the database and issue queries. Is any one facing the same problem?

[Pradeep Nayak]

Hi Reihane,

I worked with Neo4j last semester as part of my 685 project. Are you
encapsulating all your code within transactions ?


Regards,
Pradeep Nayak

http://pradeepnayak.in
http://twitter.com/_prdp

> --
> You received this message because you are subscribed to the Google Groups
> "BG Social Benchmark" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to BG-SocialBenchm...@googlegroups.com.
> Visit this group at http://groups.google.com/group/BG-SocialBenchmark.
> For more options, visit https://groups.google.com/groups/opt_out.

[Reihane Boghrati]

Yes, all of different codes have their own transaction, like:

Transaction tx = graphDb.beginTx();

try {

    /* some code */

        tx.success();

}

finally{

        tx.finish();

}

I can't even connect to Neo4j with BG, when "Neo4j Community" is running on my laptop.

> email to BG-SocialBenchmark+unsub...@googlegroups.com.

[Pradeep Nayak]

>>I can't even connect to Neo4j with BG, when "Neo4j Community" is running on my laptop.

You should not be running neo4j server while you run BG.

Regards,
Pradeep Nayak

http://pradeepnayak.in
http://twitter.com/_prdp

>> > email to BG-SocialBenchm...@googlegroups.com.

>> > Visit this group at http://groups.google.com/group/BG-SocialBenchmark.
>> > For more options, visit https://groups.google.com/groups/opt_out.
>

> --
> You received this message because you are subscribed to the Google Groups
> "BG Social Benchmark" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to BG-SocialBenchm...@googlegroups.com.

[Reihane Boghrati]

OK, I stop the server, but still I'm getting that exception if I want to load BG with more than 1 thread. 

This is how I connect to Neo4j from init() function in BG:

static GraphDatabaseService graphDb;

private static final String DB_PATH = "C:/Users/Reihane/Documents/Neo4j/default.graphdb"; 

init (){

     graphDb = new GraphDatabaseFactory().newEmbeddedDatabase(DB_PATH);

     //registerShutdownHook(graphDb);

     engine = new ExecutionEngine(graphDb);

>> > email to BG-SocialBenchmark+unsub...@googlegroups.com.

>> > Visit this group at http://groups.google.com/group/BG-SocialBenchmark.
>> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google Groups
> "BG Social Benchmark" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to BG-SocialBenchmark+unsub...@googlegroups.com.

[Pradeep Nayak]

Hi Reihane,

static GraphDatabaseService graphDb;
private static final String DB_PATH = "C:/Users/Reihane/Documents/Neo4j/default.graphdb";

public boolean init() {
        if (graphDb == null) {
            graphDb = new GraphDatabaseFactory().newEmbeddedDatabase(DB_PATH);
        }
        System.out.println("Created graphdb object");
        return true;
}

This should do the trick. The line "engine = new ExecutionEngine" is not required. Also could you let me know how does your cleanup function look like ?

Regards,
Pradeep Nayak

http://pradeepnayak.in
http://twitter.com/_prdp

To unsubscribe from this group and stop receiving emails from it, send an email to BG-SocialBenchm...@googlegroups.com.

[Reihane Boghrati]

I Initialized an execution engine to use for executing Cypher queries (in other functions ).

Still not working with more than one thread :(

This is how shutdown function look like:

void shutDown() {

System.out.println();

System.out.println("Shutting down database ...");

graphDb.shutdown();

}

Also I saw in blogs that they recommended to use function registerShutdownHook(graphDb) to appropriately shut down the db!

This is how its body looks like :

private static void registerShutdownHook(final GraphDatabaseService graphDb) {

// Registers a shutdown hook for the Neo4j instance so that it

// shuts down nicely when the VM exits (even if you "Ctrl-C" the

// running application).

Runtime.getRuntime().addShutdownHook(new Thread() {

@Override

public void run() {

graphDb.shutdown();

}

});

}

I also used this function, but no difference.

Thanks for you following up :)

>> > email to BG-SocialBenchmark+unsubscribe@googlegroups.com.

>> > Visit this group at http://groups.google.com/group/BG-SocialBenchmark.
>> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google Groups
> "BG Social Benchmark" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to BG-SocialBenchmark+unsubscribe@googlegroups.com.

> Visit this group at http://groups.google.com/group/BG-SocialBenchmark.
> For more options, visit https://groups.google.com/groups/opt_out.

[Pradeep Nayak]

Hi Reihane,

Can you try commenting the graphDb.shutdown() line in your cleanup function ? I think I know what may be happening. Let me know if this works ?

>>I Initialized an execution engine to use for executing Cypher queries (in other functions ).

You can get the job done without using Cypher Queries. You could use the Java API from Neo4j to implement all the BG functions.

Regards,
Pradeep Nayak

http://pradeepnayak.in
http://twitter.com/_prdp

To unsubscribe from this group and stop receiving emails from it, send an email to BG-SocialBenchm...@googlegroups.com.

[Reihane Boghrati]

Yay! it worked. Thank you so much :)

But do you know the reason? (Now, I don't have any code to explicitly state shutting down the db!)

[Reihane Boghrati]

In order to shut down the database , I used the shutdown hook and it worked :)

This is the function:

private static void registerShutdownHook(final GraphDatabaseService graphDb) {

// Registers a shutdown hook for the Neo4j instance so that it

// shuts down nicely when the VM exits (even if you "Ctrl-C" the

// running application).

Runtime.getRuntime().addShutdownHook(new Thread() {

@Override

public void run() {

graphDb.shutdown();

}

});

}

You can find the answer here.