Poor performance.

[HasanB]

Hi again,

I now have my extension with some of the functionality I want, but
find it to be unusably slow.

I load an owl file of 140 vertices and create 450 edges.

** It takes a minute and a half! **

I ran it with VisualVM monitoring execution and discovered the
following :

68 seconds --
org.neo4j.kernel.impl.transaction.xaframework.DirectMappedLogBuffer.force()

65 seconds --
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run()

13 seconds --
org.apache.lucene.store.FSDirectory.deleteFile(String)

7 seconds --
org.apache.lucene.store.FSDirectory
$FSIndexOutput.<init>(org.apache.lucene.store.FSDirectory, String)

5 seconds --
sun.nio.ch.FileChannelImpl$Unmapper.run()

This is for a single curl command to Rexster in localhost. WinXP
64bit. Core2 Quad CPU. 8GB RAM.

Obviously I'm making a fundamental mistake. Does the above point to
what my mistake might be?

Is the "ConnectionHandler" lag between curl and Rexster, or between
Blueprints and Neo4j ??

Thanks,
Hasan

[Marko Rodriguez]

Hi,

> I load an owl file of 140 vertices and create 450 edges.

How did you load it?

Marko.

http://markorodriguez.com

[Peter Neubauer]

Hi there,
this looks like some IO problem on the underlying machine
configuration. Let me ask some Neo4j people also ...

Cheers,

/peter neubauer

GTalk:      neubauer.peter
Skype       peter.neubauer
Phone       +46 704 106975
LinkedIn   http://www.linkedin.com/in/neubauer
Twitter      http://twitter.com/peterneubauer

http://www.neo4j.org               - Your high performance graph database.
http://startupbootcamp.org/    - Öresund - Innovation happens HERE.
http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.

[HasanB]

On Jul 20, 2:26 pm, Marko Rodriguez <okramma...@gmail.com> wrote:
> > I load an owl file of 140 vertices and create 450 edges.
>
> How did you load it?

curl -v -T .\src\main\resources\data\family.swrl.owl
-H "Content-Type: text/plain"
"http://localhost:8182/neo4jsample/mowa/stevens"

The PUT handler of my extension :
- saves the file locally
- instantiates an IndexableGraph
- instantiates a com.hp.hpl.jena.rdf.model.Model
- reads the local file into the model
- creates a single "file index" representing file members
- creates a single "file vertex" representing the whole file and adds
it to the "file index"
- loops through the Jena model statements treating each as a
com.hp.hpl.jena.graph.Triple
- creates a vertex for each subject and adds it to the "file index"
- creates an edge between each subject and the "file vertex"
- creates a vertex for each object and adds it to the "file index"
- creates an edge between each object and the "file vertex"
- creates an edge between each object and each subject from each
predicate.

[HasanB]

On Jul 20, 3:02 pm, Peter Neubauer <peter.neuba...@neotechnology.com>
wrote:

> this looks like some IO problem on the underlying machine
> configuration. Let me ask some Neo4j people also ...

Much obliged Peter.

[Peter Neubauer]

Hasan,
one immediate thought is that every operation in this code is running
in its own transaction, that is

Every createNode()
Every createRelationshipTo()
Every setProperty()
run within their own transactions.

So what your measured is that his filesystem can handle 140+450+(a
bunch of properties, I'm going to assume 5 per node/rel) ~= 3500
flushes in 90 seconds, that is almost 40 flushes per second (don't
remember if that is good or bad).

Tons of transactions would in combination with slow IO configuration
completely dominate the interaction. Could you try setting your graph
into manual transaction mode and repeat this? Not sure if this is a
fault in the Sail implementaiton or can be tweaked outside. Josh?

Thanks to Tobias Ivarsson for the fast analysis!

Cheers,

/peter neubauer

GTalk:      neubauer.peter
Skype       peter.neubauer
Phone       +46 704 106975
LinkedIn   http://www.linkedin.com/in/neubauer
Twitter      http://twitter.com/peterneubauer

http://www.neo4j.org               - Your high performance graph database.
http://startupbootcamp.org/    - Öresund - Innovation happens HERE.
http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.

[HasanB]

On Jul 20, 4:28 pm, Peter Neubauer <peter.neuba...@neotechnology.com>
wrote:

> Could you try setting your graph into manual transaction mode and repeat this?

Wow! 65 times faster! 90 down to 1.4

* * Thanks a million * *

[Marko Rodriguez]

Hey,

Could you try setting your graph into manual transaction mode and repeat this?

Wow!  65 times faster!  90 down to 1.4

For follow up reading, see:

https://github.com/tinkerpop/blueprints/wiki/Graph-Transactions

https://github.com/tinkerpop/blueprints/wiki/Graph-Transactions-Helpers

Good luck with your project,

Marko.

http://markorodriguez.com

[Peter Neubauer]

Nice!

With proper tweaking you should be able to get that down more, but I
think this is good for a start. Let us know if you have more
questions!

Cheers,

/peter neubauer

GTalk:      neubauer.peter
Skype       peter.neubauer
Phone       +46 704 106975
LinkedIn   http://www.linkedin.com/in/neubauer
Twitter      http://twitter.com/peterneubauer

http://www.neo4j.org               - Your high performance graph database.
http://startupbootcamp.org/    - Öresund - Innovation happens HERE.
http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.

[HasanB]

On Jul 20, 11:43 pm, Marko Rodriguez <okramma...@gmail.com> wrote:
> For follow up reading, see:
>        https://github.com/tinkerpop/blueprints/wiki/Graph-Transactions
>                https://github.com/tinkerpop/blueprints/wiki/Graph-Transactions-Helpers

CommitManager is now part of my Rexster extension demo.

Thanks for the tip!