Neo4j crashes when running a query

[Alx]

I have a neo4j db with 1M nodes and 10M relationships running on my local computer (16GB RAM, 4-core i7). The configuration of the server is as follows:

neo4j.properties:

neostore.nodestore.db.mapped_memory=25M

neostore.relationshipstore.db.mapped_memory=500M

neostore.propertystore.db.mapped_memory=900M

neostore.propertystore.db.strings.mapped_memory=1300M

neostore.propertystore.db.arrays.mapped_memory=1300M

cache_type=strong

neo4j-wrapper.conf:

wrapper.java.initmemory=1512

wrapper.java.maxmemory=11120

I am running the following query:

OPTIONAL MATCH (m:User)-[:REL1]->() - [:REL2]-> ()<-[:REL1]-(n) RETURN  m.id AS From ,n.id AS To, count(n) AS Number

The query makes the heap size increase until it hits 11Gib on the memory. Then the server just struggles (loses connection, re-connects) for a few hours and eventually it dies with either this error:

ERROR (-v for expanded information):

        Error unmarshaling return header; nested exception is: 

        java.net.SocketException: Operation timed out

or this error:

ERROR (-v for expanded information):

          Error occurred in server thread; nested exception is: 

           java.lang.OutOfMemoryError: Java heap space

I know that [:REL2] are roughly 8M. I don't know what else I can do to fix this. Perhaps the query needs to be optimized but I don't know how I can make it simpler because [:REL2] is important to use. Any thoughts or suggestions would be much appreciated.

[Michael Hunger]

Don't use cache type strong 

It will just gc all the time

I changed your settings inline

Which version are you using?

Sent from mobile device

Am 24.04.2014 um 16:11 schrieb Alx <akara...@gmail.com>:

I have a neo4j db with 1M nodes and 10M relationships running on my local computer (16GB RAM, 4-core i7). The configuration of the server is as follows:

neo4j.properties:

neostore.nodestore.db.mapped_memory=25M

neostore.relationshipstore.db.mapped_memory=500M

neostore.propertystore.db.mapped_memory=300M

neostore.propertystore.db.strings.mapped_memory=300M

neostore.propertystore.db.arrays.mapped_memory=0M

cache_type=weak

neo4j-wrapper.conf:

wrapper.java.initmemory=1512

wrapper.java.maxmemory=8000

I am running the following query:

MATCH (m:User)-[:REL1]->(a) - [:REL2]-> (b)

With m, b, count(*) as cnt

Match (b)<-[:REL1]-(n) RETURN  m.id AS From ,n.id AS To, count(*)*cnt AS Number

Reduce the intermediate count of b's with the aggregation

The query makes the heap size increase until it hits 11Gib on the memory. Then the server just struggles (loses connection, re-connects) for a few hours and eventually it dies with either this error:

ERROR (-v for expanded information):

        Error unmarshaling return header; nested exception is: 

        java.net.SocketException: Operation timed out

or this error:

ERROR (-v for expanded information):

          Error occurred in server thread; nested exception is: 

           java.lang.OutOfMemoryError: Java heap space

I know that [:REL2] are roughly 8M. I don't know what else I can do to fix this. Perhaps the query needs to be optimized but I don't know how I can make it simpler because [:REL2] is important to use. Any thoughts or suggestions would be much appreciated.

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alx]

Hi Michael,

Thanks for the suggestions.  So I followed them but unfortunately same error like before. The database occupies 16GB on my hard disk. I wonder if it should be "wrapper.java.maxmemory=160000". Unfortunately my machine has 16GB RAM all in all.  I am running the latest version of Neo4j (2.0.2).

Is there a "trick" to have the cypher query stream the results as they come into a file? Currently I am running the cypher query from the command line like this:

"./bin/neo4j-shell -file query.cql > result.txt"

[Michael Hunger]

Streaming doesn't help when you aggregate, as it has to go through all data to finish the aggregation.

What is the stacktrace for the error?

It should not run into an OOM anymore.

Can you share your graph.db/messages.log?

[Alx]

I emailed you the messages.log privately on your email address. I hope it helps. Thanks!