Community edition performance with more than 4 cores

[Santiago Videla]

Hi,

I'm using Neo 3.0.2 community edition and running the import tool I just found that on a machine with 4 cores and 32GB of RAM the import is actually ~2x faster than on another (virtual) machine with 38 cores and ~60GB of RAM.

With 4 cores and 32GB:

Available memory:
  Free machine memory: 16.88 GB
  Max heap memory : 6.97 GB
...
...
IMPORT DONE in 1m 32s 624ms. Imported:
  4135534 nodes
  43160516 relationships
  51504726 properties

With 38 cores and ~60 GB

Available memory:
  Free machine memory: 24.08 GB
  Max heap memory : 13.14 GB
...
...
IMPORT DONE in 2m 45s 299ms. Imported:
  4135534 nodes
  43160516 relationships
  51504726 properties

Is this the expected behavior? I found in [1] that the community edition scales up to 4 cores (it says for version 2.3 but I assume the same holds for 3.0) but with more cores I'd expected at least a similar performance, not 2x slower.

Should I use a machine with maximum 4 cores? Or can I configure Neo4j somehow to avoid such a decrease in performance? Also, is there any constraint about available memory and how the community edition scales or more memory should lead to better performance?

Regards,

[1] http://neo4j.com/blog/graphs-to-production-at-scale/

--

Santiago Videla
http://www.linkedin.com/in/svidela

[Michael Hunger]

You can provide the number of processors to the import tool.

I presume it's mostly your disk performance.

The import tool is unrelated to the enterprise core scalability.

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Santiago Videla]

Thanks Michael. I also thought about disk performance after I sent the email but is good to have a confirmation that is unrelated to community/enterprise edition. Still, apart from the import tool, I will appreciate if you could clarify me what does it mean that the community edition "scales up to 4 cores"? Having more than 4 cores just doesn't help to the community edition or it could actually yield a worse performance? What about available memory? more is always expected to be better?

Regards,

[Michael Hunger]

The enterprise edition has a lock manager and some other infrastructure that scales linearly across cores. Community levels off more after 4 cores. 

Von meinem iPhone gesendet

[VineetP]

Thanks Michael.
We are experimenting the Neo4j Community Edition 2.3.1 with 1 write thread and multiple read-only threads. We have just one LOAD CSV session regularly writing into Neo4j and multiple concurrent sessions reading Neo4j data. We also have more then 4 cores on the server where Community Edition is deployed. We do not notice any performance impact with more than 4 cores.
Q1: Given our use-case, is there a limit on cores that can be used on Community Edition before we can expect to see a performance degrade?
Q2: Given our use-case, is there a legal limit on cores that can be used on Community Edition?
Q3: Can we expect a performance improvement if we simply migrate to Neo4j Community v3.0?

rgds,

[VineetP]

Hello,

Any update on my queries? 
Thanks in advance.

[Michael Hunger]

Good questions, as so much depends on your data (model) and use cases I recommend to best try it out.

Most performance improvements come from many concurrent reading and writing threads.

AFAIK there is no legal restriction on cores.

In our tests 3.0.x is faster than 2.3.x

Michael