Re: [Neo4j] OutOfMemeryError (GC) on creating a large number of nodes
51 views
Skip to first unread message
Michael Hunger
Jul 6, 2012, 1:32:40 PM7/6/12
to ne...@googlegroups.com
Can you please share your code?
Michael
Am 06.07.2012 um 19:28 schrieb pt:
> Hi,
>
> On my PC, I can only have -Xmx2048m for max heap for JVM allowed. I tried to import an XML data which is quite huge, contains data about 7.5 million nodes. The XML file itself is about 120MB in bz2 format ( from the original size 1.5GB.)
>
> I am only able to create about 600,000 nodes but no further. JAVA report the heap problem from GC exceeding limit. What the code does until now is read the XML data and creates the nodes and indexes the properties of these nodes. And the code performs a transaction every 1000 nodes created.
>
> How can I fix this problem that I am able to reach the end of the file ? I am not sure if data import using batch insertion could solve this problem. I thought that any transaction which writes graph would do also the clean-up for the written nodes ?
>
>
> I will be so grateful for any suggestion.
>
> Thank you!
> Pattreeya
>
>
Michael
Am 06.07.2012 um 19:28 schrieb pt:
> Hi,
>
> On my PC, I can only have -Xmx2048m for max heap for JVM allowed. I tried to import an XML data which is quite huge, contains data about 7.5 million nodes. The XML file itself is about 120MB in bz2 format ( from the original size 1.5GB.)
>
> I am only able to create about 600,000 nodes but no further. JAVA report the heap problem from GC exceeding limit. What the code does until now is read the XML data and creates the nodes and indexes the properties of these nodes. And the code performs a transaction every 1000 nodes created.
>
> How can I fix this problem that I am able to reach the end of the file ? I am not sure if data import using batch insertion could solve this problem. I thought that any transaction which writes graph would do also the clean-up for the written nodes ?
>
>
> I will be so grateful for any suggestion.
>
> Thank you!
> Pattreeya
>
>
pt
Jul 6, 2012, 2:02:36 PM7/6/12
to ne...@googlegroups.com
Michael, thank you very much.
I think that I did something not right that prevent the JVM to claim the memory from the heap. I am sure that the problem is not from the Neo4j itself, I just run a test and it is creating more than 2 million nodes now.
I will check about memory in JAVA.
I think that I did something not right that prevent the JVM to claim the memory from the heap. I am sure that the problem is not from the Neo4j itself, I just run a test and it is creating more than 2 million nodes now.
I will check about memory in JAVA.
Michael Hunger
Jul 6, 2012, 2:05:03 PM7/6/12
to ne...@googlegroups.com
Make sure that your tx contain about 10000 elements thats a good size
Sent from mobile device
Sent from mobile device
pt
Jul 12, 2012, 10:11:45 AM7/12/12
to ne...@googlegroups.com
Michael,
could you tell me roughly how lucene transaction for node and the lucene index transaction () handle the memory ?
Many thanks!
Pattreeya.
could you tell me roughly how lucene transaction for node and the lucene index transaction () handle the memory ?
Many thanks!
Pattreeya.
Mattias Persson
Jul 14, 2012, 10:49:05 AM7/14/12
to ne...@googlegroups.com
Neo4j builds up the data structure around the added/removed index entries in a transaction as maps and sets (for exact index), in-memory lucene index for fulltext indexes and will write those to lucene at commit.
For exact indexes see https://github.com/neo4j/community/blob/master/lucene-index/src/main/java/org/neo4j/index/impl/lucene/ExactTxData.java
For fulltext/custom indexes see https://github.com/neo4j/community/blob/master/lucene-index/src/main/java/org/neo4j/index/impl/lucene/FullTxData.java
--
Mattias Persson, [mat...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
For exact indexes see https://github.com/neo4j/community/blob/master/lucene-index/src/main/java/org/neo4j/index/impl/lucene/ExactTxData.java
For fulltext/custom indexes see https://github.com/neo4j/community/blob/master/lucene-index/src/main/java/org/neo4j/index/impl/lucene/FullTxData.java
2012/7/12 pt <patt...@gmail.com>
--
Mattias Persson, [mat...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
Reply all
Reply to author
Forward
0 new messages