Data Insertion into Neo4j

[yayat...@innoplexus.com]

Hi,

I am trying to read records from MongoDB and insert into Neo4j. Now i have 2.6 million records in mongoDb and I am inserting the records since Friday[I am from India] into my Neo4j Community Edition. so far I have processed 45k records resulting in 2.9 million nodes.

can anyone suggest a faster way for insertion? 

PS: Neo4j does not support Multiprocessing in Py2neo

[Michael Hunger]

You should share what you have done so far otherwise no one can help you

There are docs on the mongodb connector and data import on neo4j.com/developer

Most likely you forgot to create some indexes/constraints 

Von meinem iPhone gesendet

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joseph Guhlin]

I've had lots of luck using the BatchInserters.  More info at their API Doc's.

http://neo4j.com/docs/2.3.5/javadocs/

[Yayati Sule]

Hi Michael,

I am reading records from mongoDB and i have a written a generic function to create Nodes and relations.
i am appending the parameters from mongoDb into a Query String and executing the query via run() api

Is this causing a bottle neck?

Regards,
Yayati Sule
Associate Data Scientist
Innoplexus Consulting Services Pvt. Ltd.
www.innoplexus.com
Mob : +91-9527459407

Landline: +91-20-66527300

© 2011-16 Innoplexus Consulting Services Pvt. Ltd.

Unless otherwise explicitly stated, all rights including those in copyright in the content of this e-mail are owned by Innoplexus Consulting Services Pvt Ltd. and all related legal entities. The contents of this e-mail shall not be copied, reproduced, or transmitted in any form without the written permission of Innoplexus Consulting Services Pvt Ltd or that of the copyright owner. The receipt of this mail is the acknowledgement of the receipt of contents; if the recipient is not the intended addressee then the recipient shall notify the sender immediately.

The contents are provided for information only and no opinions expressed should be relied on without further consultation with Innoplexus Consulting Services Pvt Ltd. and all related legal entities. While all endeavors have been made to ensure accuracy, Innoplexus Consulting Services Pvt. Ltd. makes no warranty or representation to its accuracy, completeness or fairness and persons who rely on it do so entirely at their own risk. The information herein may be changed or withdrawn at any time without notice. Innoplexus Consulting Services Pvt. Ltd. will not be liable to any client or third party for the accuracy of the information supplied through this service.

Innoplexus Consulting Services Pvt. Ltd. accepts no responsibility or liability for the contents of any other site, whether linked to this site or not, or any consequences from your acting upon the contents of another site.

Please Consider the environment before printing this email.

[Michael Hunger]

More detail please:

0. share your query & code

0. do you use labels

1. do you use real query parameters

2. do you have indexes / constraints for the keys you merge on

3. do you batch transactions

[Yayati Sule]

Hi Michael,

I cannot share the code as it belongs to my firm. i can tell you that 

1. I use Labels,

2. i use values from MongoDB keys as parameters, 

3. I have Uniqueness constraints on the property keys that i merge on

4. i do not use Batch transactions

Regards,
Yayati Sule
Associate Data Scientist
Innoplexus Consulting Services Pvt. Ltd.
www.innoplexus.com
Mob : +91-9527459407

Landline: +91-20-66527300

© 2011-16 Innoplexus Consulting Services Pvt. Ltd.

Unless otherwise explicitly stated, all rights including those in copyright in the content of this e-mail are owned by Innoplexus Consulting Services Pvt Ltd. and all related legal entities. The contents of this e-mail shall not be copied, reproduced, or transmitted in any form without the written permission of Innoplexus Consulting Services Pvt Ltd or that of the copyright owner. The receipt of this mail is the acknowledgement of the receipt of contents; if the recipient is not the intended addressee then the recipient shall notify the sender immediately.

The contents are provided for information only and no opinions expressed should be relied on without further consultation with Innoplexus Consulting Services Pvt Ltd. and all related legal entities. While all endeavors have been made to ensure accuracy, Innoplexus Consulting Services Pvt. Ltd. makes no warranty or representation to its accuracy, completeness or fairness and persons who rely on it do so entirely at their own risk. The information herein may be changed or withdrawn at any time without notice. Innoplexus Consulting Services Pvt. Ltd. will not be liable to any client or third party for the accuracy of the information supplied through this service.

Innoplexus Consulting Services Pvt. Ltd. accepts no responsibility or liability for the contents of any other site, whether linked to this site or not, or any consequences from your acting upon the contents of another site.

Please Consider the environment before printing this email.

[Michael Hunger]

Can you at least share the output of PROFILE  of your query ?

Do you have a fast DISK (SSD) and can you share the config for heap and page-cache

Otherwise there is little I can do without information.