Best way to get data into Neo4j?

[Rodger]

Dear Experts, 

I'm trying to insert a number of nodes into NEO4J. 

With Oracle, there are three basic methods:

SQL statements

SQL Loader

import/export 

Lately there is also data pump

SQL statements: insert a single row into a single table. 

SQL Loader: (sqlldr) That takes a comma delimited file, logs into Oracle and inserts the rows into a single table. Simple and fast. 

Import:  Executable Utility imp. Imports binary files created by the export utility, exp. 

Can import multiple tables. Even entire database schemas, including tables, indexes, and all objects. 

All of these come with Oracle standard issue RDBMS. 

 

-----------

For Neo4j, so far I've found:

-------

batch-import:

https://github.com/jexp/batch-import

jexp/batch-import

Seems to also require Apache Maven (mvn).

And java. 

These prerequisites are not mentioned anywhere. 

Any other prerequisites?

----

Max De Marzi's batch import 

http://maxdemarzi.com/2012/02/28/batch-importer-part-1/

http://maxdemarzi.com/2012/02/28/batch-importer-part-2/

Also looks to have the prerequisites: 

- apache maven

- psql (postgres) 

- java

- and rake, which ruby make

Any other prerequisites?

----------

http://docs.neo4j.org/chunked/milestone/batchinsert.html

Batch Insertion

- from the Neo4j manual.

It has transactions: 

{

  "method" : "POST",

  "to" : "/node",

  "id" : 0,

  "body" : {

    "name" : "bob"

  }

But are these transactions:

commands to be on a (cypher?) command line? 

Entries in a unix like script?

In a CSV like file? To be run via another command? 

???

--------

java rest binding:

http://stackoverflow.com/questions/16048073/remote-neo4j-data-base-import-large-amount-of-data

https://github.com/neo4j/java-rest-binding

Seems to also require:  maven / ivy

Also has some kind of file for <dependency> 

But what kind of file is that?  

Which software reads the file with the  <dependency> ??

--------

Rvan Bruggen Uses a spreadsheet:

http://blog.neo4j.org/2013/03/importing-data-into-neo4j-spreadsheet.html

The Spreadsheet is actually cypher commands.  

When you think of it, the cypher commands could simply be run manually. 

--------

Are there any other methods that I've missed?

What is everyone's experience?

What is the best way to get a number of data points into Neo4j?

Which other software packages are prerequisites? 

How big are the learning curves?

Which documentation has the best instantly digestible examples?

----

I'm thinking that a simple to use utility will be very important.

Get the data out of an RDBMS system, and into NEO4J nodes. 

Then, create connections between the nodes. 

Informatica, Ab Initio and others made a very profitable industry out of ETL software.

Thanks a lot! 

[Michael Hunger]

Hi Rodger,

there are some options. But it all depends on what you want to do?

#1 how much data do you want to import ?

#2 how is it structured, do you have already thought about your graph model?

#3 do you have it to run against a Neo4j server or can you access the database disk files?

there is an overview: http://neo4j.org/develop/import

you can directly import cypher statements with bin/neo4j-shell -path /path/to/database -f import.cql

which resembles your sql loader in an rdbms

In neo4j 2.0 there is an dump command in the shell which exports the database or a subgraph into a cypher statement.

I wrote the first part of a series about importing with cypher: http://jexp.de/blog/2013/05/on-importing-data-in-neo4j-blog-series/

I just started some days ago to work on some import tools for the Neo4j shell, which should work both for the commandline shell (bin/neo4j-shell and the shell in server).

https://github.com/jexp/neo4j-shell-tools

There is also this tool from peter for importing from SQL: https://github.com/peterneubauer/sql-import

And a writeup from Luanne and Nigel:

http://blog.neo4j.org/2013/04/data-migration-between-mysql-and-neo4j.html

There is also a Talend connector: https://github.com/Zenika/talend-neo4j-connector

http://eon.businesswire.com/news/eon/20130624005116/en/Talend/Neo-Technology/Neo4j

One of the many discussions in this group around this topic: https://groups.google.com/forum/#!topic/neo4j/0KT_yF3HnU8

These you've already found:

There is the simple spreadsheet way for a few hundred nodes / relationships: http://blog.neo4j.org/2013/03/importing-data-into-neo4j-spreadsheet.html

My CSV batch-importer (which builds upon the neo4j batch inserter): http://github.com/jexp/batch-import

Michael

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jean Villedieu]

I'm happy to reply too, even if I'm no expert.

You can also Import a Dataset into Neo4j, using Gephi plugin.

Rik has done a pretty cool video showcasing that : http://vimeo.com/57207925

Jean

[Nicolas Clairon]

Personally, I'm building my data into the GraphSON format and import it via gremlin:

g.loadGraphSON("/path/to/file.graphson")

https://github.com/tinkerpop/blueprints/wiki/GraphSON-Reader-and-Writer-Library

You can also build your data into the GraphML format (xml like):

http://graphml.graphdrawing.org/

[Rodger]

Hi all,

Thanks for responding.

I decided to learn in a similar way as when I learned SQL.

Start with SQL queries. Later, use the batch import utilities.

So, at this time, I'm just using Cypher to insert the data, and create the relationships.

Best,

R.

 

[Yasser Moukhtar]

Hi Roger,

Here is a simple way to import data into Neo4j, would appreciate if everyone have a look, and give me a feedback if possible.

Import Data into Neo4j Directly using Talend

yass

[Peter Neubauer]

Yasser,

that is a nice blog post - thanks for the good visuals!

/peter

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Open Data    - @mapillary
Open Source - @neo4j
Open Future  - @coderdojo

--

You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Rodger]

Hello,

That is a really good blog post.

If you've read my other posts, you know I'm 

big on clear documentation. 

I'm not familiar with DrawExpress Diagram, or 

Talend Open Studio for Big Data. 

But you've documented it really well, 

with the screen shots and the exact code.

Bravo! 

I'll take a look when I have a chance.

Best,

Rodger

[Rodger]

BTW, this past spring, I was thinking of the 

NEO4J data load, and Oracle's SQL Loader.

I got on the white board, and designed a GUI app

for a data loader. It would take a CSV file for input. 

And have a number of options for output.

Convert date to long. 

Format for date in text. 

Separator character. Create Unique. 

Generate script only. Check for errors only. 

Input and output directories. And so on.

But I got caught up in some of the details of

working with java. This was bigger than what I

did in the java class, and wanted to get the

architecture right. Unfortunately, I didn't find much 

good local guidance to my questions. 

Then I got sidetracked by other things. 

And discovered that NEO4J has a new loader. 

Perhaps I'll revisit it in the future. 

On Tuesday, August 5, 2014 2:04:25 AM UTC-5, Yasser Moukhtar wrote:

[Yasser Moukhtar]

Thank you Peter for Neo4j and your contributions to the community, I am a BIG fan.

I am really flattered, it is actually my first blog post.

yasser

[Yasser Moukhtar]

Thank you Roger, I hope you find it useful.

yass

[Rodger]

Hello Yasser,

I began to take you up up on your suggestion.

But I had yet another problem Linux/java installation.  :(

The instructions to install Talend Big Data on Redhat 6.4 is here:
http://rodgersnotes.wordpress.com/2014/09/20/installing-talend-open-studio-for-big-data-5-5-1-on-redhat-6-4/

Hope this helps anyone who needs it.

Best,

Rodger