> I have a question regarding loading huge graphs into the Neo4j DB. I've
> read that Neo4j is capable of handling even a few billion vertices. On the
> other hand all the code examples I've found use a cache (tipically
> java.util.HashMap) for the nodes when loading the edges and I'm not sure
> that Java can handle such a big HashMap. So what is the best way to load
> such a huge graph?

> Thanks!

> Greg

> --

> Hi,

> I have a question regarding loading huge graphs into the Neo4j DB. I've
> read that Neo4j is capable of handling even a few billion vertices. On the
> other hand all the code examples I've found use a cache (tipically
> java.util.HashMap) for the nodes when loading the edges and I'm not sure
> that Java can handle such a big HashMap. So what is the best way to load
> such a huge graph?

> Thanks!

> Greg

> -  James

> On Tuesday, October 2, 2012 9:16:15 AM UTC-5, Gergely Svigruha wrote:

>> Hi,

>> I have a question regarding loading huge graphs into the Neo4j DB. I've
>> read that Neo4j is capable of handling even a few billion vertices. On the
>> other hand all the code examples I've found use a cache (tipically
>> java.util.HashMap) for the nodes when loading the edges and I'm not sure
>> that Java can handle such a big HashMap. So what is the best way to load
>> such a huge graph?

>> Thanks!

>> Greg

> --

> Thanks James,
> I had written exactly the same answer but it got stuck in the outbox :)

> Cheers,

> /peter neubauer

> G:  neubauer.peter
> S:  peter.neubauer
> P:  +46 704 106975
> L:   http://www.linkedin.com/in/neubauer
> T:   @peterneubauer

> Wanna learn something new? Come to http://graphconnect.com

> On Tue, Oct 2, 2012 at 6:27 PM, James Thornton <james.t...@gmail.com<javascript:>>
> wrote:
> > If you are preprocessing your data, I typically use Redis. And the
> fastest
> > way to load data into Neo4j is to use the batch importer
> > (https://github.com/jexp/batch-import), which imports from a CSV file.

> > -  James

> > On Tuesday, October 2, 2012 9:16:15 AM UTC-5, Gergely Svigruha wrote:

> >> Hi,

> >> I have a question regarding loading huge graphs into the Neo4j DB. I've
> >> read that Neo4j is capable of handling even a few billion vertices. On
> the
> >> other hand all the code examples I've found use a cache (tipically
> >> java.util.HashMap) for the nodes when loading the edges and I'm not
> sure
> >> that Java can handle such a big HashMap. So what is the best way to
> load
> >> such a huge graph?

> >> Thanks!

> >> Greg

> > --

> Thank you, I think what I need is the batch - CSV importer:)

> Greg

> 2012. október 3., szerda 2:24:09 UTC+7 időpontban Peter Neubauer a
> következőt írta:

>> Thanks James,
>> I had written exactly the same answer but it got stuck in the outbox :)

>> Cheers,

>> /peter neubauer

>> G:  neubauer.peter
>> S:  peter.neubauer
>> P:  +46 704 106975
>> L:   http://www.linkedin.com/in/neubauer
>> T:   @peterneubauer

>> Wanna learn something new? Come to http://graphconnect.com

>> On Tue, Oct 2, 2012 at 6:27 PM, James Thornton <james.t...@gmail.com>
>> wrote:
>> > If you are preprocessing your data, I typically use Redis. And the
>> fastest
>> > way to load data into Neo4j is to use the batch importer
>> > (https://github.com/jexp/batch-import), which imports from a CSV file.

>> > -  James

>> > On Tuesday, October 2, 2012 9:16:15 AM UTC-5, Gergely Svigruha wrote:

>> >> Hi,

>> >> I have a question regarding loading huge graphs into the Neo4j DB.
>> I've
>> >> read that Neo4j is capable of handling even a few billion vertices. On
>> the
>> >> other hand all the code examples I've found use a cache (tipically
>> >> java.util.HashMap) for the nodes when loading the edges and I'm not
>> sure
>> >> that Java can handle such a big HashMap. So what is the best way to
>> load
>> >> such a huge graph?

>> >> Thanks!

>> >> Greg

>> > --

> Can you help me what can be the cause of this?
> Thanks.

> Greg

> 2012. október 3., szerda 8:51:18 UTC+7 időpontban Gergely Svigruha a következőt írta:
> Thank you, I think what I need is the batch - CSV importer:)

> Greg

> 2012. október 3., szerda 2:24:09 UTC+7 időpontban Peter Neubauer a következőt írta:
> Thanks James,
> I had written exactly the same answer but it got stuck in the outbox :)

> Cheers,

> /peter neubauer

> G:  neubauer.peter
> S:  peter.neubauer
> P:  +46 704 106975
> L:   http://www.linkedin.com/in/neubauer
> T:   @peterneubauer

> Wanna learn something new? Come to http://graphconnect.com

> On Tue, Oct 2, 2012 at 6:27 PM, James Thornton <james.t...@gmail.com> wrote:
> > If you are preprocessing your data, I typically use Redis. And the fastest
> > way to load data into Neo4j is to use the batch importer
> > (https://github.com/jexp/batch-import), which imports from a CSV file.

> > -  James

> > On Tuesday, October 2, 2012 9:16:15 AM UTC-5, Gergely Svigruha wrote:

> >> Hi,

> >> I have a question regarding loading huge graphs into the Neo4j DB. I've
> >> read that Neo4j is capable of handling even a few billion vertices. On the
> >> other hand all the code examples I've found use a cache (tipically
> >> java.util.HashMap) for the nodes when loading the edges and I'm not sure
> >> that Java can handle such a big HashMap. So what is the best way to load
> >> such a huge graph?

> >> Thanks!

> >> Greg

> > --

> --

> Can you share the code you used.

> Michael

> Am 03.10.2012 um 07:38 schrieb Gergely Svigruha:

> I just had another issue. After creating the nodes I try to create the
> edges using the BatchInserter.createRelationShip(nodeId1, nodeId2,
> relationType, properties) function but got an exception:
> java.io.IOException: The process cannot access the file because another
> process ha locked the portion of the file.

> Can you help me what can be the cause of this?
> Thanks.

> Greg

> 2012. október 3., szerda 8:51:18 UTC+7 időpontban Gergely Svigruha a
> következőt írta:

>> Thank you, I think what I need is the batch - CSV importer:)

>> Greg

>> 2012. október 3., szerda 2:24:09 UTC+7 időpontban Peter Neubauer a
>> következőt írta:

>>> Thanks James,
>>> I had written exactly the same answer but it got stuck in the outbox :)

>>> Cheers,

>>> /peter neubauer

>>> G:  neubauer.peter
>>> S:  peter.neubauer
>>> P:  +46 704 106975
>>> L:   http://www.linkedin.com/in/neubauer
>>> T:   @peterneubauer

>>> Wanna learn something new? Come to http://graphconnect.com

>>> On Tue, Oct 2, 2012 at 6:27 PM, James Thornton <james.t...@gmail.com>
>>> wrote:
>>> > If you are preprocessing your data, I typically use Redis. And the
>>> fastest
>>> > way to load data into Neo4j is to use the batch importer
>>> > (https://github.com/jexp/batch-import), which imports from a CSV
>>> file.

>>> > -  James

>>> > On Tuesday, October 2, 2012 9:16:15 AM UTC-5, Gergely Svigruha wrote:

>>> >> Hi,

>>> >> I have a question regarding loading huge graphs into the Neo4j DB.
>>> I've
>>> >> read that Neo4j is capable of handling even a few billion vertices.
>>> On the
>>> >> other hand all the code examples I've found use a cache (tipically
>>> >> java.util.HashMap) for the nodes when loading the edges and I'm not
>>> sure
>>> >> that Java can handle such a big HashMap. So what is the best way to
>>> load
>>> >> such a huge graph?

>>> >> Thanks!

>>> >> Greg

>>> > --

> --

> import java.io.BufferedReader;
> import java.io.File;
> import java.io.FileNotFoundException;
> import java.io.FileReader;
> import java.io.IOException;
> import java.util.HashMap;
> import java.util.Map;

> import org.neo4j.graphdb.RelationshipType;
> import org.neo4j.helpers.collection.MapUtil;
> import org.neo4j.kernel.impl.util.FileUtils;
> import org.neo4j.unsafe.batchinsert.BatchInserter;
> import org.neo4j.unsafe.batchinsert.BatchInserters;

> public class GraphImporter_v2 {

>    private long nodeIdx=0;
>    private Map<Long,Long> idxMap = new HashMap<Long, Long>();

>    enum RelType implements RelationshipType {
>            CALLS
>    }

>    private void createNode(long pnum, BatchInserter db) {
>            if(!idxMap.containsKey(pnum)) {
>                    nodeIdx++;
>                    idxMap.put(pnum,  nodeIdx);
>                    Map<String, Object> prop = new HashMap<String, Object>();
>                    prop.put("Number", pnum);
>                db.createNode(nodeIdx, prop);
>            }
>    }

>    private long getNodeNum(long pnum) throws Exception {
>            if(idxMap.containsKey(pnum)) {
>                    return idxMap.get(pnum);
>            } else {
>                    throw new Exception("Missing number: "+pnum);
>            }
>    }

>    public static void main(String[] args) {

>            GraphImporter_v2 importer = new GraphImporter_v2();
>            importer.load(args[0], args[1]);
>    }

>    private void load(String inputFile, String dbpath) {
>            try {
>                    File graphDb = new File(dbpath);
>                    if (graphDb.exists()) {
>                FileUtils.deleteRecursively(graphDb);
>            }

>                    long edges = 0;
>                    long errorRows = 0;
>                    Map<String, String> config = new HashMap<String, String>();
>            config = MapUtil.load( new File( "batch.properties" ) );
>            BatchInserter db =  BatchInserters.inserter(dbpath, config);

>                    BufferedReader reader = new BufferedReader(new FileReader(new File(inputFile)));
>                    reader.readLine();
>             String line = null;
>                    while ((line = reader.readLine()) != null) {
>                            String[] lineData = line.split(",");
>                            try {
>                                    createNode(Long.valueOf(lineData[0].replace("\"", "")), db);
>                                    createNode(Long.valueOf(lineData[1].replace("\"", "")), db);
>                            } catch (NumberFormatException e) {
>                                    errorRows++;
>                            }

>                edges++;

>            }

>                    System.out.println("Total edges: "+edges);
>                    System.out.println("Error edges: "+errorRows);

>                    reader.close();
>                    reader = new BufferedReader(new FileReader(new File(inputFile)));
>                    System.out.println("Loading edges..");
>                    long node1 = 0;
>                    long node2 = 0;
>                    reader.readLine();
>             line = null;
>                    while ((line = reader.readLine()) != null) {
>                            String[] lineData = line.split(",");
>                            try {
>                                    node1 = getNodeNum(Long.valueOf(lineData[0].replace("\"", "")));
>                                    node2 = getNodeNum(Long.valueOf(lineData[1].replace("\"", "")));
>                                    Map<String, Object> prop = new HashMap<String, Object>();
>                                    prop.put("Duration", Integer.valueOf(lineData[2]));
>                                    prop.put("Cnt", Integer.valueOf(lineData[3]));
>                                    prop.put("Charge", Integer.valueOf(lineData[4]));
>                        db.createRelationship(node1, node2, RelType.CALLS, prop);
>                            } catch (NumberFormatException e) {}      
>            }

>                    db.shutdown();
>                    reader.close();
>            } catch (FileNotFoundException e) {
>                    e.printStackTrace();
>            } catch (IOException e) {
>                    e.printStackTrace();
>            } catch (Throwable e) {
>                    e.printStackTrace();
>            }

>    }

> }

> 2012. október 3., szerda 14:27:31 UTC+7 időpontban Michael Hunger a következőt írta:

>> Can you share the code you used.

>> Michael

>> Am 03.10.2012 um 07:38 schrieb Gergely Svigruha:

>>> I just had another issue. After creating the nodes I try to create the edges using the BatchInserter.createRelationShip(nodeId1, nodeId2, relationType, properties) function but got an exception:
>>> java.io.IOException: The process cannot access the file because another process ha locked the portion of the file.

>>> Can you help me what can be the cause of this?
>>> Thanks.

>>> Greg

>>> 2012. október 3., szerda 8:51:18 UTC+7 időpontban Gergely Svigruha a következőt írta:

>>>> Thank you, I think what I need is the batch - CSV importer:)

>>>> Greg

>>>> 2012. október 3., szerda 2:24:09 UTC+7 időpontban Peter Neubauer a következőt írta:

>>>>> Thanks James,
>>>>> I had written exactly the same answer but it got stuck in the outbox :)

>>>>> Cheers,

>>>>> /peter neubauer

>>>>> G:  neubauer.peter
>>>>> S:  peter.neubauer
>>>>> P:  +46 704 106975
>>>>> L:   http://www.linkedin.com/in/neubauer
>>>>> T:   @peterneubauer

>>>>> Wanna learn something new? Come to http://graphconnect.com

>>>>> On Tue, Oct 2, 2012 at 6:27 PM, James Thornton <james.t...@gmail.com> wrote:
>>>>> > If you are preprocessing your data, I typically use Redis. And the fastest
>>>>> > way to load data into Neo4j is to use the batch importer
>>>>> > (https://github.com/jexp/batch-import), which imports from a CSV file.

>>>>> > -  James

>>>>> > On Tuesday, October 2, 2012 9:16:15 AM UTC-5, Gergely Svigruha wrote:

>>>>> >> Hi,

>>>>> >> I have a question regarding loading huge graphs into the Neo4j DB. I've
>>>>> >> read that Neo4j is capable of handling even a few billion vertices. On the
>>>>> >> other hand all the code examples I've found use a cache (tipically
>>>>> >> java.util.HashMap) for the nodes when loading the edges and I'm not sure
>>>>> >> that Java can handle such a big HashMap. So what is the best way to load
>>>>> >> such a huge graph?

>>>>> >> Thanks!

>>>>> >> Greg

>>>>> > --

>>> --

> --

> Probably a leftover file lock from a previous run.

> Try to do the close of the readers and shutdown of db in try ... Finally

> Is this windows?

> Sent from mobile device

> Am 03.10.2012 um 09:51 schrieb Gergely Svigruha <sger...@gmail.com<javascript:>
> >:

> This is the code I use. Unfortunately the input I have doesn't contain
> node ids as requested in the CSV importer previously recommended, so I have
> to create the id's myself. I have a previous version which reads the input
> only once and creates the nodes and edges simultaneously but I had the same
> error with that after ~30M edges / 90M.

> import java.io.BufferedReader;
> import java.io.File;
> import java.io.FileNotFoundException;
> import java.io.FileReader;
> import java.io.IOException;
> import java.util.HashMap;
> import java.util.Map;

> import org.neo4j.graphdb.RelationshipType;
> import org.neo4j.helpers.collection.MapUtil;
> import org.neo4j.kernel.impl.util.FileUtils;
> import org.neo4j.unsafe.batchinsert.BatchInserter;
> import org.neo4j.unsafe.batchinsert.BatchInserters;

> public class GraphImporter_v2 {

> private long nodeIdx=0;
> private Map<Long,Long> idxMap = new HashMap<Long, Long>();
>  enum RelType implements RelationshipType {
> CALLS
> }
>  private void createNode(long pnum, BatchInserter db) {
> if(!idxMap.containsKey(pnum)) {
> nodeIdx++;
> idxMap.put(pnum,  nodeIdx);
> Map<String, Object> prop = new HashMap<String, Object>();
> prop.put("Number", pnum);
>     db.createNode(nodeIdx, prop);
> }
> }
>  private long getNodeNum(long pnum) throws Exception {
> if(idxMap.containsKey(pnum)) {
> return idxMap.get(pnum);
> } else {
> throw new Exception("Missing number: "+pnum);
> }
> }
>  public static void main(String[] args) {
>  GraphImporter_v2 importer = new GraphImporter_v2();
> importer.load(args[0], args[1]);
> }

> private void load(String inputFile, String dbpath) {
> try {
> File graphDb = new File(dbpath);
> if (graphDb.exists()) {
>             FileUtils.deleteRecursively(graphDb);
>         }
>  long edges = 0;
> long errorRows = 0;
> Map<String, String> config = new HashMap<String, String>();
>      config = MapUtil.load( new File( "batch.properties" ) );
>      BatchInserter db =  BatchInserters.inserter(dbpath, config);
>  BufferedReader reader = new BufferedReader(new FileReader(new
> File(inputFile)));
> reader.readLine();
>             String line = null;
> while ((line = reader.readLine()) != null) {
> String[] lineData = line.split(",");
> try {
> createNode(Long.valueOf(lineData[0].replace("\"", "")), db);
> createNode(Long.valueOf(lineData[1].replace("\"", "")), db);
> } catch (NumberFormatException e) {
> errorRows++;
> }
>             edges++;

>         }

> System.out.println("Total edges: "+edges);
> System.out.println("Error edges: "+errorRows);
>  reader.close();
> reader = new BufferedReader(new FileReader(new File(inputFile)));
> System.out.println("Loading edges..");
> long node1 = 0;
> long node2 = 0;
> reader.readLine();
>             line = null;
> while ((line = reader.readLine()) != null) {
> String[] lineData = line.split(",");
> try {
> node1 = getNodeNum(Long.valueOf(lineData[0].replace("\"", "")));
> node2 = getNodeNum(Long.valueOf(lineData[1].replace("\"", "")));
> Map<String, Object> prop = new HashMap<String, Object>();
> prop.put("Duration", Integer.valueOf(lineData[2]));
> prop.put("Cnt", Integer.valueOf(lineData[3]));
> prop.put("Charge", Integer.valueOf(lineData[4]));
>             db.createRelationship(node1, node2, RelType.CALLS, prop);
> } catch (NumberFormatException e) {}      
>         }

> db.shutdown();
> reader.close();
> } catch (FileNotFoundException e) {
> e.printStackTrace();
> } catch (IOException e) {
> e.printStackTrace();
> } catch (Throwable e) {
> e.printStackTrace();
> }
>  }
> }

> 2012. október 3., szerda 14:27:31 UTC+7 időpontban Michael Hunger a
> következőt írta:

>> Can you share the code you used.

>> Michael

>> Am 03.10.2012 um 07:38 schrieb Gergely Svigruha:

>> I just had another issue. After creating the nodes I try to create the
>> edges using the BatchInserter.createRelationShip(nodeId1, nodeId2,
>> relationType, properties) function but got an exception:
>> java.io.IOException: The process cannot access the file because another
>> process ha locked the portion of the file.

>> Can you help me what can be the cause of this?
>> Thanks.

>> Greg

>> 2012. október 3., szerda 8:51:18 UTC+7 időpontban Gergely Svigruha a
>> következőt írta:

>>> Thank you, I think what I need is the batch - CSV importer:)

>>> Greg

>>> 2012. október 3., szerda 2:24:09 UTC+7 időpontban Peter Neubauer a
>>> következőt írta:

>>>> Thanks James,
>>>> I had written exactly the same answer but it got stuck in the outbox :)

>>>> Cheers,

>>>> /peter neubauer

>>>> G:  neubauer.peter
>>>> S:  peter.neubauer
>>>> P:  +46 704 106975
>>>> L:   http://www.linkedin.com/in/neubauer
>>>> T:   @peterneubauer

>>>> Wanna learn something new? Come to http://graphconnect.com

>>>> On Tue, Oct 2, 2012 at 6:27 PM, James Thornton <james.t...@gmail.com>
>>>> wrote:
>>>> > If you are preprocessing your data, I typically use Redis. And the
>>>> fastest
>>>> > way to load data into Neo4j is to use the batch importer
>>>> > (https://github.com/jexp/batch-import), which imports from a CSV
>>>> file.

>>>> > -  James

>>>> > On Tuesday, October 2, 2012 9:16:15 AM UTC-5, Gergely Svigruha wrote:

>>>> >> Hi,

>>>> >> I have a question regarding loading huge graphs into the Neo4j DB.
>>>> I've
>>>> >> read that Neo4j is capable of handling even a few billion vertices.
>>>> On the
>>>> >> other hand all the code examples I've found use a cache (tipically
>>>> >> java.util.HashMap) for the nodes when loading the edges and I'm not
>>>> sure
>>>> >> that Java can handle such a big HashMap. So what is the best way to
>>>> load
>>>> >> such a huge graph?

>>>> >> Thanks!

>>>> >> Greg

>>>> > --

>> --

>>  --

> Yes, this is Windows 7.

> 2012. október 3., szerda 15:43:15 UTC+7 időpontban Michael Hunger a következőt írta:

>> Probably a leftover file lock from a previous run.

>> Try to do the close of the readers and shutdown of db in try ... Finally

>> Is this windows?

>> Sent from mobile device

>> Am 03.10.2012 um 09:51 schrieb Gergely Svigruha <sger...@gmail.com>:

>>> This is the code I use. Unfortunately the input I have doesn't contain node ids as requested in the CSV importer previously recommended, so I have to create the id's myself. I have a previous version which reads the input only once and creates the nodes and edges simultaneously but I had the same error with that after ~30M edges / 90M.

>>> import java.io.BufferedReader;
>>> import java.io.File;
>>> import java.io.FileNotFoundException;
>>> import java.io.FileReader;
>>> import java.io.IOException;
>>> import java.util.HashMap;
>>> import java.util.Map;

>>> import org.neo4j.graphdb.RelationshipType;
>>> import org.neo4j.helpers.collection.MapUtil;
>>> import org.neo4j.kernel.impl.util.FileUtils;
>>> import org.neo4j.unsafe.batchinsert.BatchInserter;
>>> import org.neo4j.unsafe.batchinsert.BatchInserters;

>>> public class GraphImporter_v2 {

>>>    private long nodeIdx=0;
>>>    private Map<Long,Long> idxMap = new HashMap<Long, Long>();

>>>    enum RelType implements RelationshipType {
>>>            CALLS
>>>    }

>>>    private void createNode(long pnum, BatchInserter db) {
>>>            if(!idxMap.containsKey(pnum)) {
>>>                    nodeIdx++;
>>>                    idxMap.put(pnum,  nodeIdx);
>>>                    Map<String, Object> prop = new HashMap<String, Object>();
>>>                    prop.put("Number", pnum);
>>>                db.createNode(nodeIdx, prop);
>>>            }
>>>    }

>>>    private long getNodeNum(long pnum) throws Exception {
>>>            if(idxMap.containsKey(pnum)) {
>>>                    return idxMap.get(pnum);
>>>            } else {
>>>                    throw new Exception("Missing number: "+pnum);
>>>            }
>>>    }

>>>    public static void main(String[] args) {

>>>            GraphImporter_v2 importer = new GraphImporter_v2();
>>>            importer.load(args[0], args[1]);
>>>    }

>>>    private void load(String inputFile, String dbpath) {
>>>            try {
>>>                    File graphDb = new File(dbpath);
>>>                    if (graphDb.exists()) {
>>>                FileUtils.deleteRecursively(graphDb);
>>>            }

>>>                    long edges = 0;
>>>                    long errorRows = 0;
>>>                    Map<String, String> config = new HashMap<String, String>();
>>>            config = MapUtil.load( new File( "batch.properties" ) );
>>>            BatchInserter db =  BatchInserters.inserter(dbpath, config);

>>>                    BufferedReader reader = new BufferedReader(new FileReader(new File(inputFile)));
>>>                    reader.readLine();
>>>             String line = null;
>>>                    while ((line = reader.readLine()) != null) {
>>>                            String[] lineData = line.split(",");
>>>                            try {
>>>                                    createNode(Long.valueOf(lineData[0].replace("\"", "")), db);
>>>                                    createNode(Long.valueOf(lineData[1].replace("\"", "")), db);
>>>                            } catch (NumberFormatException e) {
>>>                                    errorRows++;
>>>                            }

>>>                edges++;

>>>            }

>>>                    System.out.println("Total edges: "+edges);
>>>                    System.out.println("Error edges: "+errorRows);

>>>                    reader.close();
>>>                    reader = new BufferedReader(new FileReader(new File(inputFile)));
>>>                    System.out.println("Loading edges..");
>>>                    long node1 = 0;
>>>                    long node2 = 0;
>>>                    reader.readLine();
>>>             line = null;
>>>                    while ((line = reader.readLine()) != null) {
>>>                            String[] lineData = line.split(",");
>>>                            try {
>>>                                    node1 = getNodeNum(Long.valueOf(lineData[0].replace("\"", "")));
>>>                                    node2 = getNodeNum(Long.valueOf(lineData[1].replace("\"", "")));
>>>                                    Map<String, Object> prop = new HashMap<String, Object>();
>>>                                    prop.put("Duration", Integer.valueOf(lineData[2]));
>>>                                    prop.put("Cnt", Integer.valueOf(lineData[3]));
>>>                                    prop.put("Charge", Integer.valueOf(lineData[4]));
>>>                        db.createRelationship(node1, node2, RelType.CALLS, prop);
>>>                            } catch (NumberFormatException e) {}      
>>>            }

>>>                    db.shutdown();
>>>                    reader.close();
>>>            } catch (FileNotFoundException e) {
>>>                    e.printStackTrace();
>>>            } catch (IOException e) {
>>>                    e.printStackTrace();
>>>            } catch (Throwable e) {
>>>                    e.printStackTrace();
>>>            }

>>>    }

>>> }

>>> 2012. október 3., szerda 14:27:31 UTC+7 időpontban Michael Hunger a következőt írta:

>>>> Can you share the code you used.

>>>> Michael

>>>> Am 03.10.2012 um 07:38 schrieb Gergely Svigruha:

>>>>> I just had another issue. After creating the nodes I try to create the edges using the BatchInserter.createRelationShip(nodeId1, nodeId2, relationType, properties) function but got an exception:
>>>>> java.io.IOException: The process cannot access the file because another process ha locked the portion of the file.

>>>>> Can you help me what can be the cause of this?
>>>>> Thanks.

>>>>> Greg

>>>>> 2012. október 3., szerda 8:51:18 UTC+7 időpontban Gergely Svigruha a következőt írta:

>>>>>> Thank you, I think what I need is the batch - CSV importer:)

>>>>>> Greg

>>>>>> 2012. október 3., szerda 2:24:09 UTC+7 időpontban Peter Neubauer a következőt írta:

>>>>>>> Thanks James,
>>>>>>> I had written exactly the same answer but it got stuck in the outbox :)

>>>>>>> Cheers,

>>>>>>> /peter neubauer

>>>>>>> G:  neubauer.peter
>>>>>>> S:  peter.neubauer
>>>>>>> P:  +46 704 106975
>>>>>>> L:   http://www.linkedin.com/in/neubauer
>>>>>>> T:   @peterneubauer

>>>>>>> Wanna learn something new? Come to http://graphconnect.com

>>>>>>> On Tue, Oct 2, 2012 at 6:27 PM, James Thornton <james.t...@gmail.com> wrote:
>>>>>>> > If you are preprocessing your data, I typically use Redis. And the fastest
>>>>>>> > way to load data into Neo4j is to use the batch importer
>>>>>>> > (https://github.com/jexp/batch-import), which imports from a CSV file.

>>>>>>> > -  James

>>>>>>> > On Tuesday, October 2, 2012 9:16:15 AM UTC-5, Gergely Svigruha wrote:

>>>>>>> >> Hi,

>>>>>>> >> I have a question regarding loading huge graphs into the Neo4j DB. I've
>>>>>>> >> read that Neo4j is capable of handling even a few billion vertices. On the
>>>>>>> >> other hand all the code examples I've found use a cache (tipically
>>>>>>> >> java.util.HashMap) for the nodes when loading the edges and I'm not sure
>>>>>>> >> that Java can handle such a big HashMap. So what is the best way to load
>>>>>>> >> such a huge graph?

>>>>>>> >> Thanks!

>>>>>>> >> Greg

>>>>>>> > --

>>>>> --

>>> --

> --

> Do you have the full exception and you graphdb/messages.log ?

> Sent from mobile device

> Am 03.10.2012 um 11:14 schrieb Gergely Svigruha <sger...@gmail.com<javascript:>
> >:

> I put them in the finally block but the problem still occurs...I even
> started a new db so I think there cannot be any leftover locks...The
> problem always occurs at the same edge (row in the input file).

> Yes, this is Windows 7.

> 2012. október 3., szerda 15:43:15 UTC+7 időpontban Michael Hunger a
> következőt írta:

>> Probably a leftover file lock from a previous run.

>> Try to do the close of the readers and shutdown of db in try ... Finally

>> Is this windows?

>> Sent from mobile device

>> Am 03.10.2012 um 09:51 schrieb Gergely Svigruha <sger...@gmail.com>:

>> This is the code I use. Unfortunately the input I have doesn't contain
>> node ids as requested in the CSV importer previously recommended, so I have
>> to create the id's myself. I have a previous version which reads the input
>> only once and creates the nodes and edges simultaneously but I had the same
>> error with that after ~30M edges / 90M.

>> import java.io.BufferedReader;
>> import java.io.File;
>> import java.io.FileNotFoundException;
>> import java.io.FileReader;
>> import java.io.IOException;
>> import java.util.HashMap;
>> import java.util.Map;

>> import org.neo4j.graphdb.RelationshipType;
>> import org.neo4j.helpers.collection.MapUtil;
>> import org.neo4j.kernel.impl.util.FileUtils;
>> import org.neo4j.unsafe.batchinsert.BatchInserter;
>> import org.neo4j.unsafe.batchinsert.BatchInserters;

>> public class GraphImporter_v2 {

>> private long nodeIdx=0;
>> private Map<Long,Long> idxMap = new HashMap<Long, Long>();
>>  enum RelType implements RelationshipType {
>> CALLS
>> }
>>  private void createNode(long pnum, BatchInserter db) {
>> if(!idxMap.containsKey(pnum)) {
>> nodeIdx++;
>> idxMap.put(pnum,  nodeIdx);
>> Map<String, Object> prop = new HashMap<String, Object>();
>> prop.put("Number", pnum);
>>     db.createNode(nodeIdx, prop);
>> }
>> }
>>  private long getNodeNum(long pnum) throws Exception {
>> if(idxMap.containsKey(pnum)) {
>> return idxMap.get(pnum);
>> } else {
>> throw new Exception("Missing number: "+pnum);
>> }
>> }
>>  public static void main(String[] args) {
>>  GraphImporter_v2 importer = new GraphImporter_v2();
>> importer.load(args[0], args[1]);
>> }

>> private void load(String inputFile, String dbpath) {
>> try {
>> File graphDb = new File(dbpath);
>> if (graphDb.exists()) {
>>             FileUtils.deleteRecursively(graphDb);
>>         }
>>  long edges = 0;
>> long errorRows = 0;
>> Map<String, String> config = new HashMap<String, String>();
>>      config = MapUtil.load( new File( "batch.properties" ) );
>>      BatchInserter db =  BatchInserters.inserter(dbpath, config);
>>  BufferedReader reader = new BufferedReader(new FileReader(new
>> File(inputFile)));
>> reader.readLine();
>>             String line = null;
>> while ((line = reader.readLine()) != null) {
>> String[] lineData = line.split(",");
>> try {
>> createNode(Long.valueOf(lineData[0].replace("\"", "")), db);
>> createNode(Long.valueOf(lineData[1].replace("\"", "")), db);
>> } catch (NumberFormatException e) {