Neo4j Server takes very long to start with new DB file

[Johannes Tveitan]

Hi, 

the problem I am facing is that when I create a new DB file with the batch-import tool and then tries to start the Neo4j Server, it takes an enormous amount of time to start. The last time I did this I importer about 78M nodes and aborted the starting of the server after 3h of it trying to start. It gets to the point of saying "In just a few seconds, Neo4j will be ready...", on the windows version and the same stage on the linux version. It is at this stage i killed the application after 3h. 

Any ideas? Does it just take a really long time or is something obviously wrong?

[Michael Hunger]

Which version are you using?

I think this is solved in 2.0

The index / label store population is there moved to batchinserter-shutdown

Sent from mobile device

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Johannes Tveitan]

I am using version 2.0 (both server and batch-import). 

Altought I now see after running the batch-importer that it got an exception on shutting down (it was in the shutdown phase for about 2h when importing 78M nodes). The time could be that i ran it with the configuration that came with the download.

The exception is as follows:

G:\batch-import-20>import.bat ../Neo4j/db ../CSV/Node_Author.csv

Usage: Importer data/dir nodes.csv relationships.csv [node_index node-index-name fulltext|exact nodes_index.csv rel_inde

x rel-index-name fulltext|exact rels_index.csv ....]

Using: Importer ../Neo4j/db ../CSV/Node_Author.csv

Using Existing Configuration File

.................................................................................................... 140480 ms for 10000

000

.................................................................................................... 162812 ms for 10000

000

.................................................................................................... 173473 ms for 10000

000

.................................................................................................... 186817 ms for 10000

000

.................................................................................................... 188060 ms for 10000

000

.................................................................................................... 192955 ms for 10000

000

.................................................................................................... 158847 ms for 10000

000

........................................................................................

Importing 78819880 Nodes took 1327 seconds

Exception in thread "main" org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: Unable to close store G:\batch

-import-20\..\Neo4j\db\neostore.propertystore.db.strings

        at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:603)

        at org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:122)

        at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:546)

        at org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:254)

        at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:546)

        at org.neo4j.unsafe.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:1062)

        at org.neo4j.batchimport.Importer.finish(Importer.java:88)

        at org.neo4j.batchimport.Importer.doImport(Importer.java:239)

        at org.neo4j.batchimport.Importer.main(Importer.java:83)

Caused by: java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open

        at sun.nio.ch.FileDispatcherImpl.truncate0(Native Method)

        at sun.nio.ch.FileDispatcherImpl.truncate(Unknown Source)

        at sun.nio.ch.FileChannelImpl.truncate(Unknown Source)

        at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:583)

        ... 8 more

Configuration:

use_memory_mapped_buffers=true

neostore.nodestore.db.mapped_memory=100M

neostore.relationshipstore.db.mapped_memory=500M

neostore.propertystore.db.mapped_memory=1G

neostore.propertystore.db.strings.mapped_memory=200M

neostore.propertystore.db.arrays.mapped_memory=0M

neostore.propertystore.db.index.keys.mapped_memory=15M

neostore.propertystore.db.index.mapped_memory=15M

batch_import.csv.quotes=false

I am also using 

i:id

and one 

type:label

other each node has 7 properties.

Thanx in advance!

[Mattias Persson]

I've seen that exception on Windows a couple of times, but I haven't been able to nail down what it is.

About the slow shutdown, there's indexing happening there and I hope 2.0.1 will address performance there.

2014/1/10 Johannes Tveitan <johannes...@monocl.com>

--
Mattias Persson, [mat...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com

[Steven Suing]

I'm getting this exact same issue. I'm brining in 5.5 mill nodes and 4 million relationships.

My box is 

        windows 8.1 64 bit

        jdk1.7.0_55 64 bit

        14 gb ram

        Neo4J 2.0.3 and Importer 2.0.

Same data set worked on windows 7, but then when doing queries it was really slow so I moved it to this bigger box in the cloud.

[Michael Hunger]

Can you share your messages log and how you imported the data (import log) did the importer take a long time at shutdown?

Sent from mobile device

--

You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Steven Suing]

Here is the output from the import run.

Using Existing Configuration File
.......................
Importing 2381482 Nodes took 74 seconds


Importing 6 Nodes took 0 seconds
...........................
Importing 2799354 Nodes took 122 seconds
..............................
Importing 3075420 Relationships skipped (4783) took 312 seconds
............................
Importing 2800134 Relationships took 251 seconds
Exception in thread "main" org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: Unable to close store C:\Users\Steven.Suing\Documents\Neo4j\ClientGraph\neostore.propertystore.db
        at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:603)

        at org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:254)
        at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:546)
        at org.neo4j.unsafe.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:1062)
        at org.neo4j.batchimport.Importer.finish(Importer.java:88)
        at org.neo4j.batchimport.Importer.doImport(Importer.java:239)
        at org.neo4j.batchimport.Importer.main(Importer.java:83)
Caused by: java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open
        at sun.nio.ch.FileDispatcherImpl.truncate0(Native Method)

        at sun.nio.ch.FileDispatcherImpl.truncate(FileDispatcherImpl.java:92)
        at sun.nio.ch.FileChannelImpl.truncate(FileChannelImpl.java:346)
        at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:583)
        ... 6 more

After the line that says Importing 2800134 Relationships took 251 seconds  it takes 15 minutes before the exception is thrown 

Here is my batch.properties i took some from one of your other posts.

cache_type=none
use_memory_mapped_buffers=true
# 9 bytes per node
neostore.nodestore.db.mapped_memory=500M
# 33 bytes per relationships
neostore.relationshipstore.db.mapped_memory=3G
# 38 bytes per property
neostore.propertystore.db.mapped_memory=500M
# 60 bytes per long-string block
neostore.propertystore.db.strings.mapped_memory=500M
neostore.propertystore.db.index.keys.mapped_memory=5M
neostore.propertystore.db.index.mapped_memory=5M


batch_import.node_index.clients=exact
batch_import.node_index.systems=exact
batch_import.node_index.technicalobjectkey=exact
batch_import.rel_index.clientrelation=exact

My messages file is here

All my data files are here

Here is the command I use to import them

Import.bat C:\Users\USERNAME\Documents\Neo4j\ClientGraph data/ClientNodes.csv,data/SystemNodes.csv,data/ObjectNodes.csv data/MainRelations.csv,data/ObjectToSystemRel.csv

This does work somewhat. I can do a couple basic queries but something doesn't seem right because the Neo4J process takes 3.5 gb of memory and queries are really slow. I'm afraid I have something wrong with my indexing.  I noticed a couple dependencies in the readme of the batch importer so I recently sorted my nodes and relationships, but that didn't help.

Thanks for any help you give.

Steve

[Michael Hunger]

If you google for the error message you'll see that it is a windows related problem of not releasing buffers quickly enough

"The requested operation cannot be performed on a file with a user-mapped section open"

So it seems during the automatic label store and index generation at the end it aborted leaving a partially inconsistent store.

Would you be able to try the import on linux or mac while we sort this out?

Sent from mobile device

[Steven Suing]

Since I'm a windows .net dev this took me a bit. I finally started up a Ubuntu instance on Azure and was able to import everything in 424 seconds.  Now to finally play with my data!

--

You received this message because you are subscribed to a topic in the Google Groups "Neo4j" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/neo4j/Xll7RP0lPX0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to neo4j+un...@googlegroups.com.