Mongoimport of large data

[Nicolas]

Hi everybody,

We try to store a graph by using MongoDB.
We want to store our graph as an adjacency list : each node has an
identifier and a list of nodes which he's linked to.
{ id : identifier_of_the_node,
out : [ list_of_identifier_of_the_nodes_he_s_linkedto]
}

However, the link of nodes stored in the "out" field can be really
large : 500 000 nodes for example.
Thus, we don't know how to load efficiently our graph into mongoDB.
Using mongoimport allows us to load really fast our data (Json
format) but it seems not to work with large "out" array : User
Assertion: 10263:unknown error reading file mongodb
And using the JAVA API, it's too slow...
How can we do ?

Moreover, if we want to add a field "in" in our object to know
which node are linked to the current node :
{ id : identifier_of_the_node,
out : [ list_of_identifier_of_the_nodes_he_s_linkedto],
in :
[ list_of_identifier_of_the_nodes_which_are_linked_to_this_one],
}
we may have 3,5millions of integer in the "in" array.
Is it possible to store it in MongoDB ? I saw that we may use
GridFS for heavy documents... should we use it ? Can we do Map/reduce
on GridFsFile, can we query it easily and fastly ?

Thanks a lot

[A. Jesse Jiryu Davis]

Be aware of the 16MB-per-document limit -- if you have 3.5M integers in an array, each int will require a type header (1 byte), an index (about 8 bytes, if your indexes are in the millions), and the int (4 bytes), resulting in around 16 bytes to store each int. So you can't store more than about 1M integers in a single document's array. See http://bsonspec.org/#/specification

This may be the reason for your "Assertion: 10263:unknown error reading file mongodb " message.

How is your graph structured? Is it a tree, a directed acyclic graph, or can it be cyclic?

[Nicolas]

Thanks for your answer Jesse.

Yes, we are aware of the size limit. that's why, currently, we don't
use the "in" array with 3,5M integers. We just use the "out" array
which maximum size is 500 000 integers.
So, we're still wondering why there is an error, it may work.

Our graph is directed and contains cycles.

On 23 mai, 19:02, "A. Jesse Jiryu Davis" <je...@10gen.com> wrote:
> Be aware of the 16MB-per-document limit -- if you have 3.5M integers in an
> array, each int will require a type header (1 byte), an index (about 8
> bytes, if your indexes are in the millions), and the int (4 bytes),
> resulting in around 16 bytes to store each int. So you can't store more
> than about 1M integers in a single document's array.

> Seehttp://bsonspec.org/#/specification

[A. Jesse Jiryu Davis]

Unfortunately, mongoimport *also* has a limit: it can only read lines up to 16MB long. This limit is unrelated to the 16MB limit on bson documents.

You could change BUF_SIZE in our source code, in import.cpp, recompile mongoimport, and try again. But I think this problem is an indication that you should redesign your schema.

[Nicolas]

Maybe we should redesign our schema, I don't know. But the adjacency
list seems to be the most efficient way to query the graph in our use
case :/

I'll try what you suggest tomorrow, thanks for your answer.

[A. Jesse Jiryu Davis]

There are various ways you could work around the 16MB limit in mongoimport and in MongoDB itself. But MongoDB isn't very good at storing and querying graphs, especially the very complex graphs you're talking about. You should consider Neo4j or another specialized graph database for this purpose. MongoDB might still be useful in conjunction with the graph database.