BSON Size limit reached

[chris]

Hello,

Some of the structures now seem to reach the limits of MongoDB again ("com.mongodb.MongoInternalException: DBObject of size 21253420 is over Max BSON size 16777216"). When this happended first a year ago or so, when the limit was still 4 MB, I modified my code to use object references and collections for the sub-documents. While this would be possible again for the sub-documents of the sub-documents, it would require a large amount of development effort and in addition in the general case would severe performance. As this error happens not predictable, after all, I don't know for sure, what the attribute values of the documents will be and thus it is unclear at what point and for which structures this limit will be a problem, I'd be glad to hear some suggestions on how to solve / work around the problem. 

I'd gladly prefer to have such kind of limit optional. I am storing more than 8 TerraByte in the database, and I don't see a problem in an object size of 21 MB instead of 16 MB.

Thanks for any hints!

Chris

[Octavian Covalschi]

Are you storing files/bytes in each record? If yes, perhaps you should use GridFS, if not... then, wow it's a lot of data... 

Chris

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb

[chris]

I was a bit in a hurry when posting. Yes, it's mostly binary files already in GridFS, but also a lot of numeric data and meta data (where this error now is happening).

I now know that this error happens when some meta data is missing and the sub documents are not split into different categories. It is a kind of user error where the users are not following my expectations, but it is also a kind of problem of MongoDB, that you need to be sure where which amount of data can be expected, as soon as there is more data at an unexpected position, the application breaks.

[Sam Millman]

You *can* edit Mongo to actually accept a larger document, however memory management etc is orientated around 16meg (there is a reason for 16meg, there is a relative explanation here: http://stackoverflow.com/questions/4667597/understanding-mongodb-bson-document-size-limit).

Best thing to do is: when you save check the size of the document you have by converting it to BSON and then measuring it's string size or byte length. If the document is too big you might wanna notify the user.

That is probably the best thing to do to solve the problem atm, there is JIRA for increasing the doc size: https://jira.mongodb.org/browse/SERVER-5923