Re: Array length

[Jenna deBoisblanc]

Hello,

Your concern about unbounded array growth is well placed.  If the document grows beyond the maximum allocated space- 16MB - the document will have to be moved, which is an expensive process.  One solution may be to keep track of the array length, possibly by storing a counter field that you $inc every time an element is added to the array, and when the array reaches a certain size, you can create a new document.

On Thursday, October 18, 2012 10:33:12 PM UTC-4, uchoaaa wrote:

Hi,

My collection has a array data type (always update with $addToSet to be only unique elements) and from only 2 days ago got 12.000 elements..

Everything is ok, but I'm a little concerned.. is there a length limit to arrays? It could generate some performance issue? Is there a "best practice" to big data array?

Regards,

[uchoaaa]

Hi Jenna, thanks for the answer..

Yeah, I'm using a counter field update with $inc.. But how do I know the time to create a new document? I mean, how do I know the size of my array field?

I'm wondering if there is other approach.. If I move the array data to a collection? A collection with 15 docs is better than a array field with 15 elements??

[Asya Kamsky]

Great question: you can certainly have N documents associated with a common parent entity rather than having N element array in the parent document.  In many cases that is advisable - those include cases where these array can grow unbounded, when you might want to query across all array elements, etc.

Rather than using $addToSet you can use $push that's conditional on the array element not already being present in the array.

Something like:   db.collection.update({doc.array:{$ne:NEWVALUE}}, {$push:{array:NEWVALUE},$inc:{counter:1}})

If you describe what your array represents in your schema design and how you use it, it's possible we may be able to suggest another approach to modeling it.

[uchoaaa]

Thanks Asya.

Is there difference between using $addToSet and $push conditional on the array element not already being present in it?

Actually my collection represents a Page and Unique Visitors, so every time any system page is loaded, I $inc visits field,  put the visitor ID on unique_visitors array with $addToSet and $inc visitors_total. There is Pages with  unique_visistors array size 15.000..

I'm not interested to list all visitors ID or to query it.

It's growing quite fast, I concern if there is the time to change my schema...

Best,

[Jenna deBoisblanc]

You can estimate doc size in the shell using the command, Object.bsonsize(doc), which will give you a way to estimate how large your array can grow before it approaches the max document size.

http://www.mongodb.org/display/DOCS/Documents

The important part of the conditional $push is the query portion of the update command.  {doc.array:{$ne:NEWVALUE}} ensures that you only $inc the counter field when new values are added to the array.

> db.collection.update({doc.array:{$ne:NEWVALUE}}, {$push:{array:NEWVALUE},$inc:{counter:1}})

won't find the document if NEWVALUE is already in the array, so it won't update the the document by $inc'ing the counter field. 

> db.collection.update({doc.array:{$ne:NEWVALUE}}, {$addToSet:{array:NEWVALUE},$inc:{counter:1}})

will have the same result

To help provide schema design advice, could you show us a sample document, your most common queries, and/or common commands?