Namespace limits

[Phillip B Oldham]

In the document "Using a Large Number of Collections" there is the
following paragraph:

> Currently MongoDB has a limit of approximately 24,000 namespaces per
> database. Each collection counts as a namespace, as does each index.
> Thus if every collection had one index, we can create up to 12,000 collections.

Elsewhere in the docs it states that _id is indexed by default. Is it
the case then that technically there are is already a limit to 12,000
collections? Or is the _id index count outside of the namespace limit?

[1] http://www.mongodb.org/display/DOCS/Using+a+Large+Number+of+Collections

[Dwight Merriman]

Right. 12,000 effectively as _id index counts.

(by the way this limit is a constant and could be changed with a recompile)

[Phillip B Oldham]

On Jul 29, 8:04 pm, Dwight Merriman <dwi...@exch.alleycorp.com> wrote:
> Right.  12,000 effectively as _id index counts.
>
> (by the way this limit is a constant and could be changed with a recompile)

Heh, 12k is probably enough for most people. However, I would suggest
that the docs be changed to acknowledge that _id indexes counts
towards the total.

[Nicolas Fouché]

In another thread, I read that it's a good idea to make as much as
collection as possible instead of putting all data in one single
collection. For example, making one collection per user. But if you
have let's say 10 indexes per collection (+ the _id), you'll be able
to create: 12,000 / 10 = 1,200 collections ?

The documentation says that setting a big limit in the MongoDB code
should work fine. http://www.mongodb.org/display/DOCS/Using+a+Large+Number+of+Collections

Would you consider that having like 50,000 users, which means having
50,000 collection in the same database, is relevant ? Or should I make
one database per x users ? Does the incoming sharding feature will
allow not to worrying about this ?

If I change the value of LEN in namespace.cpp afterwards, will
existing databases still be readable ?

[Eliot Horowitz]

Where did you see a recommendation of 1 collection per user? I don't
think thats a good idea. Assuming you mean user of your wesbite or
something like that.

[Nicolas Fouché]

I finally found the original thread :
http://groups.google.com/group/mongodb-user/browse_thread/thread/87baaf84bec3f20b/1663a6e8cbb34111?lnk=gst&q=collection+database#1663a6e8cbb34111

I guessed that having one collection per user (yes, user of the
application) could be a good idea. So we can manage data per user more
easily : import/export, data migration . It would also be useful if a
repair could be applied to single collections.
Possible corruption would not impact every users.

Having a lot of collections means less data to retrieve for each
query, less data and/or indexes to scan.

Migrating users to new features can be done progressively by migrating
data one user after another.

It also allows to add indexes at runtime for a specific user once he
logs in, instead of adding index for all the data we have at once. It
would mean that users that do not come back don't waste time, CPU and
disk space for new indexes.

Also, I don't totally understand the upcoming sharding feature. Is it
made only to shard one big collection ?
From http://www.mongodb.org/display/DOCS/Sharding+Design: "A chunk can
be though of as (collectionname,fieldname,lowvalue,highvalue"
In this case my "one collection per user" setup would not work :)

Why do you think that's not a good idea ? I thought it was easier to
scale when you have the chance to be able to split up data in multiple
as little collections/tables as possible. In our case, users do not
share data with each other, and they can easily have a collection of
several hundred of megabytes.

On Aug 11, 2:29 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> Where did you see a recommendation of 1 collection per user?  I don't
> think thats a good idea.  Assuming you mean user of your wesbite or
> something like that.
>
>
>
> On Tue, Aug 11, 2009 at 5:43 AM, Nicolas Fouché<nicolas.fou...@gmail.com> wrote:
>
> > In another thread, I read that it's a good idea to make as much as
> > collection as possible instead of putting all data in one single
> > collection. For example, making one collection per user. But if you
> > have let's say 10 indexes per collection (+ the _id), you'll be able
> > to create: 12,000 / 10 = 1,200 collections ?
>
> > The documentation says that setting a big limit in the MongoDB code

> > should work fine.http://www.mongodb.org/display/DOCS/Using+a+Large+Number+of+Collections

[Eliot Horowitz]

Maybe I'm not undestanding your problem.

If you have thousands of users, having 1 collection (or db) per user
doesn't make sense. If you have 1 million users, you certainly don't
want a collection or db per user.

What are you storing per user? Is this more a virtual db hosting
thing rather than site users?

If you have 100 users, and 1M records per user, than 1 collection per
user could make sense.
If you have 1 record per user, and millions of user, than certainly 1
collection for all users make sense.


sharding is per collection, but you can shard as many collections per
database as you want.

[Nicolas Fouché]

Users have between 50,000 and 500,000 records, and currently several
thousand people are waiting for an invitation (we're in private
alpha). So you can guess the number of users once we fully open... It
seems that it does not fit in your scenarii :)

It's a kind of a personal archive of messages: emails, sms, etc. A
collection (for one user) can take from 50MB up to 500MB.

Having all users in one single collection would mean handling one
collection of several terabytes. Not quite easy to manage I guess.

Knowing the numbers, what would you recommend ? I suppose at least
have one database per group of users. And then, about the
collections ...

On Aug 11, 6:53 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> Maybe I'm not undestanding your problem.
>
> If you have thousands of users, having 1 collection (or db) per user
> doesn't make sense.  If you have 1 million users, you certainly don't
> want a collection or db per user.
>
> What are you storing per user?  Is this more a virtual db hosting
> thing rather than site users?
>
> If you have 100 users, and 1M records per user, than 1 collection per
> user could make sense.
> If you have 1 record per user, and millions of user, than certainly 1
> collection for all users make sense.
>
> sharding is per collection, but you can shard as many collections per
> database as you want.
>
> On Tue, Aug 11, 2009 at 12:49 PM, Nicolas
>
>
>
> Fouché<nicolas.fou...@gmail.com> wrote:
>
> > I finally found the original thread :

> >http://groups.google.com/group/mongodb-user/browse_thread/thread/87ba...

>
> > I guessed that having one collection per user (yes, user of the
> > application) could be a good idea. So we can manage data per user more
> > easily : import/export, data migration . It would also be useful if a
> > repair could be applied to single collections.
> > Possible corruption would not impact every users.
>
> > Having a lot of collections means less data to retrieve for each
> > query, less data and/or indexes to scan.
>
> > Migrating users to new features can be done progressively by migrating
> > data one user after another.
>
> > It also allows to add indexes at runtime for a specific user once he
> > logs in, instead of adding index for all the data we have at once. It
> > would mean that users that do not come back don't waste time, CPU and
> > disk space for new indexes.
>
> > Also, I don't totally understand the upcoming sharding feature. Is it
> > made only to shard one big collection ?

> > Fromhttp://www.mongodb.org/display/DOCS/Sharding+Design:"A chunk can

[Eliot Horowitz]

I think having 1 massive collection wouldn't be so bad.
Unless you think you'll have massively different indexing needs for
different users, there isn't really a performance penalty.
Once sharding is out, you can shard on user, and everything should work nicely

The other option might be to put 1000 users per database, and then
have 1 collection in that database.
That just keeps any given database smaller, but the total size is
still big, so not sure how much that really gets you.

-Eliot

[Nicolas Fouché]

Thanks a lot for the advices.

I'll have to find what operations put read/write locks at collection/
database level. For example, I don't know the status of the "When I
call remove() then it's blocking reads from the entire database"
problem
http://groups.google.com/group/mongodb-user/browse_thread/thread/810f410f1c997a52/e91566b5f940f226?lnk=gst&q=remove#e91566b5f940f226.

Also, being able to have statistics per collection (so per user) is
very useful http://www.mongodb.org/display/DOCS/Validate+Command.
About the "migrate user to new versions" feature, I could also move
users from one database to another with a conditional export...

Anyway, can you explain why you would not create one collection per
user in this case ?
Would it be about performance ? Too much namespaces ? Switch between
collections is costly ? Inneficient caching ? Waste RAM/disk space ?

Nicolas

[Eliot Horowitz]

1 collection per user just becomes a bit of a management headache.
Given that you'll need indexes, you probably won't want to put more
than 1000 users in a db, so then you'll need hundreds of dbs.

So from a performance standpoint is fine, its really just a management
issue in my mind.
Though it should work fine, and there are some advantages as you point out.
Another advantages is that we'll be doing more concurrency work, so
having separate collections will be slightly beneficial.
So if you're willing to take the management hit, might be good for you.

[Nicolas Fouché]

Thanks a lot for the details. We'll keep one collection per user for
now, and we'll see what happens when we get 1000 users.

By curiosity, does changing LEN in namespace.cpp break the
compatibility with existing databases ?

On Aug 11, 10:55 pm, Eliot Horowitz <eliothorow...@gmail.com> wrote:
> 1 collection per user just becomes a bit of a management headache.
> Given that you'll need indexes, you probably won't want to put more
> than 1000 users in a db, so then you'll need hundreds of dbs.
>
> So from a performance standpoint is fine, its really just a management
> issue in my mind.
> Though it should work fine, and there are some advantages as you point out.
> Another advantages is that we'll be doing more concurrency work, so
> having separate collections will be slightly beneficial.
> So if you're willing to take the management hit, might be good for you.
>
>
>
> On Tue, Aug 11, 2009 at 3:26 PM, Nicolas Fouché<nicolas.fou...@gmail.com> wrote:
>
> > Thanks a lot for the advices.
>
> > I'll have to find what operations put read/write locks at collection/
> > database level. For example, I don't know the status of the "When I
> > call remove() then it's blocking reads from the entire database"
> > problem

> >http://groups.google.com/group/mongodb-user/browse_thread/thread/810f....

>
> > Also, being able to have statistics per collection (so per user) is

> > very usefulhttp://www.mongodb.org/display/DOCS/Validate+Command.

[Eliot Horowitz]

Yes. If you change that, the database won't be able to read old files.
You would have to dump/restore.