Re: [mongodb-user] Simple MongoDb aggregation sum time

[Sam Millman]

The speed could easily be improved however one thing to note is that your summing 1m docs as such MongoDB must iterate all 1m docs, it doesn't matter if they're in memory or not so it won't be the fastest thing in the universe even when optimised.

On 17 November 2012 20:52, Sam <sau...@medgle.com> wrote:

I did a vanilla install of mongodb on my AWS large instance with ~8GB of RAM.  I inserted 1M docs, with about 2-3 keys, around 60 bytes / doc as reported by Mongo's internal stats.  I then do a aggregate sum over one of the keys over all 1M records in the mongo shell.  The query takes ~2.3 seconds.  I was wondering why it takes this long for Mongo to compute this sum given that I'd expect most of my docs to reside in memory.  Note, these are the only docs in mongo.

I'm trying to use Mongo as my analytics engine but wanted to run some tests.

Thanks.

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb

[Stephen Steneker]

if the all the docs were stored in memory I'd expect the sum to be a magnitude faster.  Could someone explain why I'm in the seconds and not the msecs?  Is there a parameter of some sort I can toggle to get mongo to store the docs in memory?

Hi Shiv,

There isn't any specific parameter to toggle; MongoDB uses memory-mapped files and will keep indexes and documents in memory based on your working set (ie. what you actively query).  Over time this should grow to fit available memory: http://www.mongodb.org/display/DOCS/Caching.

Some suggestions for you to test:
 - run mongostat (http://docs.mongodb.org/manual/reference/mongostat/) while your aggregation is running to monitor activity

 - also helpful to monitor I/O with iostat (http://www.mongodb.org/display/DOCS/iostat) and memory usage with vmstat (http://www.mongodb.org/display/DOCS/Checking+Server+Memory+Usage)

 - compare the performance for the initial run versus successive runs; perhaps the data has not been loaded into memory yet

 - on a cold server start you could use the new touch() command in MongoDB 2.2 to forcibly load indexes/data into memory

    http://docs.mongodb.org/manual/reference/command/touch/

Can you also share the aggregation query you are using?

Cheers,

Stephen