Mongoose query (find()) doesn't seem to be using Index

[thoughtAddict]

Hello. Wondering if someone could point me in the right direction. I
have a feeling it's something simple that I'm missing....

I'm seeing that when I run a "find" on one of my collections, the
MongoDB Shell response is in milliseconds, while the Mongoose response
is in seconds.

I'm wondering if it's:
------------------------------------------
1. Due to my creation of the objects initially without an "index"
1a. I later indexed the same collection on the fields in the "find"
2. Something I'm doing incorrectly within the Mongoose "find" command


Background
============================================

** Schema
**------------------------------------------
var SessionSchema = new Schema({
customerId : { type: String }
, startTimeMS : { type: Number }
, ...
, ...
, ...
});

** Same Schema (Changed later to indexing)
**------------------------------------------
var SessionSchema = new Schema({
customerId : { type: String, index: true }
, startTimeMS : { type: Number, index: true }
, ...
, ...
, ...
});

** Indexes (Added at same time with the Schema indexing above, but not
initially used)
**------------------------------------------
SessionSchema.index({ "customerId": 1 });
SessionSchema.index({ "startTimeMS": 1 });


** I also tried a Compound index, which didn't appear to make a
difference
**------------------------------------------
SessionSchema.index({ "customerId": 1, "startTimeMS": 1 });


** Init the model
**------------------------------------------
mongoose.model('Session', SessionSchema)
var Session = mongoose.model('Session');


** Then I have a script that randomly generates some test Sessions for
me
**------------------------------------------

var session = new Session();
session.startTime = (new Date()).setTime(startDateMS);
session.startTimeMS = startDateMS;
session.customerId = customerId;
session.etc
session.etc
session.etc
session.etc

session.save(function (err) {
if (!err) {
console.log("SESSION Creationg: (" + session.startTime +
")");
} else {
console.log(err);
}
});


** I have 60,000 of these Session objects in a "sessions" collection
** When executing in the MongoDB Shell:

db.sessions.find( { "customerId":"4e2fcc15a05e8e7f57000001",
"startTimeMS":{ "$gte":1312135543649, "$lte":1312394743649 } } )

** I get

{
"cursor" : "BtreeCursor startTimeMS_1",
"nscanned" : 6867,
"nscannedObjects" : 6867,
"n" : 6867,
"millis" : 60,
"nYields" : 0,
"nChunkSkips" : 0,
"isMultiKey" : false,
"indexOnly" : false,
"indexBounds" : {
"startTimeMS" : [
[
1312135543649,
1312394743649
]
]
}
}

** 60 milliseconds is fine with me, but when I try the same with a
Mongoose "find"


Session.find(
{
"customerId":cid,
"startTimeMS":{
"$gte":startTimeMS,
"$lte":endTimeMS
}
}, callback);

** or

Session.where("customerId",
cid).where("startTimeMS").gte(startTimeMS).lte(endTimeMS).find(callback)

** it takes about 9 to 11 seconds.


I've reindexed my collection on:

customerId
startTimeMS
and both (compound)


Any ideas?

Thanks,

Jon

[Aaron Heckmann]

The mongo shell is returning a cursor which you use to iterate your docs 20 at a time. The mongoose query is loading all documents into memory at once. How many results are you returning?

> --
> http://mongoosejs.com
> http://github.com/learnboost/mongoose
> You received this message because you are subscribed to the Google
> Groups "Mongoose Node.JS ORM" group.
> To post to this group, send email to mongoo...@googlegroups.com
> To unsubscribe from this group, send email to
> mongoose-orm...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/mongoose-orm?hl=en

[thoughtAddict]

I'll be returning anywhere from 3000 to 18000 (estimated) records.

My server-side code is hitting a MongoDB server to return a series of
data that will be used to populate a number of charts on the client-
side. There could be anywhere from 1 to 6 charts displayed at any
one time.

Hm. Looks like I might have to redesign how the client-side is
getting that data.

* So for large data sets such as these, would it be recommended to
just send them in chunks to the UI?
* Does Mongoose store *all* responses in memory?
* Are there examples that will show optimal ways for using that query
memory?

Thanks for the help.

Jon

On Aug 4, 8:00 pm, Aaron Heckmann <aaron.heckm...@gmail.com> wrote:
> The mongo shell is returning a cursor which you use to iterate your docs 20 at a time. The mongoose query is loading all documents into memory at once. How many results are you returning?
>

[Justin]

Try this:

https://github.com/LearnBoost/mongoose/blob/master/test/model.querying.test.js#L1350

You can use 'each' to stream each doc instead of loading all the docs
into memory at once. As the docs return, you can chunk them out to the
client.

[thoughtAddict]

Thanks much, Justin. 4 pigs with 1 bird, that solved not only the
current issue, but a few down the line...

I had to change the front-end code a bit, (actually, quite a lot), but
in doing so now there's less on the server-side.

Jon


On Aug 8, 11:28 am, Justin <jwcoo...@gmail.com> wrote:
> Try this:
>
> https://github.com/LearnBoost/mongoose/blob/master/test/model.queryin...