Why is populate this slow?

[Robbie Clarken]

I have two collections: authors and books. The authors collection has a _books field which is an array of ObjectIds from the books collection. There are 100 authors and each references 25 books. If I do an Author.find().populate('_books') query it takes 12 seconds to execute. Is this to be expected?

I would think that mongoose only needs to do 100 * 25 = 2500 queries for the population. This number of queries only takes 0.2 seconds when I do it through the mongo shell.

Any suggestions for improving performance would be appreciated.

I am running:

• node v0.10.4
• mongoose v3.6.4
  
• mongodb v2.4.1

Example code:

var mongoose = require('mongoose');

var db = mongoose.connect('mongodb://localhost/poptest');
var authorSchema = new mongoose.Schema({
  name: {type: String, default: '?'},
  _books: [{type: mongoose.Schema.Types.ObjectId, ref: 'Book'}]
});
var bookSchema = new mongoose.Schema({
  title: {type: String, default: '?'}
});
var Author = mongoose.model('Author', authorSchema);
var Book = mongoose.model('Book', bookSchema);

/*
// Populate the database
for(var i = 0; i < 100; i += 1) {
  var books = [];
  for(var j = 0; j < 25; j += 1) {
    var book = new Book();
    book.save();
    books.push(book);
  }
  author = new Author({_books: books});
  author.save();
}
*/

// The following takes about 12 seconds to run
var startTime = (new Date()).getTime();
Author.find().populate('_books').exec(function(err, authors) {
  var endTime = (new Date()).getTime();
  console.log('Time taken:', endTime - startTime);
});

[Nathan White]

Differences between the shell and your program can be dramatic. It really depends on a few factors. Is your app running on the same machine as mongo? The serialization in the native driver will have a penalty. Depending on how complex your models are this can have a huge impact when documents are hydrating. You could also be hitting pool size issues with that many queries running in parallel.

Personally, I think this is a prime example of how not to use populate. I would query all authors, iterate and build a unique list of book ids. I would then query for all book ids. Build a hash of id => book doc. Loop through authors again and inject the docs into place. Then 2501 queries become 2.

--
Documentation - http://mongoosejs.com/
Plugins - http://plugins.mongoosejs.com/
Bug Reports - http://github.com/learnboost/mongoose
Production Examples - http://mongoosejs.tumblr.com/
StackOverflow - http://stackoverflow.com/questions/tagged/mongoose
Google Groups - https://groups.google.com/forum/?fromgroups#!forum/mongoose-orm
Twitter - https://twitter.com/mongoosejs
IRC - #mongoosejs
---
You received this message because you are subscribed to the Google Groups "Mongoose Node.JS ODM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongoose-orm...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Aaron Heckmann]

Thats a nasty regression in 3.6. There is a pull request to fix this but I'm off until tomorrow. https://github.com/LearnBoost/mongoose/pull/1426

--
Aaron

@aaronheckmann

[Aaron Heckmann]

Fixed in 3.6.5 which was just released.

[Robbie Clarken]

Thanks Aaron! That reduced my real project's page load time from over 30 seconds to 0.7 seconds.

--
Documentation - http://mongoosejs.com/
Plugins - http://plugins.mongoosejs.com/
Bug Reports - http://github.com/learnboost/mongoose
Production Examples - http://mongoosejs.tumblr.com/
StackOverflow - http://stackoverflow.com/questions/tagged/mongoose
Google Groups - https://groups.google.com/forum/?fromgroups#!forum/mongoose-orm
Twitter - https://twitter.com/mongoosejs
IRC - #mongoosejs
---

You received this message because you are subscribed to a topic in the Google Groups "Mongoose Node.JS ODM" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongoose-orm/lNbxx4oId_0/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to mongoose-orm...@googlegroups.com.

[Aaron Heckmann]

We badly need reliable automated benchmarks.

On Monday, April 15, 2013, Robbie Clarken wrote:

Thanks Aaron! That reduced my real project's page load time from over 30 seconds to 0.7 seconds.

On Tue, Apr 16, 2013 at 5:08 AM, Aaron Heckmann <aaron.h...@gmail.com> wrote:

Fixed in 3.6.5 which was just released.

On Monday, April 15, 2013 6:12:02 AM UTC-7, Robbie Clarken wrote:

I have two collections: authors and books. The authors collection has a _books field which is an array of ObjectIds from the books collection. There are 100 authors and each references 25 books. If I do an Author.find().populate('_books') query it takes 12 seconds to execute. Is this to be expected?

I would think that mongoose only needs to do 100 * 25 = 2500 queries for the population. This number of queries only takes 0.2 seconds when I do it through the mongo shell.

Any suggestions for improving performance would be appreciated.

I am running:

• node v0.10.4
• mongoose v3.6.4
  
• mongodb v2.4.1

Example code:

var mongoose = require('mongoose');

var db = mongoose.connect('mongodb://localhost/poptest');
var authorSchema = new mongoose.Schema({
  name: {type: String, default: '?'},
  _books: [{type: mongoose.Schema.Types.ObjectId, ref: 'Book'}]
});
var bookSchema = new mongoose.Schema({
  title: {type: String, default: '?'}
});
var Author = mongoose.model('Author', authorSchema);
var Book = mongoose.model('Book', bookSchema);

You received this message because you are subscribed to a topic in the Google Groups "Mongoose Node.JS ODM" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongoose-orm/lNbxx4oId_0/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to mongoose-orm...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Documentation - http://mongoosejs.com/
Plugins - http://plugins.mongoosejs.com/
Bug Reports - http://github.com/learnboost/mongoose
Production Examples - http://mongoosejs.tumblr.com/
StackOverflow - http://stackoverflow.com/questions/tagged/mongoose
Google Groups - https://groups.google.com/forum/?fromgroups#!forum/mongoose-orm
Twitter - https://twitter.com/mongoosejs
IRC - #mongoosejs
---

You received this message because you are subscribed to the Google Groups "Mongoose Node.JS ODM" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongoose-orm...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Aaron

@aaronheckmann

[Aaron Heckmann]

Thank github/vedmalex !

On Monday, April 15, 2013, Robbie Clarken wrote:

Thanks Aaron! That reduced my real project's page load time from over 30 seconds to 0.7 seconds.

On Tue, Apr 16, 2013 at 5:08 AM, Aaron Heckmann <aaron.h...@gmail.com> wrote:

Fixed in 3.6.5 which was just released.

On Monday, April 15, 2013 6:12:02 AM UTC-7, Robbie Clarken wrote:

I have two collections: authors and books. The authors collection has a _books field which is an array of ObjectIds from the books collection. There are 100 authors and each references 25 books. If I do an Author.find().populate('_books') query it takes 12 seconds to execute. Is this to be expected?

I would think that mongoose only needs to do 100 * 25 = 2500 queries for the population. This number of queries only takes 0.2 seconds when I do it through the mongo shell.

Any suggestions for improving performance would be appreciated.

I am running:

• node v0.10.4
• mongoose v3.6.4
  
• mongodb v2.4.1

Example code:

var mongoose = require('mongoose');

var db = mongoose.connect('mongodb://localhost/poptest');
var authorSchema = new mongoose.Schema({
  name: {type: String, default: '?'},
  _books: [{type: mongoose.Schema.Types.ObjectId, ref: 'Book'}]
});
var bookSchema = new mongoose.Schema({
  title: {type: String, default: '?'}
});
var Author = mongoose.model('Author', authorSchema);
var Book = mongoose.model('Book', bookSchema);

You received this message because you are subscribed to a topic in the Google Groups "Mongoose Node.JS ODM" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongoose-orm/lNbxx4oId_0/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to mongoose-orm...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Documentation - http://mongoosejs.com/
Plugins - http://plugins.mongoosejs.com/
Bug Reports - http://github.com/learnboost/mongoose
Production Examples - http://mongoosejs.tumblr.com/
StackOverflow - http://stackoverflow.com/questions/tagged/mongoose
Google Groups - https://groups.google.com/forum/?fromgroups#!forum/mongoose-orm
Twitter - https://twitter.com/mongoosejs
IRC - #mongoosejs
---

You received this message because you are subscribed to the Google Groups "Mongoose Node.JS ODM" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongoose-orm...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Aaron

@aaronheckmann