Re: [mongoose] Objects not found after multiple insert

[Aaron Heckmann]

Javascript closures + async i/o are biting you.

for(var i=0;i<10;i++) {

  var file = new FileModel({servername: "hello"+i, originalname: "world"+i});

  file.save(function(serr) {

Here's whats happening, all files are being created and saved as you intend and `i` is incremented each time.

  file.save(function(serr) {

 FileModel.findOne({servername: "hello10"}

Firstly, you should handle any potential `serr` returned. 

Second, and more importantly, by the time any save callback executes, the `for()` loop has already exited and `i` has been incremented again, which leaves it set to 10. So all of your callbacks are executing the same query for a document that doesn't exist:

   FileModel.findOne({servername: "hello10"}

Try this in your node console and see for yourself:

    for (var i = 0; i < 10; i++) console.log('i is %d', i );

  console.log('afterward i is %d', i) // 10

  

Sometimes it helps to enable mongoose debug mode to inspect what is getting sent to the db: 

  mongoose.set('debug', true)

On Wed, Nov 21, 2012 at 3:05 AM, Tug <deke...@gmail.com> wrote:

When I do multiple insert, objects are not found when callbacks are called.

Take this short code for instance : https://gist.github.com/4124267

Execution output is :

Connected to MongoDB

ERROR File not found

ERROR File not found

ERROR File not found

ERROR File not found

ERROR File not found

ERROR File not found

ERROR File not found

ERROR File not found

ERROR File not found

ERROR File not found

At the end of the execution, every objects are in the collection!

Is it some well known behavior or am I missing something ?

It's not limited to mongoose actually, I asked for help about the same thing in the node-mongod-native google group.

--
--
http://mongoosejs.com - docs
http://plugins.mongoosejs.com - plugins search
http://github.com/learnboost/mongoose - source code
 
You received this message because you are subscribed to the Google
Groups "Mongoose Node.JS ORM" group.
To post to this group, send email to mongoo...@googlegroups.com
To unsubscribe from this group, send email to
mongoose-orm...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/mongoose-orm?hl=en
 
 

--
Aaron

@aaronheckmann

[Tug]

Damn you are right! I forgot I was using the increment in the callback.

Thank you a lot for pointing me this out :-)

So, that was a dumb attempt of me trying to reproduce an issue I had when inserting a document in a replicaset configuration. Querying it just after would return null sometimes, even with w set to the number of servers. I'll have to have a closer look at it now.

Anyway, thanks again Aaron!

[Tug]

Ok I finally reproduced my original issue :

https://gist.github.com/4137177

Looks like it's related to the read_secondary option which is set to false by default.

But the thing is, even with w set to the number of nodes and poolSize set to 1, I still have read errors!

Is it a bug? What did I do wrong?

More generally, how can I distribute read load in MongoDB then?

Thanks !

[Aaron Heckmann]

I'm not sure what you mean by "read errors". Can you be more specific?

To read from secondary nodes use readPreferences: http://mongoosejs.com/docs/guide.html#read

I don't recommend setting `w` to the number of nodes in your replica set unless absolutely necessary. If any of your nodes go down or a network partition occurs the write will hang.

[Tug]

Well, when I run this code I almost always have some objects which are not found just after being inserted. Probably because the driver is reading from a secondary which hasn't the object yet.

Sure I won't set `w` to the max in production but I don't see how I can have safe reads otherwise?

Basically, the problem I'm having is that I want to scale mongodb gridfs as the number of reads per file globally grows, and not really as the number of files grows.

In fact, I think I don't really need sharding as I won't have many files in database at a given moment.
Any idea what could be a solution in my case ?

Thanks :)

[Aaron Heckmann]

Great find! 

mongoose has its default "safe" setting stored at the schema level (http://mongoosejs.com/docs/guide.html#safe). this setting is passed on to all writes which override the options specified in the call to connect(). this is not obvious or documented and surprising indeed.

solution:

  move the "safe" setting from the db option to your schema option OR set your schemas safe option to undefined.

short term:

  I'll get the documentation updated shortly.

long term:

  since acknowledged writes are now the default in the latest driver releases (http://blog.mongodb.org/post/36666163412/introducing-mongoclient), we'll review the mongoose implementation and preferably remove the default setting on schemas.

[Tugdual de Kerviler]

Thanks Aaron, this fixed my issue !

The safe object option seems to work as expected now.

However, I just found queries to be a LOT slower when j is set to 1. I'm going to investigate ;)

[Aaron Heckmann]

Enabling the j option forces write acknowledgements to wait until the write is in the journal. Journal commits happen every ~100ms but using the j flag forces a group commit sooner.

How much sooner isn't really specified.

[Aaron Heckmann]

long term fix is now in master: https://github.com/LearnBoost/mongoose/commit/e3fe3a3366a859d8602ad304f771f6c0fec26770

--
Aaron

@aaronheckmann