nodejs mongoose bulk update

[Claudio Pomo]

I've a collection of data and I need to add for every documents a new field. If I run a query to get all documents and the update every single one node.js is stopped, may be for memory leak

This is my code

   

var express = require('express');
   
    var geocoderProvider = 'google';
    var httpAdapter = 'http';
   
    var People = require("./models/people").collection.initializeOrderedBulkOp();
   
    var app = express();
   
    var geocoder = require('node-geocoder').getGeocoder(geocoderProvider, httpAdapter, {});
   
    app.get('/', function (req, res) {
      People.find({}, function (err, docs) {
        if (err) {
          res.send(err);
        }else{
          docs.forEach( function (doc){
            geocoder.geocode({address: doc.address, country: 'Italy', zipcode: doc.cap}, function(error, value) {
              doc.loc.coordinates[0]=value[0].latitude;
              doc.loc.coordinates[1]=value[0].longitude;
              People.update({ _id: doc._id }, { $set: { loc: doc.loc }}, { multi: true }, function (error){
                if(error){
                  console.error('ERROR!');
                }
              });
            });
          });
        }
      });
    });
   
    var server = app.listen(3000, function () {
      var host = server.address().address
      var port = server.address().port
      console.log('Example app listening at http://%s:%s', host, port)
    });

There is any way to bulk update with mongoose?

Thanks in advance

[Jason Crawford]

I don't think there's a way to bulk update if you're setting a different value on every record.

Try a query stream instead. That will return the items one at a time, you can process each one and then move on:

http://mongoosejs.com/docs/api.html#query_Query-stream

Hope that helps,

Jason

--
Blog: http://blog.jasoncrawford.org  |  Twitter: @jasoncrawford

--
Documentation - http://mongoosejs.com/
Plugins - http://plugins.mongoosejs.com/
Bug Reports - http://github.com/learnboost/mongoose
Production Examples - http://mongoosejs.tumblr.com/
StackOverflow - http://stackoverflow.com/questions/tagged/mongoose
Google Groups - https://groups.google.com/forum/?fromgroups#!forum/mongoose-orm
Twitter - https://twitter.com/mongoosejs
IRC - #mongoosejs
---
You received this message because you are subscribed to the Google Groups "Mongoose Node.JS ODM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongoose-orm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ryan Wheale]

If you don't go the query stream route, I would recommend batching the updates instead of doing one at a time.  Load 1000 records, calculate the coordinates, build an in-memory object, and update all in a single call... rinse and repeat.

Also, do lean queries to save a lot of overhead.  Your code is not really relying on mongoose documents (eg. you're not using doc.save()), so lean makes sense.  I actually recommend always doing lean queries until you actually need do use the mongoose document API.  Even further, you may want to write updates like this against the native mongo API and skip mongoose.  

~Ryan

[Claudio Pomo]

I've changed my code

app.get('/', function (req, res) {

  People.find().stream().concurrency(2).work(
    function (doc, done) {

        geocoder.geocode({address: doc.address, country: 'Italy', zipcode: doc.cap}, function(error, value) {

          if(!error){

            doc.loc.coordinates[0]=value[0].latitude;
            doc.loc.coordinates[1]=value[0].longitude;

            doc.save(function (err, doc, numberAffected) {
                if (err){
                  console.error("!!!!!!!!!"+error+"!!!!!!!!!!!!!\n"+doc.surname+" "+doc.name+" "+doc.city);
                }
            });
            console.log(doc.surname+" "+doc.name+" "+doc.city);
            done();
          }else{
            console.error("!!!!!!!!!"+error+"!!!!!!!!!!!!!\n"+doc.surname+" "+doc.name+" "+doc.city);
          }
        });
    },
    function (err) {
      console.log(doc.surname+" "+doc.name+" "+doc.city+" completed!");
    }
  );
});

Now I can't understand how to save my update. When I call

doc.sav(....);

any error is returned but into my db all documets are the same. I'm desperate!

[Valeri Karpov]

Enter code here...

Mongoose doesn't really support the "initializeOrderedBulkOp()" function atm, because it doesn't work with mongoose's internal buffering system - you have to wait for mongoose to successfully connect to the db before using "initializeOrderedBulkOp()".

Also, please make sure you're using the correct syntax for mongoose query streams. See SO: http://stackoverflow.com/questions/7372626/how-to-stream-mongodb-query-results-with-nodejs, for some examples.

...