Re: NodeJS MongoDB native driver issue during inserting lot of records. Is it the driver memory leak issue?

[Gianfranco]

This seems more like a memory limit on NodeJS' V8 engine.

See this:

http://stackoverflow.com/questions/7193959/memory-limit-in-node-js-and-chrome-v8

On Tuesday, October 30, 2012 12:17:15 PM UTC, Oleg Seriaga wrote:

Hi, all

I will really appreciate if someone can explain me the reason of issue. 

I faced the memory leak issue during inserting 10 million records into MongoDB using NodeJS native driver (Windows 7 / 64 bit). I had only 1.5 million inserted, node.exe process took more than 1GB RAM and the following error occurred FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory. 

Please find the code of nodeunit test below. 

var mongo = require('../code/node_modules/mongodb');

var MONGO_URL = "mongo://127.0.0.1:27017/mongoTests?safe=false";

module.exports = {

    "insert 10 million records into mongo": function (test) {

        mongo.Db.connect(MONGO_URL, function (err, db) {

            if (err) {

                test.fail(err);

                return;

            }

            var countOfRecordsToBeInserted = 10000000;

            var countOfInsertedRecords = 0;

            var collection = db.collection('test');

            var onInsert = function (err) {

                if (err) {

                    db.close();

                    test.fail(err);

                    test.done();

                    return;

                }

                countOfInsertedRecords++;

                if (countOfInsertedRecords >= countOfRecordsToBeInserted) {

                    db.close();

                    test.done();

                }

            };

            var insertRecord = function (data) {

                collection.insert(data, onInsert);

            };

            for (var i = 0; i < countOfRecordsToBeInserted; i++) {

                var data = {

                    a: 'text ' + i,

                    b: i,

                    d: new Date()

                };

                insertRecord(data);

            }

        });

    }

};

[Graeme Foster]

Ah - just dug around a bit and found this issue: https://github.com/mongodb/node-mongodb-native/issues/778#issuecomment-10712665

Without the optional callback parameter the function gives the impression that it's synchronous (even if still fire & forget) and so I assumed it was safe to call like that. Phew!

G.

 

On Thursday, January 3, 2013 1:52:04 PM UTC, Graeme Foster wrote:

Yes - I've just run into this with an extremely simple node program that just loops calling collection.insert() with an object. If I remove the call to collection.insert() then it runs fine, otherwise I get the same error. This is on OS X.

G.

On Monday, November 5, 2012 4:24:45 PM UTC, Jeroen Janssen wrote:

Has anyone else seen similar 'behaviour'? Or maybe knows what exactly is causing this?

[Matt]

Do you have an updated link on this? Or can share any info from that page?

Looks like the issues on github have been pulled and can't seem to locate it on JIRA.

Matt

[Bharath S]

Can somebody post the JIRA links please ?

Thanks.

[Stephen Steneker]

On Tuesday, 29 December 2015 05:01:50 UTC+11, Bharath S wrote:

Can somebody post the JIRA links please ?

Hi Bharath,

You're replying to a discussion thread from several years ago; versions of the driver and Node.js have changed significantly since then.

If you have a specific question, can you please start a new discussion with details relevant to your environment (driver version, code snippet, etc).

Thanks,

Stephen