Error: connect EADDRNOTAVAIL while processing big async loop

[Jochen Brüggemann]

Hi,

I am experiencing a very strange problem. I am importing some big xml-files and store them into mongoDB. The algorythm is a typical async loop:

doLoop = function( it, callback_loop ) {

    if( it < no_of_items ) {

        storeToMongo( ..., function( err, result ) {

                ...

                doLoop( it+1, callback_loop );

        });

    } else {

        callback_loop();

    }

};

doLoop( 0, function() {

    ...

});

Now (suddenly without any remarkable change in the code) I get the following error while performing the loop:

events.js:72

        throw er; // Unhandled 'error' event

              ^

Error: connect EADDRNOTAVAIL

    at errnoException (net.js:901:11)

    at connect (net.js:764:19)

    at net.js:842:9

    at dns.js:72:18

    at process._tickCallback (node.js:415:13)

The error happens after approximately a minute. The number of items processed in the meantime is quite the same, but not exactly. 

I tried to find out what connect/net causes the error, but I am lost. There is not socket-connection in my code. I have a connection to redis, but that is o.k. Is it the mongoDB-connection? But why does it get lost suddenly?

The only thing that helps to run through the whole loop is to perform the recursive loop call within the mongo-callback like this:

setTimeout( function() {

    doLoop( it+1, callback_loop );

},1);

Anyone out there who has an idea what is going wrong here?

Thanks,

Jochen

[Ben Noordhuis]

I suspect that it makes a new connection on each call and that you're
calling it so often, that it exhausts the ephemeral ports on your
machine.

[Jochen Brüggemann]

Good idea, but no, that is not the case. I am calling 

    svr = ew mongodb.Server(...) and

    db = new mongodb.Db(...)

only one time at the beginning of the process. And even 

    coll = db.collection(...) 

is only called once per collection, because I am storing them in an array. From the second call on I use the array members.

[Jochen Brüggemann]

You have been right (in an way). I have been providing Solr also with the data in parallel (and I did it by error), so Solr has been unreachable after a while when beeing under massive add/commit attack!

Thanks for your hint :-)

Am Donnerstag, 11. Juli 2013 12:32:05 UTC+2 schrieb Ben Noordhuis: