Error: EMFILE, Too many open files

[kindy]

$ cat tt.js

var sys = require("sys"), 

   http = require("http");

http.createServer(function (request, response) {

  response.writeHead(200, {"Content-Type": "text/plain"});

  response.write("Hello World\n");

  response.close();

}).listen(8000);

sys.puts("Server running at http://127.0.0.1:8000/");

$ node tt.js

$ ab -c 1000 -n 3000 http://127.0.0.1:8000/

Server running at http://127.0.0.1:8000/

Error: EMFILE, Too many open files

    at IOWatcher.callback (net:810:22)

    at node.js:815:9

anybody know this?

--
Regards,

林青(Kindy Lin)

[Ryan Dahl]

(I love descriptive stack traces!)

In the C version, when I got this error (retuned from accept()) I
would disable the server until another fd was closed. Apparently I
forgot to do this in net2. It should be an easy patch - anyone want to
try? (Play with test/pummel/test-tcp-many-clients.js to demonstrate.)

[kindy]

it seems that the latest node(v0.1.33-134-g6f84063) is OK.

:)

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

--
Regards,

林青(Kindy Lin)

[Ryan Dahl]

On Tue, Mar 23, 2010 at 8:31 AM, kindy <kin...@gmail.com> wrote:
> it seems that the latest node(v0.1.33-134-g6f84063) is OK.

No, I believe it is broken with respect to dieing on EMFILE.

[Jorge]

No, but I'm getting this err after a few runs of ab -n 5000 -c 10 http://127.0.0.1:12345/ :-|

$ cat emptyServer.js

var sys= require("sys");

var http= require("http");

var port= 12345;

http.createServer(function (request, response) {

  response.writeHeader(200, {

    "Content-Type": "text/plain",

    "server":"Node.js"

  });

  response.write(" ");

  response.close();

}).listen(port);

sys.puts("Server running at http://localhost:"+ port+ "/");

$ node emptyServer.js

$ ab -n 5000 -c 10 http://127.0.0.1:12345/

Benchmarking 127.0.0.1 (be patient)

Completed 500 requests

Completed 1000 requests

apr_poll: The timeout specified has expired (70007)

Total of 1379 requests completed

?

I'm running node-v0.1.33 on OSX 10.6.2

[Jorge]

On Mar 25, 1:28 am, Jorge <jo...@jorgechamorro.com> wrote:
> $ ab -n 5000 -c 10http://127.0.0.1:12345/

>
> Benchmarking 127.0.0.1 (be patient)
> Completed 500 requests
> Completed 1000 requests
> apr_poll: The timeout specified has expired (70007)
> Total of 1379 requests completed
>
> ?
>
> I'm running node-v0.1.33 on OSX 10.6.2

If instead I do an

$ ab -n 50000 -c 1 http://127.0.0.1:12345/

It errs at the very suspicious number of 16383 requests:

Benchmarking 127.0.0.1 (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests

apr_poll: The timeout specified has expired (70007)

Total of 16383 requests completed

?
--
Jorge.

[Peter Sunde]

Just ran into the same error.

Running now with sudo gdb --args /software/node/node_g server.js and i
will provide a stack trace when/if this occurs again.

Regards,
Peter Sunde.

On 25 Mar, 02:55, Jorge <jo...@jorgechamorro.com> wrote:
> On Mar 25, 1:28 am, Jorge <jo...@jorgechamorro.com> wrote:
>
> > $ ab -n 5000 -c 10http://127.0.0.1:12345/
>
> > Benchmarking 127.0.0.1 (be patient)
> > Completed 500 requests
> > Completed 1000 requests
> > apr_poll: The timeout specified has expired (70007)
> > Total of 1379 requests completed
>
> > ?
>
> > I'm running node-v0.1.33 on OSX 10.6.2
>
> If instead I do an
>

> $ ab -n 50000 -c 1http://127.0.0.1:12345/

[Peter Sunde]

I can confirm this issue:

uname -a:
Linux 2.6.24-24-server #1 SMP Fri Sep 18 16:47:05 UTC 2009 x86_64 GNU/
Linux

--------------------------------------------------
stack trace.
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/
gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(gdb) run
Starting program: /software/node/node_g animal_server.js
[Thread debugging using libthread_db enabled]
[New Thread 0x7fd1c8df26f0 (LWP 13321)]
[New Thread 0x40e8e950 (LWP 13324)]
The 'tcp' module is now called 'net'. Otherwise it should have a
similar interface.
sys.exec has moved to the "child_process" module. Please update your
source code.

Error: EMFILE, Too many open files

at net:622:19
at dns:100:7
at IdleWatcher.callback (node.js:256:5)
at node.js:813:9

Program exited with code 01.
(gdb)

On 23 Mar, 15:54, kindy <kind...@gmail.com> wrote:
> $ cat tt.js
> var sys = require("sys"),
>    http = require("http");
> http.createServer(function (request, response) {
>   response.writeHead(200, {"Content-Type": "text/plain"});
>   response.write("Hello World\n");
>   response.close();}).listen(8000);
>

> sys.puts("Server running athttp://127.0.0.1:8000/");

>
> $ node tt.js
>
> $ ab -c 1000 -n 3000http://127.0.0.1:8000/
>

> Server running athttp://127.0.0.1:8000/

[Jorge]

On 09/04/2010, at 10:52, Peter Sunde wrote:
> I can confirm this issue:

(:::)

> The 'tcp' module is now called 'net'. Otherwise it should have a
> similar interface.
> sys.exec has moved to the "child_process" module. Please update your
> source code.
> Error: EMFILE, Too many open files
> at net:622:19
> at dns:100:7
> at IdleWatcher.callback (node.js:256:5)
> at node.js:813:9
>
> Program exited with code 01.
> (gdb)

Hi Peter,

That's what happens once you get past maxfiles, ISTM. You may need to queue file operations and dispatch them orderly and in such a way that the number of opened files never gets dangerously close to maxfiles:

$ sysctl -a | grep files
kern.maxfiles = 12288
kern.maxfilesperproc = 10240
kern.maxfiles: 12288
kern.maxfilesperproc: 10240
kern.num_files: 639
--
Jorge.

[Jorge Chamorro]

On 09/04/2010, at 15:24, Jorge wrote:

(...) You may need to queue file operations (...)

Here's how I have done the queue in my gzipStr module, in order to avoid err-ing due to spawning too many simultaneous processes: if at any given time there are more than kMaxAtOnce children running at once, the calls to gzipStr() are just saved into the queue to be processed later. 

//20100402 jo...@jorgechamorro.com

//For NodeJS > v 0.1.33

var spawn = require("child_process").spawn;

var kMaxAtOnce= 15;

var queue= [];

var spawned= 0;

function dispatch (e) {

  var child= spawn("gzip", ["-c", "-f", "-n"]);

  var stdout= "", stdin= "";

  spawned++;

  

  child.stdout.addListener("data", function (chunk) {

    if (chunk) stdout+= chunk;

  });

  child.stderr.addListener("data", function (chunk) {

    if (chunk) stderr+= chunk;

  });

  child.addListener("exit", function (code) {

    e.callback(code ? "" : stdout);

    spawned--;

    if (queue.length) {

      dispatch(queue.shift());

    }

  });

  child.stdout.setEncoding('binary');

  child.stdin.write(e.str, "utf8");

  child.stdin.close();

}

exports.gzipStr= function gzipStr (str, callback) {

  var e= { str: str, callback: callback};

  if (spawned < kMaxAtOnce) {

    dispatch(e);

  } else {  

    queue.push(e);

  }

};

-- 

Jorge

[Ryan Dahl]

On Tue, Mar 23, 2010 at 6:54 AM, kindy <kin...@gmail.com> wrote:

Should be fixed in 9331218449921290d699fc8eaaa77a9b7155223b

[Peter Sunde]

Yes, the problem seems to be resolved now. Before commit
9331218449921290d699fc8eaaa77a9b7155223b (Thank you for the tip
Jorge) it was crashing every 3-4 minutes or so.

Thanks, Ryan.

Regards,
Peter Sunde.

On 9 apr, 20:12, Ryan Dahl <coldredle...@gmail.com> wrote:

> On Tue, Mar 23, 2010 at 6:54 AM, kindy <kind...@gmail.com> wrote:
> > $ cat tt.js
> > var sys = require("sys"),
> >    http = require("http");
> > http.createServer(function (request, response) {
> >   response.writeHead(200, {"Content-Type": "text/plain"});
> >   response.write("Hello World\n");
> >   response.close();
> > }).listen(8000);

> > sys.puts("Server running athttp://127.0.0.1:8000/");
> > $ node tt.js
> > $ ab -c 1000 -n 3000http://127.0.0.1:8000/
>
> > Server running athttp://127.0.0.1:8000/

[aleks raiden]

Hmm,
at version v0.1.93-37-gab723d0 i have this error after 2 - 2 hour of
running script with periodical http connecting (as client, to remote
host at 3000 ms inyterval) and two persistent connection by Redis (at
local machine).