tcp.createServer vs. http.createServer

[pete]

I lifted the tcp and http server creation samples from node's
documentation and subjected each to 10k nonconcurrent requests, i.e.:

$ cat tcp-test.js
var server = node.tcp.createServer(function (socket) {
socket.setEncoding("utf8");
socket.addListener("connect", function () {
socket.send("Hello World!");
socket.close();
});}, {backlog: 1024});
server.listen(7000, "localhost");
puts("Server running at http://127.0.0.1:7000/");

$ cat client/netcat10k.sh
#!/bin/bash
for i in `seq 1 10000`;
do
echo $i
netcat localhost 7000
done
$

$ cat httpd-test.js
node.http.createServer(function (request, response) {
response.sendHeader(200, [["Content-Type", "text/plain"]]);
response.sendBody("Hello World!");
response.finish();
}).listen(8000);
puts("Server running at http://127.0.0.1:8000/");

$ cat client/curl10k.sh
#!/bin/bash
for i in `seq 1 10000`;
do
echo $i
curl http://localhost:8000
done

When I start up the tcp example and run the client script, their
interaction goes as one would expect:

$ node ./tcp-test.js
Server running at http://127.0.0.1:7000/

$ client/netcat10k.sh
1
Hello World!2
Hello World!3
...
Hello World!10000
$

But with the http example, curl appears to hang on request 1019:

$ node ./httpd-test.js
Server running at http://127.0.0.1:8000/

$ client/curl10k.sh
1
Hello World!2
Hello World!3
....
Hello World!1014
Hello World!1015
Hello World!1016
Hello World!1017
Hello World!1018
Hello World!1019

And the server begins to complain about too many open connections:

....
accept(): Too many open files
accept(): Too many open files
accept(): Too many open files
accept(): Too many open files
accept(): Too many open file
....

Is their some contrivance in the example I've setup here that's
responsible for the apparently misbehaving http or are connections
perhaps being left open in this case? Having the call to setTimeout()
in the case of the http example does not seem to make a difference.

I went through this exercise just as a first attempt to diagnose a
more elaborate http.createServer app that keeps dying overnight with
"CALL_AND_RETRY_2 Allocation failed - process out of memory". I'll
soon confirm but suspect the long-polling XHR clients attaching to it
are causing the node process to grow and grow in resident memory until
V8 reaches its limit. In any case I'm not a v8 developer but will
keep digging and see what I can determine.

Thanks in advance and thanks for node!

-pete

[Brian Hammond]

Check your resource limits (e.g. ulimit -n).

[Urban Hafner]

pete wrote:
> I lifted the tcp and http server creation samples from node's
> documentation and subjected each to 10k nonconcurrent requests, i.e.:

I don't really have the expertise (yet), but I don't think that the
requests are non-concurrent from the node side. It doesn't really block
when it receives a request (though it's not multi-threaded either). So
it might well be that the requests are handles at the same time.

What happens if there's a (very) small delay between the requests? Maybe
node just isn't fast enough in handling the request.

Or, you just found a bug ;)

Urban

[Hagen]

I don't really have the expertise (yet), but I don't think that the
requests are non-concurrent from the node side. It doesn't really block
when it receives a request (though it's not multi-threaded either). So
it might well be that the requests are handles at the same time.

From my understanding, node is single threaded, but all i/o is async. I.e. node.js has cooperative multitasking, with i/o being the only context switch (hence, the callbacks).

If you run into "too many open files", this can have only two reasons:

1. node.js is too slow to clear the incoming buffer
2. node.js does not release the connection as soon as possible

I guess we have to wait for Ryan to return to get the correct answer. But since sockets works, it looks more like 1), as for http there is header parsing required. Anybody know, if that is done in C or in JavaScript?

[Urban Hafner]

Hagen wrote:

> I guess we have to wait for Ryan to return to get the correct answer.
> But since sockets works, it looks more like 1), as for http there is
> header parsing required. Anybody know, if that is done in C or in
> JavaScript?

IIRC, the HTTP parser is written in C or C++.

Urban

[pete]

On Jul 8, 3:21 am, Urban Hafner <ur...@bettong.net> wrote:

> What happens if there's a (very) small delay between the requests?

I could not see how to introduce a sub-second delay within curl so
just added a "sleep 0.1" after that call in the bash script. This
delay did not seem to make a difference; the loop still has the http-
serving process bump up against maximum open file descriptors as it
approaches receipt of the 1024th request.

Also just to see if it would make any difference I built node 0.1.0
with v8 1.2.12 instead of the 1.2.10 and witnessed the same result.

-pete

[Urban Hafner]

That's too bad :( Unfortunately Ryan (the creator of node.js) is on
holiday until the 20th. And I guess, no one else knows node.js good
enough to help you here.

Urban

[Brian Hammond]

Sorry, I'm not sure I follow. Are you not expecting this limit of
1024? Have you modified your resource limits? AFAIK, 1024 is the
default for 'nofiles' in Linux.

What does 'ulimit -n' output?

[pete]

On Jul 8, 10:19 am, Brian Hammond <or.else.it.gets.the.h...@gmail.com>
wrote:

> Sorry, I'm not sure I follow. Are you not expecting this limit of
> 1024? Have you modified your resource limits? AFAIK, 1024 is the
> default for 'nofiles' in Linux.

Apologies for not being clear with my gripe, I'll try one more time
here and as someone suggested maybe Ryan can help me get it straight
upon his return...

It's definitely true that the loop over curl appears to makes
node.http.createServer quickly run up against the default 1024 file
descriptor limit (on Ubuntu 8.10 Server). What's unexpected is that
such a process would approach this limit at all in response to a
simple loop over a single call to curl. Certainly to affect the
symptom Ubuntu's limit could be bumped up manually and then the loop
of http requests could be seen to fail at a higher number of
iterations, but that exercise would essentially skirt the behavior I'm
trying to either understand or file as a bug.

The behavior boils down to node.http.createServer being seen to grow
file descriptors up to the default maximum (1024) while that trivial
loop over curl is executing. That behavior is observable with
something like:

$ while true;do lsof -p <node's pid here> | wc -l; sleep 1;done

Oddly enough Perl's POE will do this, too. Run the perl script in
place of the node program and employ lsof again to count the file
descriptors and you'll only have to wait a moment before it bombs out
in the same fashion:

use strict;
use POE qw(Component::Server::HTTP);
POE::Component::Server::HTTP->new(
Port => 6000,
ContentHandler => {"/" => sub {
my ($request, $response) = @_;
$response->code(RC_OK);
$response->content_type("text/html");
$response->content('Hello World!');
return RC_OK;
}},
);
print "Server running at http://127.0.0.1:6000/\n";
$poe_kernel->run();
exit;

But monitoring apache, nginx, etc. or trying other trival server
samples with some dynamic code shows file descriptors going away when
they're no longer being used and thus keeping the process safely under
the max. Samples like these in place of the node program + counting
their descriptors will show fd release after the connection goes away

(perl)

use strict;
use HTTP::Daemon;
use HTTP::Response;
my $x = HTTP::Response->new(200, 'OK', undef, 'Hello World!');
my $d = HTTP::Daemon->new(LocalPort => 6500) || die;
print "Please contact me at: <URL:", $d->url, ">\n";
while (my $c = $d->accept) {
while (my $r = $c->get_request) {
$c->send_response($x);
}
$c->close;
undef($c);
}

(python)

import socket
host = ''
port = 9000
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
sock.bind((host,port))
sock.listen(1)
while 1:
csock,caddr = sock.accept()
cfile = csock.makefile('rw',0)
while 1:
line = cfile.readline().strip()
if line == '':
cfile.write("HTTP/1.0 200 OK\n\n")
cfile.write("Hello World!")
cfile.close()
csock.close()
break

(ruby)

require 'webrick'
s = WEBrick::HTTPServer.new(:Port => 7000)
s.mount_proc("/") { |req,resp| resp.body = 'Hello World!' }
trap('INT') { s.stop; s.shutdown }
s.start

So it seems like these non-evented web servers feature sockets that
are created and then destroyed but if you go with node or Perl's
POE::Component::Server::HTTP (an event dispatcher, of course) you see
file descriptors that never get released.

The POE behavior has caused headaches before (http://osdir.com/ml/
lang.perl.poe/2007-03/msg00051.html) but this last tidbit is what
makes me suspect I may just be making bad assumptions about how node
and other event-based frameworks deal with the sockets they create -
but then that in turn makes me wonder on the node side of things why
node's "raw" tcp example destroys its descriptors while the http
example apparently does not.

I'll try running the node http thing again with v8's --gc_global and --
always_compact options enabled, see if I detect any differences.
Otherwise I'm fine waiting for advice from Ryan and other experts on
how to either fix or better understand this behavior.

-pete

[Connor Dunn]

It seems that node doesn't close the connections very quickly (if at
all) with the http client, if you try with the following code it seems
to work fine.

node.http.createServer(function (request, response) {
response.sendHeader(200, [["Content-Type", "text/plain"]]);
response.sendBody("Hello World!");
response.finish();

request.connection.close();
}).listen(8000);
puts("Server running at http://127.0.0.1:8000/");

Not that I think this should be required, but it seems that this is
simply a bug/oversight in node.

Connor

[Pete Thomas]

Thank you, this is a good workaround.  I just reran the example with the explicit call to close and node starts out with 16 descriptors, keeping between 16-17 for the duration of the loop.  -pete

[ry]

This seems to be fixed in the latest repo version? I fixed a few bugs
over vacation that might have contributed to it.

On Jul 8, 4:26 am, pete <p...@pollenware.com> wrote:
> I lifted the tcp and http server creation samples from node's
> documentation and subjected each to 10k nonconcurrent requests, i.e.:
>
>     $ cat tcp-test.js
>     var server = node.tcp.createServer(function (socket) {
>       socket.setEncoding("utf8");
>       socket.addListener("connect", function () {
>         socket.send("Hello World!");
>         socket.close();
>       });}, {backlog: 1024});
>     server.listen(7000, "localhost");

>     puts("Server running athttp://127.0.0.1:7000/");

>
>     $ cat client/netcat10k.sh
>     #!/bin/bash
>     for i in `seq 1 10000`;
>     do
>       echo $i
>       netcat  localhost 7000
>     done
>     $
>
>     $ cat httpd-test.js
>     node.http.createServer(function (request, response) {
>       response.sendHeader(200, [["Content-Type", "text/plain"]]);
>       response.sendBody("Hello World!");
>       response.finish();
>     }).listen(8000);

>     puts("Server running athttp://127.0.0.1:8000/");

>
>     $ cat client/curl10k.sh
>     #!/bin/bash
>     for i in `seq 1 10000`;
>     do
>       echo $i

>       curlhttp://localhost:8000

>     done
>
> When I start up the tcp example and run the client script, their
> interaction goes as one would expect:
>
>     $ node ./tcp-test.js

>     Server running athttp://127.0.0.1:7000/

>
>     $ client/netcat10k.sh
>     1
>     Hello World!2
>     Hello World!3
>     ...
>     Hello World!10000
>     $
>
> But with the http example, curl appears to hang on request 1019:
>
>     $ node ./httpd-test.js

>     Server running athttp://127.0.0.1:8000/

[Brian Hammond]

good stuff

$ ab -n 1000000 -c 1000 http://127.0.0.1:8000/

Requests per second: 3204.16 [#/sec] (mean)