Segmentation Fault

[Felix Geisendörfer]

Just started playing around with node.js, absolutely impressive &
fantastic so far.

Anyway, I think I found an easily reproducible segmentation fault:

seg_fault.js
------------------------------------------------------
new node.http.Server(function (req, res) {
var ls = new node.Process("ls -lah");
ls.onOutput = function(chunk) {
res.sendHeader(200, [["Content-Type", "text/plain"]]);
res.sendBody(chunk || "");
res.finish();
};
}).listen(8001);
puts("Server running at http://127.0.0.1:8001/");
------------------------------------------------------

triggering the seg fault:
------------------------------------------------------
$ ab -n 1000 http://localhost:8001/
Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests

Test aborted after 10 failures

apr_socket_connect(): Connection refused (61)
Total of 379 requests completed
------------------------------------------------------

The fact that no matter how often I run this I always gets the
segfault at exactly 379 connections makes me think it's probably some
system process limit being hit. Speaking of which, I used Ubuntu 9.04
Server Edition for my tests.

Anyway, let me know if I can be of any help providing more data. Keep
up the fantastic work!

-- Felix Geisendörfer aka the_undefined

[Felix Geisendörfer]

More observations:

Changing the process to "ls" (less output) causes the segfault to move
up to 449 requests.

So I am now more thinking that garbage collection might be the issue.
However, putting a "ls.close()" and/or a "delete ls" statement in my
onOutput closure seems to have no effect on the segfault - not sure if
that's expected or not.

-- Felix

On Jun 26, 8:43 pm, Felix Geisendörfer <haimu...@gmail.com> wrote:
> Just started playing around with node.js, absolutely impressive &
> fantastic so far.
>
> Anyway, I think I found an easily reproducible segmentation fault:
>
> seg_fault.js
> ------------------------------------------------------
> new node.http.Server(function (req, res) {
>         var ls = new node.Process("ls -lah");
>         ls.onOutput = function(chunk) {
>                 res.sendHeader(200, [["Content-Type", "text/plain"]]);
>                 res.sendBody(chunk || "");
>                 res.finish();
>         };}).listen(8001);
>

> puts("Server running athttp://127.0.0.1:8001/");

> ------------------------------------------------------
>
> triggering the seg fault:
> ------------------------------------------------------

> $ ab -n 1000http://localhost:8001/

[ry]

Thanks -
It would be more proper to buffer the output and send the response in
the "onExit" callback:

new node.http.Server(function (req, res) {
var ls = new node.Process("ls -lah");

var output = "";

ls.onOutput = function(chunk) {
if (chunk) output += chunk;
};

ls.onExit = function () {

res.sendHeader(200, [["Content-Type", "text/plain"]]);

res.sendBody(output);
res.finish();
};
}).listen(8001);

As the "onOutput" will be called more than once. Nevertheless, this is
still segfaulting.

[ryan dahl]

Fixed in 5ab9350

[Brian Hammond]

I'm wondering what's going on in the following example.

It seems related to this.

I'm using the 5ab9350 version.

Running the sample from tinyclouds.org/node

new node.http.Server(function (req, res) {
setTimeout(function () {

res.sendHeader(200, [["Content-Type", "text/plain"]]);

res.sendBody("Hello World");
res.finish();
}, 2000);
}).listen(8000, {backlog:8192});
puts("Server running at http://127.0.0.1:8000/");

Then benchmarking with 'ab' I see the following:

$ ab -n 10000 -c 1000 http://myhost/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking myhost (be patient)
Completed 1000 requests
Completed 2000 requests
apr_socket_recv: Connection reset by peer (54)
Total of 2589 requests completed


Sometimes the 'ab' completes all calls and sometimes fails with
connection reset by peer.

The backlog I'm playing with was 1024 and 8192. Same erratic behavior
with both.

Any idea what's going on? Sure, I'm stressing it out but I'd like to
know what's being broken by this level of stress at least.

Thanks,
Brian


On Jun 26, 2009, at 6:16 PM, ryan dahl wrote:

>
> Fixed in 5ab9350
>
> >

[ryan dahl]

> Benchmarking myhost (be patient)
> Completed 1000 requests
> Completed 2000 requests
> apr_socket_recv: Connection reset by peer (54)
> Total of 2589 requests completed

My first inclination is to suspect some operating system boundary (eg
on number of open connections).On my system, for example, this works.

[Brian Hammond]

Hmmm I have nofiles set to 256000 and ulimit -n 256000.

What system are you running on and how do you have such limits
configured? I'd like it to work too!

:)

[Felix Geisendörfer]

Hey ryan,

thanks for the incredibly fast fix. I just ran the test on a fresh
build from GitHub and the segfault is gone indeed. However, I now am
seeing a not quite as consistently reproducible error when running the
same test. Only this time node reports this:

root@vbox:~# node example.js

Server running at http://127.0.0.1:8001/

V8 FATAL ERROR. v8::External::Cast() Could not convert to external

This usually happens after running the ab test with a few thousand
connections, but sometimes as early as a few hundred.

Let me know if there is anything else I might be able to provide you
with!

Best Regards,
-- Felix Geisendörfer

root@vbox:~# node example.js

Server running at http://127.0.0.1:8001/

V8 FATAL ERROR. v8::External::Cast() Could not convert to external




On Jun 27, 12:16 am, ryan dahl <coldredle...@gmail.com> wrote:
> Fixed in 5ab9350

[ryan dahl]

2009/6/27 Felix Geisendörfer <haim...@gmail.com>:

> thanks for the incredibly fast fix. I just ran the test on a fresh
> build from GitHub and the segfault is gone indeed. However, I now am
> seeing a not quite as consistently reproducible error when running the
> same test. Only this time node reports this:
>
> root@vbox:~# node example.js
> Server running at http://127.0.0.1:8001/
> V8 FATAL ERROR. v8::External::Cast() Could not convert to external
>
> This usually happens after running the ab test with a few thousand
> connections, but sometimes as early as a few hundred.
>
> Let me know if there is anything else I might be able to provide you
> with!

Hi -

You should use my modified version - there is an error in the original.

Node ought to raise an exception (and not crash with fatal error) when
one tries to call sendHeader but already has done so - I've opened an
issue for this: http://github.com/ry/node/issues/#issue/13