Node.js process exits

[Serge S. Koval]

Hi,

I'll post it here, maybe someone had same problem.

So, I have SockJS server running on top of node.js 0.8.5. It works
like a broadcast server with redis as a data source. Server is running
under 'forever' supervisor.

Everything is fine, except that it "crashes" once per day. There's no
stack trace. But node.js process exits and forever restarts it.

This is all I have in the log:
POST /api/624/0haf56ne/xhr_send 1ms 204
GET /api/466/fc3v1ro_/websocket 1ms (unfinished)
warn: Forever detected script exited with code: null
warn: Forever restarting script for 12 tim

Memory usage is constant, file descriptors are not leaking.
Obviously, there's no process.exit() anywhere.

Any help will be appreciated.

Serge.

[Marek Majkowski]

Whooa! Sounds like fun.

1) dmesg. Does it say that the process actually crashed, for example:
[2074115.763411] step2[409]: segfault at 9012ebc3 ip 9012ebc3 sp
bff2e88c error 14

2) if so, make it to save the core dump: "ulimit -c unlimited"
_before_ the process is started

3) if so, consider attaching GDB to it (gdb -p) and just let it run
there. This is risky though as gdb will stop the process on signal,
etc. (core dump better) On the other hand in GDB you could set a
breakpoint on exit() :)

4) in every case (crashes or not), you may want to record "strace"
output to see what happens just before the process exits / crashes

5) if the process does a graceful exit (ie: nothing in dmesg), what's
the return code?

6) Did you review changelog since node 0.8.5? Have you tried
recompiling it with safer flags (-O2 instead of -O3,
-fstack-protector-all, -D_FORTIFY_SOURCE=2 etc). Did you try different
versions of node?

Marek

[Marek Majkowski]

> 2) if so, make it to save the core dump: "ulimit -c unlimited"
> _before_ the process is started

Actually, that's more complex nowadays. Google it.

Marek

[Serge S. Koval]

Thanks for feedback. Comments are inline.

вторник, 26 марта 2013 г., 18:33:27 UTC+2 пользователь majek написал:

1) dmesg. Does it say that the process actually crashed, for example:
[2074115.763411] step2[409]: segfault at 9012ebc3 ip 9012ebc3 sp
bff2e88c error 14

Will take a look.

4) in every case (crashes or not), you may want to record "strace"
output to see what happens just before the process exits / crashes

This will create huge log file. It crashes approximately after 24h uptime.

 

5) if the process does a graceful exit (ie: nothing in dmesg), what's
the return code?

According to 'forever' it is 'null'. I guess it stands for 0.

 

6) Did you review changelog since node 0.8.5? Have you tried
recompiling it with safer flags (-O2 instead of -O3,
-fstack-protector-all, -D_FORTIFY_SOURCE=2 etc). Did you try different
versions of node?

Yes, upgraded to 0.8.22 and let it run for a day. We'll see how it'll behave.

Serge.

[Marek Majkowski]

On Tue, Mar 26, 2013 at 12:38 PM, Serge S. Koval <serge...@gmail.com> wrote:
> Thanks for feedback. Comments are inline.
>
> вторник, 26 марта 2013 г., 18:33:27 UTC+2 пользователь majek написал:
>>
>> 1) dmesg. Does it say that the process actually crashed, for example:
>> [2074115.763411] step2[409]: segfault at 9012ebc3 ip 9012ebc3 sp
>> bff2e88c error 14
>
> Will take a look.
>>
>>
>> 4) in every case (crashes or not), you may want to record "strace"
>> output to see what happens just before the process exits / crashes
>
> This will create huge log file. It crashes approximately after 24h uptime.

How about piping it to a pipe and showing it on a console? (ie: rolling buffer).