long-running node process unexpectedly exits without error

[jfhbrook]

Hi,

I have a long-running node script doing sustained reads and writes against multiple HTTP APIs. Most of the work is being done with mikeal/request and caolan/async. After an hour or two of grinding against all the same code paths, my node process unexpectedly exits without any error messages, stack traces, etc and an exit status of 0. I've more or less ruled out dropped callbacks because the script exits at unpredictable points in the code and on paths that have already ran hundreds of times with similar inputs without incident.

Other things I’ve looked at that haven’t appeared problematic: Ephemeral ports, /var/log/messages and resource usage.

Any suggestions in terms of figuring out what's happening? None of my normal tools (like console.log, the debugger, core dumps) seem directly applicable. This isn't a lot to go on given that I don't have a succinct repro case OR an error message.

—Josh

[Sam Roberts]

On Tue, Jun 23, 2015 at 3:03 PM, jfhbrook <josh.h...@gmail.com> wrote:
> Hi,
>
> I have a long-running node script doing sustained reads and writes against
> multiple HTTP APIs. Most of the work is being done with mikeal/request and
> caolan/async. After an hour or two of grinding against all the same code
> paths, my node process unexpectedly exits without any error messages, stack
> traces, etc and an exit status of 0. I've more or less ruled out dropped
> callbacks because the script exits at unpredictable points in the code and
> on paths that have already ran hundreds of times with similar inputs without
> incident.

Is it possible that your application briefly has no work to do? If it
has no active handles, it would just exit. You could try setting an
interval timer at every 5 minutes, and seeing if that keeps node alive
forever. If so, that's your problem.

You can also replace process.exit with a hook:
process.exit = function() {
console.trace('EXIT, whoa...!')
};

to see if some part of the code is explicitly calling exit when it shouldn't be.

You can also run using `strace` (assuming you are on linux), and the
output at the end should give an indication of why node is exiting,
and what happened in terms of system calls just before the exit. The
output will be copious, of course, but you only care about the end!
:-)

Sam

--
http://StrongLoop.com makes it easy to develop APIs in Node, plus get
DevOps capabilities like monitoring, debugging and clustering.