Handling uncaught exceptions

[Ian Petzer]

Hello NodeCT,

I'm trying to understand best practice for deploying a node application. I know that an uncaught exception will result in the process crashing. I've included a shorter and longer description of the problem.. any help much appreciated.

TLDR;

I'm worried that an uncaught exception takes down my whole Node process including other longer running requests that are being processed concurrently which will leave my data in an indeterminate state.

I'm deploying on modulus.io which uses forever to restart the process, is there a way to ensure longer running processes can complete first?

Longer version:

On Nodejitsu/Heroku/Modulus they use the forever module (or something similar) to restart the process when an uncaught exception occurs.

I'd just like to know what impact that will have on other concurrent requests that are being processed.

Our app consists of backend node.js server exposing a RESTful interface which also uses Express to serve up a web application.

The web application is able to make queries to the back-end server.. Some of them can take a few seconds to complete as they require integration to third party servers or the generation of PDF's using PhantomJS.  We also have processes that are kicked off on a regular basis using the node-cron module.

So.. if I have a longer running request in progress, and a separate request comes in which results in a runtime exception, is there any way that I can keep the node process going in order to complete the longer running job.

Maybe get the node.js process to stop accepting new requests while it waits for existing ones to complete?

Thanks,

Ian

[Len Weincier]

Hi Ian

Its a tricky situation in general when things go bad and how to handle them. To stop the single thread dying this is the common practice : 

// catch the uncaught errors that weren't wrapped in a domain or try catch statement
// do not use this in modules, but only in applications, as otherwise we could have multiple of these bound
process.on('uncaughtException', function(err) {
    // handle the error safely
    console.log(err);
});

This answer on StackOverflow is a god summary : 

http://stackoverflow.com/a/7313005/41292

Cheers

Len

--

CloudAfrica

0834006106

--
You received this message because you are subscribed to the Google Groups "Node Cape Town" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodecpt+u...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.
Visit this group at http://groups.google.com/group/nodecpt.
For more options, visit https://groups.google.com/groups/opt_out.

[Rudolf Meijering]

Node.js domains were created for this purpose. You want to wrap every request in it's own domain so that when a single request throws an error, you can just dispose the domain and the rest of your app is unaffected.

It's still best practise to use a cluster of node processes so that anything that your worker is restarted if a domain didn't catch an exception. (You can still use forever to run your cluster master process, but you probably don't want a single threaded node process on an 8 core server).

--
Rudolf Meijering

[Ian Petzer]

Thanks Rudolf and Len,

I've been reading up on Domains and it seems the way to go.

I did notice that the stability of Domains is marked as 'Unstable' according to http://nodejs.org/api/domain.html

I guess people have been using them since they were 'experimental' so hopefully it will work for me.

[Len Weincier]

Had to laugh at this : 

http://dshaw.github.io/2012-05-jsday/#/11/1

Rest of the talk is good too.

Len

--

CloudAfrica

0834006106

[Marcin Jekot]

Len, that whole dshaw presentation was super educational for me. Thanks!!

[Len Weincier]

Hi Marcin

Len, that whole dshaw presentation was super educational for me. Thanks!!

@dshaw sent us the video link as well 

http://lanyrd.com/2012/jsday/stgxk/

Len

[Ruan Botha]

Hi Rudolph,

I have tried clusters and domains and it's worked quite well. Just wanted to find out if anyone has used clusters successfully with socket.io? (I've tried socket.io-clusterhub and redisStore to manage connection pooling of the websocket connections across the different running processes, but had intermittent results)

Also http keep-alive keeps some of the cluster.workers connections open, as Ian mentioned this morning on gtalk, which means the cluster.worker doesn't close properly, after an uncaughtException (unless process.exit(1) is called in that cluster.worker, with a setTimeout).

Has anyone had a working implementation of nodejs clusters and socket.io and which socketio store (io.set('store', store)) did you use to pool the websocket connections?

[Rudolf Meijering]

Sorry Ruan, we haven't used socket.io with clusters. I know strongloop has a node socket.io store implementation: http://docs.strongloop.com/display/DOC/Socket+IO+Store+for+Clusters

But I'd rather try to iron out problems with something like redisStore cause else you're still only synchronising on a single machine and eventually you'd probably want to scale out horizontally.

--
You received this message because you are subscribed to the Google Groups "Node Cape Town" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodecpt+u...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.
Visit this group at http://groups.google.com/group/nodecpt.

For more options, visit https://groups.google.com/d/optout.

--
Rudolf Meijering

[Ruan Botha]

Hi Rudolf,

Thank you for the link and info. Going to try redisStore again. I realised that if you leave the serverURL parameter out of io.connect() in the client, it ascertains the server itself and works best for heroku servers, better than specifying the serverURL explicitly as the first parameter.

Will try clustering again in the coming week as one heroku dyno has 4 CPUs available, so it's a noticeable performance increase and helps with error handling.

Best,
Ruan

You received this message because you are subscribed to a topic in the Google Groups "Node Cape Town" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodecpt/yS4Zh_UGuZ8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodecpt+u...@googlegroups.com.