How to "kill" callback by timeout?

[Alexey Petrushin]

There are N Workers and one Master. 

Workers are unstable and frequently can hang up or die. 

If worker dies - it's ok because there will be error in callback in master. 

But if worker hangs - there's a problem - callback in master also will hangs indefinitely.

And this is a problem, I need to kill those hanging callbacks in master.

Is it possible to do something like this?

    function safeCallToSomeRemoteService(callback){

      var remoteServiceResponded = false      

      unsafeCallToSomeRemoteService(function(err, data){

        remoteServiceResponded = true

        callback(err, data)

      })

      

      setTimeout(function(){

        if(!remoteServiceResponded){

          somehow kill callback waiting for remote service

          callback(new Error("Remote service hanged!"))

        }

      }, 3000)

    }

Or, if it's not possible, is it safe to just ignore such hanged callbacks, call callback on timeout and move on (while leaving hanging callback active)?

[// ravi]

On Apr 16, 2014, at 2:17 PM, Alexey Petrushin <alexey.p...@gmail.com> wrote:

There are N Workers and one Master. 

Workers are unstable and frequently can hang up or die. 

If worker dies - it's ok because there will be error in callback in master. 

But if worker hangs - there's a problem - callback in master also will hangs indefinitely.

And this is a problem, I need to kill those hanging callbacks in master.

Assuming I understand your need correctly, your problem is that there are actions in the master’s callback function that will never occur because the callback is never called back. I am rephrasing it this way because the callback isn’t really “hanging” since it never ran.

If the actions performed in the callback are purely local (and do not alter global state) then there is little danger in just leaving it as-is (apart from concerns of potential memory leaks caused by closure). OTOH, if the callback alters global state or in other ways is responsible for execution flow in the master (e.g: recursion), then you may need to do something like what you suggest below. In which case, you have to ensure that the callback guards itself from being fired twice, should the hanging unsafeCallToSomeRemoteService() suddenly return after a long time. For example, within safeCallToSomeRemoteService() you could not just check for remoteServiceResponded, but also for callbackCalled before you call it.

All sorts of caveats and cautions related to data integrity (but beyond the scope of this answer) apply,

—ravi

[Alexey Petrushin]

Thanks for reply. A little more about the project - it's a web crawler that uses lots of selenium nodes. Selenium nodes are very unstable and hangs frequently.

Yes you are right my code isnt finished and need to ensure that callback will be called only once. But, my primary concern is memory leeks (maybe also descriptors leeks as it won't be released also). Is there a way to prevent those leaks and terminate hanging callbacks and release allocated resources?

[Ryan Graham]

On Wed, Apr 16, 2014 at 11:17 AM, Alexey Petrushin <alexey.p...@gmail.com> wrote:

Is it possible to do something like this?

    function safeCallToSomeRemoteService(callback){

      var remoteServiceResponded = false      

      unsafeCallToSomeRemoteService(function(err, data){

        remoteServiceResponded = true

        callback(err, data)

      })

      

      setTimeout(function(){

        if(!remoteServiceResponded){

          somehow kill callback waiting for remote service

          callback(new Error("Remote service hanged!"))

        }

      }, 3000)

    }

Or, if it's not possible, is it safe to just ignore such hanged callbacks, call callback on timeout and move on (while leaving hanging callback active)?

Ignoring whether there are better options available through the API you are using, the generic timeout code would look something like this:

    function safeCallToSomeRemoteService(callback) {

      var timer = setTimeout(function() {

        callback(new Error('Remote service timed out!'));

      }, 3000);

      unsafeCallToSomeRemoteService(function(err, data){

        clearTimeout(timer);

        callback(err, data);

      });

    }

~Ryan

--
http://twitter.com/rmgraham

[Francesco Mari]

Maybe you also want to prevent callback to execute twice if the timer
triggers and the remote service sends a response after the time limit.

function safeCallToSomeRemoteService(callback) {
var done = callback;

var timer = setTimeout(function() {
var temp = done;
done = function () {};
temp(new Error('Remote service timed out!'));

}, 3000);

unsafeCallToSomeRemoteService(function(err, data){
clearTimeout(timer);

done(err, data);
});

> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nodejs+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Alexey Petrushin]

Yes, I'm agree with your notes about the code and preventing callback from being called twice. 

But what about memory and descriptors leaks, is there a way to terminate hanging callback and clear it? Or I can just safely ignore those issues and node.js somehow will clear it internally?

[Francesco Mari]

I think that most of your answer depends on what
unsafeCallToSomeRemoteService does, and why the remote service is
unreliable. Can you provide more information about it?

[Alexey Petrushin]

I use wd.js (RESTful client to Selenium WebDriver) to connect to remote server with running Selenium WebDriver controlling Chrome Browser. 

It's used for web crawling and sometimes there are sites with bugs that take too much memory or somehow other crush running Chrome Browser Process.

In such cases (when Selenium Node stop responding) - there are background processes that will kill whole Selenium & Chrome processes and restart it.

The crawler itself don't know about all this stuff, it only knows list of IP addresses with Selenium Nodes and uses it. And when it hangs - it can do nothing about by himself, only wait until such hanging Selenium Node will be discovered by other processes and rebooted.

[Francesco Mari]

Maybe you can check if the client has a timeout parameter or other fallback strategies. If it does, you should trust the client that it performs the appropriate cleanup of the resources it uses.

[Alex Kocharin]

 

... and if it doesn't, you can run it in the separate process, and kill the entire process in case of failure. This way OS will clean up all resources automatically.

 

 

17.04.2014, 15:33, "Francesco Mari" <mari.fr...@gmail.com>:

[Alexey Petrushin]

Separate process is too costly. Well, it seems the way to go is to just ignore resource leaks (it should be small), and periodically restart the master process.