Asynchronous exception handling

[Serge S. Koval]

Hi,

 I have following problem in regards to exception handling:

1. I have piece of code which polls some data source, say every second

2. If there's some data, it will process it and it might take, say, a minute

3. While processing data, it might crash. In this case it should retry from the beginning second later.

Here's gist to illustrate approach: https://gist.github.com/3840620

Even though there's only 'raise' inside of 'with' statement, lets pretend that this is some code that fails and it is asynchronous: using gen.Task to make some asynchronous calls - you can't use try/catch.

This code contains one big problem: it leaks ExceptionStackContext every time exception handler re-queues next iteration. So on next iteration, _exc will be called twice: once for outer ExceptionStackContext, once for inner ExceptionStackContext. Next time, will be called 3 times, etc.

I can't suppress exceptions either - if my exception handler returns True, I will see '.. success' printed but it was not successful operation.

I have feeling that wrapping with NullStackContext won't help either - it will keep growing stack anyway.

What is the best way to accomplish this task?

Thank you,

Serge.

[Serge S. Koval]

As a small update - looks like I need to read documentation more often.

Since python 2.5, you can mix try/finally with generators, so you don't have to use ExceptionStackContext with tornado.gen to handle exceptions.

However, if you need to handle exceptions with callbacks, you need to wrap code that reschedules next run with NullContext to prevent stack context overflow and multiple calls to error handler.

Serge.

[A. Jesse Jiryu Davis]

Could you post your fixed code for our edification? I'm still learning about StackContexts.

[Serge S. Koval]

Sure, here's gist which illustrates exception handling with ExceptionStackContext and try/catch: https://gist.github.com/3841339

Serge.

[A. Jesse Jiryu Davis]

OK, I think I mostly understand now. So Motor is working correctly with StackContexts, you think?

Your second example, with a simple try / finally, is definitely the way I'd intended people to handle exceptions from Motor.

[Ben Darnell]

Instead of NullContext, you might try structuring things so that the
scheduler is in a separate StackContext "layer" than the worker
function (like ioloop.PeriodicCallback). Note that PeriodicCallback
will mostly work fine as-is with async functions, but for overrun
protection you need something that knows its function is asynchronous.

class PeriodicAsyncCallback(PeriodicCallback):
def _run(self):
if not self._running:
return
with ExceptionStackContext(functools.partial(self.handle_exc,
self.counter)):
self.callback(callback=functools.partial(self.handle_finished,
self.counter))

def handle_exc(self, counter, typ, value, tb):
logging.error("error in PeriodicAsyncCallback", exc_info=(typ,value,tb))
if self.counter == counter:
self._schedule_next()

def handle_finished(self, counter):
if self.counter == counter:
self._schedule_next()

def _schedule_next(self)
# counter ensures that even if a callback generates multiple
exceptions in its
# stack context we won't get multiple copies rescheduled.
self.counter += 1
super(PeriodicCallback, self)._schedule_next()