I have following problem in regards to exception handling:
1. I have piece of code which polls some data source, say every second
2. If there's some data, it will process it and it might take, say, a minute
3. While processing data, it might crash. In this case it should retry from
the beginning second later.
Even though there's only 'raise' inside of 'with' statement, lets pretend
that this is some code that fails and it is asynchronous: using gen.Task to
make some asynchronous calls - you can't use try/catch.
This code contains one big problem: it leaks ExceptionStackContext every
time exception handler re-queues next iteration. So on next iteration, _exc
will be called twice: once for outer ExceptionStackContext, once for inner
ExceptionStackContext. Next time, will be called 3 times, etc.
I can't suppress exceptions either - if my exception handler returns True,
I will see '.. success' printed but it was not successful operation.
I have feeling that wrapping with NullStackContext won't help either - it
will keep growing stack anyway.
As a small update - looks like I need to read documentation more often.
Since python 2.5, you can mix try/finally with generators, so you don't
have to use ExceptionStackContext with tornado.gen to handle exceptions.
However, if you need to handle exceptions with callbacks, you need to wrap
code that reschedules next run with NullContext to prevent stack context
overflow and multiple calls to error handler.
Serge.
On Fri, Oct 5, 2012 at 6:56 PM, Serge S. Koval <serge.ko...@gmail.com>wrote:
> I have following problem in regards to exception handling:
> 1. I have piece of code which polls some data source, say every second
> 2. If there's some data, it will process it and it might take, say, a
> minute
> 3. While processing data, it might crash. In this case it should retry
> from the beginning second later.
> Even though there's only 'raise' inside of 'with' statement, lets pretend
> that this is some code that fails and it is asynchronous: using gen.Task to
> make some asynchronous calls - you can't use try/catch.
> This code contains one big problem: it leaks ExceptionStackContext every
> time exception handler re-queues next iteration. So on next iteration, _exc
> will be called twice: once for outer ExceptionStackContext, once for inner
> ExceptionStackContext. Next time, will be called 3 times, etc.
> I can't suppress exceptions either - if my exception handler returns True,
> I will see '.. success' printed but it was not successful operation.
> I have feeling that wrapping with NullStackContext won't help either - it
> will keep growing stack anyway.
On Friday, October 5, 2012 9:20:16 AM UTC-7, Serge S. Koval wrote:
> As a small update - looks like I need to read documentation more often.
> Since python 2.5, you can mix try/finally with generators, so you don't > have to use ExceptionStackContext with tornado.gen to handle exceptions.
> However, if you need to handle exceptions with callbacks, you need to wrap > code that reschedules next run with NullContext to prevent stack context > overflow and multiple calls to error handler.
> Serge.
> On Fri, Oct 5, 2012 at 6:56 PM, Serge S. Koval <serge...@gmail.com<javascript:> > > wrote:
>> Hi,
>> I have following problem in regards to exception handling: >> 1. I have piece of code which polls some data source, say every second >> 2. If there's some data, it will process it and it might take, say, a >> minute >> 3. While processing data, it might crash. In this case it should retry >> from the beginning second later.
>> Even though there's only 'raise' inside of 'with' statement, lets pretend >> that this is some code that fails and it is asynchronous: using gen.Task to >> make some asynchronous calls - you can't use try/catch.
>> This code contains one big problem: it leaks ExceptionStackContext every >> time exception handler re-queues next iteration. So on next iteration, _exc >> will be called twice: once for outer ExceptionStackContext, once for inner >> ExceptionStackContext. Next time, will be called 3 times, etc.
>> I can't suppress exceptions either - if my exception handler returns >> True, I will see '.. success' printed but it was not successful operation.
>> I have feeling that wrapping with NullStackContext won't help either - it >> will keep growing stack anyway.
> Could you post your fixed code for our edification? I'm still learning
> about StackContexts.
> On Friday, October 5, 2012 9:20:16 AM UTC-7, Serge S. Koval wrote:
>> As a small update - looks like I need to read documentation more often.
>> Since python 2.5, you can mix try/finally with generators, so you don't
>> have to use ExceptionStackContext with tornado.gen to handle exceptions.
>> However, if you need to handle exceptions with callbacks, you need to
>> wrap code that reschedules next run with NullContext to prevent stack
>> context overflow and multiple calls to error handler.
>> Serge.
>> On Fri, Oct 5, 2012 at 6:56 PM, Serge S. Koval <serge...@gmail.com>wrote:
>>> Hi,
>>> I have following problem in regards to exception handling:
>>> 1. I have piece of code which polls some data source, say every second
>>> 2. If there's some data, it will process it and it might take, say, a
>>> minute
>>> 3. While processing data, it might crash. In this case it should retry
>>> from the beginning second later.
>>> Even though there's only 'raise' inside of 'with' statement, lets
>>> pretend that this is some code that fails and it is asynchronous: using
>>> gen.Task to make some asynchronous calls - you can't use try/catch.
>>> This code contains one big problem: it leaks ExceptionStackContext every
>>> time exception handler re-queues next iteration. So on next iteration, _exc
>>> will be called twice: once for outer ExceptionStackContext, once for inner
>>> ExceptionStackContext. Next time, will be called 3 times, etc.
>>> I can't suppress exceptions either - if my exception handler returns
>>> True, I will see '.. success' printed but it was not successful operation.
>>> I have feeling that wrapping with NullStackContext won't help either -
>>> it will keep growing stack anyway.
On Friday, October 5, 2012 10:59:15 AM UTC-7, Serge S. Koval wrote:
> Sure, here's gist which illustrates exception handling with > ExceptionStackContext and try/catch: https://gist.github.com/3841339
> Serge.
> On Fri, Oct 5, 2012 at 8:03 PM, A. Jesse Jiryu Davis <ajess...@gmail.com<javascript:> > > wrote:
>> Could you post your fixed code for our edification? I'm still learning >> about StackContexts.
>> On Friday, October 5, 2012 9:20:16 AM UTC-7, Serge S. Koval wrote:
>>> As a small update - looks like I need to read documentation more often.
>>> Since python 2.5, you can mix try/finally with generators, so you don't >>> have to use ExceptionStackContext with tornado.gen to handle exceptions.
>>> However, if you need to handle exceptions with callbacks, you need to >>> wrap code that reschedules next run with NullContext to prevent stack >>> context overflow and multiple calls to error handler.
>>> Serge.
>>> On Fri, Oct 5, 2012 at 6:56 PM, Serge S. Koval <serge...@gmail.com>wrote:
>>>> Hi,
>>>> I have following problem in regards to exception handling: >>>> 1. I have piece of code which polls some data source, say every second >>>> 2. If there's some data, it will process it and it might take, say, a >>>> minute >>>> 3. While processing data, it might crash. In this case it should retry >>>> from the beginning second later.
>>>> Even though there's only 'raise' inside of 'with' statement, lets >>>> pretend that this is some code that fails and it is asynchronous: using >>>> gen.Task to make some asynchronous calls - you can't use try/catch.
>>>> This code contains one big problem: it leaks ExceptionStackContext >>>> every time exception handler re-queues next iteration. So on next >>>> iteration, _exc will be called twice: once for outer ExceptionStackContext, >>>> once for inner ExceptionStackContext. Next time, will be called 3 times, >>>> etc.
>>>> I can't suppress exceptions either - if my exception handler returns >>>> True, I will see '.. success' printed but it was not successful operation.
>>>> I have feeling that wrapping with NullStackContext won't help either - >>>> it will keep growing stack anyway.
>>>> What is the best way to accomplish this task?
Instead of NullContext, you might try structuring things so that the
scheduler is in a separate StackContext "layer" than the worker
function (like ioloop.PeriodicCallback). Note that PeriodicCallback
will mostly work fine as-is with async functions, but for overrun
protection you need something that knows its function is asynchronous.
class PeriodicAsyncCallback(PeriodicCallback):
def _run(self):
if not self._running:
return
with ExceptionStackContext(functools.partial(self.handle_exc,
self.counter)):
self.callback(callback=functools.partial(self.handle_finished,
self.counter))
def handle_exc(self, counter, typ, value, tb):
logging.error("error in PeriodicAsyncCallback", exc_info=(typ,value,tb))
if self.counter == counter:
self._schedule_next()
def handle_finished(self, counter):
if self.counter == counter:
self._schedule_next()
def _schedule_next(self)
# counter ensures that even if a callback generates multiple
exceptions in its
# stack context we won't get multiple copies rescheduled.
self.counter += 1
super(PeriodicCallback, self)._schedule_next()
On Fri, Oct 5, 2012 at 9:20 AM, Serge S. Koval <serge.ko...@gmail.com> wrote:
> As a small update - looks like I need to read documentation more often.
> Since python 2.5, you can mix try/finally with generators, so you don't have
> to use ExceptionStackContext with tornado.gen to handle exceptions.
> However, if you need to handle exceptions with callbacks, you need to wrap
> code that reschedules next run with NullContext to prevent stack context
> overflow and multiple calls to error handler.
> Serge.
> On Fri, Oct 5, 2012 at 6:56 PM, Serge S. Koval <serge.ko...@gmail.com>
> wrote:
>> Hi,
>> I have following problem in regards to exception handling:
>> 1. I have piece of code which polls some data source, say every second
>> 2. If there's some data, it will process it and it might take, say, a
>> minute
>> 3. While processing data, it might crash. In this case it should retry
>> from the beginning second later.
>> Even though there's only 'raise' inside of 'with' statement, lets pretend
>> that this is some code that fails and it is asynchronous: using gen.Task to
>> make some asynchronous calls - you can't use try/catch.
>> This code contains one big problem: it leaks ExceptionStackContext every
>> time exception handler re-queues next iteration. So on next iteration, _exc
>> will be called twice: once for outer ExceptionStackContext, once for inner
>> ExceptionStackContext. Next time, will be called 3 times, etc.
>> I can't suppress exceptions either - if my exception handler returns True,
>> I will see '.. success' printed but it was not successful operation.
>> I have feeling that wrapping with NullStackContext won't help either - it
>> will keep growing stack anyway.
