Generators and propagation of exceptions
r
work-around, but I'd like to ask you if there is a better/nicer
looking solution. Perhaps I'm missing something obvious.
The code looks like this:
stream-of-tokens = token-generator(stream-of-characters)
stream-of-parsed-expressions = parser-generator(stream-of-tokens)
stream-of-results = evaluator-generator(stream-of-parsed-expressions)
each of the above functions consumes and implements a generator:
def evaluator-generator(stream-of-tokens):
for token in stream-of-tokens:
try:
yield token.evaluate() # evaluate() returns a Result
except Exception as exception:
yield ErrorResult(exception) # ErrorResult is a subclass of Result
The problem is that, when I use the above mechanism, the errors
propagate to the output embedded in the data streams. This means, I
have to make them look like real data (in the example above I'm
wrapping the exception with an ErrorExpression object) and raise them
and intercept them again at each level until they finally trickle down
to the output. It feels a bit like writing in C (checking error codes
and propagating them to the caller).
OTOH, if I don't intercept the exception inside the loop, it will
break the for loop and close the generator. So the error no longer
affects a single token/expression but it kills the whole session. I
guess that's because the direction flow of control is sort of
orthogonal to the direction of flow of data.
Any idea for a working and elegant solution?
Thanks,
-r
Ian Kelly
I guess that depends on what you would want to do with the exceptions
instead. Collect them out-of-band? Revising your pseudo-code:
errors = []
stream-of-tokens = token-generator(stream-of-characters, errors)
stream-of-parsed-expressions = parser-generator(stream-of-tokens, errors)
stream-of-results = evaluator-generator(stream-of-parsed-expressions, errors)
def evaluator-generator(stream-of-tokens, errors):
for token in stream-of-tokens:
try:
yield token.evaluate() # evaluate() returns a Result
except Exception as exception:
errors.append(exception)
# or:
# errors.append(EvaluatorExceptionContext(exception, ...))
Cheers,
Ian
Terry Reedy
> I had a problem for which I've already found a "satisfactory"
> work-around, but I'd like to ask you if there is a better/nicer
> looking solution. Perhaps I'm missing something obvious.
>
> The code looks like this:
>
> stream-of-tokens = token-generator(stream-of-characters)
> stream-of-parsed-expressions = parser-generator(stream-of-tokens)
> stream-of-results = evaluator-generator(stream-of-parsed-expressions)
>
> each of the above functions consumes and implements a generator:
>
> def evaluator-generator(stream-of-tokens):
According to the above, that should be stream-of-parsed-expressions.
> for token in stream-of-tokens:
> try:
> yield token.evaluate() # evaluate() returns a Result
> except Exception as exception:
> yield ErrorResult(exception) # ErrorResult is a subclass of Result
The question which you do not answer below is what, if anything, you
want to do with error? If nothing, just pass. You are now, in effect,
treating them the same as normal results (at least sending them down the
same path), but that does not seem satisfactory to you. If you want them
treated separately, then send them down a different path. Append the
error report to a list or queue or send to a consumer generator
(consumer.send).
> The problem is that, when I use the above mechanism, the errors
> propagate to the output embedded in the data streams. This means, I
> have to make them look like real data (in the example above I'm
> wrapping the exception with an ErrorExpression object) and raise them
> and intercept them again at each level until they finally trickle down
> to the output. It feels a bit like writing in C (checking error codes
> and propagating them to the caller).
>
> OTOH, if I don't intercept the exception inside the loop, it will
> break the for loop and close the generator. So the error no longer
> affects a single token/expression but it kills the whole session. I
> guess that's because the direction flow of control is sort of
> orthogonal to the direction of flow of data.
--
Terry Jan Reedy
r
On Sat, Apr 9, 2011 at 1:30 AM, Terry Reedy <tjr...@udel.edu> wrote:
[...]
> According to the above, that should be stream-of-parsed-expressions.
Good catch.
> The question which you do not answer below is what, if anything, you want to
> do with error? If nothing, just pass. You are now, in effect, treating them
> the same as normal results (at least sending them down the same path), but
> that does not seem satisfactory to you. If you want them treated separately,
> then send them down a different path. Append the error report to a list or
> queue or send to a consumer generator (consumer.send).
The code above implements an interactive session (a REPL). Therefore,
what I'd like to get is an error information printed out at the output
as soon as it becomes available. Piping the errors together with data
works fine (it does what I need) but it feels a bit clunky [1]. After
all exceptions were specifically invented to simplify error handling
(delivering errors to the caller) and here they seem to chose a
"wrong" caller.
Ignoring the errors or collecting them out of band are both fine ideas
but they don't suit the interactive mode of operation.
[1] It's actually not _that_ bad. Exceptions still are very useful
inside of each of these procedures. Errors only have to be handled
manually in that main data path with generators.
Thanks,
-r
Raymond Hettinger
> I had a problem for which I've already found a "satisfactory"
> work-around, but I'd like to ask you if there is a better/nicer
> looking solution. Perhaps I'm missing something obvious.
>
> The code looks like this:
>
> stream-of-tokens = token-generator(stream-of-characters)
> stream-of-parsed-expressions = parser-generator(stream-of-tokens)
> stream-of-results = evaluator-generator(stream-of-parsed-expressions)
>
> each of the above functions consumes and implements a generator:
>
> def evaluator-generator(stream-of-tokens):
> for token in stream-of-tokens:
> try:
> yield token.evaluate() # evaluate() returns a Result
> except Exception as exception:
> yield ErrorResult(exception) # ErrorResult is a subclass of Result
>
> The problem is that, when I use the above mechanism, the errors
> propagate to the output embedded in the data streams. This means, I
> have to make them look like real data (in the example above I'm
> wrapping the exception with an ErrorExpression object) and raise them
> and intercept them again at each level until they finally trickle down
> to the output. It feels a bit like writing in C (checking error codes
> and propagating them to the caller).
You could just let the exception go up to an outermost control-loop
without handling it at all on a lower level. That is what exceptions
for you: terminate all the loops, unwind the stacks, and propagate up
to some level where the exception is caught:
while 1:
try:
results = evaluator-generator(stream-of-parsed-expressions)
for result in results:
print(result)
except Exception as e:
handle_the_exception(e)
OTOH, If you want to catch the exception at the lowest level and wrap
it up as data (the ErrorResult in your example), there is a way to
make it more convenient. Give the ErrorResult object some identify
methods that correspond to the methods being called by upper levels.
This will let the object float through without you cluttering each
level with detect-and-reraise logic.
class ErrorResult:
def __iter__(self):
# pass through an enclosing iterator
yield self
Here's a simple demo of how the pass through works:
>>> from itertools import *
>>> list(chain([1,2,3], ErrorResult(), [4,5,6]))
[1, 2, 3, <__main__.ErrorResult object at 0x2250f70>, 4, 5, 6]
> Any idea for a working and elegant solution?
Hope these ideas have helped.
Raymond
Ethan Furman
> The code above implements an interactive session (a REPL). Therefore,
> what I'd like to get is an error information printed out at the output
> as soon as it becomes available.
Couple ideas:
1) Instead of yielding the error, call some global print function, then
continue on; or
2) Collect the errors, then have the top-most consumer check for errors
and print them out before reading the next generator output.
~Ethan~
r
>
> You could just let the exception go up to an outermost control-loop
> without handling it at all on a lower level. That is what exceptions
> for you: terminate all the loops, unwind the stacks, and propagate up
> to some level where the exception is caught:
>
> while 1:
> try:
> results = evaluator-generator(stream-of-parsed-expressions)
> for result in results:
> print(result)
> except Exception as e:
> handle_the_exception(e)
I've been thinking about something like this but the problem with
shutting down the generators is that I lose all the state information
associated with them (line numbers, have to reopen files if in batch
mode etc.). It's actually more difficult than my current solution.
> OTOH, If you want to catch the exception at the lowest level and wrap
> it up as data (the ErrorResult in your example), there is a way to
> make it more convenient. Give the ErrorResult object some identify
> methods that correspond to the methods being called by upper levels.
> This will let the object float through without you cluttering each
> level with detect-and-reraise logic.
I'm already making something like this (that is, if I understand you
correctly). In the example below (an "almost" real code this time, I
made too many mistakes before) all the Expressions (including the
Error one) implement an 'eval' method that gets called by one of the
loops. So I don't have to do anything to detect the error, just have
to catch it and reraise it at each stage so that it propagates to the
next level (what I have to do anyway as the next level generates
errors as well).
class Expression(object):
def eval(self):
pass
class Error(Expression):
def __init__(self, exception):
self.exception = exception
def eval(self):
raise self.exception
def parseTokens(self, tokens):
for token in tokens:
try:
yield Expression.parseToken(token, tokens)
except ExpressionError as e:
# here is where I wrap exceptions raised during parsing and embed them
yield Error(e)
def eval(expressions, frame):
for expression in expressions:
try:
# and here (.eval) is where they get unwrapped and raised again
yield unicode(expression.eval(frame))
except ExpressionError as e:
# here they are embedded again but because it is the last stage
# text representation is fine
yield unicode(e)
> class ErrorResult:
> def __iter__(self):
> # pass through an enclosing iterator
> yield self
>
> Here's a simple demo of how the pass through works:
>
> >>> from itertools import *
> >>> list(chain([1,2,3], ErrorResult(), [4,5,6]))
> [1, 2, 3, <__main__.ErrorResult object at 0x2250f70>, 4, 5, 6]
I don't really understand what you mean by this example. Why would
making the Error iterable help embedding it into data stream? I'm
currently using yield statement and it seems to work well (?).
Anyway, thank you all for helping me out and bringing some ideas to
the table. I was hoping there might be some pattern specifically
designed for thiskind of job (exception generators anyone?), which
I've overlooked. If not anything else, knowing that this isn't the
case, makes me feel better about the solution I've chosen.
Thanks again,
-r
Raymond Hettinger
> Anyway, thank you all for helping me out and bringing some ideas to
> the table. I was hoping there might be some pattern specifically
> designed for thiskind of job (exception generators anyone?), which
> I've overlooked. If not anything else, knowing that this isn't the
> case, makes me feel better about the solution I've chosen.
Sounds like you've gathered a bunch of good ideas and can now be
pretty sure of your design choices.
While it doesn't help your current use case, you might be interested
in the iter_except() recipe in http://docs.python.org/py3k/library/itertools.html#itertools-recipes
Raymond
twitter: @raymondh
Kent Johnson
> I'm already making something like this (that is, if I understand you
> correctly). In the example below (an "almost" real code this time, I
> made too many mistakes before) all the Expressions (including the
> Error one) implement an 'eval' method that gets called by one of the
> loops. So I don't have to do anything to detect the error, just have
> to catch it and reraise it at each stage so that it propagates to the
> next level (what I have to do anyway as the next level generates
> errors as well).
>
> class Expression(object):
> def eval(self):
> pass
>
> class Error(Expression):
> def __init__(self, exception):
> self.exception = exception
>
> def eval(self):
> raise self.exception
Perhaps, instead of raising exceptions at each level, you could return
an Error object that implements the eval() (or whatever appropriate
protocol) to return an appropriate new Error object. In this case, you
could have
class Error(Expression);
....
def eval(self):
return unicode(self.exception)
Error would itself be created by Expression.parseToken(), instead of
raising an exception.
The idea is that at each level of parsing, instead of raising an
exception you return an object which, when interpreted at the next
outer level, will again return an appropriate error object for that
level. You might be able to have a single Error object which
implements the required methods at each level to just return self.
I don't know if this really works when you start nesting but perhaps
it is worth a try.
Kent