[Python-ideas] x=(yield from) confusion [was:Yet another alternative name for yield-from]
Jim Jewett
I can understand a generator that returns something useful with each yield.
I can understand a generator that is used only for cooperative
multi-tasking, and returns nothing useful until the end.
yield from *as an expression* only really makes sense if the generator
is sending useful information *both* ways. I can understand that sort
of generator only while reading the PEP; the code smell is strong
enough that I forget it by the next day.
Details:
On 4/2/09, Nick Coghlan <ncog...@gmail.com> wrote:
> Jim Jewett wrote:
>> If the "gencall" exhausts the generator f ...
>> If the "return value" of the generator really is important ...
> The intermediate values aren't necessarily discarded by "yield from"
> though: they're passed out to whoever is consuming the values yielded by
> the outermost generator.
I think this may be where I start to have trouble. When I see the
statement form:
def outer_g():
yield from inner_g()
I expect inner_g to suspend execution, and it isn't that hard to
remember that it might do so several times. I also expect the results
of inner_g.next() to be passed on out to outer_g's own caller, thereby
suspending outer_g. So far, so good.
But when yield from is used as an expression, and I see:
x = yield from g()
I somehow expect only a single call to g.next(), whose value gets
assigned to x, and not passed out. I did read the PEP, in several
versions, and understood it at the time ... and still managed (several
times) to forget and misinterpret by a day or two later.
And yes, I realize it doesn't make too much sense to call g.next()
only once -- so I kept looking for a loop around that yield from
statement. When there wasn't one, I shrugged it off like a with
clause -- and still didn't remember the actual PEP-intended meaning.
The times I did remember that (even) the expression form looped, I was
still boggled that it would return something other than None after it
was exhausted. Greg's answer was that it was for threading, and the
final return was the real value. This seems like a different category
of generator, but I could get my head around it -- so long as I forgot
that the yield itself was returning anything useful.
-jJ
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas
Guido van Rossum
> yield from *as an expression* only really makes sense if the generator
> is sending useful information *both* ways. I can understand that sort
> of generator only while reading the PEP; the code smell is strong
> enough that I forget it by the next day.
Read Dave Beazley's coroutines tutorial (dabeaz.com/couroutines) and
check out the contortions in the scheduler to support subgenerators
(Part 8).
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Greg Ewing
> yield from *as an expression* only really makes sense if the generator
> is sending useful information *both* ways.
No, that's not the only way it makes sense. In my
multitasking example, none of the yields send or
receive any values. But they're still needed,
because they define the points at which the task
can be suspended.
> The times I did remember that (even) the expression form looped,
The yield-from expression itself doesn't loop. What
it does do is yield multiple times, if the generator
being called yields multiple times. But it has to
be driven by whatever is calling the whole thing
making a sufficient number of next() or send() calls,
in a loop or otherwise.
In hindsight, the wording in the PEP about the
subiterator being "run to exhaustion" might be a bit
misleading. I'l see if I can find a better way to
phrase it.
--
Greg
Nick Coghlan
> The times I did remember that (even) the expression form looped, I was
> still boggled that it would return something other than None after it
> was exhausted. Greg's answer was that it was for threading, and the
> final return was the real value. This seems like a different category
> of generator, but I could get my head around it -- so long as I forgot
> that the yield itself was returning anything useful.
Greg tried to clarify this a bit already, but I think Jacob's averager
example is an interesting case where it makes sense to both yield
multiple times and also "return a value".
While it is just a toy example, I believe it does a great job of
illustrating the control flow expectations. (Writing this email has
certainly clarified a lot of things about the PEP in my *own* mind).
The following reworking of Jacob's example assumes a couple of things
that differ from the current PEP:
- the particular colour my bikeshed is painted when it comes to
returning values from a generator is "return finally" (the idea being to
emphasise that this represents a special "final" value for the generator
that happens only after all of the normal yields are done).
- rather than trying to change the meaning of GeneratorExit and close(),
3 new generator methods would be added: next_return(), send_return() and
throw_return(). The new methods have the same signatures as their
existing counterparts, but if the generator raises GeneratorReturn, they
trap it and return the associated value instead. Like close(), they
complain with a RuntimeError if the generator doesn't finish. For example:
def throw_return(self, *exc_info):
try:
self.throw(*exc_info)
raise RuntimeError("Generator did not terminate")
except GeneratorReturn as gr:
return gr.value
(Note that I've also removed the 'yield raise' idea from the example -
if next() or send() triggers termination of the generator with an
exception other than StopIteration, then that exception is already
propagated into the calling scope by the existing generator machinery. I
realise Jacob was trying to make it possible to "yield an exception"
without terminating the coroutine, but that idea is well beyond the
scope of the PEP)
You then get:
class CalcAverage(Exception): pass
def averager(start=0):
# averager that maintains a running average
# and returns the final average when done
count = 0
exc = None
sum = start
while 1:
avg = sum / count
try:
val = yield avg
except CalcAverage:
return finally avg
sum += val
count += 1
avg = averager()
avg.next() # start coroutine
avg.send(1.0) # yields 1.0
avg.send(2.0) # yields 1.5
print avg.throw_return(CalcAverage) # prints 1.5
Now, suppose I want to write another toy coroutine that calculates the
averages of two sequences and then returns the difference:
def average_diff(start=0):
avg1 = yield from averager(start)
avg2 = yield from averager(start)
return finally avg2 - avg1
diff = average_diff()
diff.next() # start coroutine
# yields 0.0
avg.send(1.0) # yields 1.0
avg.send(2.0) # yields 1.5
diff.throw(CalcAverage) # Starts calculation of second average
# yields 0.0
diff.send(2.0) # yields 2.0
diff.send(3.0) # yields 2.5
print diff.throw_return(CalcAverage) # Prints 1.0 (from "2.5 - 1.5")
The same example could be rewritten to use "None" as a sentinel value
instead of throwing in an exception (average_diff doesn't change, so I
haven't rewritten that part):
def averager(start=0):
count = 0
exc = None
sum = start
while 1:
avg = sum / count
val = yield avg
if val is None:
return finally avg
sum += val
count += 1
# yielded values are same as the throw_return() approach
diff = average_diff()
diff.next() # start coroutine
diff.send(1.0)
diff.send(2.0)
diff.send(None) # Starts calculation of second average
diff.send(2.0)
diff.send(3.0)
print diff.send_return(None) # Prints 1.0 (from "2.5 - 1.5")
Notice how the coroutines in this example can be thought of as simple
state machines that the calling code needs to know how to drive. That
state sequence is as much a part of the coroutine's signature as are the
arguments to the constructor and the final return value.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Jacob Holm
Your reworking of my "averager" example has highlighted another issue
for me, which I will get to below. First a few comments on your message.
Nick Coghlan wrote:
> [snip]
>
> The following reworking of Jacob's example assumes a couple of things
> that differ from the current PEP:
>
> - the particular colour my bikeshed is painted when it comes to
> returning values from a generator is "return finally" (the idea being to
> emphasise that this represents a special "final" value for the generator
> that happens only after all of the normal yields are done).
>
We should probably drop that particular bikeshed discussion until we
actually know the details of what the construct should do, esp in the
context of close(). I am starting to lose track of all the different
possible versions.
> - rather than trying to change the meaning of GeneratorExit and close(),
> 3 new generator methods would be added: next_return(), send_return() and
> throw_return(). The new methods have the same signatures as their
> existing counterparts, but if the generator raises GeneratorReturn, they
> trap it and return the associated value instead. Like close(), they
> complain with a RuntimeError if the generator doesn't finish. For example:
>
> def throw_return(self, *exc_info):
> try:
> self.throw(*exc_info)
> raise RuntimeError("Generator did not terminate")
> except GeneratorReturn as gr:
> return gr.value
>
I don't much like the idea of adding these methods, but that is not the
point of this mail.
> (Note that I've also removed the 'yield raise' idea from the example -
> if next() or send() triggers termination of the generator with an
> exception other than StopIteration, then that exception is already
> propagated into the calling scope by the existing generator machinery. I
> realise Jacob was trying to make it possible to "yield an exception"
> without terminating the coroutine, but that idea is well beyond the
> scope of the PEP)
>
I think it was pretty clearly marked as out of scope for this PEP, but I
still like the idea.
> You then get:
>
> class CalcAverage(Exception): pass
>
> def averager(start=0):
> # averager that maintains a running average
> # and returns the final average when done
> count = 0
> exc = None
> sum = start
> while 1:
> avg = sum / count
> try:
> val = yield avg
> except CalcAverage:
> return finally avg
> sum += val
> count += 1
>
> avg = averager()
> avg.next() # start coroutine
> avg.send(1.0) # yields 1.0
> avg.send(2.0) # yields 1.5
> print avg.throw_return(CalcAverage) # prints 1.5
>
This version has a bug. It will raise ZeroDivisionError on the initial
next() call used to start the generator. A better version if you insist
on yielding the running average, would be:
def averager(start=0):
# averager that maintains a running average
# and returns the final average when done
count = 0
sum = start
avg = None
while 1:
try:
val = yield avg
except CalcAverage:
return finally avg
sum += val
count += 1
avg = sum/count
> Now, suppose I want to write another toy coroutine that calculates the
> averages of two sequences and then returns the difference:
>
> def average_diff(start=0):
> avg1 = yield from averager(start)
> avg2 = yield from averager(start)
> return finally avg2 - avg1
>
> diff = average_diff()
> diff.next() # start coroutine
> # yields 0.0
> avg.send(1.0) # yields 1.0
> avg.send(2.0) # yields 1.5
> diff.throw(CalcAverage) # Starts calculation of second average
> # yields 0.0
> diff.send(2.0) # yields 2.0
> diff.send(3.0) # yields 2.5
> print diff.throw_return(CalcAverage) # Prints 1.0 (from "2.5 - 1.5")
>
>
(There is another minor bug here: the two avg.send() calls should have
been diff.send()).
Now for my problem. The original averager example was inspired by the
tutorial http://dabeaz.com/coroutines/ that Guido pointed to. (Great
stuff, btw). One pattern that is recommended by the tutorial and used
throughout is to decorate all coroutines with a decorator like:
def coroutine(func):
def start(*args,**kwargs):
cr = func(*args,**kwargs)
cr.next()
return cr
return start
The idea is that it saves you from the initial next() call used to start
the coroutine. The problem is that you cannot use such a decorated
coroutine in any flavor of the yield-from expression we have considered
so far, because the yield-from will start out by doing an *additional*
next call and yield that value.
I have a few vague ideas of how we might change "yield from" to support
this, but nothing concrete enough to put here. Is this a problem we
should try to fix, and if so, how?
not-trying-to-be-difficult-ly yours
- Jacob
Nick Coghlan
>> - the particular colour my bikeshed is painted when it comes to
>> returning values from a generator is "return finally" (the idea being to
>> emphasise that this represents a special "final" value for the generator
>> that happens only after all of the normal yields are done).
>>
>
> We should probably drop that particular bikeshed discussion until we
> actually know the details of what the construct should do, esp in the
> context of close(). I am starting to lose track of all the different
> possible versions.
Note that the syntax for returning values from generators is largely
independent of the semantics. Guido has pointed out that disallowing the
naive "return EXPR" in generators is an important learning tool for
inexperienced generator users, and I think he's right.
"return finally" reads pretty well and doesn't add a new keyword, while
still allowing generator return values to be written easily. I haven't
seen other suggestions I particularly like, so I figured I'd run with
that one for the revised example :)
>> - rather than trying to change the meaning of GeneratorExit and close(),
>> 3 new generator methods would be added: next_return(), send_return() and
>> throw_return(). The new methods have the same signatures as their
>> existing counterparts, but if the generator raises GeneratorReturn, they
>> trap it and return the associated value instead. Like close(), they
>> complain with a RuntimeError if the generator doesn't finish. For
>> example:
>>
>> def throw_return(self, *exc_info):
>> try:
>> self.throw(*exc_info)
>> raise RuntimeError("Generator did not terminate")
>> except GeneratorReturn as gr:
>> return gr.value
>>
>
> I don't much like the idea of adding these methods, but that is not the
> point of this mail.
They don't have to be generator methods - they could easily be functions
in a coroutine module. However, I definitely prefer the idea of new
methods or functions that support a variety of interaction styles over
trying to redefine generator finalisation tools (i.e. GeneratorExit and
close()) to cover this completely different use case. Why create a
potential backwards compatibility problem for ourselves when there are
equally clean alternative solutions?
I also don't like the idea of imposing a specific coroutine return idiom
in the PEP - better to have a system that supports both sentinel values
(via next_return() and send_return()) and sentinel exceptions (via
send_throw()).
> Now for my problem. The original averager example was inspired by the
> tutorial http://dabeaz.com/coroutines/ that Guido pointed to. (Great
> stuff, btw). One pattern that is recommended by the tutorial and used
> throughout is to decorate all coroutines with a decorator like:
>
> def coroutine(func):
> def start(*args,**kwargs):
> cr = func(*args,**kwargs)
> cr.next()
> return cr
> return start
>
>
> The idea is that it saves you from the initial next() call used to start
> the coroutine. The problem is that you cannot use such a decorated
> coroutine in any flavor of the yield-from expression we have considered
> so far, because the yield-from will start out by doing an *additional*
> next call and yield that value.
>
> I have a few vague ideas of how we might change "yield from" to support
> this, but nothing concrete enough to put here. Is this a problem we
> should try to fix, and if so, how?
Hmm, that's a tricky one. It sounds like it is definitely an issue the
PEP needs to discuss, but I don't currently have an opinion as to what
it should say.
> not-trying-to-be-difficult-ly yours
We have a long way to go before we even come close to consuming as many
pixels as PEP 308 or PEP 343 - a fact for which Greg is probably grateful ;)
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Jim Jewett
> Jim Jewett wrote:
>> yield from *as an expression* only really makes sense if the generator
>> is sending useful information *both* ways.
> No, that's not the only way it makes sense. In my
> multitasking example, none of the yields send or
> receive any values.
err... I didn't mean both directions, I meant "from the callee to the
caller as a yielded value" and "from the callee to the caller as a
final return value that can't be yielded normally."
> But they're still needed,
> because they define the points at which the task
> can be suspended.
If they don't send or receive values, then why do they need to be
expressions instead of statements?
>> The times I did remember that (even) the expression form looped,
> The yield-from expression itself doesn't loop. What
> it does do is yield multiple times,
That sounds to me like an implicit loop.
yield from iter <==> for val in iter: yield val
So the outside generator won't progress to its own next line (and
subsequent yield) until it has finished looping over the inner
generator.
-jJ
Jim Jewett
> Greg tried to clarify this a bit already, but I think Jacob's averager
> example is an interesting case where it makes sense to both yield
> multiple times and also "return a value".
> def averager(start=0):
> # averager that maintains a running average
> # and returns the final average when done
> count = 0
> exc = None
> sum = start
> while 1:
> avg = sum / count
> try:
> val = yield avg
> except CalcAverage:
> return finally avg
> sum += val
> count += 1
It looks to me like it returns (or yields) the running average either way.
I see a reason to send in a sentinel value, saying "Don't update the
average, just tell me the current value."
I don't see why that sentinel has to terminate the generator, nor do I
see why that final average has to be returned rather than yielded.
-jJ
Jim Jewett
> On Thu, Apr 2, 2009 at 6:43 PM, Jim Jewett <jimjj...@gmail.com> wrote:
>> yield from *as an expression* only really makes sense if the generator
>> is sending useful information *both* ways. I can understand that sort
>> of generator only while reading the PEP; the code smell is strong
>> enough that I forget it by the next day.
> Read Dave Beazley's coroutines tutorial (dabeaz.com/couroutines) and
> check out the contortions in the scheduler to support subgenerators
> (Part 8).
I have. I still don't see it really helping with anything except
maybe (in http://dabeaz.com/coroutines/pyos8.py) the Task.run method.
Even there, I don't see an expression form helping very much. Passing
through the intermediate yields could be nice, but a statement can do
that. Grabbing a separate final value could change the "try ...
except StopIteration" into an "x=yield from", but ... I'm not sure
that could really work, because I don't see how all the code inside
the try block goes away. (It might go away if you moved that logic
from the task to the scheduler, but then the protocol seems to get
even more complicated, as the scheduler has to distinguish between
"I've used up this time slot", as well as "I have intermediate
results, but might be called again", and "I'm done".)
-jJ
Guido van Rossum
+1 on adding this as a recipe to the docs.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Jacob Holm
> It looks to me like it returns (or yields) the running average either way.
>
You can see my original example at:
http://mail.python.org/pipermail/python-ideas/2009-April/003841.html
and a few arguments why I think it is better at:
http://mail.python.org/pipermail/python-ideas/2009-April/003847.html
> I see a reason to send in a sentinel value, saying "Don't update the
> average, just tell me the current value."
>
> I don't see why that sentinel has to terminate the generator, nor do I
> see why that final average has to be returned rather than yielded.
>
Yielding the current value on each send was not part of the original
example because I was thinking in terms of well-behaved coroutines as
described in http://dabeaz.com/coroutines/. I agree that it makes sense
for running averages, but it is not that hard to come up with similar
examples where the intermediate state is not really useful and/or may be
expensive to compute.
The reason for closing would be that once you have computed the final
result, you want whatever resources the coroutine is using to be freed.
Since only the final result is assumed to be useful, it makes perfect
sense to close the coroutine at the same time as you are requesting the
final result.
- Jacob
Jim Jewett
> Yielding the current value on each send was not part of the original
> example ...
So in the original you cared about the final value, but not the
intermediate yields. My question is whether there is a sane case
where you care about *both*, *and* you care about distinguishing them.
And is this case common really enough that we don't want it marked up
with something more explicit, like a sentinel or a raise?
> The reason for closing would be that once you have computed the final
> result, you want whatever resources the coroutine is using to be freed.
> Since only the final result is assumed to be useful, it makes perfect
> sense to close the coroutine at the same time as you are requesting the
> final result.
def outer(unfinished=object()):
g=inner(unfinished)
for result in g:
yield unfinished # cooperative multi-tasking, so co-operate
if result is not unfinished: break
...
I see some value in simplifying that, or adding more power.
But I'm not convinced the current proposal actually is much simpler,
or that the extra power wouldn't be better written in a more explicit
manner. I think the above still translates into
def outer(unfinished=object()):
# # now also need to set an initial value of result
# # *OR* distinguish intermediate from final results.
# result=unfinished
g=inner(unfinished)
# # loop is now implicit
# while result is unfinished:
# result = yield from g
result = yield from g
...
-jJ
Guido van Rossum
> You can see my original example at:
>
> http://mail.python.org/pipermail/python-ideas/2009-April/003841.html
>
> and a few arguments why I think it is better at:
>
> http://mail.python.org/pipermail/python-ideas/2009-April/003847.html
> The reason for closing would be that once you have computed the final
> result, you want whatever resources the coroutine is using to be freed.
> Since only the final result is assumed to be useful, it makes perfect sense
> to close the coroutine at the same time as you are requesting the final
> result.
Hm. I am beginning to see what you are asking for. Your averager
example is somewhat convincing.
Interestingly, in a sense it seems unrelated to yield-from: the
averager doesn't seem to need yield-from, it receives values sent to
it using send(). An alternative version (that works today), which I
find a bit clearer, uses exceptions instead of a sentinel value: when
you are done with sending it the sequence of values, you throw() a
special exception into it, and in response it raises another exception
back with a value attribute. I'm showing the usage example first, then
the support code.
Usage example:
@coroutine
def summer():
sum = 0
while True:
try:
value = yield
except Terminate:
raise Done(sum)
else:
sum += value
def main():
a = summer()
a.send(1)
a.send(2)
print finalize(a)
Support code:
class Terminate(Exception):
"""Exception thrown into the generator to ask it to stop."""
class Done(Exception):
"""Exception raised by the generator when it catches Terminate."""
def __init__(self, value=None):
self.value = value
def coroutine(func):
"""Decorator around a coroutine, to make the initial next() call."""
def wrapper(*args, **kwds):
g = func(*args, **kwds)
g.next()
return g
return wrapper
def finalize(g):
"""Throw Terminate into a couroutine and extract the value from Done."""
try:
g.throw(Terminate)
except Done as e:
return e.value
else:
g.close()
raise RuntimeError("Expected Done(<value>)")
I use a different exception to throw into the exception as what it
raises in response, so that mistakes (e.g. the generator not catching
Terminate) are caught, and no confusion can exist with the built-in
exceptions StopIteration and GeneratorExit.
Now I'll compare this manual version with your (Jacob Holm's) proposal:
- instead of Done you use GeneratorExit
- hence, instead of g.throw(Done) you can use g.close()
- instead of Terminate you use StopException
- you want g.close() to extract and return the value from StopException
- you use "return value" instead of "raise Done(value)"
The usage example then becomes, with original version indicated in comments:
@coroutine
def summer():
sum = 0
while True:
try:
value = yield
except GeneratorExit: # except Terminate:
return sum # raise Done(sum)
else:
sum += value
def main():
a = summer()
a.send(1)
a.send(2)
print a.close() # print finalize(a)
At this point, I admin that I am not yet convinced. On the one hand,
my support code melts away, except for the @coroutine decorator. On
the other hand:
- the overloading of GeneratorExit and StopIteration reduces
diagnostics for common beginner's mistakes when writing regular
(iterator-style) generator code
- the usage example isn't much simpler
- the support code isn't rocket science
- the coroutine use case is specialized enough that a little support
seems okay (you still need @coroutine anyway)
The last three points don't sway me either way: they pit minor
conveniences against minor inconveniences.
However, the first point worries me a lot. The concern over
StopIteration can be dealt with by introducing a new exception that is
raised only by "return value" inside a generator.
But I'm also worried that the mere need to catch GeneratorExit for a
purpose other than resource cleanup will cause examples using it to
pop up on the web, which will then be copied and modified by clueless
beginners, and *increase* the probability of bad code being written.
(That's why I introduce *new* exceptions in my support code -- they
don't have predefined meanings in other contexts.)
Finally, I am not sure of the connection with "yield from". I don't
see a way to exploit it for this example. As an exercise, I
constructed an "averager" generator out of the above "summer" and a
similar "counter", and I didn't see a way to exploit "yield from". The
only connection seems to be PEP 380's proposal to turn "return value"
inside a generator into "raise StopIteration(value)", and that's the
one part of the PEP with which I have a problem anyway (the beginner's
issues above). Oh, and "yield from" competes with @couroutine over
when the initial next() call is made, which again suggests the two
styles (yield-from and coroutines) are incompatible.
All in all, I think I would be okay with turning "return value" inside
a generator into raising *some* exception, as long as that exception
is not StopIteration (nor derives from it, nor from GeneratorExit).
PEP 380 and its implementation would become just a tad more complex,
but I think that's worth it. Generators used as iterators would raise
a (normally) uncaught exception if they returned a value, and that's
my main requirement. I'm still not convince that more is needed, in
particular I'm still -0 on catching this value in gen_close() and
returning the value attribute from there.
As I've said before, I don't care whether "return None" would be
treated more like "return" or more like "return value" -- for
beginners' code I don't think it matters, and for advanced code they
should be equivalent.
I'll stop arguing for new syntax to return a value from a generator
(like Phillip Eby's proposed "return from yield with <value>"): I
don't think it adds enough to overcome the pain for the parser and
other tools.
Finally, as far as a name for the new exception, I think something
long like ReturnFromGenerator would be fine, since most of the time it
is handled implicitly by coroutine support code (whether this is user
code or gen_close()) or the yield-from implementation.
I'm sorry for being so long winded and yet somewhat inconclusive. I
wouldn't have bothered if I didn't think there was something worth
pursuing. But it sure seems elusive.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Jacob Holm
Thank you for taking the time. I'll try to be brief so I don't spend
more of it than necesary... (I'll probably fail though)
Guido van Rossum wrote:
> Hm. I am beginning to see what you are asking for. Your averager
> example is somewhat convincing.
>
> Interestingly, in a sense it seems unrelated to yield-from: the
> averager doesn't seem to need yield-from, it receives values sent to
> it using send().
It is related to yield-from only because the "return value from
generator" concept is introduced there. I was trying to work out the
details of how I would like that to work in the context of GeneratorExit
and needed a near-trivial example.
[snip "summer" example using existing features and comparison with
version using suggested new features]
> At this point, I admin that I am not yet convinced. On the one hand,
> my support code melts away, except for the @coroutine decorator. On
> the other hand:
>
> - the overloading of GeneratorExit and StopIteration reduces
> diagnostics for common beginner's mistakes when writing regular
> (iterator-style) generator code
> - the usage example isn't much simpler
> - the support code isn't rocket science
> - the coroutine use case is specialized enough that a little support
> seems okay (you still need @coroutine anyway)
>
> The last three points don't sway me either way: they pit minor
> conveniences against minor inconveniences.
>
> However, the first point worries me a lot. The concern over
> StopIteration can be dealt with by introducing a new exception that is
> raised only by "return value" inside a generator.
>
I am still trying to get a clear picture of what kind of mistakes you
are trying to protect against.
If it is people accidently writing return in a generator when they
really mean yield, that is what I thought the proposal for an alternate
syntax was for. That sounds like a good idea to me, especially if we
could also ban or discourage the use of normal return. But the
alternate syntax doesn't have to mean a different exception.
Since you are no longer pushing an alternative syntax for return but
still want a different exception, I'll assume there is some other
beginner mistake you are worried about. My guess is it is some mistake
at the places where the generator is used, but I am having a hard time
figuring out where the mistake could be in ignoring the returned value.
Perhaps you (or someone who has more time) can provide an example where
this is a bad thing?
> But I'm also worried that the mere need to catch GeneratorExit for a
> purpose other than resource cleanup will cause examples using it to
> pop up on the web, which will then be copied and modified by clueless
> beginners, and *increase* the probability of bad code being written.
> (That's why I introduce *new* exceptions in my support code -- they
> don't have predefined meanings in other contexts.)
>
That worry I can understand.
> Finally, I am not sure of the connection with "yield from". I don't
> see a way to exploit it for this example. As an exercise, I
> constructed an "averager" generator out of the above "summer" and a
> similar "counter", and I didn't see a way to exploit "yield from". The
> only connection seems to be PEP 380's proposal to turn "return value"
> inside a generator into "raise StopIteration(value)", and that's the
> one part of the PEP with which I have a problem anyway (the beginner's
> issues above).
Yes, the only connection is that this is where "return value" is
introduced. I could easily see "return value" as a separate PEP,
except PEP 380 provides one of the better reasons for its inclusion. It
might be good to figure out how this feature should work by itself
before complicating things by integrating it in the yield-from semantics.
> Oh, and "yield from" competes with @couroutine over
> when the initial next() call is made, which again suggests the two
> styles (yield-from and coroutines) are incompatible.
>
It is a serious problem, because one of the major points of the PEP is
that it should be useful for refactoring coroutines. As a matter of
fact, I started another thread on this specific issue earlier today
which only Nick has so far responded to. I think it is solvable, but
requires some more work.
> All in all, I think I would be okay with turning "return value" inside
> a generator into raising *some* exception, as long as that exception
> is not StopIteration (nor derives from it, nor from GeneratorExit).
> PEP 380 and its implementation would become just a tad more complex,
> but I think that's worth it. Generators used as iterators would raise
> a (normally) uncaught exception if they returned a value, and that's
> my main requirement. I'm still not convince that more is needed, in
> particular I'm still -0 on catching this value in gen_close() and
> returning the value attribute from there.
>
As long as close is not catching it without also returning the value.
That would be *really* annoying.
> As I've said before, I don't care whether "return None" would be
> treated more like "return" or more like "return value" -- for
> beginners' code I don't think it matters, and for advanced code they
> should be equivalent.
>
> I'll stop arguing for new syntax to return a value from a generator
> (like Phillip Eby's proposed "return from yield with <value>"): I
> don't think it adds enough to overcome the pain for the parser and
> other tools.
>
And here I was starting to think that a new syntax for return could
solve the problem of beginner mistakes without needing a new exception.
> Finally, as far as a name for the new exception, I think something
> long like ReturnFromGenerator would be fine, since most of the time it
> is handled implicitly by coroutine support code (whether this is user
> code or gen_close()) or the yield-from implementation.
>
GeneratorReturn is the name we have been using for this beast so far,
but I really don't care what it is called.
> I'm sorry for being so long winded and yet somewhat inconclusive. I
> wouldn't have bothered if I didn't think there was something worth
> pursuing. But it sure seems elusive.
>
And I thank you again for your time.
Best regards
- Jacob
Jacob Holm
> Jacob Holm wrote:
>
>> We should probably drop that particular bikeshed discussion until we
>> actually know the details of what the construct should do, esp in the
>> context of close(). I am starting to lose track of all the different
>> possible versions.
>>
>
> Note that the syntax for returning values from generators is largely
> independent of the semantics. Guido has pointed out that disallowing the
> naive "return EXPR" in generators is an important learning tool for
> inexperienced generator users, and I think he's right.
>
I agree that a separate syntax for returning a value from a
generator/coroutine is propably a good idea. (I am still not convinced
we need a separate exception for it, but that is a separate
discussion). I even think it would be a good idea to deprecate the use
of the normal "return" in generators, but that is probably not going to
happen.
> "return finally" reads pretty well and doesn't add a new keyword, while
> still allowing generator return values to be written easily. I haven't
> seen other suggestions I particularly like, so I figured I'd run with
> that one for the revised example :)
>
You can call it whatever you want, as long as it works predictably for
the use cases we are finding.
[snip]
>> Now for my problem. The original averager example was inspired by the
>> tutorial http://dabeaz.com/coroutines/ that Guido pointed to. (Great
>> stuff, btw). One pattern that is recommended by the tutorial and used
>> throughout is to decorate all coroutines with a decorator like:
>>
>> def coroutine(func):
>> def start(*args,**kwargs):
>> cr = func(*args,**kwargs)
>> cr.next()
>> return cr
>> return start
>>
>>
>> The idea is that it saves you from the initial next() call used to start
>> the coroutine. The problem is that you cannot use such a decorated
>> coroutine in any flavor of the yield-from expression we have considered
>> so far, because the yield-from will start out by doing an *additional*
>> next call and yield that value.
>>
>> I have a few vague ideas of how we might change "yield from" to support
>> this, but nothing concrete enough to put here. Is this a problem we
>> should try to fix, and if so, how?
>>
>
> Hmm, that's a tricky one. It sounds like it is definitely an issue the
> PEP needs to discuss, but I don't currently have an opinion as to what
> it should say.
>
Here is one possible fix, never mind the syntax. We could change the
yield from expression from the current:
RESULT = yield from EXPR
by adding an extra form, possibly one of:
RESULT = yield STARTEXPR from EXPR
RESULT = yield from EXPR with STARTEXPR
RESULT = yield from EXPR as NAME starting with STARTEXPR(NAME)
And letting STARTEXPR if given take the place of the initial _i.next()
in the expansion(s). The point is we need to yield *something* first,
before rerouting all send(), next() and throw() calls to the subiterator.
>> not-trying-to-be-difficult-ly yours
>>
>
> We have a long way to go before we even come close to consuming as many
> pixels as PEP 308 or PEP 343 - a fact for which Greg is probably grateful ;)
>
Him and everybody else I would think. But AFAICT we are not even close
to finished, so we may get there yet.
- Jacob
Nick Coghlan
> On 4/3/09, Nick Coghlan <ncog...@gmail.com> wrote:
>> Greg tried to clarify this a bit already, but I think Jacob's averager
>> example is an interesting case where it makes sense to both yield
>> multiple times and also "return a value".
>
>> def averager(start=0):
>> # averager that maintains a running average
>> # and returns the final average when done
>> count = 0
>> exc = None
>> sum = start
>> while 1:
>> avg = sum / count
>> try:
>> val = yield avg
>> except CalcAverage:
>> return finally avg
>> sum += val
>> count += 1
>
> It looks to me like it returns (or yields) the running average either way.
>
> I see a reason to send in a sentinel value, saying "Don't update the
> average, just tell me the current value."
>
> I don't see why that sentinel has to terminate the generator, nor do I
> see why that final average has to be returned rather than yielded.
It doesn't *have* to do anything - you could set it up that way if you
wanted to. However, when the running average is only yielded, then the
location providing the numbers (i.e. the top level code in my example)
is also the only location which can receive the running average. That's
why anyone using coroutines now *has* to have a top-level scheduler to
handle the "coroutine stack" by knowing which coroutines are calling
each other and detecting when one has "returned" (i.e. yielded a special
sentinel value that the scheduler recognises) so the return value can be
passed back to the calling coroutine.
I hoped to show the advantage of the separate return value with the
difference calculating coroutine: in that example, having the separate
return value makes it easy to identify the values which need to be
returned to a location *other* than the source of the numbers being
averaged. The example goes through some distinct stages:
- top level code sending numbers to first averager via send() and
receiving running averages back via yield
- top level code telling first averager to finish up (via either throw()
or send() depending on implementation), first averager returning final
average value to the difference calculator
- top level code sending numbers to second averager and receiving
running averages back
- top level code telling second averager to finish up, second averager
returning final average value to the difference calculator, difference
calculator returning difference between the two averages to the top
level code
That is, yield, send() and throw() involve communication between the
currently active subcoroutine and the client of the whole coroutine.
They bypass the current stack in the coroutine itself. The return
values, on the other hand, *do* involve unwinding the coroutine stack,
just like they do with normal function calls.
I'll have another go, this time comparing the "normal" calls to the
coroutine version:
def average(seq, start=0):
if seq:
return sum(seq, start) / len(seq)
return start
def average_diff(seq1, seq2):
avg1 = average(seq1)
avg2 = average(seq2)
return avg2 - avg1
OK, simple and straightforward. The idea of 'yield from' is to allow
"average_diff" to be turned into a coroutine that receives the values to
be averaged one by one *without* having to inline the actual average
calculation.
I'll also go back to Jacob's original averaging example that *doesn't*
yield a running average - it is more inline with examples where the
final calculation is expensive, so you won't do it until you're told you
have all the data.
What might that look like as 'yield from' style coroutines?
class CalcAverage(Exception): pass
def average_cr(start=0):
count = 0
exc = None
sum = start
while 1:
try:
# The yield surrenders control to the code
# that is supplying the numbers to be
# averaged
val = yield
except CalcAverage:
# The return finally passes the final
# average back to the code that was
# asking for the average (which is NOT
# necessarily the same code that was
# supplying the numbers
if count:
return finally sum / count
return finally start
sum += val
count += 1
# Note how similar this is to the normal version above
def average_diff_cr(start):
avg1 = yield from average_cr(start, seq1)
avg2 = yield from average_cr(start, seq2)
return avg2 - avg1
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Nick Coghlan
> Since you are no longer pushing an alternative syntax for return but
> still want a different exception, I'll assume there is some other
> beginner mistake you are worried about. My guess is it is some mistake
> at the places where the generator is used, but I am having a hard time
> figuring out where the mistake could be in ignoring the returned value.
> Perhaps you (or someone who has more time) can provide an example where
> this is a bad thing?
I can't speak for Guido, but the two easy beginner mistakes I think are
worth preventing:
- using 'return' where you meant 'yield' (however, if even 'return
finally' doesn't appeal to Guido as alternative syntax for "no, this is
a coroutine, I really mean it" then I'm fine with that)
- trying to iterate normally over a coroutine instead of calling it
appropriately (raising GeneratorReturn instead of StopIteration means
that existing iterative code will let the new exception escape rather
than silently suppressing the return exception)
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Greg Ewing
> "return finally" reads pretty well and doesn't add a new keyword
Still doesn't mean anything, though. Ordinary returns
happen "finally" too (most of the time, anyway), so
what's the difference?
--
Greg
Greg Ewing
> err... I didn't mean both directions, I meant "from the callee to the
> caller as a yielded value" and "from the callee to the caller as a
> final return value that can't be yielded normally."
I think perhaps we're misunderstanding each other. You
seemed to be saying that the only time you would want
a generator to return a value is when you were also
using it to either send or receive values using yield,
and I was just pointing out that's not true.
If that's not what you meant, you'll have to explain
more clearly what you do mean.
--
Greg
Nick Coghlan
> Nick Coghlan wrote:
>
>> "return finally" reads pretty well and doesn't add a new keyword
>
> Still doesn't mean anything, though. Ordinary returns
> happen "finally" too (most of the time, anyway), so
> what's the difference?
It needs to be mnemonic, not literal. The difference between a generator
return and a normal function return is that with a generator you will
typically have at least one yield before the actual return, whereas with
a normal function, return or an exception are the only way to leave the
function's frame. So "return finally" is meant to help you remember that
it happens only after all the yields are done.
Guido has said he is OK with losing the novice assistance on this one
though, so it's probably a moot point.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Jacob Holm
> Jacob Holm wrote:
>
>> Since you are no longer pushing an alternative syntax for return but
>> still want a different exception, I'll assume there is some other
>> beginner mistake you are worried about. My guess is it is some mistake
>> at the places where the generator is used, but I am having a hard time
>> figuring out where the mistake could be in ignoring the returned value.
>> Perhaps you (or someone who has more time) can provide an example where
>> this is a bad thing?
>>
>
> I can't speak for Guido, but the two easy beginner mistakes I think are
> worth preventing:
>
> - using 'return' where you meant 'yield' (however, if even 'return
> finally' doesn't appeal to Guido as alternative syntax for "no, this is
> a coroutine, I really mean it" then I'm fine with that)
>
> - trying to iterate normally over a coroutine instead of calling it
> appropriately (raising GeneratorReturn instead of StopIteration means
> that existing iterative code will let the new exception escape rather
> than silently suppressing the return exception)
>
But this one I still don't get. Let me try a couple of cases:
1) We have a coroutine that expects you to call send and/or throw with
specific values, and ends up returning a value. A beginner may try to
iterate over it, but will most likely get an exception on the first
next() call because the input is not valid. Or he would get an infinite
loop because None is not changing the state of the coroutine. In any
case, it is unlikely that he will get to see either StopIteration or the
new exception, because the input is not what the coroutine expects. The
new exception doesn't help here.
2) We have a generator that e.g. pulls values from a file, yielding the
processed values as it goes along, and returning some form of summary at
the end. If I iterate over it with a for-loop, I get all the values
asn usual ... followed by an exception. Why do I have to get an
exception there just because the generator has some information that its
implementer thought I might want? Ignoring the value in this case
seems perfectly reasonable, so having to catch an exception is just
noise here.
3) We have a coroutine that computes something expensive, occationally
yielding to let other code run. It neither sends or receives values,
just uses yield for cooperative multitasking. When it is done it
returns a value. If you loop over this coroutine, you will get a bunch
of Nones, followed by the new exception. You could argue that the new
exception helps you here. One way of accessing the returned value would
be to catch it and look at an attribute. However, for this case I would
prefer to just call close on the generator to get the value afterwards.
A beginner might be helped by the unexpected exception, but I think even
a beginner would find that something strange was going on when the only
value he gets for the loop variable is None. He might even look up the
documentation for the coroutine he was calling and see how it was
supposed to be used.
4) ... ?
Do you have other concrete use cases I haven't thought of where a the
new exception would help?
- Jacob
Jacob Holm
> Here is one possible fix, never mind the syntax. We could change the
> yield from expression from the current:
>
> RESULT = yield from EXPR
>
> by adding an extra form, possibly one of:
>
> RESULT = yield STARTEXPR from EXPR
> RESULT = yield from EXPR with STARTEXPR
> RESULT = yield from EXPR as NAME starting with STARTEXPR(NAME)
>
> And letting STARTEXPR if given take the place of the initial _i.next()
> in the expansion(s). The point is we need to yield *something* first,
> before rerouting all send(), next() and throw() calls to the subiterator.
>
Another possible fix would be to have new syntax for specifying that the
initial call to the coroutine should be using send or throw instead.
This could be seen as a restriction on what could be used as
STARTEXPR(NAME) in the earlier syntax idea.
Yet another fix that requires no extra syntax would be to store the
latest value yielded by the generator in a property on the generator
(raising AttributeError before the first yield). Then the initial:
_y = _i.next()
could be replaced with:
try:
_y = _i.gi_latest_yield # or whatever its name would be.
except AttributeError:
_y = _i.next()
The benefit of this version is that it requires no new syntax, it avoids
the extra next() call for coroutines, and it opens some new ways of
using generators. It also supports almost everything that would be
possible with the syntax-based fix. (Everything if the property is
actually writable, but I don't really see a use for that except perhaps
for deleting it).
I can even remember that I have wanted such a property before, although
I don't recall the exact use case.
One bad thing about it is that the initial yield made by the yield-from
is then the value that the coroutine decorator was meant to discard
(usually None). That might be a reason for allowing the property to be
writable, or for a change in syntax after all. On the other hand, if
this is a problem you can manually call the coroutine the way you want
before using it in yield-from, which would then initialize the value
exactly like with the second syntax idea.
Of the three ideas so far, I much prefer the one without extra syntax.
Nick Coghlan
This issue is still bouncing around in my brain, so I don't have a lot
say about it yet, but a special attribute on the generator-iterator
object that the yield from expression could check was the first possible
approach that occurred to me.
Although, rather than it being the "latest yield" from the generator, I
was thinking more of just an ordinary attribute that a @coroutine
decorator could set to indicate what to yield when firing it up with
'yield from'.
On your syntax ideas, note that the parser can't do anything tricky with
expressions of the form "yield EXPR" - the parser will treat that as a
normal yield and get confused if you try to add anything after it.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Jacob Holm
> This issue is still bouncing around in my brain, so I don't have a lot
> say about it yet, but a special attribute on the generator-iterator
> object that the yield from expression could check was the first possible
> approach that occurred to me.
>
Ok, keep it bouncing.
> Although, rather than it being the "latest yield" from the generator, I
> was thinking more of just an ordinary attribute that a @coroutine
> decorator could set to indicate what to yield when firing it up with
> 'yield from'.
>
I made it the latest yield because I have had a use case for that in the
past, and it seemed like a natural thing to do.
> On your syntax ideas, note that the parser can't do anything tricky with
> expressions of the form "yield EXPR" - the parser will treat that as a
> normal yield and get confused if you try to add anything after it.
>
I don't really like the idea of new syntax anyway, now that it seems
there is a way to avoid it. But thanks for the reminder.
- Jacob
Guido van Rossum
> Another possible fix would be to have new syntax for specifying that the
> initial call to the coroutine should be using send or throw instead. This
> could be seen as a restriction on what could be used as STARTEXPR(NAME) in
> the earlier syntax idea.
All, please stop making more proposals. I've got it all in my head but
no time to write it up right now. Hopefully before the weekend is over
I'll find the time.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum
On Fri, Apr 3, 2009 at 4:25 PM, Jacob Holm <j...@improva.dk> wrote:
> I am still trying to get a clear picture of what kind of mistakes you are
> trying to protect against.
> If it is people accidently writing return in a generator when they really
> mean yield, that is what I thought the proposal for an alternate syntax was
> for. That sounds like a good idea to me, especially if we could also ban or
> discourage the use of normal return. But the alternate syntax doesn't have
> to mean a different exception.
I want to name the new exception ReturnFromGenerator to minimize the similarity with GeneratorExit: if we had both GeneratorExit and GeneratorReturn there would be endless questions on the newbie forums about the differences between the two, and people might use the wrong one. Since ReturnFromGenerator goes *out* of the generator and GeneratorExit goes *in*, there really are no useful parallels, and similar names would cause confusion.
> I could easily see "return value" as a separate PEP, except PEP 380
> provides one of the better reasons for its inclusion. It might be good to
> figure out how this feature should work by itself before complicating things
> by integrating it in the yield-from semantics.
Normal loops (for-loops, list comprehensions, other implied loops) only catch StopIteration, so that returning a value is still wrong here. But some other contexts treat ReturnFromGenerator similar as StopIteration except the latter conveys None and the former conveys an explicit value. This applies to yield-from as well as to explicit or implied closing of the generator (close() or deallocation).
So g.close() returns the value (I think I earlier said I didn't like that -- I turned around on this one). It's pseudo-code is roughly:
def close(it):
try:
it.throw(GeneratorExit)
except (GeneratorExit, StopIteration):
return None
except ReturnFromGenerator as e: # This block is really the only new thing
return e.value
# Other exceptions are passed out unchanged
else:
# throw() yielded a value -- unchanged
raise RuntimeError(.....)
Deleting a generator is like closing and printing (!) a traceback (to stderr) if close() raises an exception. A returned value it is just ignored. Explicit pseudo-code without falling back to close():
def __del__(it):
try:
it.throw(GeneratorExit)
except (GeneratorExit, StopIteration, ReturnFromGenerator):
pass
except:
# Some other exception happened
<print traceback>
else:
# throw() yielded another value
<print traceback>
I have also worked out what I want yield-from to do, see end of this message.
[Guido]
>> Oh, and "yield from" competes with @couroutine over
>> when the initial next() call is made, which again suggests the two
>> styles (yield-from and coroutines) are incompatible.
>
> It is a serious problem, because one of the major points of the PEP is that
> it should be useful for refactoring coroutines. As a matter of fact, I
> started another thread on this specific issue earlier today which only Nick
> has so far responded to. I think it is solvable, but requires some more
> work.
-----
So now let me develop my full thoughts on yield-from. This is unfortunately long, because I want to show some intermediate stages. I am using a green font for new code. I am using stages, where each stage provides a better approximation of the desired semantics. Note that each stage *adds* some semantics for corner cases that weren't handled the same way in the previous stage. Each stage proposes an expansion for "RETVAL = yield from EXPR". I am using Py3k syntax.
1. Stage one uses the for-loop equivalence:
for x in EXPR:
yield x
RETVAL = None
2. Stage two expands the for-loop into an explicit while-loop that has the same meaning. It also sets RETVAL when breaking out of the loop. This prepares for the subsequent stages. Note that we have an explicit iter(EXPR) call here, since that is what a for-loop does:
it = iter(EXPR)
while True:
try:
x = next(it)
except StopIteration:
RETVAL = None; break
yield x
3. Stage three further rearranges stage 2 without making semantic changes, Again this prepares for later stages:
it = iter(EXPR)
try:
x = next(it)
except StopIteration:
RETVAL = e.value
else:
while True:
yield x
try:
x = next(x)
except StopIteration:
RETVAL = None; break
4. Stage four adds handling for ReturnFromGenerator, in both places where next() is called:
it = iter(EXPR)
try:
x = next(it)
except StopIteration:
RETVAL = e.value
except ReturnFromGenerator as e:
RETVAL = e.value; break
else:
while True:
yield x
try:
x = next(it)
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break
yield x
5. Stage five shows what should happen if "yield x" above returns a value: it is passed into the subgenerator using send(). I am ignoring for now what happens if it is not a generator; this will be cleared up later. Note that the initial next() call does not change into a send() call, because there is no value to send before before we have yielded:
it = iter(EXPR)
try:
x = next(it)
except StopIteration:
RETVAL = None
except ReturnFromGenerator as e:
RETVAL = e.value
else:
while True:
v = yield x
try:
x = it.send(v)
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break
6. Stage six adds more refined semantics for when "yield x" raises an exception: it is thrown into the generator, except if it is GeneratorExit, in which case we close() the generator and re-raise it (in this case the loop cannot continue so we do not set RETVAL):
it = iter(EXPR)
try:
x = next(it)
except StopIteration:
RETVAL = None
except ReturnFromGenerator as e:
RETVAL = e.value
else:
while True:
try:
v = yield x
except GeneratorExit:
it.close()
raise
except:
try:
x = it.throw(*sys.exc_info())
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break
else:
try:
x = it.send(v)
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break
7. In stage 7 we finally ask ourselves what should happen if it is not a generator (but some other iterator). The best answer seems subtle: send() should degenerator to next(), and all exceptions should simply be re-raised. We can conceptually specify this by simply re-using the for-loop expansion:
it = iter(EXPR)
if <it is not a generator>:
for x in it:
yield next(x)
RETVAL = None
else:
try:
x = next(it)
except StopIteration:
RETVAL = None
except ReturnFromGenerator as e:
RETVAL = e.value
else:
while True:
try:
v = yield x
except GeneratorExit:
it.close()
raise
except:
try:
x = it.throw(*sys.exc_info())
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break
else:
try:
x = it.send(v)
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break
Note: I don't mean that we literally should have a separate code path for non-generators. But writing it this way adds the generator test to one place in the spec, which helps understanding why I am choosing these semantics. The entire code of stage 6 degenerates to stage 1 if we make the following substitutions:
it.send(v) -> next(v)
it.throw(sys.exc_info()) -> raise
it.close() -> pass
(Except for some edge cases if the incoming exception is StopIteration or ReturnFromgenerator, so we'd have to do the test before entering the try/except block around the throw() or send() call.)
We could do this based on the presence or absence of the send/throw/close attributes: this would be duck typing. Or we could use isinstance(it, types.GeneratorType). I'm not sure there are strong arguments for either interpretation. The type check might be a little faster. We could even check for an exact type, since GeneratorType is final. Perhaps the most important consideration is that if EXPR produces a file stream object (which has a close() method), it would not consistently be closed: it would be closed if the outer generator was closed before reaching the end, but not if the loop was allowed to run until the end of the file. So I'm leaning towards only making the generator-specific method calls if it is really a generator.
Guido van Rossum
> 1) We have a coroutine that expects you to call send and/or throw with
> specific values, and ends up returning a value. A beginner may try to
> iterate over it, but will most likely get an exception on the first next()
> call because the input is not valid. Or he would get an infinite loop
> because None is not changing the state of the coroutine. In any case, it is
> unlikely that he will get to see either StopIteration or the new exception,
> because the input is not what the coroutine expects. The new exception
> doesn't help here.
>
> 2) We have a generator that e.g. pulls values from a file, yielding the
> processed values as it goes along, and returning some form of summary at the
> end. If I iterate over it with a for-loop, I get all the values asn usual
> ... followed by an exception. Why do I have to get an exception there just
> because the generator has some information that its implementer thought I
> might want? Ignoring the value in this case seems perfectly reasonable, so
> having to catch an exception is just noise here.
Sorry, I read this message after writing a long response to an earlier
message of you where I rejected the idea of new syntax for returning a
value from a generator. I find this example somewhat convincing, and
more so because the extra processing of ReturnFromGenerator makes my
proposal a bit messy: there are three "except StopIteration" clauses,
all with parallel "except ReturnFromGenerator" clauses.
Though the real implementation would probably merge all that code into
a single C-level function.
And new syntax *is* a much bigger burden than a new exception. I think
I need to ponder this for a while and think more about how important
it really is to hold the hand of newbies trying to write a vanilla
generator, vs. how important this use case really is (it's easily
solved with a class, for example).
> 3) We have a coroutine that computes something expensive, occationally
> yielding to let other code run. It neither sends or receives values, just
> uses yield for cooperative multitasking. When it is done it returns a
> value. If you loop over this coroutine, you will get a bunch of Nones,
> followed by the new exception. You could argue that the new exception helps
> you here. One way of accessing the returned value would be to catch it and
> look at an attribute. However, for this case I would prefer to just call
> close on the generator to get the value afterwards. A beginner might be
> helped by the unexpected exception, but I think even a beginner would find
> that something strange was going on when the only value he gets for the loop
> variable is None. He might even look up the documentation for the coroutine
> he was calling and see how it was supposed to be used.
Well if you have nothing else to do you could just use "yield from"
over this coroutine and get the return value through that syntax.
And if you're going to call next() on it with other activities in
between, you have to catch StopIteration from that next() call anyway
-- you would have to also catch ReturnFromGenerator to extract the
value.
I don't believe that once the generator has raised StopIteration or
ReturnFromGenerator, the return value should be saved somewhere to be
retrieved with an explicit close() call -- I want to be able to free
all resources once the generator frame is dead.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Greg Ewing
> RESULT = yield STARTEXPR from EXPR
> RESULT = yield from EXPR with STARTEXPR
> RESULT = yield from EXPR as NAME starting with STARTEXPR(NAME)
>
> And letting STARTEXPR if given take the place of the initial _i.next()
No, that's not satisfactory at all, because it introduces
a spurious value into the stream of yielded values seen
by the user of the outer generator.
For refactoring to work correctly, the first value yielded
by the yield-from expression *must* be the first value
yielded by the subgenerator. There's no way of achieving
that when using the Beazley decorator, because it thinks
the first yielded value is of no interest and discards it.
>> We have a long way to go before we even come close to consuming as many
>> pixels as PEP 308 or PEP 343 - a fact for which Greg is probably
>> grateful ;)
At least Guido will know if someone manages to unsubscribe
him from the list -- he won't be getting 500 messages about
yield-from every day. :-)
--
Greg
Nick Coghlan
> [Guido]
>>> Oh, and "yield from" competes with @couroutine over
>>> when the initial next() call is made, which again suggests the two
>>> styles (yield-from and coroutines) are incompatible.
>>
>> It is a serious problem, because one of the major points of the PEP is
> that
>> it should be useful for refactoring coroutines. As a matter of fact, I
>> started another thread on this specific issue earlier today which only
> Nick
>> has so far responded to. I think it is solvable, but requires some more
>> work.
>
> I think that's the thread where I asked you and Nick to stop making more
> proposals.I a worried that a solution would become too complex, and I
> want to keep the "naive" interpretation of "yield from EXPR" to be as
> close as possible to "for x in EXPR: yield x". I think the @coroutine
> generator (whether built-in or not) or explicit "priming" by a next()
> call is fine.
The trick is that if the definition of "yield from" *includes* the
priming step, then we are saying that coroutines *shouldn't* be primed
in a decorator. I don't actually have a problem with that, so long as we
realise that existing coroutines that are automatically primed when
created won't work unmodified with "yield from" (since they would get
primed twice - once by the wrapper function and once by the "yield from"
expression).
To be honest, I see that "auto-priming" behaviour as similar to merging
creation of threading.Thread instances with calling t.start() on them -
while it is sometimes convenient to do that, making it impossible to
separate the creation from the activation the way a @coroutine decorator
does actually seems like an undesirable thing to do.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Greg Ewing
> We could do this based on the presence or absence of the
> send/throw/close attributes: this would be duck typing. Or we could use
> isinstance(it, types.GeneratorType).
I don't like the idea of switching the entire behaviour
based on a blanket generator/non-generator distinction.
Failure to duck-type is unpythonic unless there's a very
good reason for it, and I don't see any strong reason here.
Why shouldn't an iterator be able to emulate a generator
by providing all the necessary methods?
On the other hand, if we're looking at the presence of
methods, what happens if e.g. it has a throw() method
but not a send() method? Do we treat it as though the
throw() method didn't exist just because it doesn't have
the full complement of generator methods? That doesn't
seem very pythonic either.
> if EXPR produces
> a file stream object (which has a close() method), it would not
> consistently be closed: it would be closed if the outer generator was
> closed before reaching the end, but not if the loop was allowed to run
> until the end of the file.
I don't think this is a serious problem, for the following
reasons:
1. We've already more or less decided that yield-from is not
going to address the case of shared subiterators, so if anything
else would care about the file being closed unexpectedly, you
shouldn't be using yield-from on it.
2. It's well known that you can't rely on automatic closing
of files in non-refcounting implementations, so code wanting
to ensure the file is closed will need to do so explicitly
using a finally clause or something equivalent, which will
get triggered by closing the outer generator.
--
Greg
Leif Walsh
time for anything else if I did, it seems), but I think I understand
the problem with priming a coroutine, and then trying to use it in
yield from, and I may have a solution. I don't understand what it
means to 'yield from' a coroutine, but I'll here's my proposed fix:
Give all generators/coroutines a 'prime' (or better named) function.
This prime function can set some 'is_primed' internal variable so that
it never primes more than once. Now, yield from and @coroutine can
(and this is the hazy part because I don't know what yield from is
really doing under the hood) both use prime(), so yielding from a
non-decorated coroutine will have the same effect as yielding from a
decorated coroutine.
--
Cheers,
Leif
Leif Walsh
> but I'll here's my proposed fix:
hoo-ray copyediting!
Greg Ewing
> how important this use case really is (it's easily
> solved with a class, for example).
Yes, it's my feeling that a class would be better for
this kind of thing too.
Let's not lose sight of the fundamental motivation
for all this, the way I see it at least: yield-from is
primarily to permit factoring of generator code. Any
proposals for enhancements or extensions ought to be
justified in relation to that.
> I don't believe that once the generator has raised StopIteration or
> ReturnFromGenerator, the return value should be saved somewhere to be
> retrieved with an explicit close() call -- I want to be able to free
> all resources once the generator frame is dead.
I agree with that.
As a corollary, I *don't* think that close() should
return the value of a ReturnFromGenerator even if it
gets one, because unless the value is stored, you'll
only get it the first time close() is called, and
only if the generator has not already completed
normally. That would make it too unreliable for any
practical use as far as I can see.
--
Greg
Terry Reedy
> So now let me develop my full thoughts on yield-from. This is
> unfortunately long, because I want to show some intermediate stages.
I found this extremely helpful. Whatever expansion you finally decide
on (pun intended ;-), an explanation like this in the PEP would be nice.
tjr
Jacob Holm
I like the way you are building the description up from the simple case, but I think you are missing a few details along the way.
Those details are what has been driving the discussion, so I think it is important to get them handled. I'll comment on each point as I get to it.
Guido van Rossum wrote:
I want to name the new exception ReturnFromGenerator to minimize the similarity with GeneratorExit [...]
Fine with me, assuming we can't get rid of it altogether.
[Snipped description of close() and __del__(), which I intend to comment on in the other thread]
[Guido]
>> Oh, and "yield from" competes with @couroutine over
>> when the initial next() call is made, which again suggests the two
>> styles (yield-from and coroutines) are incompatible.
>
> It is a serious problem, because one of the major points of the PEP is that
> it should be useful for refactoring coroutines. As a matter of fact, I
> started another thread on this specific issue earlier today which only Nick
> has so far responded to. I think it is solvable, but requires some more
> work.
I think that's the thread where I asked you and Nick to stop making more proposals.I a worried that a solution would become too complex, and I want to keep the "naive" interpretation of "yield from EXPR" to be as close as possible to "for x in EXPR: yield x". I think the @coroutine generator (whether built-in or not) or explicit "priming" by a next() call is fine.
So now let me develop my full thoughts on yield-from. This is unfortunately long, because I want to show some intermediate stages. I am using a green font for new code. I am using stages, where each stage provides a better approximation of the desired semantics. Note that each stage *adds* some semantics for corner cases that weren't handled the same way in the previous stage. Each stage proposes an expansion for "RETVAL = yield from EXPR". I am using Py3k syntax.
4. Stage four adds handling for ReturnFromGenerator, in both places where next() is called:
it = iter(EXPR)
try:
x = next(it)
except StopIteration:
RETVAL = e.value
except ReturnFromGenerator as e:
RETVAL = e.value; break
else:
while True:
yield x
try:
x = next(it)
except StopIteration:
RETVAL = None; break
except ReturnFromGenerator as e:
RETVAL = e.value; break
yield x
5. Stage five shows what should happen if "yield x" above returns a value: it is passed into the subgenerator using send(). I am ignoring for now what happens if it is not a generator; this will be cleared up later. Note that the initial next() call does not change into a send() call, because there is no value to send before before we have yielded:
The argument that we have no value to send before we have yielded is wrong. The generator containing the "yield-from" could easily have a value to send (or throw), and if iter(EXPR) returns a coroutine or a non-generator it could easily be ready to accept it. That is the idea behind my attempted fixes to the @coroutine issue.
6. Stage six adds more refined semantics for when "yield x" raises an exception: it is thrown into the generator, except if it is GeneratorExit, in which case we close() the generator and re-raise it (in this case the loop cannot continue so we do not set RETVAL):
This is where the fun begins. In an earlier thread we concluded that if the thrown exception is a StopIteration and the *same* StopIteration instance escapes the throw() call, it should be reraised rather than caught and turned into a RETVAL. The reasoning was the following example:
def inner():
for i in xrange(10):
yield i
def outer():
yield from inner()
print "if StopIteration is thrown in we shouldn't get here"
Which we wanted to be equivalent to:
def outer():
for i in xrange(10):
yield i
print "if StopIteration is thrown in we shouldn't get here"
The same argument goes for ReturnFromGenerator, so the expansion at this stage should be more like:
it = iter(EXPR) try: x = next(it) except StopIteration: RETVAL = None except ReturnFromGenerator as e: RETVAL = e.value else: while True: try: v = yield x
except GeneratorExit:
it.close()
raise
except BaseException as e:
try:
x = it.throw(e) # IIRC this includes the correct traceback in 3.x so we don't need to use sys.exc_info
except StopIteration as r:
if r is e:
raise
RETVAL = None; break
except ReturnFromGenerator as r:
if r is e:
raise
RETVAL = r.value; break
else:
try: x = it.send(v) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break
Next issue is that the value returned by it.close() is thrown away by yield-from. Here is a silly example:
def inner():
i = 0
while True
try:
yield
except GeneratorExit:
return i
i += 1
def outer():
try:
yield from inner()
except GeneratorExit:
# nothing I can write here will get me the value returned from inner()
Also the trivial:
def outer():
return yield from inner()
Would swallow the return value as well.
I have previously suggested attaching the return value to the (re)raised GeneratorExit, and/or saving the return value on the generator and making close return the value each time it is called. We could also choose to define this as broken behavior and raise a RuntimeError, although it seems a bit strange to have yield-from treat it as an error when close doesn't. Silently having the yield-from construct swallow the returned value is my least favored option.
x = it.close() -->
m = getattr(it, 'close', None)
if m is not None:
x = it.close()
else:
x = None
x = it.send(v) -->
if v is None:
x = next(it)
else:
try:
m = it.send
except AttributeError:
m = getattr(it, 'close', None)
if m is not None:
it.close() # in this case I think it is ok to ignore the return value
raise
else:
x = m(v)
x = throw(e) -->
m = getattr(it, 'throw', None)
if m is not None:
x = m()
else:
m = getattr(it, 'close', None)
if m is not None:
it.close() # in this case I think it is ok to ignore the return value
raise e
In this version it is easy enough to wrap the final iterator if you
want different behavior. With your version it becomes difficult to
replace a generator that is used in a yield-from with an iterator.
(You would have to wrap the iterator in a generator that mostly
consisted of the expansion from this PEP with the above substitution).I don't think we need to worry about performance at this stage. AFAICT from the patch I was working on, the cost of a few extra checks is negligible compared to the savings you get from using yield-from in the first place.
Best regards
- Jacob
Guido van Rossum
> The argument that we have no value to send before we have yielded is wrong.
> The generator containing the "yield-from" could easily have a value to send
> (or throw), and if iter(EXPR) returns a coroutine or a non-generator it
> could easily be ready to accept it. That is the idea behind my attempted
> fixes to the @coroutine issue.
I think it's simpler to refrain from yield-from in that case and spell
it out. If the value to send doesn't come from outside the outer
generator, yield-from is not the solution.
> This is where the fun begins. In an earlier thread we concluded that if the
> thrown exception is a StopIteration and the *same* StopIteration instance
> escapes the throw() call, it should be reraised rather than caught and
> turned into a RETVAL. The reasoning was the following example:
>
> def inner():
> for i in xrange(10):
> yield i
>
> def outer():
> yield from inner()
> print "if StopIteration is thrown in we shouldn't get here"
>
> Which we wanted to be equivalent to:
>
> def outer():
> for i in xrange(10):
> yield i
> print "if StopIteration is thrown in we shouldn't get here"
>
> The same argument goes for ReturnFromGenerator, so the expansion at this
> stage should be more like:
[snip]
This example and reasoning are invalid. You shouldn't be throwing
StopIteration (or ReturnFromGenerator) *into* a generator. That's
something that should only come *out*.
> Next issue is that the value returned by it.close() is thrown away by
> yield-from. Here is a silly example:
>
> def inner():
> i = 0
> while True
> try:
> yield
> except GeneratorExit:
> return i
> i += 1
>
> def outer():
> try:
> yield from inner()
> except GeneratorExit:
> # nothing I can write here will get me the value returned from inner()
>
> Also the trivial:
>
> def outer():
> return yield from inner()
>
> Would swallow the return value as well.
>
> I have previously suggested attaching the return value to the (re)raised
> GeneratorExit, and/or saving the return value on the generator and making
> close return the value each time it is called. We could also choose to
> define this as broken behavior and raise a RuntimeError, although it seems a
> bit strange to have yield-from treat it as an error when close doesn't.
> Silently having the yield-from construct swallow the returned value is my
> least favored option.
Attaching it to the GeneratorExit is just plain wrong -- this is an
exception you throw *in*, not something that is thrown out (except
when it bounces back).
One solution is not to use yield-from but write it out using yield and
send (just like the full expansion, but you can probably drop most of
the complexity for any particular example).
Another solution is not to use close() and GeneratorExit but some
application-specific exception to signal the end.
But perhaps it would be okay to change the GeneratorExit handler in
the expansion so that it passes through the return value with a
StopIteration exception:
rv = it.close()
if rv is None:
raise StopIteration(rv) # Or ReturnFromGenerator(rv)
else:
raise
Alternatively, simpler:
it.throw(GeneratorExit)
# We only get here if it yielded a value
raise RuntimeError(...)
(Though this isn't exactly if we were to use duck typing.)
We could then write the first version of outer() like this:
def outer():
try:
yield from inner()
except StopIteration as e:
...access return value as e.value...
and I think the second (trivial) outer() will return inner()'s return
value just fine, since it just passes through as a StopIteration
value.
> Like Greg, I am in favor of duck-typing this as closely as possible.
OK, noted. I think it's probably fine.
FWIW, I'm beginning to think that ReturnFromGenerator is a bit of a
nuisance, and that it's actually fine to allow "return value" inside a
generator to mean "raise StopIteration(value)" (well not quite at that
point in the code but once we are about to clean up the frame). Maybe
I've overstated the case for preventing beginners' mistakes. After all
they'll notice that their generator returns prematurely when they
include any kind of return value. Also if the StopIteration ends being
printed as a traceback the value will be printed, which is the kind of
hint newbies love.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Jacob Holm
> Guido van Rossum wrote:
>> I don't believe that once the generator has raised StopIteration or
>> ReturnFromGenerator, the return value should be saved somewhere to be
>> retrieved with an explicit close() call -- I want to be able to free
>> all resources once the generator frame is dead.
>
> I agree with that.
I don't think it is common to keep the generator object alive long after
the generator is closed, so I don't see the problem in keeping the value
so it can be returned by the next close() call.
>
> As a corollary, I *don't* think that close() should
> return the value of a ReturnFromGenerator even if it
> gets one, because unless the value is stored, you'll
> only get it the first time close() is called, and
> only if the generator has not already completed
> normally. That would make it too unreliable for any
> practical use as far as I can see.
>
And I think it is a mistake to have close() swallow the return value. If
it catches ReturnFromGenerator, it should also return the value or raise
a RuntimeError.
In my order of preference:
1. "return value" in a generator raises StopIteration(value). Any
exception raised by next, send or throw sets a return value on the
generator on the way out. If the exception is a StopIteration or
GeneratorExit (see below) that was not the argument to throw, the
value is taken from there, else it is None. Any next(), send(), or
throw() operation on a closed generator raises a new StopIteration
using the saved value. When close catches a StopIteration or
GeneratorExit it returns the value. After yield-from calls close
as part of its GeneratorExit handling, it raises a new
GeneratorExit with the returned value.
The GeneratorExit part lets "def outer(): return yield from
inner()" behave as expected.
2. Same as #1 but using ReturnFromGenerator(value) instead of
StopIteration.
3. Same as #1 but without attaching return value to GeneratorExit in
yield-from.
4. Same as #3 but using ReturnFromGenerator(value) instead of
StopIteration.
5. Same as #1 but without storing the value on the generator.
6. Same as #5 but using ReturnFromGenerator(value) instead of
StopIteration.
7. "return value" in a generator raises ReturnFromGenerator(value).
close() doesn't catch it.
8. "return value" in a generator raises ReturnFromGenerator(value).
close() catches ReturnFromGenerator and raises a RuntimeError.
9. Anything else that has been suggested. In particular anything
where close() catches ReturnFromGenerator without either returning
the value or raising another exception.
Too many options? This is just a small subset of what has been discussed.
- Jacob
Guido van Rossum
> Let's not lose sight of the fundamental motivation
> for all this, the way I see it at least: yield-from is
> primarily to permit factoring of generator code. Any
> proposals for enhancements or extensions ought to be
> justified in relation to that.
I still don't think that refactoring should drive the design
exclusively. Refactoring is *one* thing that becomes easier with
yield-from. But I want the design to look pretty from as many angles
as possible.
>> I don't believe that once the generator has raised StopIteration or
>> ReturnFromGenerator, the return value should be saved somewhere to be
>> retrieved with an explicit close() call -- I want to be able to free
>> all resources once the generator frame is dead.
>
> I agree with that.
>
> As a corollary, I *don't* think that close() should
> return the value of a ReturnFromGenerator even if it
> gets one, because unless the value is stored, you'll
> only get it the first time close() is called, and
> only if the generator has not already completed
> normally. That would make it too unreliable for any
> practical use as far as I can see.
Throwing in GeneratorExit and catching the ReturnFromGenerator
exception would have the same problem though, so I'm not sure I buy
this argument.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Jacob Holm
> On Sun, Apr 5, 2009 at 7:46 AM, Jacob Holm <j...@improva.dk> wrote:
>
>> The argument that we have no value to send before we have yielded is wrong.
>> The generator containing the "yield-from" could easily have a value to send
>> (or throw), and if iter(EXPR) returns a coroutine or a non-generator it
>> could easily be ready to accept it. That is the idea behind my attempted
>> fixes to the @coroutine issue.
>>
>
> I think it's simpler to refrain from yield-from in that case and spell
> it out. If the value to send doesn't come from outside the outer
> generator, yield-from is not the solution.
>
But it *could* come from outside. If it is a coroutine calling another
coroutine, it could have done any number of yields first, the last of
which would return the value to be sent to the inner one.
It bothers me a lot if you cannot use yield-from with coroutines,
because most other uses I can see are just as easily written as
for-loops. I'll think a bit more about this.
>> This is where the fun begins. In an earlier thread we concluded that if the
>> thrown exception is a StopIteration and the *same* StopIteration instance
>> escapes the throw() call, it should be reraised rather than caught and
>> turned into a RETVAL. The reasoning was the following example:
>>
>> def inner():
>> for i in xrange(10):
>> yield i
>>
>> def outer():
>> yield from inner()
>> print "if StopIteration is thrown in we shouldn't get here"
>>
>> Which we wanted to be equivalent to:
>>
>> def outer():
>> for i in xrange(10):
>> yield i
>> print "if StopIteration is thrown in we shouldn't get here"
>>
>> The same argument goes for ReturnFromGenerator, so the expansion at this
>> stage should be more like:
>>
> [snip]
>
> This example and reasoning are invalid. You shouldn't be throwing
> StopIteration (or ReturnFromGenerator) *into* a generator. That's
> something that should only come *out*.
>
I am not claiming that you *should* be throwing StopIteration to a
generator, just that there is nothing that prevents you from doing it,
so we need to consider what should happen if you do. The above
reasoning based on the refactoring principle lead to one choice, which I
happen to like. If you only focus on getting the expansion in the PEP
as simple as possible you will probably make another choice.
Note that if you don't handle StopIteration this way but just treat it
as a normal StopIteration you open up for interesting ways to abuse
yield-from, exactly by throwing StopIteration. In particular, if you
use an iterator without throw or close methods you can break out of the
innermost yield-from and even set the value to be returned.
I don't mind either way. I just thought I would mention this in case
you missed it.
[snip my examples where the return value was swallowed]
>> I have previously suggested attaching the return value to the (re)raised
>> GeneratorExit, and/or saving the return value on the generator and making
>> close return the value each time it is called. We could also choose to
>> define this as broken behavior and raise a RuntimeError, although it seems a
>> bit strange to have yield-from treat it as an error when close doesn't.
>> Silently having the yield-from construct swallow the returned value is my
>> least favored option.
>>
>
> Attaching it to the GeneratorExit is just plain wrong -- this is an
> exception you throw *in*, not something that is thrown out (except
> when it bounces back).
>
I expanded a little bit on the idea in my reply to Greg that must have
crossed your mail. Listed a number of possible solutions that had been
discussed in my order of preference. I don't see a problem in having
the language construct "yield-from" raise GeneratorExit with a value as
a result of GeneratorExit.
> One solution is not to use yield-from but write it out using yield and
> send (just like the full expansion, but you can probably drop most of
> the complexity for any particular example).
>
> Another solution is not to use close() and GeneratorExit but some
> application-specific exception to signal the end.
>
That doesn't really solve the issue of what should happen if you write
such code. What bothers me most is that the return value is silently
swallowed.
> But perhaps it would be okay to change the GeneratorExit handler in
> the expansion so that it passes through the return value with a
> StopIteration exception:
>
> rv = it.close()
> if rv is None:
> raise StopIteration(rv) # Or ReturnFromGenerator(rv)
> else:
> raise
>
> Alternatively, simpler:
>
> it.throw(GeneratorExit)
> # We only get here if it yielded a value
> raise RuntimeError(...)
>
> (Though this isn't exactly if we were to use duck typing.)
>
> We could then write the first version of outer() like this:
>
> def outer():
> try:
> yield from inner()
> except StopIteration as e:
> ...access return value as e.value...
>
> and I think the second (trivial) outer() will return inner()'s return
> value just fine, since it just passes through as a StopIteration
> value.
>
I don't mind catching an exception to get the value in this case. I
just think it should be GeneratorExit i should catch.
This is related to the question of what should happen if you throw
StopIteration. If you don't special-case StopIteration in throw, using
StopIteration for this is fine.
> FWIW, I'm beginning to think that ReturnFromGenerator is a bit of a
> nuisance, and that it's actually fine to allow "return value" inside a
> generator to mean "raise StopIteration(value)" (well not quite at that
> point in the code but once we are about to clean up the frame). Maybe
> I've overstated the case for preventing beginners' mistakes. After all
> they'll notice that their generator returns prematurely when they
> include any kind of return value. Also if the StopIteration ends being
> printed as a traceback the value will be printed, which is the kind of
> hint newbies love.
>
Ok, then I will stop worrying about ReturnFromGenerator until it is
brought up again.
Best regards
- Jacob
Guido van Rossum
whoever else wants to join) find agreement on a next version of the
PEP.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Nick Coghlan
> Greg Ewing wrote:
>> Guido van Rossum wrote:
>>> I don't believe that once the generator has raised StopIteration or
>>> ReturnFromGenerator, the return value should be saved somewhere to be
>>> retrieved with an explicit close() call -- I want to be able to free
>>> all resources once the generator frame is dead.
>>
>> I agree with that.
>
> I don't think it is common to keep the generator object alive long after
> the generator is closed, so I don't see the problem in keeping the value
> so it can be returned by the next close() call.
I don't think close() means to me what it means to you... close() to me
means "I'm done with this, it should have been exhausted already, but
just to make sure all the resources held by the internal frame are
released properly, I'm shutting it down explicitly"
In other words, the *normal* flow for close() should be the "frame has
already terminated, so just return immediately" path, not the "frame
hasn't terminated yet, so throw GeneratorExit in and complain if the
frame doesn't terminate" path.
You're trying to move throwing GeneratorExit into the internal frame
from the exceptional path to the normal path and I don't think that is a
good idea. Far better to attach the return value to StopIteration as in
Greg's original proposal (since Guido no longer appears to be advocating
a separate exception for returning a value, we should be able to just go
back to Greg's original approach) and use a normal next(), send() or
throw() call along with a helper function to catch the StopIteration.
Heck, with that approach, you can even write a context manager to catch
the result for you:
@contextmanager
class cr_result(object):
def __init__(self, cr):
self.coroutine = cr
self.result = None
def __enter__(self):
return self
def __exit__(self, et, ev, tb):
if et is StopIteration:
self.result = ev.value
return True # Trap StopIteration
# Anything else is propagated
with cr_result(a) as x:
# Call a.next()/a.send()/a.throw() as you like
# Use x.result to retrieve the coroutine's value
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Jim Jewett
> Jim Jewett wrote:
>> err... I didn't mean both directions, I meant "from the callee to the
>> caller as a yielded value" and "from the callee to the caller as a
>> final return value that can't be yielded normally."
> I think perhaps we're misunderstanding each other. You
> seemed to be saying that the only time you would want
> a generator to return a value is when you were also
> using it to either send or receive values using yield,
Almost exactly the opposite. I can see cases where you want
the interim yields, and I can see cases where you want the final
result -- but I'm asking how common it is to need *both*, and
whether we should really be going to such effort for it.
If you only need one or the other, I think the PEP can be greatly simplified.
-jJ
Nick Coghlan
> Almost exactly the opposite. I can see cases where you want
> the interim yields, and I can see cases where you want the final
> result -- but I'm asking how common it is to need *both*, and
> whether we should really be going to such effort for it.
>
> If you only need one or the other, I think the PEP can be greatly simplified.
One of the things you may want to use coroutines with is a trampoline
scheduler for handling asynchronous IO. In that case, the inner
coroutine may want to yield an IO wait object so the scheduler can add
the relevant descriptor to the main select() loop.
In such a case, the interim yielded values would go back to the
scheduler, while the final result would go back to the calling coroutine.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Greg Ewing
> The argument that we have no value to send before we have yielded is
> wrong. The generator containing the "yield-from" could easily have a
> value to send (or throw)
No, Guido is right here. You *can't* send a value (other
than None) into a generator that hasn't reached its first
yield (try it and you'll get an exception). The first
call has to be next().
> and if iter(EXPR) returns a coroutine or a
> non-generator it could easily be ready to accept it.
If it's ready to accept a send, it must have already
yielded a value, which has been lost, when it should have
been yielded to the caller of the delegating generator.
> In an earlier thread we concluded that if
> the thrown exception is a StopIteration and the *same* StopIteration
> instance escapes the throw() call, it should be reraised rather than
> caught and turned into a RETVAL.
That part is right.
> Next issue is that the value returned by it.close() is thrown away by
> yield-from.
Since I don't believe that close() should be expected to
return a useful value anyway, that's not a problem.
--
Greg
Greg Ewing
> This example and reasoning are invalid. You shouldn't be throwing
> StopIteration (or ReturnFromGenerator) *into* a generator. That's
> something that should only come *out*.
Okay, if you're happy to take that views, then so
am I. It'll make my expansion simpler.
--
Greg
Greg Ewing
> I'm all of round tuits for a while, so I recommend that you all (and
> whoever else wants to join) find agreement on a next version of the
> PEP.
Just one thing before you go -- did you reach a
decision on whether you want a ReturnFromGenerator
exception?
--
Greg
Greg Ewing
> That doesn't really solve the issue of what should happen if you write
> such code.
I think the point is that if it's something you
shouldn't be doing in the first place, it doesn't
really matter what happens if you do.
--
Greg
Greg Ewing
> Throwing in GeneratorExit and catching the ReturnFromGenerator
> exception would have the same problem though, so I'm not sure I buy
> this argument.
I'm not advocating doing that. My view is that both
calling close() on the generator and throwing
GeneratorExit into it are things you only do to
make sure the generator cleans up. You can't
expect to get a meaningful return value either
way.
--
Greg
Greg Ewing
> I can see cases where you want
> the interim yields, and I can see cases where you want the final
> result -- but I'm asking how common it is to need *both*, and
> whether we should really be going to such effort for it.
>
> If you only need one or the other, I think the PEP can be greatly simplified.
How, exactly? I'd need convincing that you wouldn't
just end up with the same amount of complexity
arranged differently.
Greg Ewing
> I still don't think that refactoring should drive the design
> exclusively. Refactoring is *one* thing that becomes easier with
> yield-from. But I want the design to look pretty from as many angles
> as possible.
Certainly. But I feel that any extra features should
be in some sense extensions or generalizations of
what is needed to support refactoring. Otherwise the
scope of the proposal can expand without bound.
--
Greg
Jacob Holm
> Jacob Holm wrote:
>
>> I don't think it is common to keep the generator object alive long after
>> the generator is closed, so I don't see the problem in keeping the value
>> so it can be returned by the next close() call.
>>
>
> I don't think close() means to me what it means to you... close() to me
> means "I'm done with this, it should have been exhausted already, but
> just to make sure all the resources held by the internal frame are
> released properly, I'm shutting it down explicitly"
>
> In other words, the *normal* flow for close() should be the "frame has
> already terminated, so just return immediately" path, not the "frame
> hasn't terminated yet, so throw GeneratorExit in and complain if the
> frame doesn't terminate" path.
>
That is why #1-6 in my list took care to extract the value from
StopIteration and attach it to the generator. Doing it like that allows
you to ask for the value after the generator is exhausted normally,
using either next() or close(). This is interesting because it allows
you to loop over the generator with a normal for-loop and *still* get
the return value after the loop if you want it. (You have to construct
the generator before the loop instead of in the for-loop statement
itself, and call close() on it afterwards, but that is easy). It also
makes it possible for close to reliably return the value.
The idea of saving the value on the generator is more basic than the
idea of having close return a value. It means that calling next on an
exhausted generator will keep raising StopIteration with the same
value. If you don't save the return value on the generator, only the
first StopIteration will have a value, the rest will always have None as
their value.
> You're trying to move throwing GeneratorExit into the internal frame
> from the exceptional path to the normal path and I don't think that is a
> good idea.
I think it is exacltly the right thing for the use cases I have.
Anything else requires extra support code to get a similar api. (Extra
exceptions to throw in and/or out, an alternative close function to
catch the extra exceptions, probably other things as well).
Whether or not it is a good idea to use GeneratorExit for this, I think
it is important that a "return value from GeneratorExit" does not
silently throw away the value. In other words, if close does *not*
return the value it gets from StopIteration, it should raise an
exception if that value is not None.
One option is to let close() reraise the StopIteration if it has a
non-None value. This matches Guidos suggestion for a way to access the
return value after a GeneratorExit in yield-from without changing his
suggested expansion. If the return value from the generator isn't
stored and I can't have close() return the value, this would be my
preference.
Another option (if you insist that it is an error to return a value
after a GeneratorExit) is to let close() raise a RuntimeError when it
catches a StopIteration with a non-None value.
- Jacob
Jacob Holm
> Jacob Holm wrote:
>
>> The argument that we have no value to send before we have yielded is
>> wrong. The generator containing the "yield-from" could easily have a
>> value to send (or throw)
>
> No, Guido is right here. You *can't* send a value (other
> than None) into a generator that hasn't reached its first
> yield (try it and you'll get an exception). The first
> call has to be next().
The whole idea of the coroutine pattern is to replace this restriction
on the caller with a restriction about the first yield in the coroutine.
This would probably be a lot clearer if the coroutine decorator was
written as:
def coroutine(func):
def start(*args,**kwargs):
cr = func(*args,**kwargs)
v = cr.next()
if v is not None:
raise RuntimeError('first yield in coroutine was not None')
return cr
return start
The first call *from user code* to a generator decorated with @coroutine
*can* be a send() or throw(), and in most cases probably should be.
>> and if iter(EXPR) returns a coroutine or a non-generator it could
>> easily be ready to accept it.
>
> If it's ready to accept a send, it must have already
> yielded a value, which has been lost, when it should have
> been yielded to the caller of the delegating generator.
No, in the coroutine pattern it absolutely should not. The first value
yielded by the generator of every coroutine is None and should be thrown
away.
>> Next issue is that the value returned by it.close() is thrown away by
>> yield-from.
>
> Since I don't believe that close() should be expected to
> return a useful value anyway, that's not a problem.
>
It is a problem in the sense that it is surprising behavior. If close()
doesn't return the value, it should at least raise some exception. I am
warming to the idea of reraising the StopIteration if it has a non-None
value. This matches Guidos suggestion for how to retrieve the value
after a yield-from that was thrown a GeneratorExit. If you insist it is
an error to return a value as response to GeneratorExit, raise
RuntimeError. But *please* don't just swallow the value.
- Jacob
Nick Coghlan
> Another option (if you insist that it is an error to return a value
> after a GeneratorExit) is to let close() raise a RuntimeError when it
> catches a StopIteration with a non-None value.
Why do you consider it OK for close() to throw away all of the values
the generator might have yielded in the future, but not OK for it to
throw away the generator's return value? The objection I have to having
close() return a value is that it encourages people to start using
GeneratorExit in their normal generator control flow and I think that's
a really bad idea (on par with calling sys.exit() and then trapping
SystemExit to terminate a search loop - perfectly legal from a language
point of view, but a really bad plan nonetheless).
Now, the fact that repeatedly calling next()/send()/throw() on a
finished generator is meant to keep reraising the same StopIteration
that was thrown when the generator first terminated is a *much* better
justification for preserving the return value on the generator object.
But coupling that with the idea of close() doing anything more than
giving an unfinished generator a final chance to release any resources
it is holding is mixing two completely different ideas.
Better to just add a "value" property to generators that raises a
RuntimeError if the generator frame hasn't terminated yet (probably
along with a "finished" property to allow LBYL interrogation of the
generator state).
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Guido van Rossum
> Guido van Rossum wrote:
>
>> I'm all of round tuits for a while, so I recommend that you all (and
>> whoever else wants to join) find agreement on a next version of the
>> PEP.
>
> Just one thing before you go -- did you reach a
> decision on whether you want a ReturnFromGenerator
> exception?
Let's do without it.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Nick Coghlan
> If you insist it is
> an error to return a value as response to GeneratorExit, raise
> RuntimeError. But *please* don't just swallow the value.
As I asked in the other thread (but buried in a longer message): why do
you see it as OK for close() to throw away every later value a generator
may have yielded, but not OK for it to throw away the return value?
close() is for finalisation so that generators can have a __del__ method
and hence we can allow yield inside try-finally. That's it. Don't break
that by trying to turn close() into something it isn't.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Jacob Holm
> Jacob Holm wrote:
>
>> Another option (if you insist that it is an error to return a value
>> after a GeneratorExit) is to let close() raise a RuntimeError when it
>> catches a StopIteration with a non-None value.
>>
>
> Why do you consider it OK for close() to throw away all of the values
> the generator might have yielded in the future, but not OK for it to
> throw away the generator's return value?
Because the return value is actually computed and returned from the
generator, *then* thrown away. If there is no way to access the value,
it should be considered an error to return it and flagged as such. What
the generator might have yielded if close wasn't called doesn't interest
me the slightest.
> The objection I have to having
> close() return a value is that it encourages people to start using
> GeneratorExit in their normal generator control flow and I think that's
> a really bad idea (on par with calling sys.exit() and then trapping
> SystemExit to terminate a search loop - perfectly legal from a language
> point of view, but a really bad plan nonetheless).
>
Yes, I understand that this is how you think of GeneratorExit.
> Now, the fact that repeatedly calling next()/send()/throw() on a
> finished generator is meant to keep reraising the same StopIteration
> that was thrown when the generator first terminated is a *much* better
> justification for preserving the return value on the generator object.
>
Ok
> But coupling that with the idea of close() doing anything more than
> giving an unfinished generator a final chance to release any resources
> it is holding is mixing two completely different ideas.
>
> Better to just add a "value" property to generators that raises a
> RuntimeError if the generator frame hasn't terminated yet (probably
> along with a "finished" property to allow LBYL interrogation of the
> generator state).
>
Why not a single property raising AttributeError until the frame is
terminated? (Not that I really care as long as I can access the value
without having access to the original StopIteration).
If the value is stored, I am fine with close not returning it or raising
an exception.
- Jacob
Greg Ewing
> No, in the coroutine pattern it absolutely should not. The first value
> yielded by the generator of every coroutine is None and should be thrown
> away.
That only applies to the *top level* of a coroutine. If
you factor some code out of a coroutine and call it using
yield-from, the first value yielded by the factored-out
code is needed and mustn't be thrown away.
So I stand by what I said before. If you're using such
a decorator, you only apply it to the top level generator
of the coroutine, and you don't call the top level
using yield-from.
--
Greg
Jacob Holm
> Jacob Holm wrote:
>
>> No, in the coroutine pattern it absolutely should not. The first
>> value yielded by the generator of every coroutine is None and should
>> be thrown away.
>
> That only applies to the *top level* of a coroutine. If
> you factor some code out of a coroutine and call it using
> yield-from, the first value yielded by the factored-out
> code is needed and mustn't be thrown away.
One major reason for factoring something out of a coroutine would be if
the factored-out code was independently useful as a coroutine. But I
cannot actually *make* it a coroutine if I want to call it using
yield-from because it is not always at the "top level".
>
> So I stand by what I said before. If you're using such
> a decorator, you only apply it to the top level generator
> of the coroutine, and you don't call the top level
> using yield-from.
>
So one @coroutine can't call another using yield-from. Why shouldn't it
be possible? All we need is a way to avoid the first next() call and
substitute some other value.
Here is a silly example of two coroutines calling each other using one
of the syntax-based ideas I have for handling this. (I don't care about
the actual syntax, just about the ability to do this)
@coroutine
def avg2():
a = yield
b = yield
return (a+b)/2
@coroutine
def avg_diff():
a = yield from avg2() start None # "start EXPR" means use EXPR for first value to yield instead of next()
b = yield from avg2() start a # "start EXPR" means use EXPR for first value to yield instead of next()
yield b
return a-b
a = avg2()
a.send(41)
a.send(43) # raises StopIteration(42)
d = avg_diff()
d.send(1.0)
d.send(2.0) # returns from first yield-from, yields 1.5 as part of starting second yield-from
d.send(3.0)
d.send(4.0) # returns from second yield-from. yields 3.5
d.next() # returns from avg_diff. raises StopIteration(-2.0)
The important things to note here are that both avg2 and avg_diff are
independently useful coroutines (yeah ok, not that useful), and that the
"natural" value to yield from the "d.send(2.0)" line does not come from
calling next() on the subgenerator, but rather from the outer generator.
I don't think there is any way to achieve this without some way of
substituting the initial next() call in yield-from.
Regards
- Jacob
Greg Ewing
> One major reason for factoring something out of a coroutine would be if
> the factored-out code was independently useful as a coroutine.
So provide another entry point for using that part
as a top level, or manually apply the wrapper when
it's appropriate.
I find it extroardinary that people seem to have
latched onto David Beazley's idiosyncratic definition
of a "coroutine" and decided that it's written on
a stone tablet from Mt. Sinai that we must always
wrap them in his decorator.
> @coroutine
> def avg_diff():
> a = yield from avg2() start None # "start EXPR" means use EXPR for
> first value to yield instead of next()
> b = yield from avg2() start a # "start EXPR" means use EXPR for
> first value to yield instead of next()
I'm going to need a less abstract example to see
why you might want to do something like that.
--
Greg
Jacob Holm
> Jacob Holm wrote:
>
>> One major reason for factoring something out of a coroutine would be
>> if the factored-out code was independently useful as a coroutine.
>
> So provide another entry point for using that part
> as a top level, or manually apply the wrapper when
> it's appropriate.
It is inconvenient to have to keep separate versions around, and it
doesn't solve the problem. See example below.
>
> I find it extroardinary that people seem to have
> latched onto David Beazley's idiosyncratic definition
> of a "coroutine" and decided that it's written on
> a stone tablet from Mt. Sinai that we must always
> wrap them in his decorator.
I find it extraordinary that my critical perspective on this issue makes
you think I am taking his tutorial as dogma. There is a real issue here
IMNSHO, and the @coroutine examples just happens to be the easiest way I
can see of explaining it.
>
>> @coroutine
>> def avg_diff():
>> a = yield from avg2() start None # "start EXPR" means use EXPR
>> for first value to yield instead of next()
>> b = yield from avg2() start a # "start EXPR" means use EXPR
>> for first value to yield instead of next()
>
> I'm going to need a less abstract example to see
> why you might want to do something like that.
>
Ok, below you will find a modified version of your parser example taken
from
http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt
The modification consists of applying the @coroutine decorator to
parse_items and parse_elem and changing them to yield a stream of
2-tuples describing how each token sent to the coroutine was interpreted.
The expected output is:
Feeding: '<foo>'
Yielding: ('Open', 'foo')
Feeding: 'This'
Yielding: ('Data', 'This')
Feeding: 'is'
Yielding: ('Data', 'is')
Feeding: 'a'
Yielding: ('Data', 'a')
Feeding: '<b>'
Yielding: ('Open', 'b')
Feeding: 'foo'
Yielding: ('Data', 'foo')
Feeding: 'file'
Yielding: ('Data', 'file')
Feeding: '</b>'
Yielding: ('Close', 'b')
Feeding: 'you'
Yielding: ('Data', 'you')
Feeding: 'know.'
Yielding: ('Data', 'know.')
Feeding: '</foo>'
Yielding: ('Close', 'foo')
[('foo', ['This', 'is', 'a', ('b', ['foo', 'file']), 'you', 'know.'])]
I can't see a nice way to get the same sequence of send() calls to yield
the same values without the ability to override the way the value for
the initial yield in yield-from is computed. Even avoiding the use of
@coroutine, I will still need to pass extra arguments to the generator
functions to control the initial yield.
- Jacob
------------------------------------------------------------------------
# Support code from example at http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt
import re
pat = re.compile(r"(\S+)|(<[^>]*>)")
def scanner(text):
for m in pat.finditer(text):
token = m.group(0)
print "Feeding:", repr(token)
yield token
yield None # to signal EOF
text = "<foo> This is a <b> foo file </b> you know. </foo>"
token_stream = scanner(text)
def is_opening_tag(token):
return token.startswith("<") and not token.startswith("</")
# Coroutine decorator copied from earlier mail, based on the one in http://dabeaz.com/coroutines/
def coroutine(func):
def start(*args, **kwargs):
cr = func(*args, **kwargs)
v = cr.next()
if v is not None:
raise RuntimeError('first yield from coroutine was not None')
return cr
return start
# Runner modified from example at http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt
# to also print the yielded values.
def run():
parser = parse_items()
# The original forgot to call next() here. That is not necessary in this version since parse_items uses
# the @coroutine decorator.
try:
for m in pat.finditer(text):
token = m.group(0)
print "Feeding:", repr(token)
v = parser.send(token)
print "Yielded:", v
parser.send(None) # to signal EOF
except StopIteration, e:
tree = e.args[0]
print tree
# parse_elem modified from example at http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt
# to make it a coroutine and to yield a sequence of 2-tuples describing how the recieved data was interpreted.
# (Does not yield the final ('Close', <tagname>) because it returns the tree instead).
@coroutine
def parse_elem():
opening_tag = yield
name = opening_tag[1:-1]
closing_tag = "</%s>" % name
items = yield from parse_items(closing_tag) start ('Open', name)
return (name, items)
# parse_items modified from example at http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt
# to make it a coroutine and to yield a sequence of 2-tuples describing how the recieved data was interpreted.
@coroutine
def parse_items(closing_tag = None):
elems = []
token = yield
while token != closing_tag:
if is_opening_tag(token):
subtree = yield from parse_elem() as p start p.send(token)
elems.append(subtree)
out = ('Close', subtree[0])
else:
elems.append(token)
out = ('Data', token)
token = yield out
return elems
Nick Coghlan
> So one @coroutine can't call another using yield-from. Why shouldn't it
> be possible? All we need is a way to avoid the first next() call and
> substitute some other value.
>
> Here is a silly example of two coroutines calling each other using one
> of the syntax-based ideas I have for handling this. (I don't care about
> the actual syntax, just about the ability to do this)
>
> @coroutine
> def avg2():
> a = yield
> b = yield
> return (a+b)/2
>
> @coroutine
> def avg_diff():
> a = yield from avg2() start None # "start EXPR" means use EXPR for
> first value to yield instead of next()
> b = yield from avg2() start a # "start EXPR" means use EXPR for
> first value to yield instead of next()
> yield b
> return a-b
You can fix this without syntax by changing the way avg2 is written.
@coroutine
def avg2(start=None):
a = yield start
b = yield
return (a+b)/2
@coroutine
def avg_diff(start=None):
a = yield from avg2(start)
b = yield from avg2(a)
yield b
return a-b
a = avg2()
a.send(41)
a.send(43) # raises StopIteration(42)
d = avg_diff()
d.send(1.0)
d.send(2.0) # returns from first yield-from, yields 1.5 as part of
starting second yield-from
d.send(3.0)
d.send(4.0) # returns from second yield-from. yields 3.5
d.next() # returns from avg_diff. raises StopIteration(-2.0)
So it just becomes a new rule of thumb for coroutines: a yield-from
friendly coroutine will accept a "start" argument that defaults to None
and is returned from the first yield call.
And just like threading.Thread, it will leave the idea of defining the
coroutine and starting the coroutine as separate activities.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Jacob Holm
> Jacob Holm wrote:
>
>> So one @coroutine can't call another using yield-from. Why shouldn't it
>> be possible? All we need is a way to avoid the first next() call and
>> substitute some other value.
>>
> You can fix this without syntax by changing the way avg2 is written.
>
>
> So it just becomes a new rule of thumb for coroutines: a yield-from
> friendly coroutine will accept a "start" argument that defaults to None
> and is returned from the first yield call.
>
That is not quite enough. Your suggested rule of thumb is to replace
def example(*args, **kw):
...
x = yield # first yield
with
def example(start=None, *args, **kw):
...
x = yield start # first yield
That would have been correct if my statement about what was needed
wasn't missing a bit. What you actually need to replace with is
something like:
def example(start, *args, **kw):
...
if 'throw' in start:
raise start['throw'] # simulate a throw() on first next()
elif 'send' in start:
x = start['send'] # simulate a send() on first next()
else:
x = yield start.get('yield') # use specified value for first next()
This allows you to set up so the first next() call skips the yield and
acts like a send() or a throw() was called. Actually, I think that can
be refactored to:
def cr_init(start):
if 'throw' in start:
raise start['throw']
if 'send' in start:
return start['send']
return yield start.get('yield')
def example(start, *args, **kw):
...
x = yield from cr_init(start)
Which makes it almost bearable.
It is also possible to write a @coroutine decorator that can be used
with this, the trick is to make the undecorated function available as an
attribute of the wrapper so it is available for use in yield-from. The
wrapper can also hide the existence of the start argument from top-level
users.
def coroutine(func):
def start(*args, **kwargs):
cr = func({}, *args, **kwargs)
v = cr.next()
if v is not None:
raise RuntimeError('first yield from coroutine was not None')
return cr
start.raw = func
return start
Using such a coroutine in yield-from then becomes:
# Yield None as first value
yield from example.raw({}, *args, **kwargs)
# Yield 42 as first value
yield from example.raw({'yield':42}, *args, **kwargs)
# Skip the first yield and treat the first next() as a send(42)
yield from example.raw({'send':42}, *args, **kwargs)
# Skip the first yield and treat the first next() as a throw(ValueError(42))
yield from example.raw({'throw':ValueError(42)}, *args, **kwargs)
While using it in other contexts is exactly like people are used to.
So it turns out a couple of support routines and a simple convention can
work around most of the problems with using @coroutines in yield-from.
I still think it would be nice if yield-from didn't insist on treating
its iterator as if it was new.
- Jacob
Nick Coghlan
> That would have been correct if my statement about what was needed
> wasn't missing a bit. What you actually need to replace with is
> something like:
>
> def example(start, *args, **kw):
> ...
> if 'throw' in start:
> raise start['throw'] # simulate a throw() on first next()
> elif 'send' in start:
> x = start['send'] # simulate a send() on first next()
> else:
> x = yield start.get('yield') # use specified value for first next()
This elaboration strikes me as completely unecessary, since next() or
the equivalent send(None) are the only legitimate ways to start a generator:
>>> def gen():
... print "Generator started"
... yield
...
>>> g = gen()
>>> g.next()
Generator started
A generator that receives a throw() first thing never executes at all:
>>> g = gen()
>>> g.throw(AssertionError)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in gen
AssertionError
Similarly, sending a non-None value first thing triggers an exception
since there is nowhere for the value to go:
>>> g = gen()
>>> g.send(42)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't send non-None value to a just-started generator
For the PEP, I think the solution to this issue is a couple of conventions:
1. Either don't implicitly call next() when creating coroutines or else
make that behaviour easy to bypass). This is a bad idea for the same
reason that implicitly calling start() on threads is a bad idea:
sometimes the user will want to separate definition from activation, and
it is a pain when the original author makes that darn near impossible in
order to save one line in the normal case.
2. Coroutines intended for use with yield-from should take a "start"
argument that is used for the value of their first yield. This allows
coroutines to be nested without introducing spurious "None" values into
the yield stream.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Jacob Holm
> Jacob Holm wrote:
>
>> That would have been correct if my statement about what was needed
>> wasn't missing a bit. What you actually need to replace with is
>> something like:
>>
>> def example(start, *args, **kw):
>> ...
>> if 'throw' in start:
>> raise start['throw'] # simulate a throw() on first next()
>> elif 'send' in start:
>> x = start['send'] # simulate a send() on first next()
>> else:
>> x = yield start.get('yield') # use specified value for first next()
>>
>
> This elaboration strikes me as completely unecessary, since next() or
> the equivalent send(None) are the only legitimate ways to start a generator:
>
>
"send" parts badly. The point is that the following two examples have
the same effect:
g = example({'send':42})
g.next()
g = example({})
g.next()
g.send(42)
Same effect meaning same last value returned and same internal state.
You can't do this by just providing a value to yield on the first
next(). The modified parser example I sent to Greg shows that there is
a use case for it (although it is written using one of the syntax-based
ideas).
> For the PEP, I think the solution to this issue is a couple of conventions:
>
> 1. Either don't implicitly call next() when creating coroutines or else
> make that behaviour easy to bypass). This is a bad idea for the same
> reason that implicitly calling start() on threads is a bad idea:
> sometimes the user will want to separate definition from activation, and
> it is a pain when the original author makes that darn near impossible in
> order to save one line in the normal case.
>
I don't agree that it is a bad idea to call next automatically. I can
see that it is necessary to keep a version around that doesn't do it,
but that is because of limitations in yield-from.
> 2. Coroutines intended for use with yield-from should take a "start"
> argument that is used for the value of their first yield. This allows
> coroutines to be nested without introducing spurious "None" values into
> the yield stream.
>
For the coroutine writer, it is just as easy to write:
x = yield start
As it is to write:
x = yield from cr_init(start)
The difference is that the second version is much more useful in yield-from.
Even assuming every relevant object implemented the pattern I suggest,
it is still not possible to use yield-from to write something like
itertools.dropwhile and have it delegate all send and throw calls
correctly. To make that possible, you need exactly the same thing that
you need for pre-started coroutines: The ability to replace the next()
call made by the yield-from expression with something else. Give me
that, and you will also have removed the need for a special pattern for
coroutines that should be usable with yield-from.
Still-hoping-to-avoid-the-need-for-a-special-pattern-ly yours
- Jacob
Jacob Holm
> Even assuming every relevant object implemented the pattern I suggest,
> it is still not possible to use yield-from to write something like
> itertools.dropwhile and have it delegate all send and throw calls
> correctly. To make that possible, you need exactly the same thing that
> you need for pre-started coroutines: The ability to replace the next()
> call made by the yield-from expression with something else. Give me
> that, and you will also have removed the need for a special pattern
> for coroutines that should be usable with yield-from.
To be clear, I think the best way of handling this is to add a read-only
property to generator objects holding the latest value yielded, and let
yield-from use that when present instead of calling next(). (This is not
a new idea, I am just explaining the consequences as I see them). The
property can be cleared when the frame is released, so there should be
no issues with that.
With that property, the dropwhile example becomes trivial:
def dropwhile(predicate, iterable):
it = iter(iterable)
v = next(it)
while predicate(v):
v = next(it)
return yield from it # Starts by yielding the last value checked, which is v.
More interesting (to me) is that the following helpers allow you to call
a pre-started generator using yield-from in the 3 special ways I
mentioned *without* needing the generator constructor to take any magic
arguments.
def first_yielding(value, iterable):
it = iter(iterable)
try:
s = yield value
except GeneratorExit:
it.close()
except BaseException as e:
it.throw(e) # sets the property so yield-from will use that first
else:
it.send(s) # sets the property so yield-from will use that first
return yield from it
def first_sending(value, iterable):
it = iter(iterable)
it.send(value) # sets the property so yield-from will use that first
return yield from it
def first_throwing(exc, iterable):
it = iter(iterable)
it.throw(exc) # sets the property so yield-from will use that first
return yield from it
# Yield None (first value yielded by a @coroutine) as first value
yield from example(*args, **kwargs)
# Yield 42 as first value
yield from first_yielding(42, example(*args, **kwargs))
# Treat the first next() as a send(42)
yield from first_sending(42, example(*args, **kwargs))
# Treat the first next() as a throw(ValueError(42))
yield from first_throwing(ValueError(42), example(*args, **kwargs))
So no new syntax needed, and coroutines are easily callable without the
constructor needing extra magic arguments. Also, I am sure the property
has other unrelated uses. What's not to like?
Guido van Rossum
> Jacob Holm wrote:
>> Even assuming every relevant object implemented the pattern I suggest, it
>> is still not possible to use yield-from to write something like
>> itertools.dropwhile and have it delegate all send and throw calls correctly.
>> To make that possible, you need exactly the same thing that you need for
>> pre-started coroutines: The ability to replace the next() call made by the
>> yield-from expression with something else. Give me that, and you will also
>> have removed the need for a special pattern for coroutines that should be
>> usable with yield-from.
>
> To be clear, I think the best way of handling this is to add a read-only
> property to generator objects holding the latest value yielded, and let
> yield-from use that when present instead of calling next(). (This is not a
> new idea, I am just explaining the consequences as I see them). The property
> can be cleared when the frame is released, so there should be no issues with
> that.
Let me just respond with the recommendation that you stop pushing for
features that require storing state on the generator object. Quite
apart from implementation issues (which may well be non-existent) I
think it's a really scary thing to add any "state" to a generator that
isn't contained naturally in its stack frame.
If this means that you can't use yield-from for some of your use
cases, well, so be it. It has plenty of other use cases.
And no, I am not prepared to defend this recommendation. But I feel
very strongly about it. So don't challenge me -- it's just going to be
a waste of everyone's time to continue this line of thought.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Greg Ewing
> You can't do this by just providing a value to yield on the first
> next().
I don't see how providing an initial yield value directly
to the yield-from expression can give you any greater
functionality, though. Can you post a complete example
of that so I don't have to paste code from several messages
together?
> The modified parser example I sent to Greg shows that there is
> a use case for it
FWIW, that example doesn't fit David Beazley's definition of
a coroutine, since it uses yields to both send and receive
values. He doesn't think that's a sane thing to do.
> I don't agree that it is a bad idea to call next automatically. I can
> see that it is necessary to keep a version around that doesn't do it,
> but that is because of limitations in yield-from.
An alternative viewpoint would be the idea that a coroutine
should always start itself automatically is too simplistic.
--
Greg
Greg Ewing
> I think the best way of handling this is to add a read-only
> no issues with that.
It will still keep the value alive longer than it would
be otherwise. Some people might take issue with that.
> def dropwhile(predicate, iterable):
> it = iter(iterable)
> v = next(it)
> while predicate(v):
> v = next(it)
> return yield from it # Starts by yielding the last value checked,
> which is v.
In my view this constitutes a shared iterator, and is
therefore outside the scope of yield-from.
I also don't think this generalizes well enough to be
worth going out of our way to support. It only works
because you're making a "tail call" to the iterator,
which is a rather special case. Most itertools-style
functions won't have that property.
> What's not to like?
The fact that yield-from and/or generator behaviour
is being complexified to support things that are
outside the scope of my proposal.
This is why I want to keep focused on refactoring,
to prevent this kind of feature creep.
--
Greg
Jacob Holm
> On Wed, Apr 8, 2009 at 8:04 AM, Jacob Holm <j...@improva.dk> wrote:
>
>> Jacob Holm wrote:
>>
>>> Even assuming every relevant object implemented the pattern I suggest, it
>>> is still not possible to use yield-from to write something like
>>> itertools.dropwhile and have it delegate all send and throw calls correctly.
>>> To make that possible, you need exactly the same thing that you need for
>>> pre-started coroutines: The ability to replace the next() call made by the
>>> yield-from expression with something else. Give me that, and you will also
>>> have removed the need for a special pattern for coroutines that should be
>>> usable with yield-from.
>>>
>> To be clear, I think the best way of handling this is to add a read-only
>> property to generator objects holding the latest value yielded, and let
>> yield-from use that when present instead of calling next(). (This is not a
>> new idea, I am just explaining the consequences as I see them). The property
>> can be cleared when the frame is released, so there should be no issues with
>> that.
>>
>
> Let me just respond with the recommendation that you stop pushing for
> features that require storing state on the generator object. Quite
> apart from implementation issues (which may well be non-existent) I
> think it's a really scary thing to add any "state" to a generator that
> isn't contained naturally in its stack frame.
>
Does storing it as part of the frame object count as "naturally in the
stack frame"? Because that is probably the most natural place to put
this, implementation-wise. If storing on the frame object is also out,
I will have to start thinking about new syntax again. Oh well.
I was going to push for saving the final return value from a generator
somewhere so that each StopIteration raised by an operation on the
closed generator could have the same value as the StopIteration that
closed it (or None if it was closed by another exception). If that
value can't live on the generator object, I guess that idea is dead. Am
I right?
> If this means that you can't use yield-from for some of your use
> cases, well, so be it. It has plenty of other use cases.
>
That is exactly what I am worried about. I think the number of use
cases will be severely limited if we don't have a way to replace the
initial next() made by yield-from. This is different from the close()
issues we debated earlier, where there is a relatively simple
workaround. The full workaround for the "initial next()" issue is big,
ugly, and slow.
Anyway, I still hope the workaround won't be needed.
> And no, I am not prepared to defend this recommendation. But I feel
> very strongly about it. So don't challenge me -- it's just going to be
> a waste of everyone's time to continue this line of thought.
>
No, I am not going to challenge you on this. Once I have your answer
to the questions at the beginning of this mail, I will try to adjust my
future proposals accordingly.
Best regards
- Jacob
Guido van Rossum
> Guido van Rossum wrote:
>>
>> On Wed, Apr 8, 2009 at 8:04 AM, Jacob Holm <j...@improva.dk> wrote:
>>
>>>
>>> Jacob Holm wrote:
>>> To be clear, I think the best way of handling this is to add a read-only
>>> property to generator objects holding the latest value yielded, and let
>>> yield-from use that when present instead of calling next(). (This is not
>>> a
>>> new idea, I am just explaining the consequences as I see them). The
>>> property
>>> can be cleared when the frame is released, so there should be no issues
>>> with
>>> that.
>>>
>>
>> Let me just respond with the recommendation that you stop pushing for
>> features that require storing state on the generator object. Quite
>> apart from implementation issues (which may well be non-existent) I
>> think it's a really scary thing to add any "state" to a generator that
>> isn't contained naturally in its stack frame.
>>
>
> Does storing it as part of the frame object count as "naturally in the stack
> frame"? Because that is probably the most natural place to put this,
> implementation-wise.
No, that's out too. My point is that I don't want to add *any* state
beyond what the user thinks of as the "normal" state in the frame
(i.e. local variables, where it is suspended, and the expression stack
and try-except stack). Nothing else.
> If storing on the frame object is also out, I will
> have to start thinking about new syntax again. Oh well.
Sorry, no go. New syntax is also out.
> I was going to push for saving the final return value from a generator
> somewhere so that each StopIteration raised by an operation on the closed
> generator could have the same value as the StopIteration that closed it (or
> None if it was closed by another exception). If that value can't live on
> the generator object, I guess that idea is dead. Am I right?
Right.
>> If this means that you can't use yield-from for some of your use
>> cases, well, so be it. It has plenty of other use cases.
>>
>
> That is exactly what I am worried about. I think the number of use cases
> will be severely limited if we don't have a way to replace the initial
> next() made by yield-from.
It doesn't matter if there is only one use case, as long as it is a
common one. And we already have that: Greg Ewing's "refactoring".
Remember, Dave Beazley in his coroutine tutorial expresses doubts
about whether it is really that useful to use generators as
coroutines.
You are fighting a losing battle here, and it would be better if we
stopped short of trying to attain perfection, and instead accepted an
imperfect solution that may be sufficient, or may have to be extended
in the future. If your hypothetical use cases really become important
you can write another PEP.
> This is different from the close() issues we
> debated earlier, where there is a relatively simple workaround. The full
> workaround for the "initial next()" issue is big, ugly, and slow.
>
> Anyway, I still hope the workaround won't be needed.
Not if you give up now.
>> And no, I am not prepared to defend this recommendation. But I feel
>> very strongly about it. So don't challenge me -- it's just going to be
>> a waste of everyone's time to continue this line of thought.
>>
>
> No, I am not going to challenge you on this. Once I have your answer to
> the questions at the beginning of this mail, I will try to adjust my future
> proposals accordingly.
Great.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Nick Coghlan
> An alternative viewpoint would be the idea that a coroutine
> should always start itself automatically is too simplistic.
That's the angle I've been taking. I can see why it can be convenient to
start a coroutine automatically, but only in the same way that a thread
creation function that also starts the thread for you can be convenient.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Jacob Holm
> On Wed, Apr 8, 2009 at 7:52 PM, Jacob Holm <j...@improva.dk> wrote:
>
>> Does storing it as part of the frame object count as "naturally in the stack
>> frame"? Because that is probably the most natural place to put this,
>> implementation-wise.
>>
>
> No, that's out too. My point is that I don't want to add *any* state
> beyond what the user thinks of as the "normal" state in the frame
> (i.e. local variables, where it is suspended, and the expression stack
> and try-except stack). Nothing else.
>
That rules out Gregs and my patches as well. They both need extra state
on the frame object to be able to implement yield-from in the first place.
>> If storing on the frame object is also out, I will
>> have to start thinking about new syntax again. Oh well.
>>
>
> Sorry, no go. New syntax is also out.
>
In that case, I give up. There is no possible way to fix the "initial
next()" issue of yield-from without one or the other.
>> I was going to push for saving the final return value from a generator
>> somewhere so that each StopIteration raised by an operation on the closed
>> generator could have the same value as the StopIteration that closed it (or
>> None if it was closed by another exception). If that value can't live on
>> the generator object, I guess that idea is dead. Am I right?
>>
>
> Right.
>
Then let me revisit my earlier statement that when close() catches a
StopIteration with a non-None value, it should either return it or raise
an exception. Since the value is not saved, a second close() will
neither be able to return it, nor raise a StopIteration with it.
Therefore I now think that raising a RuntimeError in that case is the
only right thing to do.
>>> If this means that you can't use yield-from for some of your use
>>> cases, well, so be it. It has plenty of other use cases.
>>>
>>>
>> That is exactly what I am worried about. I think the number of use cases
>> will be severely limited if we don't have a way to replace the initial
>> next() made by yield-from.
>>
>
> It doesn't matter if there is only one use case, as long as it is a
> common one. And we already have that: Greg Ewing's "refactoring".
>
I remain unconvinced that the "initial next()" issue isn't also a
problem for that use case, but I am not going to argue about this.
> Remember, Dave Beazley in his coroutine tutorial expresses doubts
> about whether it is really that useful to use generators as
> coroutines.
>
> You are fighting a losing battle here, and it would be better if we
> stopped short of trying to attain perfection, and instead accepted an
> imperfect solution that may be sufficient, or may have to be extended
> in the future. If your hypothetical use cases really become important
> you can write another PEP.
>
>
Looks to me like the battle is not so much losing as already lost, and
probably was from the start. I just wish I had understood that earlier.
One final suggestion I have is to make yield-from raise a RuntimeError
if used on a generator that already has a frame. That would a) open
some optimization possibilities, b) make it clear that the only intended
use is with a *fresh* generator or a non-generator iterable, and c)
allow us to change our minds about the initial next() later without
changing the semantics of working code.
>> This is different from the close() issues we
>> debated earlier, where there is a relatively simple workaround. The full
>> workaround for the "initial next()" issue is big, ugly, and slow.
>>
>> Anyway, I still hope the workaround won't be needed.
>>
>
> Not if you give up now.
>
Well, since I am giving up on fixing the "initial next()" issue in the
core, the workaround *will* be needed. Maybe not in the PEP, but
certainly as a recipe somewhere. If you don't mind, I will continue the
discussion about the details of such a workaround that Nick started a
couple of mails back in this thread.
>>> And no, I am not prepared to defend this recommendation. But I feel
>>> very strongly about it. So don't challenge me -- it's just going to be
>>> a waste of everyone's time to continue this line of thought.
>>>
>>>
>> No, I am not going to challenge you on this. Once I have your answer to
>> the questions at the beginning of this mail, I will try to adjust my future
>> proposals accordingly.
>>
>
> Great.
>
Frustrated-ly yours
- Jacob
Nick Coghlan
> Then let me revisit my earlier statement that when close() catches a
> StopIteration with a non-None value, it should either return it or raise
> an exception. Since the value is not saved, a second close() will
> neither be able to return it, nor raise a StopIteration with it.
> Therefore I now think that raising a RuntimeError in that case is the
> only right thing to do.
Remember, close() is designed to be about finalization. So long as the
generator indicates that it has finished (i.e. by reraising
GeneratorExit or raising StopIteration with or without a value), the
method has done its job. Raising a RuntimeError for a successfully
closed generator doesn't make any sense.
So if someone wants the return value, they'll need to either use next(),
send() or throw() and catch the StopIteration themselves, or else use
'yield from'.
That said, creating your own stateful wrapper that preserves the last
yield value and the final return value of a generator iterator is also
perfectly possible:
class CaptureGen(object):
"""Capture and preserve the last yielded value and the
final return value of a generator iterator instance"""
NOT_SET = object()
def __init__(self, geniter):
self.geniter = geniter
self._last_yield = self.NOT_SET
self._return_value = self.NOT_SET
@property
def last_yield(self):
if self._last_yield is self.NOT_SET:
raise RuntimeError("Generator has not yielded")
return self._last_yield
@property
def return_value(self):
if self._return_value is self.NOT_SET:
raise RuntimeError("Generator has not returned")
return self._return_value
def _delegate(self, meth, *args):
try:
val = meth(*args)
except StopIteration, ex:
if self._return_value is self.NOT_SET:
self._return_value = ex.value
raise
raise StopIteration(self._return_value)
self._last_yield = val
return val
def __next__(self):
return self._delegate(self.geniter.next)
next = __next__
def send(self, val):
return self._delegate(self.geniter.send, val)
def throw(self, et, ev=None, tb=None):
return self._delegate(self.geniter.throw, et, ev, tb)
def close(self):
self.geniter.close()
return self._return_value
Something like that may actually turn out to be useful as the basis for
an enhanced coroutine decorator, similar to the way one uses
contextlib.contextmanager to turn a generator object into a context
manager. The PEP is quite usable for refactoring without it though.
>> It doesn't matter if there is only one use case, as long as it is a
>> common one. And we already have that: Greg Ewing's "refactoring".
>>
>
> I remain unconvinced that the "initial next()" issue isn't also a
> problem for that use case, but I am not going to argue about this.
For refactoring, the pattern of passing in a "start" value for use in
the first yield expression in the subiterator should be adequate. That's
enough to avoid injecting spurious "None" values into the yield sequence.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Jacob Holm
> Jacob Holm wrote:
>
>> Then let me revisit my earlier statement that when close() catches a
>> StopIteration with a non-None value, it should either return it or raise
>> an exception. Since the value is not saved, a second close() will
>> neither be able to return it, nor raise a StopIteration with it.
>> Therefore I now think that raising a RuntimeError in that case is the
>> only right thing to do.
>>
>
> Remember, close() is designed to be about finalization. So long as the
> generator indicates that it has finished (i.e. by reraising
> GeneratorExit or raising StopIteration with or without a value), the
> method has done its job. Raising a RuntimeError for a successfully
> closed generator doesn't make any sense.
>
Returning a value in response to GeneratorExit is what doesn't make any
sense when there is no possible way to access that value later. Raising
a RuntimeError in this case will be a clear reminder of this. Think
newbie protection if you will.
> So if someone wants the return value, they'll need to either use next(),
> send() or throw() and catch the StopIteration themselves, or else use
> 'yield from'.
>
Yes. How is that an argument against making close raise a RuntimeError
if it catches a StopIteration with a non-None value?
I know it is possible to create a complete workaround. The problem is
that any complete workaround will break the chain of yield-from calls,
causing a massive overhead on next(), send(), and throw() compared to a
partial workaround that doesn't break the yield-from chain.
> For refactoring, the pattern of passing in a "start" value for use in
> the first yield expression in the subiterator should be adequate. That's
> enough to avoid injecting spurious "None" values into the yield sequence.
>
If you define "refactoring" narrowly enough, you are probably right.
Otherwise it depends on how you want the new subgenerator to work. If
you want it to be a @coroutine (with or without the decorator) there are
good reasons for wanting to throw away the value from the first next()
and provide an initial value to send() or throw() immediately after
that, so the first value yielded by the yield-from becomes the result of
the send() or throw(). Taking just a simple "start" value in the
constructor and making the first yield an "x = yield start" doesn't
support that use. Taking a compound "start" value in the constructor
and making the first yield an "x = yield from cr_init(start)" does, by
skipping the first yield when neccessary and simulating the send() or
throw().
- Jacob
Jim Jewett
> Jacob Holm wrote:
>> Then let me revisit my earlier statement that when close() catches a
>> StopIteration with a non-None value, it should either return it or raise
>> an exception.
This implies that the value is always important; I write plenty of
functions that don't bother to return anything, and expect that would
still be common if I were factoring out loops.
>> Since the value is not saved, a second close() will
>> neither be able to return it, nor raise a StopIteration with it.
If close is called more than once, that suggests at least one of those
calls is just freeing resources, and doesn't need a value.
>>> It doesn't matter if there is only one use case, as long as it is a
>>> common one. And we already have that: Greg Ewing's "refactoring".
>> I remain unconvinced that the "initial next()" issue isn't also a
>> problem for that use case, but I am not going to argue about this.
That does suggest that yield-from *should* accept pre-started
generators, if only because the previous line (or a decorator) may
have primed it.
> For refactoring, the pattern of passing in a "start" value for use in
> the first yield expression in the subiterator should be adequate.
Only if you know what the first value should be. If you're using a
sentinel that you plan to discard, then it would be way simpler to
just prime the generator before the yield-from.
-jJ
Jacob Holm
> On 4/9/09, Nick Coghlan <ncog...@gmail.com> wrote:
>
>> Jacob Holm wrote:
>>
>>> Then let me revisit my earlier statement that when close() catches a
>>> StopIteration with a non-None value, it should either return it or raise
>>> an exception.
>>>
>
> This implies that the value is always important; I write plenty of
> functions that don't bother to return anything, and expect that would
> still be common if I were factoring out loops.
>
If you return None (or no value) you won't get an exception with my
proposal. Only if you handle a GeneratorExit by returning a non-None
value. Since that value is not going to be visible to any other code,
it doesn't make sense to return it. I am suggesting that you should get
a RuntimeError so you become aware that returning the value doesn't make
sense.
>>> Since the value is not saved, a second close() will
>>> neither be able to return it, nor raise a StopIteration with it.
>>>
>
> If close is called more than once, that suggests at least one of those
> calls is just freeing resources, and doesn't need a value.
>
>
>>>> It doesn't matter if there is only one use case, as long as it is a
>>>> common one. And we already have that: Greg Ewing's "refactoring".
>>>>
>
>
>>> I remain unconvinced that the "initial next()" issue isn't also a
>>> problem for that use case, but I am not going to argue about this.
>>>
>
> That does suggest that yield-from *should* accept pre-started
> generators, if only because the previous line (or a decorator) may
> have primed it.
>
>
That was my argument, but since there is no sane way of handling
pre-primed generators without extending the PEP in a direction that
Guido has forbidden, I suggest raising a RuntimeError in this case
instead. That allows us to add the necessary features later if the
need is recognized.
>> For refactoring, the pattern of passing in a "start" value for use in
>> the first yield expression in the subiterator should be adequate.
>>
>
> Only if you know what the first value should be. If you're using a
> sentinel that you plan to discard, then it would be way simpler to
> just prime the generator before the yield-from.
>
>
Yeah, but yield from with pre-primed generators just ain't gonna happen. :(
- Jacob
Greg Ewing
> That rules out Gregs and my patches as well. They both need extra state
> on the frame object to be able to implement yield-from in the first place.
But that state is obviously necessary in order to support
yield-from, and it goes away as soon as the yield-from
itself finishes.
Your proposals add non-obvious extra state that persists
longer than normally expected, to support obscure features
that will rarely be used.
> One final suggestion I have is to make yield-from raise a RuntimeError
> if used on a generator that already has a frame. That would ...
> b) make it clear that the only intended
> use is with a *fresh* generator or a non-generator iterable,
But there's no way of detecting a violation of that rule
for a non-generator iterator, and I want to avoid having
special cases for generators, since it goes against duck
typing.
--
Greg
Greg Ewing
> That does suggest that yield-from *should* accept pre-started
> generators, if only because the previous line (or a decorator) may
> have primed it.
If the previous line has primed it, then it's not a
fresh iterator, so all bets are off.
Same if a decorator has primed it in such a way that
it doesn't behave like a fresh iterator.
--
Greg
Jim Jewett
> Jim Jewett wrote:
>> On 4/9/09, Nick Coghlan <ncog...@gmail.com> wrote:
>>> Jacob Holm wrote:
>>>> ... when close() catches a
>>>> StopIteration with a non-None value, it should either return it or raise
>> This implies that the value is always important; ...
> If you return None (or no value) you won't get an exception
Think of all the C functions that return a success status, or the
number of bytes read/written. Checking the return code is good, but
not checking it isn't always worth an Exception.
>> ... does suggest that yield-from *should* accept pre-started
>> generators, if only because the previous line (or a decorator) may
>> have primed it.
> That was my argument, but since there is no sane way of handling
> pre-primed generators without extending the PEP in a direction that
> Guido has forbidden,
Huh? All you need to do is to not care whether the generator is fresh
or not (let alone primed vs half-used).
If it won't be primed, *and* you can't afford to send back an extra
junk yield, then you need to prime it yourself. That can be awkward,
but so are all the syntax extensions that boil down to "and implicitly
call next for me once". (And the "oops, just this once, don't prime
it after all" extensions are even worse.)
-jJ
Guido van Rossum
> Jacob Holm wrote:
>> That rules out Gregs and my patches as well. They both need extra state
>> on the frame object to be able to implement yield-from in the first place.
>
> But that state is obviously necessary in order to support
> yield-from, and it goes away as soon as the yield-from
> itself finishes.
Another way to look at this is, "RETVAL = yield from EXPR" has an
expansion into source code where all that state is kept as either a
local variable or via the position in the code, and that's how we
define the semantics. I don't believe Jacob's proposal (the one that
doesn't require new syntax) works this way.
> Your proposals add non-obvious extra state that persists
> longer than normally expected, to support obscure features
> that will rarely be used.
>
>> One final suggestion I have is to make yield-from raise a RuntimeError if
>> used on a generator that already has a frame. That would ...
>
>> b) make it clear that the only intended
>>
>> use is with a *fresh* generator or a non-generator iterable,
>
> But there's no way of detecting a violation of that rule
> for a non-generator iterator, and I want to avoid having
> special cases for generators, since it goes against duck
> typing.
It also goes against the "for x in EXPR: yield x" expansion, which I
would like to maintain as an anchor point for the semantics. To be
precise: when the .send(), .throw() and .close() methods of the outer
generator aren't used, *or* when iter(EXPR) returns a non-generator
iterator, these semantics should be maintained precisely, and the
extended semantics if .send()/.throw()/.close() are used on the outer
generator and iter(EXPR) is a generator should be natural extensions
of this, without semantic discontinuities.
PS. On Jacob's complaint that he didn't realize earlier that his
proposal was dead on arrival: I didn't either. It took me all this
time to translate my gut feelings into rules.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Jacob Holm
> On 4/9/09, Jacob Holm <j...@improva.dk> wrote:
>
>>> ... does suggest that yield-from *should* accept pre-started
>>> generators, if only because the previous line (or a decorator) may
>>> have primed it.
>>>
>
>
>> That was my argument, but since there is no sane way of handling
>> pre-primed generators without extending the PEP in a direction that
>> Guido has forbidden,
>>
>
> Huh? All you need to do is to not care whether the generator is fresh
> or not (let alone primed vs half-used).
>
By no sane way I mean that there is no way to avoid that the first call
made by the yield-from construct is a
next(). If the pre-primed generator is expecting a send() at that point
you are screwed.
> If it won't be primed, *and* you can't afford to send back an extra
> junk yield, then you need to prime it yourself.
And then the yield-from is still starting out by calling next() so you
are still screwed.
> That can be awkward,
> but so are all the syntax extensions that boil down to "and implicitly
> call next for me once". (And the "oops, just this once, don't prime
> it after all" extensions are even worse.)
>
>
The only syntax extension that was really interesting was intended to
*avoid* this call to next() by providing a different value to yield the
first time. Avoiding the call to next() allows you to create all kinds
of wrappers that manipulate the start of the sequence, and covers all
other cases I had considered syntax for.
Anyway, this syntax discussion is moot since Guido has already ruled
that we will have no syntax for this.
- Jacob
Jacob Holm
> On Thu, Apr 9, 2009 at 5:35 PM, Greg Ewing <greg....@canterbury.ac.nz> wrote:
>
>> Jacob Holm wrote:
>>
>>> That rules out Gregs and my patches as well. They both need extra state
>>> on the frame object to be able to implement yield-from in the first place.
>>>
>> But that state is obviously necessary in order to support
>> yield-from, and it goes away as soon as the yield-from
>> itself finishes.
>>
>
> Another way to look at this is, "RETVAL = yield from EXPR" has an
> expansion into source code where all that state is kept as either a
> local variable or via the position in the code, and that's how we
> define the semantics. I don't believe Jacob's proposal (the one that
> doesn't require new syntax) works this way.
>
>
My proposal was to extend the lifetime of the yielded value until the
yield expression returned, and to make that value available on the
frame. The "keep the value alive" part can easily be written as an
expansion based on the existing yield semantics, and the "make it
available on the frame" is similar to the need that yield-from has.
That doesn't look all that different to me, but I'll let you be the
judge of that.
- Jacob
Greg Ewing
>>>>class Tricky(object):
>
> ... def __iter__(self):
> ... return self
> ... def next1(self):
> ... print "Next 1"
> ... Tricky.next = Tricky.next2
I just tried using a class like that in a for loop, and
it doesn't work. The next() method has to be defined in
the class, because it's actually a special method (it
has a slot in the type object).
> Greg - looks to me like you're going to have to disallow caching the
> method lookup in the PEP.
Even if it would work, I don't want to preclude a
valuable optimization for the sake of such an obscure
use case.
--
Greg
Greg Ewing
> I
> am saying that there are examples where it is desirable to move one of
> the arguments that this form of refactoring forces you to put in the
> constructor so it instead becomes the argument of the first send.
I'm having trouble seeing circumstances in which you
would need to do that. Can you provide an example
in the form of
(a) a piece of unfactored code
(b) a desired refactoring
(c) an explanation of why the desired refactoring
can't conveniently be done using an unprimed
generator and plain yield-from.
--
Greg
Nick Coghlan
> Nick Coghlan wrote:
>
>>>>> class Tricky(object):
>>
>> ... def __iter__(self):
>> ... return self
>> ... def next1(self):
>> ... print "Next 1"
>> ... Tricky.next = Tricky.next2
>
> I just tried using a class like that in a for loop, and
> it doesn't work. The next() method has to be defined in
> the class, because it's actually a special method (it
> has a slot in the type object).
Note that my example actually *does* modify the class as it goes along
(for exactly the reason you give - next() has to be defined on the class
or the interpreter will ignore it).
Is it possible you assigned to "self.next" instead of to "Tricky.next"
when trying it out?
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Jacob Holm
> Jacob Holm wrote:
>> I am saying that there are examples where it is desirable to move one
>> of the arguments that this form of refactoring forces you to put in
>> the constructor so it instead becomes the argument of the first send.
>
> I'm having trouble seeing circumstances in which you
> would need to do that. Can you provide an example
> in the form of
>
> (a) a piece of unfactored code
Ok, once again based on your own parser example. The parse_items
generator could have been written as:
def parse_items(closing_tag = None):
elems = []
token = yield
while token != closing_tag:
if is_opening_tag(token):
name = token[1:-1]
items = yield from parse_items("</%s>" % name)
elems.append((name, items))
else:
elems.append(token)
token = yield
return elems
>
> (b) a desired refactoring
I would like to split off a function for parsing a single element. And
I would like it to look like this:
def parse_elem():
opening_tag = yield
name = opening_tag[1:-1]
items = yield from parse_items("</%s>" % name)
return (name, items)
This differs from the version in your example by taking all the tags as
arguments to send() instead of having the opening tag as an argument to
the constructor.
Unfortunately, there is no way to actually use this version in the
implementation of parse_items.
>
> (c) an explanation of why the desired refactoring
> can't conveniently be done using an unprimed
> generator and plain yield-from.
>
The suggested subroutine cannot be used, because parse_items already has
the value that should go as the argument to the first send().
It is easy to rewrite it to the version you used in the example, but
that requires you to make the opening_tag an argument to the
constructor, whereas I want it as an argument to the first send. You
can of course make that argument optional and adjust the function to
only do the first yield if the argument is not given. That is
essentially what my "cr_init()" pattern does. Using that pattern, the
refactoring looks like this:
def cr_init(start):
if start is None:
return yield
if 'send' in start:
return start['send']
if 'throw' in start:
raise start['throw']
return yield start.get('yield')
def parse_elem(start=None):
opening_tag = yield from cr_init(start)
name = opening_tag[1:-1]
items = yield from parse_items("</%s>" % name)
return (name, items)
def parse_items(closing_tag=None, start=None):
elems = []
token = yield from cr_init(start)
while token != closing_tag:
if is_opening_tag(token):
elems.append(yield from parse_elem(start={'send':token}))
else:
elems.append(token)
token = yield
return elems
As you see it *can* be done, but I would hardly call it convenient. The
main problem is that the coroutine you want to call must be written with
this in mind or you are out of luck. While it *is* possible to write a
wrapper that lets you call the unmodified parse_elem, that wrapper
cannot use yield_from to call it so you get a rather large overhead that
way.
A convention like Nick suggested where all coroutines take an optional
"start" argument with the first value to yield doesn't help, because it
is not the value to yield that is the problem.
I hope this helps to explain why the cr_init pattern is needed even for
relatively simple refactoring now that it seems we are not fixing the
"initial next()" issue.
- Jacob
Greg Ewing
> Note that my example actually *does* modify the class as it goes along
> (for exactly the reason you give - next() has to be defined on the class
> or the interpreter will ignore it).
Sorry, I didn't notice you were doing that.
But this doesn't seem useful to me. What if you
have two instances of Tricky in use at the same
time? They're going to interfere with each other.
I suppose you could make it work if you created
a new class each time. But my earlier comment
stands -- I don't want to preclude optimizing the
common case for the sake of an uncommon case.
--
Greg
Greg Ewing
> I would like to split off a function for parsing a single element. And
> I would like it to look like this:
>
> def parse_elem():
> opening_tag = yield
> name = opening_tag[1:-1]
> items = yield from parse_items("</%s>" % name)
> return (name, items)
I don't see what you gain by writing it like that, though.
You don't even know whether you want to call this function
until you've seen the first token and realized that it's
a tag.
In other words, you need a one-token lookahead. A more
conventional parser would use a scanner that lets you
peek at the next token without absorbing it, but that's
not an option when you're receiving the tokens via
yield, so another solution must be found.
The solution I chose was to keep the lookahead token
as state in the parsing functions, and pass it to
wherever it's needed. Your parse_elem() function clearly
needs it, so it should take it as a parameter.
If there's some circumstance in which you know for
certain that there's an elem coming up, you can always
write another parsing function for dealing with that,
e.g.
def expect_elem():
first = yield
return yield from parse_elem(opening_tag = first)
I don't think there's anything inconvenient about that.
> A convention like Nick suggested where all coroutines take an
> optional "start" argument with the first value to yield doesn't
> help, because it is not the value to yield that is the problem.
I think you've confused the issue a bit yourself, because
you started out by asking for a way of specifing the first
value to yield in the yield-from expression. But it seems
that what you really want is to specify the first value
to *send* into the subiterator.
I haven't seen anything so far that convinces me it would
be a serious inconvenience not to have such a feature.
Also, it doesn't seem to generalize. What if your parser
needs a two-token lookahead? Then you'll be asking for a
way to specify the first *two* values to send in. Where
does it end?
--
Greg
Nick Coghlan
> Nick Coghlan wrote:
>
>> Note that my example actually *does* modify the class as it goes along
>> (for exactly the reason you give - next() has to be defined on the class
>> or the interpreter will ignore it).
>
> Sorry, I didn't notice you were doing that.
>
> But this doesn't seem useful to me. What if you
> have two instances of Tricky in use at the same
> time? They're going to interfere with each other.
> I suppose you could make it work if you created
> a new class each time.
I provided a more complete example in another message that showed how to
put the "start" value idiom in a helper function so it could be used
with any iterator:
def start(iterable, start):
itr = iter(iterable)
class TrapFirstNext(object):
def __iter__(self):
return self
def next(self):
TrapFirstNext.next = itr.next
return start
# Should also set send and throw to
# itr.send and itr.throw if they exist
return TrapFirstNext()
>>> for val in start(range(2), "Hello World!"):
... print val
...
Hello World!
0
1
> But my earlier comment
> stands -- I don't want to preclude optimizing the
> common case for the sake of an uncommon case.
My main concern with allowing next() to be cached is that none of the
existing looping constructs (for loop, comprehensions, generator
expressions) have ever cached the bound method, so doing it in
yield-from would make the new expression an odd special case (e.g. the
for loop above would work, but using start() in a yield-from that cached
the bound method would result in an infinite loop).
That said, the existing language reference doesn't actually *say* that
an implementation isn't allowed to cache the bound next() method in a
for loop - it's just a long-standing convention that CPython has done
the lookup every time around the loop.
If this turns out to be "too slow" for coroutine setups that use
send(val) rather than next(), then a better optimisation may be to turn
send() and throw() into proper optional elements of the iterator
protocol and give them their own slots on the type object. Deviating
from existing looping behaviour in order to allow caching seems like a
worse option, even if the details of the current behaviour were
originally just an implementation accident.
Also, for the simple case where send() and throw() aren't used we want:
yield from subiter
to do the same thing as:
for x in subiter:
yield x
For CPython, that currently means no caching of the next() method in
yield-from, since caching it would break the equivalence with the latter
expansion.
I think allowing next() to be cached in looping constructs should be
pulled out of the PEP and turned into a separate PEP seeking explicit
clarification from Guido as to whether the lack of caching of next() is
part of the definition of looping in the language, or whether it is just
an implementation detail of CPython that shouldn't be relied on.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Jacob Holm
> Jacob Holm wrote:
>
>> I would like to split off a function for parsing a single element.
>> And I would like it to look like this:
>>
>> def parse_elem():
>> opening_tag = yield
>> name = opening_tag[1:-1]
>> items = yield from parse_items("</%s>" % name)
>> return (name, items)
>
> I don't see what you gain by writing it like that, though.
A more consistent api for calling it. Instead of special-casing the
first input, all input is provided the same way. That makes it more
similar to parse_items that is already called that way.
> You don't even know whether you want to call this function
> until you've seen the first token and realized that it's
> a tag.
Not when used from parse_items, but remember that my intension was to
make parse_elem independently useful. If you are just starting to parse
a presumed valid xml stream (without self-closed tags), you know that it
consists of a single element but won't know that it is.
>
> In other words, you need a one-token lookahead. A more
> conventional parser would use a scanner that lets you
> peek at the next token without absorbing it, but that's
> not an option when you're receiving the tokens via
> yield, so another solution must be found.
>
> The solution I chose was to keep the lookahead token
> as state in the parsing functions, and pass it to
> wherever it's needed. Your parse_elem() function clearly
> needs it, so it should take it as a parameter.
>
> If there's some circumstance in which you know for
> certain that there's an elem coming up, you can always
> write another parsing function for dealing with that,
> e.g.
>
> def expect_elem():
> first = yield
> return yield from parse_elem(opening_tag = first)
>
> I don't think there's anything inconvenient about that.
>
Except you now have one extra function for exactly the same task, just
with a different calling convention. And this doesn't handle an initial
throw() correctly. Not that I see any reason to use throw in the parser
example. I'm just saying an extra function wouldn't work in that case.
>> A convention like Nick suggested where all coroutines take an
> > optional "start" argument with the first value to yield doesn't
> > help, because it is not the value to yield that is the problem.
>
> I think you've confused the issue a bit yourself, because
> you started out by asking for a way of specifing the first
> value to yield in the yield-from expression. But it seems
> that what you really want is to specify the first value
> to *send* into the subiterator.
In this case, yes. In other cases it really is the first value to yield
from the subiterator, or the first value to throw into the subiterator.
At least part of the confusion comes from the fact that if yield-from
could somehow suppress the initial next and yield a different value
instead (either an extra expression in yield-from or the last value
yielded by a primed generator), there would be a simple way to write
wrappers that could be used at the call site to handle all those cases.
So a feature that allowed specifying the first value to yield in the
yield-from expression *would* be enough, but a start argument to the
coroutine constructor isn't.
>
> I haven't seen anything so far that convinces me it would
> be a serious inconvenience not to have such a feature.
>
> Also, it doesn't seem to generalize. What if your parser
> needs a two-token lookahead? Then you'll be asking for a
> way to specify the first *two* values to send in. Where
> does it end?
>
The "suppress initial next()" feature *would* have helped, by enabling
you to write a generic wrapper to use at the call site that could do
exactly that. The wrapper could use send() as many times as needed on
the wrapped generator, then use yield-from to call it when done.
Without that feature, the wrapper can't use yield-from to call the
wrapped generator. Of course there are (slower) ways to write such a
wrapper without using yield-from.
The alternative to using a call wrapper is to rewrite the subiterator to
take the full lookahead as arguments, but how would you write functions
like parse_elem and parse_items if the lookahead is variable? (You can
safely assume that the lookahead is no more than is needed to exhaust
the generator)
I think I can probably generalize the cr_init() pattern to handle a
variable lookahead, but I think even a slow call wrapper might be faster
in that case (depending on the nesting level).
- Jacob
Nick Coghlan
> At least part of the confusion comes from the fact that if yield-from
> could somehow suppress the initial next and yield a different value
> instead (either an extra expression in yield-from or the last value
> yielded by a primed generator), there would be a simple way to write
> wrappers that could be used at the call site to handle all those cases.
> So a feature that allowed specifying the first value to yield in the
> yield-from expression *would* be enough, but a start argument to the
> coroutine constructor isn't.
I think leaving this task to wrapper classes in the initial version of
the PEP is the right way to go at this point. Adding a "skip the initial
next and yield <expr> instead" clause later will be much easier than
trying to undo something added now if it turns out to be a mistake.
Greg's basic proposal makes the easy things easy and the difficult
things possible, so it is a very good place to start. The main change I
would like from the original version of the PEP is for caching the bound
methods to be explicitly disallowed in order to match the behaviour of
normal for loops.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Jacob Holm
> Jacob Holm wrote:
>
>> At least part of the confusion comes from the fact that if yield-from
>> could somehow suppress the initial next and yield a different value
>> instead (either an extra expression in yield-from or the last value
>> yielded by a primed generator), there would be a simple way to write
>> wrappers that could be used at the call site to handle all those cases.
>> So a feature that allowed specifying the first value to yield in the
>> yield-from expression *would* be enough, but a start argument to the
>> coroutine constructor isn't.
>>
>
> I think leaving this task to wrapper classes in the initial version of
> the PEP is the right way to go at this point.
I have already given up on getting this feature in at this point. The
above paragraph was just meant to clear up some misunderstandings.
> Adding a "skip the initial
> next and yield <expr> instead" clause later will be much easier than
> trying to undo something added now if it turns out to be a mistake.
>
Note that if we decide that it is OK to use yield-from with an
already-started generator in this version, we can't later change
yield-from to use the latest value yielded in place of the initial
next(). That makes new syntax the only possibility for that future
extension. Not that this is necessarily a bad thing.
> Greg's basic proposal makes the easy things easy and the difficult
> things possible, so it is a very good place to start.
Yes. You can even write the slow version of the call-wrappers I am
talking about, and then replace them with the faster versions later if
the feature becomes available.
> The main change I
> would like from the original version of the PEP is for caching the bound
> methods to be explicitly disallowed in order to match the behaviour of
> normal for loops.
>
I am not really sure about this. It looks very much like an
implementation detail to me. On the other hand, the ability to replace
the methods mid-flight might give us a way to implement the
call-wrappers with minimal overhead. Since the current patches don't
actually do any caching, this is something I should actually be able to try.
- Jacob
Nick Coghlan
> I am not really sure about this. It looks very much like an
> implementation detail to me. On the other hand, the ability to replace
> the methods mid-flight might give us a way to implement the
> call-wrappers with minimal overhead. Since the current patches don't
> actually do any caching, this is something I should actually be able to
> try.
The part that makes me nervous is the fact that the PEP as it stands
gives the green light to an implementation having different bound method
caching behaviour between for loops and the yield-from expression.
That goes against Guido's request that the degenerate case of yield-from
have the same semantics as:
for x in subiter:
yield x
Since the language reference is actually silent on the topic of caching
the bound method when iterating over an object, I would phrase it along
the following lines:
- if for loops in a Python implementation cache next(), then yield-from
in that implementation should also cache next()
- if yield-from caches next(), it should also cache sent() and throw()
- Since CPython for loops don't cache the bound method for next(), it
won't cache the methods used by yield-from either
Who knows, maybe Guido will actually clarify the matter for us when he
gets back from his vacation :)
Cheers,
Nick.
P.S. Speaking of vacations, I'll also be offline for the next week or so
(starting tomorrow), and then my internet access for Python activities
will be sketchy for another couple of weeks as I move house. So I won't
be able to contribute much more to this discussion.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Guido van Rossum
> Jacob Holm wrote:
>> I am not really sure about this. It looks very much like an
>> implementation detail to me. On the other hand, the ability to replace
>> the methods mid-flight might give us a way to implement the
>> call-wrappers with minimal overhead. Since the current patches don't
>> actually do any caching, this is something I should actually be able to
>> try.
>
> The part that makes me nervous is the fact that the PEP as it stands
> gives the green light to an implementation having different bound method
> caching behaviour between for loops and the yield-from expression.
>
> That goes against Guido's request that the degenerate case of yield-from
> have the same semantics as:
>
> for x in subiter:
> yield x
>
> Since the language reference is actually silent on the topic of caching
> the bound method when iterating over an object, I would phrase it along
> the following lines:
>
> - if for loops in a Python implementation cache next(), then yield-from
> in that implementation should also cache next()
> - if yield-from caches next(), it should also cache sent() and throw()
> - Since CPython for loops don't cache the bound method for next(), it
> won't cache the methods used by yield-from either
In ceval.c, the FOR_ITER opcode expects the iterator on top of the
stack and calls (v->ob_type->tp_iternext)(v).
You tell me whether that is caching or not. :-)
> Who knows, maybe Guido will actually clarify the matter for us when he
> gets back from his vacation :)
Or sooner. :-)
> P.S. Speaking of vacations, I'll also be offline for the next week or so
> (starting tomorrow), and then my internet access for Python activities
> will be sketchy for another couple of weeks as I move house. So I won't
> be able to contribute much more to this discussion.
Good, let this be a general trend. :-)
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Greg Ewing
> Except you now have one extra function for exactly the same task, just
> with a different calling convention.
I don't see anything wrong with that. If you look in the
stdlib, there are plenty of places where alternative APIs
are provided for the same functionality, e.g. in the re
module you have the module-level functions as well as the
match object methods.
I would rather have a couple of functions written in a
straightforward way than rely on a magic wrapper to
artificially munge them into one. Transparent is better
than opaque.
> The "suppress initial next()" feature *would* have helped, by enabling
> you to write a generic wrapper to use at the call site that could do
> exactly that.
Now you're just moving the wrappers from one place to
another. I can write a wrapper to convert any lookahead
taking parsing function into a non-lookahead one:
def expect(f):
first = yield
return yield from f(first)
So at the cost of just one extra function, I can call
any of my parsing functions using either style.
--
Greg
Greg Ewing
> The main change I
> would like from the original version of the PEP is for caching the bound
> methods to be explicitly disallowed in order to match the behaviour of
> normal for loops.
But if I let the expansion serve as a literal specification,
it won't match the behaviour of for-loops either, because
although it doesn't cache methods, PyIter_Next isn't
exactly the same as looking up next() on the instance
either.
I definitely don't want to preclude the implementation
from using PyIter_Next, as that would be a major
performance hit in the most common case.
I also don't want to preclude caching a send() method,
because in the absence of a __send__ typeslot it's the
only way we have of improving performance.
I don't care much about throw() or close(), because
they will rarely be called anyway. But by the same
token, little would be gained by a wrapper using fancy
tricks to redirect them.
--
Greg
Greg Ewing
> Since the language reference is actually silent on the topic of caching
> the bound method when iterating over an object,
Since it's silent about that, if you write a for-loop
that relies on presence or absence of cacheing behaviour,
the result is undefined. The behaviour of yield-from
on the same iterator would also be undefined.
It's meaningless to talk about whether one undefined
construct has the same semantics as another.
--
Greg
Guido van Rossum
> Nick Coghlan wrote:
>> Since the language reference is actually silent on the topic of caching
>> the bound method when iterating over an object,
>
> Since it's silent about that, if you write a for-loop
> that relies on presence or absence of cacheing behaviour,
> the result is undefined. The behaviour of yield-from
> on the same iterator would also be undefined.
>
> It's meaningless to talk about whether one undefined
> construct has the same semantics as another.
But I wouldn't claim that the language reference being silent means
that it's undefined. If I were asked for a clarification I would say
that caching shouldn't be allowed if it changes the meaning of the
program. Python in general favors *defined* semantics over leaving
things in the gray.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Nick Coghlan
> Nick Coghlan wrote:
>
>> Since the language reference is actually silent on the topic of caching
>> the bound method when iterating over an object,
>
> Since it's silent about that, if you write a for-loop
> that relies on presence or absence of cacheing behaviour,
> the result is undefined. The behaviour of yield-from
> on the same iterator would also be undefined.
>
> It's meaningless to talk about whether one undefined
> construct has the same semantics as another.
I agree that would be true in the absence of an accepted reference
implementation (i.e. CPython) that doesn't cache the bound methods
(hence allowing one to play games with the next() method definition
while looping over an iterator).
If I understand Guido's last message correctly, this is one of the cases
where he would like the existing behaviour of the CPython implementation
to be the defined behaviour for the language as well.
Cheers,
Nick.
P.S. I created http://bugs.python.org/issue5739 as a documentation bug
pointing back to this email thread in relation to whether it is OK for a
Python implementation to cache the next() method lookup in a for loop.
P.P.S. OK, stepping away from the computer and going on vacation now... :)
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Nick Coghlan
> I definitely don't want to preclude the implementation
> from using PyIter_Next, as that would be a major
> performance hit in the most common case.
We already have a general caveat in the docs saying that an
implementation is allowed (or sometimes even required) to bypass normal
attribute lookup for special methods defined by the language. You may
want to point to that caveat from the PEP:
http://docs.python.org/reference/datamodel.html#special-method-lookup-for-new-style-classes
http://docs.python.org/3.0/reference/datamodel.html#special-method-lookup
Being able to use PyIter_Next and the various other typeslots is exactly
what that caveat is about.
One way you can make the expansion more explicit about bypassing the
instance is to write things like "type(itr).send(itr, val)" instead of
"itr.send(val)".
> I also don't want to preclude caching a send() method,
> because in the absence of a __send__ typeslot it's the
> only way we have of improving performance.
Actually, we do have another way of improving performance - add a
typeslot for it :)
That can be left until we find out whether or not the lookup of send()
becomes a performance bottleneck for yield-from usage (which I doubt
will be the case).
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------