Yielding a chain of values
Talin
output of a generator from another generator. Currently the only method
I know of to do this is to explicitly loop over the results from the
inner generator, and yield each one:
for x in inner():
yield x
I was wondering if there was a more efficient and concise way to do
this. And if there isn't, then what about extending the * syntax used
for lists, i.e.:
yield *inner()
The * would indicate that you want to iterate through the given
expression, and yield each value in turn. You could also use it on
ordinary lists:
yield *[1, 2, 3 ]
Anyway, just an idle thought on a Sunday morning :)
Peter Hansen
> I'm finding that a lot of places within my code, I want to return the
> output of a generator from another generator. Currently the only method
> I know of to do this is to explicitly loop over the results from the
> inner generator, and yield each one:
>
> for x in inner():
> yield x
>
> I was wondering if there was a more efficient and concise way to do
> this. And if there isn't, then what about extending the * syntax used
> for lists, i.e.:
>
> yield *inner()
It's not the first time it's been suggested. You can check the archives
perhaps for past discussions.
I think syntax like that could be a good idea too, for readability
reasons perhaps, but I don't think it's really important for efficiency
reasons. If you think about it, this involves one stack frame being set
up for the call to the generator, and then a series of quick context
switches (or whatever they're called in this situation) between the
current stack frame and the inner() one, as each yielded value is
produced, yield, then re-yielded up to the calling frame (with another
quick context switch). No additional stack frames are generated, and
very few byte codes are involved:
>>> def f():
... for x in inner():
... yield x
...
>>> dis.dis(f)
2 0 SETUP_LOOP 21 (to 24)
3 LOAD_GLOBAL 0 (inner)
6 CALL_FUNCTION 0
9 GET_ITER
>> 10 FOR_ITER 10 (to 23)
13 STORE_FAST 0 (x)
3 16 LOAD_FAST 0 (x)
19 YIELD_VALUE
20 JUMP_ABSOLUTE 10
...
Yes, that is some overhead, but unless you are going many levels deep
(and that's usually a design smell of some kind) this isn't likely to be
noticed amongst the rest of the code which is presumably doing something
non-trivial to produce the values in the first place, and to consume
them ultimately.
The basic loop could be handled entirely from C with an appropriate
syntax addition as you suggest, but executing those four byte code
instructions is very quick, and there is no repeated (Python) function
call overhead as you would expect if inner() were not a generator.
-Peter
Michael Hudson
> I'm finding that a lot of places within my code, I want to return the
> output of a generator from another generator. Currently the only
> method I know of to do this is to explicitly loop over the results
> from the inner generator, and yield each one:
>
> for x in inner():
> yield x
>
> I was wondering if there was a more efficient and concise way to do
> this. And if there isn't,
Greenlets, perhaps? (for which, see google).
Cheers,
mwh
--
LINTILLA: You could take some evening classes.
ARTHUR: What, here?
LINTILLA: Yes, I've got a bottle of them. Little pink ones.
-- The Hitch-Hikers Guide to the Galaxy, Episode 12
Bengt Richter
>Talin <ta...@acm.org> writes:
>
>> I'm finding that a lot of places within my code, I want to return the
>> output of a generator from another generator. Currently the only
>> method I know of to do this is to explicitly loop over the results
>> from the inner generator, and yield each one:
>>
>> for x in inner():
>> yield x
>>
>> I was wondering if there was a more efficient and concise way to do
>> this. And if there isn't,
>
>Greenlets, perhaps? (for which, see google).
>
Maybe
yield in inner()
could be sugar for the above and become something optimized?
Regards,
Bengt Richter
Reinhold Birkenfeld
The problem here is that yield isn't a statement any more. It's now an
expression, so it is not easy to find new syntax around it.
Reinhold
Bengt Richter
No, the idea was that it's still a statement, but what it
yields is "in inner()" which UIAM is illegal now, and would
signify "whatever sequence of elements is in inner()" --
really yield in seq -- I don't know what inner() was, but I assumed
an iterable.
Regards,
Bengt Richter
Peter Hansen
I believe he was referring indirectly to
http://www.python.org/peps/pep-0342.html (see item #1 in the
"Specification Summary" section), where yield will become an expression.
This PEP has been accepted, thus his use of present tense, confusing
though it is when it's not in the released version of Python yet.
Or I might be wrong. ;-)
-Peter
Bengt Richter
Well, maybe it's right both ways ;-) I.e., even though yield "is" now
an expression, it is valid to use it as an expression-statement which
evaluates the expression and discards the value. So I think you could
still use the currently illegal "yield in" token sequence to mean that
what follows is to be taken as an iterable whose full sequence is
to be yielded sequentially as if
yield in iterable
were sugar for
for _ in iterable: yield _
Regards,
Bengt Richter
Matt Hammond
> an expression, it is valid to use it as an expression-statement which
> evaluates the expression and discards the value. So I think you could
> still use the currently illegal "yield in" token sequence to mean that
> what follows is to be taken as an iterable whose full sequence is
> to be yielded sequentially as if
>
> yield in iterable
>
> were sugar for
>
> for _ in iterable: yield _
"yield in" could make sense when thought of as an expression too.
x = yield in iterable
Would behave like a list comprehension. x would be assigned a list
containing
the results of the successive yields. Equivalent to:
x = [ yield r for r in iterable ]
regards
Matt
--
| Matt Hammond
| R&D Engineer, BBC Research and Development, Tadworth, Surrey, UK.
Reinhold Birkenfeld
>> Well, maybe it's right both ways ;-) I.e., even though yield "is" now
>> an expression, it is valid to use it as an expression-statement which
>> evaluates the expression and discards the value. So I think you could
>> still use the currently illegal "yield in" token sequence to mean that
>> what follows is to be taken as an iterable whose full sequence is
>> to be yielded sequentially as if
>>
>> yield in iterable
>>
>> were sugar for
>>
>> for _ in iterable: yield _
>
> "yield in" could make sense when thought of as an expression too.
>
> x = yield in iterable
>
> Would behave like a list comprehension. x would be assigned a list
> containing
> the results of the successive yields. Equivalent to:
>
> x = [ yield r for r in iterable ]
Which is quite different from
x = (yield) in iterable
which is currently (PEP 342) equivalent to
_ = (yield)
x = _ in iterable
So, no further tinkering with yield, I'm afraid.
Reinhold
Kay Schluehr
> > x = [ yield r for r in iterable ]
>
> Which is quite different from
>
> x = (yield) in iterable
>
> which is currently (PEP 342) equivalent to
>
> _ = (yield)
> x = _ in iterable
>
> So, no further tinkering with yield, I'm afraid.
>
> Reinhold
Is the statement
yield from iterable
also in danger to be ambigous?
The resolution of "(yield) from iterable" into
_ = (yield)
x = _ from iterable
would not result in valid Python syntax.
Kay
Reinhold Birkenfeld
Right.
Problem is, how would you define the "from" syntax: Is its use as
an expression allowed? What value does it have, then?
Reinhold
Kay Schluehr
Do you mention statements like this?
x = (yield from [1,2,3])
I do think that such "yield-comprehensions" may be valid and raise a
StopIteration exception after being called three times by means of
next() or send().
Kay
vir...@gmail.com
keyword. The reasoning behind this is as follows: When calling a
function, a parameter of the form "*expression" expands to a list of
arguments. From the Python reference manual:
"If the syntax '*expression' appears in the function call, 'expression'
must evaluate to a sequence. Elements from this sequence are treated as
if they were additional positional arguments."
By (somewhat stretched) analogy, "yield *expression" would "expand" the
sequence into a series of multiple yields.
Of course, now that yield is an expression (as stated above), you would
indeed have to account for the fact that the yield statement used in
this way would yield a sequence of values:
for x in yield *[ 1, 2, 3 ]:
print x
So the '*' would not only signal that multiple values were being
yielded, but that multiple values are also being returned.
-- Talin
yai...@gmail.com
about it!
but python is all about simplicity.
with python, when I program - I don't think *about* it - I think it. or
something - don't make me think about it.
so how about a "reyield" or some other new keyword (cause reyield is
too quircky) instead of joining stuff which once ment something (one
thing)?
huh?
Yair.
yairchu a@T gmail
Kay Schluehr
auro...@gmail.com
(and that's usually a design smell of some kind)
No, its not a bug. its a feature! See the discussion in the recursive
generator thread below:
In my opinion, traversing recursive data structure is where generator
shine best. Alternative implementation using iterator is lot more
difficult and lot less elegant. Unfortunate the right now recursive
generator would carry a price tag of O(n^2) to the level of depth.
Terry Reedy
<auro...@gmail.com> wrote in message
news:1126285038....@o13g2000cwo.googlegroups.com...
The problem with asymptotic 'big O' notation is that is leaves out both the
constant multiplier and lesser terms and promotes the misperception that
'lower order' asymtotic behavior is always preferable. But much real
computation is done with small and effectively non-asymptotic values where
the omitted components *are* significant.
In this case, the O(depth^2) cost applies, I believe, to resumptions (and
yields) of suspended generators, which are *much* faster than function
calls, so that the omitted multiplier is relatively small. Given that
there is also at least one function call cost for each tree node, I expect
one would need a somewhat deep (intentionally vague without specific timing
data) and unbalanced tree for the resume price to be worrisome.
In any case, having an easily written and understood version can help in
testing a faster and more complicated version, especially on random,
non-corner case examples.
Terry J. Reedy
auro...@gmail.com
having optimized some O(n^3) SQL operations :-o !)
The things is this really should be an O(n) operation. Having a yield
all statement or expression is useful in its own right and also
potentially a way to optimized away the O(n^2) issue.