PEP 288: Generator Attributes
Bengt Richter
===================================
`PEP 288: Generator Attributes`__
===================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030321.html
Raymond Hettinger has revised `PEP 288`_ with a new proposal on how to
pass things into a generator that has already started. He has asked
for comments on the changes, so let him know what you think.
.. _PEP 288: http://www.python.org/peps/pep-0288.html
so ok, here's what I think ;-)
I like the general *idea* of __self__. I.e., I think
it could be useful for every object to be able to refer to
itself efficiently without having to do it via a global
name bound to itself.
But this particular use of __self__ seems to me suboptimal.
I gather the purpose is to be able to pass data to a generator
in a way that substitutes for not being able to do it through
the .next() method call. But you can already do something so
close to the suggested use that IMO a change is not worth it
for just that.
I.e., the example in PEP 288 is
_______________________________________________________
def mygen():
while True:
print __self__.data
yield None
g = mygen()
g.data = 1
g.next() # prints 1
g.data = 2
g.next() # prints 2
_______________________________________________________
and by just passing a mutable argument, you can write the
generator code body *exactly* the same, and write the
calling code practically the same:
>>> from __future__ import generators
>>> def mygen(__self__):
... while True:
... print __self__.data
... yield None
...
>>> class NS: pass
...
>>> d = NS()
>>> g = mygen(d)
>>> d.data = 1
>>> g.next()
1
>>> d.data = 2
>>> g.next()
2
I'd rather see generators extended in a backwards compatible way to allow you to
write the minimal example thus:
def mygen(data):
while True:
print data
yield None
g = mygen.iter() # mygen.__class__.iter() gets called with the function instance
g(1) # prints 1 (arg name data is rebound to 1, then g.next() effect is done)
g(2) # prints 2 (similarly)
but also allowing something with varying arguments, e.g.,
def mygen(*args, **kwds):
...
g = mygen.iter()
first_result = g(first_args, blah)
second_result = g(other_args, foo, bar)
third_result = g(a_kw_arg=123)
fourth_result = g() # also ok
I.e., in addition to having a .next() method for backwards compatibility,
the generator object would have a def __call__(self, *args, **kwargs) method,
to make the generator look like a callable proxy for the function.
When the generator's __call__ method was called, it would rebind the associated
function's arg locals each time, as if calling with arg unpacking like mygen(*args, **kwargs)
-- but then resuming like .next()
Possibly g() could be treated as another spelling of g.next(), skipping arg name rebinding.
A default argument should be allowable, and def mygen(x, y=123): ... would have
x and y rebound as you would expect on the first call, at least. You could argue
whether to use the original default for subsequent calls or allow a rebinding
of a default arg name to act like a default for a subsequent call, but I think
optional args should remain optional for all calls.
Note that a generator could now be also used with map, reduce, and filter as functions
as well as sequence sources. E.g., IWT you could write compile as a string filter.
For backwards compatibility. mygen.iter() would have to set a flag or whatever so
that the first g() call would not call mygen and get a generator, but would just
bind the mygen args and "resume" like .next() from the very start instead.
Calls to g.next() could be intermixed and would just bypass the rebinding of the
mygen arg names.
Another possibility you might get from using mygen.iter() rather than a strange (ISTM)
initial function call to create a generator is that you wouldn't be dependent
on yield-detecting lookahead creating a function with builtin weirdness.
I.e., maybe .iter() could be generalized as a factory method of the function class/type,
which might mean that you could set up to capture the frame of an ordinary function
and allow yields from nested calls -- which would open more multitasking possibilities
using generators.
I.e., yield would look down the stack until it found the first call of a __call__ of
a generator, and apparently return from that, while the generator kept the whole
stack state from the yield down to itself for continuing when called again.
Also, it seems like this would make a generator *relate to and control* an associated
normal function instead of *being* a weirded-up function. And yield would be decoupled
enough that you could probably write exec 'yield' and have it work from a dynamically
determined place, so long as there was the frame from a call to a generator's __call__
somewhere on the stack.
Sorry about handwave content in the above, but the possibilities seem tantalizing ;-)
[posted to c.l.p]
Regards,
Bengt Richter
Bengt Richter
[...]
>
>Note that a generator could now be also used with map, reduce, and filter as functions
>as well as sequence sources. E.g., IWT you could write compile as a string filter.
>
recursive, even involving other recursive functions recursively calling back!
This would be a consequence of changing yield to act according to its dynamic position in
execution. I.e., yield becomes like raising an exception without unwinding the stack, and
instead tracing it down to the first[1] generator __call__ frame.
[1] Hm... I wonder if a targeted yield that could find a particular generator __call__ frame
would be of use. You could specify an optional name in the call to the generator
factory method, which would default to the name of the function instance passed to it.
Then you could potentially suspend a context with one or more internal generators running.
This could be nice for coroutine multitasking, to yield back to a scheduler without having
to give up the state of active generators within tasks.
You'd need syntax to use it, e.g., maybe
yield some_value as 'name_string' # re-using 'as'
e.g., alluding to foo function-result in "as foo" when returning to the generator __call__
context that made the primary call to foo.
Regards,
Bengt Richter
Raymond Hettinger
> I'd rather see generators extended in a backwards compatible way to allow
you to
> write the minimal example thus:
>
> def mygen(data):
> while True:
> print data
> yield None
>
> g = mygen.iter() # mygen.__class__.iter() gets called with
the function instance
> g(1) # prints 1 (arg name data is rebound to 1,
then g.next() effect is done)
> g(2) # prints 2 (similarly)
So, the behavior of mygen() is radically different depending on whether it
is called with iter(mygen()) or mygen.iter(). It looks like the parameter
string is being asked to pull double duty. How would you code an equivalent
to:
def logger(afile, author):
while True:
print >> file, __self__.data, ':changed by', author
yield None
> but also allowing something with varying arguments, e.g.,
>
> def mygen(*args, **kwds):
Lesson 1 from a PEP author: the more you include in
a proposal, the less likely you are to get any of it.
> I.e., in addition to having a .next() method for backwards compatibility,
> the generator object would have a def __call__(self, *args, **kwargs)
method,
> to make the generator look like a callable proxy for the function.
Lesson 2: the more clever the proposal, the more likely
it is to be deemed unpythonic.
> When the generator's __call__ method was called, it would rebind the
associated
> function's arg locals each time, as if calling with arg unpacking like
mygen(*args, **kwargs)
> -- but then resuming like .next()
See Lesson 1.
>
> Possibly g() could be treated as another spelling of g.next(), skipping
arg name rebinding.
See Lesson 2
> A default argument should be allowable, and def mygen(x, y=123):
See Lesson 1
.> .. would have
> x and y rebound as you would expect on the first call, at least. You could
argue
> whether to use the original default for subsequent calls or allow a
rebinding
> of a default arg name to act like a default for a subsequent call, but I
think
> optional args should remain optional for all calls.
See Lesson 2 ;)
>
> Note that a generator could now be also used with map, reduce, and filter
as functions
> as well as sequence sources. E.g., IWT you could write compile as a string
filter.
>
> For backwards compatibility. mygen.iter() would have to set a flag or
whatever so
> that the first g() call would not call mygen and get a generator, but
would just
> bind the mygen args and "resume" like .next() from the very start instead.
I'm sure there's another lesson here too ;)
> Calls to g.next() could be intermixed and would just bypass the rebinding
of the
> mygen arg names.
Really?
> I.e., maybe .iter() could be generalized as a factory method of the
function class/type,
> which might mean that you could set up to capture the frame of an ordinary
function
> and allow yields from nested calls -- which would open more multitasking
possibilities
> using generators.
Yes!
See my cookbook recipe for an example.
It implements most of the PEP 288 in pure python using a factory function.
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/164044
> I.e., yield would look down the stack until it found the first call of a
__call__ of
> a generator, and apparently return from that, while the generator kept the
whole
> stack state from the yield down to itself for continuing when called
again.
>
> Also, it seems like this would make a generator *relate to and control* an
associated
> normal function instead of *being* a weirded-up function. And yield would
be decoupled
> enough that you could probably write exec 'yield' and have it work from a
dynamically
> determined place, so long as there was the frame from a call to a
generator's __call__
> somewhere on the stack.
See PEP 667, proposed mandatory drug testing for pythonistas ;)
Raymond Hettinger
Denis S. Otkidach
RH> So, the behavior of mygen() is radically different depending
RH> on whether it
RH> is called with iter(mygen()) or mygen.iter(). It looks
RH> like the parameter
RH> string is being asked to pull double duty. How would you
RH> code an equivalent
RH> to:
RH>
RH> def logger(afile, author):
RH> while True:
RH> print >> file, __self__.data, ':changed by', author
RH> yield None
class logger(object):
def __init__(self, afile, author):
self.afile = afile
self.author = author
self.data = 'default data'
self.next = self._gen().next
def _gen(self):
while True:
print >> self.afile, self.data, ':changed by', self.author
yield None
--
Denis S. Otkidach
http://www.python.ru/ [ru]
Bengt Richter
>[Bengt Richter]
>> I'd rather see generators extended in a backwards compatible way to allow
>you to
>> write the minimal example thus:
>>
>> def mygen(data):
>> while True:
>> print data
>> yield None
>>
>> g = mygen.iter() # mygen.__class__.iter() gets called with
>the function instance
>> g(1) # prints 1 (arg name data is rebound to 1,
>then g.next() effect is done)
>> g(2) # prints 2 (similarly)
>
>So, the behavior of mygen() is radically different depending on whether it
>is called with iter(mygen()) or mygen.iter(). It looks like the parameter
>string is being asked to pull double duty. How would you code an equivalent
I'm not sure how you mean "double duty," but yes backwards compatibility would
force a radical difference. (Hm, since it is in the __future__, would it be
technically possible to drop the old way altogether?)
>to:
>
>def logger(afile, author):
> while True:
> print >> file, __self__.data, ':changed by', author
> yield None
well, that's a matter of choosing how you want to pass the data
you are now passing via attribute manipulations separate from calls.
I'd probably just pass it as plain parameter with the others as
optional parameters. Remember, I am proposing to (re)bind the
parameter names each time the generator is called with parameters,
just before resuming execution on the line after the last yield.
def logger(data, file=None, author=None):
while data:
print >> file, data, ':changed by', author
yield None
g = logger.iter()
g(some_data, file('log_file.txt','a'), 'Author Name') # actual param bindings replace opt params
g(other_data) # ditto, but opt param bindings are undisturbed
...
g(None) # trigger end -- just an example of one possible way of doing it
>
>
>
>> but also allowing something with varying arguments, e.g.,
>>
>> def mygen(*args, **kwds):
>
>Lesson 1 from a PEP author: the more you include in
>a proposal, the less likely you are to get any of it.
>
Even if all the pieces are required to make the whole work nicely?
>
>> I.e., in addition to having a .next() method for backwards compatibility,
>> the generator object would have a def __call__(self, *args, **kwargs)
>method,
>> to make the generator look like a callable proxy for the function.
>
>Lesson 2: the more clever the proposal, the more likely
>it is to be deemed unpythonic.
It's not meant to be "clever" for its own sake. But if g.next(params) is
easier to accept than g(params), I guess I can live with it ;-)
However, there is the compatibility issue, and I suspect it would be easier
to leave g.next() alone and parameterless, and to get the effects of the
first weird function call (which apparently currently binds parameters for
subsequent use and returns the generator object) separately. I.e.,
"g = foo.iter(); g(parameters)" does the same parameter binding and execution
as the old "g=foo(parameters); g.next()". If you then continue with another
g.next(), you resume after the last yield and can refer to the parameter name
locals as they were when that last yield was executed. This would be the same
either way. But if instead of g.next() you did g(new_parameters), the new parameters
would be received by g.__call__ and before resuming foo at the line after the
last yield, the generator would reach into the frame state and rebind the parameter
locals as if g(new_parameters) were a foo(new_parameters) call. Then the resume
would be effected, just like g.next().
>
>
>> When the generator's __call__ method was called, it would rebind the
>associated
>> function's arg locals each time, as if calling with arg unpacking like
>mygen(*args, **kwargs)
>> -- but then resuming like .next()
>
>See Lesson 1.
But this is part of making it all hang together.
>
>>
>> Possibly g() could be treated as another spelling of g.next(), skipping
>arg name rebinding.
>
>See Lesson 2
Ok, that is an unnecessay trick. All parameters can be made optional to allow g(),
just like an ordinary function.
>
>> A default argument should be allowable, and def mygen(x, y=123):
>
>See Lesson 1
Also part of making it all work, and if the parameter name binding is
done like for an ordinary function call, combined with the persistence
of locals in the generator state, it should just naturally follow.
>
>
>.> .. would have
>> x and y rebound as you would expect on the first call, at least. You could
>argue
>> whether to use the original default for subsequent calls or allow a
>rebinding
>> of a default arg name to act like a default for a subsequent call, but I
>think
>> optional args should remain optional for all calls.
>
>See Lesson 2 ;)
Again, I think it would just follow from doing the param rebinding in a
natural way. I'm thinking rebound optional parameters should persist in
the generator state and act like they would as default values in the first
call. Also a part of making the whole thing useful.
>
>
>>
>> Note that a generator could now be also used with map, reduce, and filter
>as functions
>> as well as sequence sources. E.g., IWT you could write compile as a string
>filter.
>>
>> For backwards compatibility. mygen.iter() would have to set a flag or
>whatever so
>> that the first g() call would not call mygen and get a generator, but
>would just
>> bind the mygen args and "resume" like .next() from the very start instead.
>
>I'm sure there's another lesson here too ;)
>
Orthogonality breeds? ;-)
>
>> Calls to g.next() could be intermixed and would just bypass the rebinding
>of the
>> mygen arg names.
>
>Really?
The way I'm thinking of it, yes. But not if g.next() takes parameters and
is used in place of g.__call__. But if you used g.next() after g=foo.iter()
without either having default parameters for foo or a g(some_params) call
to make bindings, you would have a possible use-before-set error.
>
>
>> I.e., maybe .iter() could be generalized as a factory method of the
>function class/type,
>> which might mean that you could set up to capture the frame of an ordinary
>function
>> and allow yields from nested calls -- which would open more multitasking
>possibilities
>> using generators.
>
>Yes!
>See my cookbook recipe for an example.
>It implements most of the PEP 288 in pure python using a factory function.
>http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/164044
Interesting. But I want the generator to be a controlling wrapper for
a function and its potential tree of nested calls rather than conceiving
of a particular function as transmogrified by presence of yield, and all
the rest of current limitations.
I see your recipe as a clever workaround for (some) current generator shortcomings,
whereas I'd like to dream of a better generator. See PEP 667, right? ;-/
>
>
>> I.e., yield would look down the stack until it found the first call of a
>__call__ of
>> a generator, and apparently return from that, while the generator kept the
>whole
>> stack state from the yield down to itself for continuing when called
>again.
>>
>> Also, it seems like this would make a generator *relate to and control* an
>associated
>> normal function instead of *being* a weirded-up function. And yield would
>be decoupled
>> enough that you could probably write exec 'yield' and have it work from a
>dynamically
>> determined place, so long as there was the frame from a call to a
>generator's __call__
>> somewhere on the stack.
>
>See PEP 667, proposed mandatory drug testing for pythonistas ;)
Which aspect strikes you that way, the technical or the logistic?
Regards,
Bengt Richter
logistix
implements the iterator interface? I always assumed that generators
were just shorthand for an iterable (if thats a word) class.
>>> class foo:
... def __iter__(self):
... return self
... def next(self):
... print self.data
... def __init__(self):
... self.data = 1
...
>>> x = iter(foo())
>>> x.next()
1
>>> x.next()
1
>>> x.data = 3
>>> x.next()
3
>>> y = iter(foo())
>>> y.next()
1
Of course Generator Attributes would also be shorthand, but I don't
think it'd be good shorthand. It'd allow you to do many unobvious
things with your generators.
Bengt Richter
>What advantage do Generator Attributes have over a class that
>implements the iterator interface? I always assumed that generators
>were just shorthand for an iterable (if thats a word) class.
>
A generator remembers a different kind of state between calls to .next().
The class iterable remembers whatever attributes you attach to "self"
and that's it (other than globals and special stuff like default method
parameter values etc).
But a generator is like a function that remembers its local variables and
also its state of execution control at special points (namely where
the yield statements are). It remembers where it was at the last yield
that it executed to return a value. The call to .next() doesn't just
execute a defined method like the one in your class foo -- it continues
on the line after the last yield. That's a big difference!! It means you
can return a value from inside a loop and continue on, e.g., consider
a simple state machine parsing a string into a sequence of names and numbers:
(Note that "names" can have trailing digits here, and numbers are unsigned integers)
>>> from __future__ import generators
>>> def namenum(s):
... state = '?'
... sl = list(s); sl.reverse()
... while sl:
... c = sl.pop()
... if state=='?':
... if c.isspace(): continue
... if c.isdigit(): state = 'N'; n = int(c)
... else: state = 'T'; sout = [c]
... elif state == 'N':
... if not c.isdigit() or not sl:
... yield n
... state = '?'; sl.append(c) # reuse non-digit
... else:
... n = n*10 + int(c)
... else: #T
... if not c.isspace(): sout.append(c)
... if c.isspace() or not sl:
... yield ''.join(sout)
... state = '?'
...
>>> g = namenum(' hi ho 123 4 five5 6six see?')
>>> [x for x in g]
['hi', 'ho', 123, 4, 'five5', 6, 'six', 'see?']
>>>
The above could be implemented as a class supporting iter, but
it wouldn't be quite as easy to code. It could probably be
done as a one-line list comprehension too, but it's nice to
have more than one way to do it (;-) so you can choose one
that expresses the semantics best (to a human).
>>>> class foo:
>... def __iter__(self):
>... return self
>... def next(self):
>... print self.data
>... def __init__(self):
>... self.data = 1
>...
>>>> x = iter(foo())
>>>> x.next()
>1
>>>> x.next()
>1
>>>> x.data = 3
>>>> x.next()
>3
>>>> y = iter(foo())
>>>> y.next()
>1
>
>
>Of course Generator Attributes would also be shorthand, but I don't
>think it'd be good shorthand. It'd allow you to do many unobvious
>things with your generators.
Passing parameters into a running generator or iterator is a special concern
that should not obscure the other important differences and commonalities, IMO.
Regards,
Bengt Richter
Rocco Moretti
> I saw this in the python-dev summary:
>
> ===================================
> `PEP 288: Generator Attributes`__
> ===================================
The attribute passing scheme seems to be complex and cumbersome. The
process of set attributes -> call processing function -> repeat is an
akward way of achieving what you really want to do - continue the
generator with given data. I'd agree with what Guido said on
Python-Dev: there is no advantage in this scheme over passing a
mutable dummy object when the generator is created (you could even
call it __self__ if you really wanted too ...).
That said, I'm still not sure what the actual objections to the
previous (rejected) alternative are. I think that the reason for the
empty/ignored .next() is obvious if you understand how generators
work, and I question why this is a show-stopper issue (you'll never
see it if you never pass values to generators). I get the impression
that there are other reasons people have for being against using
.next() (and generator value passing in general) that aren't mentioned
in the PEP - it would be nice to get them down for posterity.
I would also quibble with RH's assertation that the unused first
.next(value) problem is "intractable." If we're open to using magic
locals (as in the __self__ value proposed), you could just use a magic
local to store the parameter passed to the generator (__genval__?,
__nextval__?). If you also allowed yield to return the value as well
(as in: invalue = yield outvalue) you could (in most cases) circumvent
Guido's complaint that "__self__ is butt-ugly."
As for GvR's issue of why generator passing is better than passing a
dummy object at the creation of the generator, it is, in essence, the
same argument for why true object orientation is better than emulating
it in C: when presented simply, the concept flows more naturally.
(Note the following argument flownders a bit for the current PEP) The
.next(value) passing method frames itself as a symetric dialog between
two functions (I give you data, you reply, I give you data ...).
Whereas the using of a dummy object presents itself more as an
asymetric, type-up-a-memo-and-post-it-on-the-bulletin-board type
arrangement. Besides- it adds another entity you have to keep track
of.
sig.__Limerick__ = '''
There once was a young man from Occam,
Whose rhetorical theories would shock them.
"When entities multiply,
Your arguments die.
So please use my razor to dock them."
'''
P.S. I believe further PEP division may be called for. Value passing
and exception throwing in generators are synergystic, yes, but can be
implemented independantly if need be. The chance of passing at least
one may be increased if split into separate PEPs, instead of having
their separate destinies thrown into a common PEP number.
John Roth
"Rocco Moretti" <roccom...@netscape.net> wrote in message
news:ff133276.0212...@posting.google.com...
> bo...@oz.net (Bengt Richter) wrote in message
news:<asjh27$80v$0...@216.39.172.122>...
> > I saw this in the python-dev summary:
> >
> > ===================================
> > `PEP 288: Generator Attributes`__
> > ===================================
>
> The attribute passing scheme seems to be complex and cumbersome. The
> process of set attributes -> call processing function -> repeat is an
> akward way of achieving what you really want to do - continue the
> generator with given data. I'd agree with what Guido said on
> Python-Dev: there is no advantage in this scheme over passing a
> mutable dummy object when the generator is created (you could even
> call it __self__ if you really wanted too ...).
I thoroughly agree. I've been looking at this entire discussion
with some bemusement, because, for me at least, the most
intuitive thing to do is make yield a built in function rather than
a statement. That way, it can return a value, which is what
was passed in on the reinvocation.
The trouble with the whole thing is that conceptually, "yield" has two
values: one that it returns to the caller, and one that it
gets from the caller. It's a two way pipe.
Making yield a statement effectively forclosed the option of
having it return anything, at least without major hackery to
what a statement can do. Of course, we've got the example
of the print statement for that kind of hackery.
Terry Reedy
"John Roth" <john...@ameritech.net> wrote in message
news:uv94kkq...@news.supernews.com...
>
> "Rocco Moretti" <roccom...@netscape.net> wrote
> > process of set attributes -> call processing function -> repeat is
an
> > akward way of achieving what you really want to do - continue the
> > generator with given data. I'd agree with what Guido said on
> > Python-Dev: there is no advantage in this scheme over passing a
> > mutable dummy object when the generator is created (you could even
> > call it __self__ if you really wanted too ...).
> I thoroughly agree.
Me too.
> with some bemusement, because, for me at least, the most
> intuitive thing to do is make yield a built in function rather than
> a statement.
But then we would need a another keyword (possible, of course) to tell
the compiler to change the generated bytecode. It would also break
the parallel between 'return something' and 'yield something'. The
only difference in execution is that yield does less -- by *not*
deleting the execution frame.
> That way, it can return a value, which is what
> was passed in on the reinvocation.
One of the design features of generators is that resumption is
extremely quick precisely because the argument processing and function
setup steps are bypassed. Just restore the execution frame and go on
with the next statement [bytecode actually].
Another design feature is that they are meant to work seemlessly with
for statements, with only one explicit call (to produce the iterator
that 'for' needs). In this context, passing in additional values is
not possible.
Another problem is with multiple yields. Consider the following
hypothetical generator with the proposed yield() 'function' (see next
comment for why the '' marks):
def genf(a):
b=yield(a)
b,c=yield(a+b)
yield(a+b+c)
gen = genf(1)
Then the documentation must say: on the first call to gen.next(), pass
one arg; on the next, pass two; finally, don't pass any. Not good,
methings.
> The trouble with the whole thing is that conceptually, "yield" has
two
> values: one that it returns to the caller, and one that it
> gets from the caller. It's a two way pipe.
(You mean, 'would be'.) Pipes and such usually have explicit read and
write methods that make the order of operations clear. The yield()
proposal hijacks function notation to squash a read and write
together -- with the order switched at the two ends.
To explain: y=f(x) usually means 'set y to a value depending on
(calculated from) x' -- this is the meaning of 'function'.
Operationally, a function call means to send x to process f to
initialize the corresponding parameter, suspend operation while f
operates, and set y to the value returned by f. The combined
next(arg)/yield() idea warps and shuffles these semantics.
Let p = pipe or whatever: Then (omitting suspend) y = yield(x) could
mean
p.write(x); y=p.read()
where (contrary to the implication of function notation) the value
read cannot depend on x since it is calculated at the other end before
the write.
The corresponding x = gen.next(y) then means
x=p.read; p.write(y)
which makes the order of value passing the opposite of what a function
call means. The only reason to write y is for a possible future call
to gen.next(). I think it much better to separate the read and write
and pair the writing of y with the reading of the x that functionally
depends on the value written.
One could reverse the meaning of x = gen.next(y) to be
p.write(y); x=p.read()
but the x read would still not depend on the y written since the
latter would have to be set aside and ignored until the predetermined
x was sent back. IE, y=yield(x) would have to mean and be implemented
as
<hidden-slot> = p.read(); p.write(x); y=<hidden-slot>
In either case, there is a synchronization problem in that the values
sent to the generator are shifted by one call. The last gen.next()
call must send a dummy that will be ignored. On the other hand, if,
for instance, there is one yield function which depends on the
variable reset by the yield function, then that variable must be
separately initialized in the initial genf() call.
So it arguably makes just as much sense to initialize via genf() with
a mutable with paired write/read, send/receive, or set/get methods or
the equivalent operations (as with dicts). One can then make explicit
calls to pass data in either direction without fake 'function' calls.
If one pairs 'yield None' with 'dummy=gen.next()' or 'for dummy in
genf(mutable):', one can even do all value passing, in both
directions, via the two-way channel or mutable and use 'yield'
strictly for suspend/resume synchronization of the coroutines.
--
This proposal come close to asking for what the original Stackless did
with continuations. These allowed some mind-boggling code. Almost
too fancy for what Python is meant to be 8-).
Terry J. Reedy
Bengt Richter
>bo...@oz.net (Bengt Richter) wrote in message news:<asjh27$80v$0...@216.39.172.122>...
>> I saw this in the python-dev summary:
>>
>> ===================================
>> `PEP 288: Generator Attributes`__
>> ===================================
>
>The attribute passing scheme seems to be complex and cumbersome. The
I'm not sure why you are quoting me saying I saw a reference to PEP 288 in
the python-dev summary ;-)
>process of set attributes -> call processing function -> repeat is an
>akward way of achieving what you really want to do - continue the
>generator with given data. I'd agree with what Guido said on
I guess you noticed I was trying to propose and alternative to the attribute
methodology? (see more detail below)
>Python-Dev: there is no advantage in this scheme over passing a
>mutable dummy object when the generator is created (you could even
>call it __self__ if you really wanted too ...).
As I showed in an example, pointing this out ;-)
>
>That said, I'm still not sure what the actual objections to the
>previous (rejected) alternative are. I think that the reason for the
>empty/ignored .next() is obvious if you understand how generators
>work, and I question why this is a show-stopper issue (you'll never
>see it if you never pass values to generators). I get the impression
Right, and you need it for compatibility with for x in foo(...)
even if you deprecate that spelling in favor of for x in foo.iter(...)
(which IMO expresses the semantics better ;-).
>that there are other reasons people have for being against using
>.next() (and generator value passing in general) that aren't mentioned
>in the PEP - it would be nice to get them down for posterity.
>
>I would also quibble with RH's assertation that the unused first
>.next(value) problem is "intractable." If we're open to using magic
>locals (as in the __self__ value proposed), you could just use a magic
>local to store the parameter passed to the generator (__genval__?,
>__nextval__?). If you also allowed yield to return the value as well
>(as in: invalue = yield outvalue) you could (in most cases) circumvent
IMO "invalue = yield outvalue" is ugly for several reasons, one of them
being special casing the first invalue, but also the lack of a parameter
signature for the invalue. E.g., should it be treated as args in foo(*args)
or what? You could say that it should match the parameter list of foo,
but does that mean you should write inx, iny = yield outvalue for a generator
based on foo(x, y)? That seems silly if you could just use the original parameter
names and expect the latest values to be bound to them (which is what I proposed ;-)
when you continue at the next line after the yield.
>Guido's complaint that "__self__ is butt-ugly."
At least used for that purpose. OTOH I think it would be nice for all objects
to be able to get references to themselves in some way without looking up a
global binding for themselves. A magic __self__ doesn't seem worse than a magic
__doc__ for that, but using it to pass parameters tacked to a function's __self__
is "butt-ugly."
>
>As for GvR's issue of why generator passing is better than passing a
^^^^^^^^^^^^^^^^^ ??
>dummy object at the creation of the generator, it is, in essence, the
>same argument for why true object orientation is better than emulating
>it in C: when presented simply, the concept flows more naturally.
>(Note the following argument flownders a bit for the current PEP) The
>.next(value) passing method frames itself as a symetric dialog between
>two functions (I give you data, you reply, I give you data ...).
>Whereas the using of a dummy object presents itself more as an
>asymetric, type-up-a-memo-and-post-it-on-the-bulletin-board type
>arrangement. Besides- it adds another entity you have to keep track
>of.
Yes, and IMHO my proposal below is comparatively butt-beautiful ;-)
If foogen.gi_frame.f_locals could be modified, I could just about write a class
to implement a new style generator without modifying the old, I think.
>>> def foo(x):
... yield 1
... yield 2
... yield x
...
>>> foo
<function foo at 0x007D96E0>
>>> foogen = foo('x value')
>>> foogen.next
<method-wrapper object at 0x007D8FE0>
>>> foogen.gi_frame.f_code.co_varnames
('x',)
>>> foogen.gi_frame.f_code.co_nlocals
1
>>> foogen.gi_frame.f_code.co_argcount
1
>>> foogen.gi_frame.f_locals
{'x': 'x value'}
all you'd need is a __call__ method in the generator class,
and a modified next that would take care of the first bindings for the
ordinary case of for x in foo.iter(param):
# untested(!!), and for concept illustration mainly
class NewGen:
def __init__(self, fun, *args, **kwargs):
self.fun = fun
self.gen = None
self.iterargs = args
def __call__(self, *args, **kwargs):
if self.gen is None:
if self.iterargs!=() or self.iterkwargs!={}:
raise ValueError, 'Initial args already specified via .iter(...)'
self.iterargs = args
self.iterkwargs = kwargs
return self.next()
fc = self.gi_frame.f_code
for i in range(fc.co_argcount):
self.gi_frame.f_locals[fc.co_varnames[i]] = args[i] # read-only prevents this now
# not sure how to handle kwargs
return self.gen.next() # the old next
def next(self):
if self.gen is None: # first call args come from func.iter(func,*args)
self.gen = self.fun(*self.iterargs, **self.iterkwargs) # make generator old way
return self.gen.next()
else:
return self.gen.next()
# Also, a new method in the function class would be added
# so you can call foo.iter(parameters)
def iter(self, *args, **kwargs): # the self instance would be the function
return NewGen(self, *args, **kwargs) # will deferred-ly handle initial nonsense for backwards
# compatibility, depending on how called
Then, you could make a generator either old or new way
foogen = foo.iter('first x value') # style for use in for x in foo.iter(...)
first_value = foogen.next()
or
foogen = foo.iter() # style for explicit co-routine style calls from start
first_value = foogen('first x value')
and then be able to continue with either
next_value = foogen.next()
(which would re-use the previously bound value foogen.gi_frame.f_locals['x'])
or
next_value = foogen('new_x_arg')
which would have the x parameter bound to 'new_x_arg' when it continue from the previous yield
Old style generators could coexist without change. (In fact the above uses them).
I probably overlooked something, but I hope the idea is getting clearer ;-)
>
>sig.__Limerick__ = '''
>There once was a young man from Occam,
>Whose rhetorical theories would shock them.
>"When entities multiply,
>Your arguments die.
>So please use my razor to dock them."
>'''
>
>P.S. I believe further PEP division may be called for. Value passing
>and exception throwing in generators are synergystic, yes, but can be
>implemented independantly if need be. The chance of passing at least
>one may be increased if split into separate PEPs, instead of having
>their separate destinies thrown into a common PEP number.
Perhaps. OTOH a two-legged stool will probably sell less well than a
three-legged one ;-)
Regards,
Bengt Richter