Finally figured out member generators...
Jeff Epler
I just tried a similar example on Python 2.2.2 using "from __future__
import generators", and there was no need to introduce a separate
method, or call it in a strange way.
>>> from __future__ import generators
>>> class R10:
... def __iter__(self):
... for i in range(10): yield i
...
>>> list(R10())
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> R10().__iter__
<bound method R10.__iter__ of <__main__.R10 instance at 0x4008088c>>
>>> R10.__iter__
<unbound method R10.__iter__>
>>> r = R10()
>>> list(r.__iter__())
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> list(R10.__iter__(r))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> iter(r)
<generator object at 0x40066c00>
Jeff
Bjorn Pettersen
a generator produce the individual items...
Here's the doc-report I filed:
This should probably go at the end of 2.2.5 (LibRef).
"""If you are designing a class that should be iterable, i.e. you want
to be able to say "for x in myClass:...", and, you also want the
convenience of using a member function that is a generator to provide
the items, your class' __iter__() method should return
"MyClass.myGenerator(self)". The object returned from this call is an
iterator-generator that implements both of the required __iter__() and
next() methods.
Example:
class Range10:
def __init__(self): self.scale = 5
def __iter__(self):
return Range10.generateItems(self)
def generateItems(self):
for i in range(10): yield i * self.scale
There are a couple of subtleties here. First, only "user-defined
functions" are converted to methods when accessed through a class or
instance, i.e.myObject.foo() will extract the foo function wrap it up as
a method object, insert myObject in the argument list and call it
(LibRef: 3.2/Callable types/User-defined methods). [thanks to Alex M.
for clarifying this issue for me]
This automatic conversion does not happen for any other type of objects
defined inside class scope. In our case, generateItems() will be a
generator-iterator, i.e. not a "user-defined function". Thus the
conversion to an instance method does not happen, and it is left as a
static method of the class. [this seems like it might be a bug to
me...].
To get around this irregularity, make sure your __iter__() calls
generateItems() as a static method, and explictly pass self.
"""
-- bjorn
Steven Taschuk
> In case anyone else has struggled to make their class iterable by making
> a generator produce the individual items...
> Example:
> class Range10:
> def __init__(self): self.scale = 5
> def __iter__(self):
> return Range10.generateItems(self)
> def generateItems(self):
> for i in range(10): yield i * self.scale
Why is the Range10.generateItems(self) dance necessary? (I didn't
follow your explanation in the part I snipped.) The obvious code
seems to work fine:
>>> from __future__ import generators
>>> class spam:
... def __iter__(self):
... return self.generateItems()
... def generateItems(self):
... for i in range(10):
... yield 5*i
...
>>> s = spam()
>>> s.generateItems
<bound method spam.generateItems of <__main__.spam instance at 0x8115de4>>
>>> iter(s)
<generator object at 0x814e228>
>>> [x for x in s]
[0, 5, 10, 15, 20, 25, 30, 35, 40, 45]
Am I misunderstanding something?
--
Steven Taschuk stas...@telusplanet.net
Every public frenzy produces legislation purporting to address it.
(Kinsley's Law)
Bjorn Pettersen
> Which version of Python are you using?
>
> I just tried a similar example on Python 2.2.2 using "from __future__
> import generators", and there was no need to introduce a separate
> method, or call it in a strange way.
>
> >>> class R10:
> ... def __iter__(self):
> ... for i in range(10): yield i
> ...
> >>> list(R10())
> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Thanks! I guess I got completely confused by the docs... According to
LibRef 2.2.5, __iter__() should return an _object_ that is "required to
support the iterator protocol". The iterator object protocol requires:
__iter__(): which should return the iterator object itself
next(): which should return the next object or raise StopIteration
I took that to mean that classes that wanted to be their own iterators
should "return self" in their __iter__() method [after all self
implements both __iter__ and next], and have next() produce successive
values on iterative calls until finally raising StopIteration. Like,
e.g.:
>>> class D:
... def __init__(self): self.v = 5
... def __iter__(self): return self
... def next(self):
... for i in range(10):
... yield i + self.v
...
>>> d = D()
>>> list(d)
which will recurse infitely. That __iter__ should return self.next(),
thus creating and returning an iterator object because of generator
semantics didn't occur to me since the whole point was to have the class
be it's own iterator... Oh, well, it was fun anyways :-)
-- bjorn
Steven Taschuk
[...]
> >>> class D:
> ... def __init__(self): self.v = 5
> ... def __iter__(self): return self
> ... def next(self):
> ... for i in range(10):
> ... yield i + self.v
> ...
> >>> d = D()
> >>> list(d)
>
> which will recurse infitely. That __iter__ should return self.next(),
> thus creating and returning an iterator object because of generator
> semantics didn't occur to me since the whole point was to have the class
> be it's own iterator... Oh, well, it was fun anyways :-)
Um. I think you're conflating iterators with iterables. In the
code above, if D.__iter__ is changed to
return self.next()
as you suggest, then instances of D will be iterable, but they
will not be iterators, since the next method does not return
elements (it returns a generator object).
Iterables must supply __iter__(), returning an iterator; iterators
must supply next(), returning successive elements of the sequence
at each call. Iterators are stateful by specification; iterables
need not be.
Iterables may be used in for loops and whatnot; the loop obtains
(by calling __iter__) an iterator for the iterable and uses that
iterator's next() method to obtain elements.
(The distinction is somewhat obscured by the fact that iterators
themselves are iterable.)
In particular, if you just want to be able to write
d = D()
for x in d:
...
then you want instances of D to be iterables, not necessarily
iterators. In this case you could write, for example:
class SpamIterable:
def __init__(self, scale):
self.scale = scale
def __iter__(self):
return SpamIterator(self)
class SpamIterator:
def __init__(self, iterable):
self.d = d
self.val = 0
def __iter__(self):
return self
def next(self):
if self.val >= 5:
raise StopIteration()
val = self.val
self.val += 1
return val*self.d.scale
Thus:
>>> x = SpamIterable(3)
>>> x
<foo.SpamIterable instance at 0x815565c>
>>> iter(x)
<foo.SpamIterator instance at 0x8155a04>
>>> list(x)
[0, 3, 6, 9, 12]
The list() call obtains (by calling SpamIterable.__iter__) a
SpamIterator object, then uses that object's next method to obtain
the elements. Note that the iterable and iterator objects are
distinct.
A generator function, as you say, is handy here, because it lets
you avoid writing a separate iterator class:
class WithGenerator:
def __init__(self, scale):
self.scale = scale
def __iter__(self):
for i in range(5):
yield i*self.scale
But the iterator is still a distinct object; it's just a generator
object instead of an instance of a class you wrote yourself:
>>> x = WithGenerator(3)
>>> x
<foo.WithGenerator instance at 0x81212a4>
>>> g = iter(x)
>>> g
<generator object at 0x8155aa0>
>>> while 1:
... g.next()
...
0
3
6
9
12
Traceback (most recent call last):
File "<stdin>", line 2, in ?
StopIteration
>>> list(x)
[0, 3, 6, 9, 12]
--
Steven Taschuk "The world will end if you get this wrong."
stas...@telusplanet.net (Brian Kernighan and Lorrinda Cherry,
"Typesetting Mathematics -- User's Guide")
Bjorn Pettersen
> Quoth Bjorn Pettersen:
> [...]
> > >>> class D:
> > ... def __init__(self): self.v = 5
> > ... def __iter__(self): return self
> > ... def next(self):
> > ... for i in range(10):
> > ... yield i + self.v
> > ...
> > >>> d = D()
> > >>> list(d)
> >
> > which will recurse infitely. That __iter__ should return
> > self.next(), thus creating and returning an iterator
> > object because of generator semantics didn't occur to
> > me since the whole point was to have the class
> > be it's own iterator... Oh, well, it was fun anyways :-)
>
> Um. I think you're conflating iterators with iterables. In the
> code above
Well, yes, I guess I gave it away when I said "the whole point was to
have the class be it's own iterator" <wink>. Although it would certainly
cause problems with multiple simultaneous iterations over the same
container, there shouldn't be anything wrong with doing it when you know
there can only be one for loop (etc.) working on the container.
The way I read the docs, the iterator protocol was simply (a) an
iterable container must have a method __iter__(self), which (b) returns
an object having a method next(self), which when repeadtedly called will
return successive items of the container, and finally (c) next() will
raise StopIteration when all items in the container have been returned.
Here's an example:
>>> class E:
... def __init__(self):
... self.value = range(10)
... self.scale = 5
... def __iter__(self):
... self.cur = 0
... return self
... def next(self):
... pos = self.cur
... self.cur += 1
... if pos < len(self.value):
... return self.value[pos] * self.scale
... else:
... raise StopIteration
...
>>> e = E()
>>> list(e)
[0, 5, 10, 15, 20, 25, 30, 35, 40, 45]
The problem with the exmple the top, was simply that by specifying self
as the __iter__, I was implying that d.next() would return the next item
in D [which I think is perfectly reasonable, I mean it says it's going
to "yield" the value right there, you'd almost think there were some
magic wrapping going on -- who was it that said it wouldn't confuse us
to reuse 'def' to declare generators?..... (tim?) <wink>]
The reality is, of course, that d.next() returns not an item in D, but
an (generator-)iterator (an object containing next() that will produce
the items).
[... how to use separate iterator classes..]
I agree that separate iterator classes are very explicit and have a
number of good qualities, they seem heavy-handed when it comes to Python
though (remember we came from the __getitem__ protocol...) Also, I would
probably make the iterator an inner class (doesn't pollute the
namespace, and just have a look at the C++ stl for what you can do when
common parts of classes are named the same :-):
class MyContainer:
class iterator:
def __init__(self):...
def __iter__(self): return iterator(self)
def next(self): ...
etc.
[...]
> A generator function, as you say, is handy here, because it lets
> you avoid writing a separate iterator class:
>
> class WithGenerator:
> def __init__(self, scale):
> self.scale = scale
> def __iter__(self):
> for i in range(5):
> yield i*self.scale
Agreed. I haven't used generators much until now (at least not in Python
:-), but I think we're going to move to 2.3 final where they're
standard, etc., etc.
Thanks,
-- bjorn
Steven Taschuk
[...]
> >>> class E:
> ... def __init__(self):
> ... self.value = range(10)
> ... self.scale = 5
> ... def __iter__(self):
> ... self.cur = 0
> ... return self
> ... def next(self):
> ... pos = self.cur
> ... self.cur += 1
> ... if pos < len(self.value):
> ... return self.value[pos] * self.scale
> ... else:
> ... raise StopIteration
> ...
> >>> e = E()
> >>> list(e)
> [0, 5, 10, 15, 20, 25, 30, 35, 40, 45]
This implementation has the unusual property of being implicitly
reset by attempts to iterate over it. This can be surprising if
one is used to the normal iterators; for instance:
>>> it = iter(range(10))
>>> for value in it:
... if value > 5:
... break
...
>>> value
6
>>> list(it)
[7, 8, 9]
Note that this normal iterator preserves its state between
iterations over it. Among other things, this makes it possible to
consume some elements from an iterator, then pass it to some other
code for processing of the remaining elements.
Instances of your class E, however, do not behave this way:
>>> it = iter(E())
>>> for value in it:
... if value > 20:
... break
...
>>> value
25
>>> list(it)
[0, 5, 10, 15, 20, 25, 30, 35, 40, 45]
Afaik the docs don't explicitly prescribe the normal behaviour,
but imho it's the most natural way for an iterator to behave, and
writing an iterator which behaves differently is asking for
trouble.
I don't mean to suggest that iterable objects should never be
their own iterators; on the contrary, for certain stateful objects
this is quite natural. But even for such objects, calling
__iter__ should imho not reset the iteration; if resetting makes
sense for the object in question, it should be a separate method
that the client code must explicitly call.
> The problem with the exmple the top, was simply that by specifying self
> as the __iter__, I was implying that d.next() would return the next item
> in D [...]
> The reality is, of course, that d.next() returns not an item in D, but
> an (generator-)iterator (an object containing next() that will produce
> the items).
Exactly.
> [... how to use separate iterator classes..]
>
> I agree that separate iterator classes are very explicit and have a
> number of good qualities, they seem heavy-handed when it comes to Python
[...]
Absolutely. That example was just for explanatory purposes, to
illustrate that the iterable and iterator contracts are
conceptually separate. In practice a generator function with its
implicit iterator is usually more pleasing (if only because, in
2.2.2 at least, it requires a future statement, which is the
closest PSU wannabes like me get to real time travel).
--
Steven Taschuk o- @
stas...@telusplanet.net 7O )
" (
Greg Ewing (using news.cis.dfn.de)
> Well, yes, I guess I gave it away when I said "the whole point was to
> have the class be it's own iterator" <wink>. Although it would certainly
> cause problems with multiple simultaneous iterations over the same
> container, there shouldn't be anything wrong with doing it when you know
> there can only be one for loop (etc.) working on the container.
But if you're using a generator to do the iterating,
you're (implicitly) creating a separate iterator
object anyway, so there's no point in trying to make
the class be its own iterator.
There would only be a point to it if you *weren't*
using a generator, and explicitly maintaining the
iteration state yourself, in which case it might
be marginally simpler to keep that state in the
iterable instead of a separate iterator object.
But I think I would only be inclined to do that if
the object was by its very nature only iterable
once.
Particularly since generators now make it so easy
to implement separate iterators properly!
--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg