Augmented generators?

[Andrew Koenig]

Can anyone think of an easy technique for creating an object that acts like
a generator but has additional methods?

For example, it might be nice to be able to iterate through an associative
container without having to index it for each element. Right now, I can say

i = iter(d)

and then repeatedly calling i.next() gives us the keys for the elements.
But to get the corresponding value requires us to look up the key.

Of course one could define a generator that yields key-value pairs, along
the following lines:

def kviter(d):
for i in d:
yield i, d[i]

to hide the lookup. But this yields a tuple even when you don't want it.
In other words, I must now write

for k, v in kviter(d):
# whatever

and I can think of situations in which I don't really want both the key and
the value all the time.

So what I really want is something like this:

it = augiter(d)
for i in it:
if <some condition on i>:
foo(it.value())

In other words, I want "it" to support both the next and value methods (or
next and something else)

Of course I can write such a beast as a class, but that prevents me from
taking advantage of the yield statement in its implementation.

So my question is: Can you think of an easy way to write something that
looks like a generator (using yield), but can also incorporate methods other
than next?

[Paul Rubin]

"Andrew Koenig" <a...@acm.org> writes:
> Can anyone think of an easy technique for creating an object that acts like
> a generator but has additional methods?

> For example, it might be nice to be able to iterate through an associative
> container without having to index it for each element.

Normally you'd define a class and give it __iter__ and next operations.
I guess that can get messy. Does it qualify as easy for your purposes?

> Of course I can write such a beast as a class, but that prevents me from
> taking advantage of the yield statement in its implementation.

You can make an internal function that's a generator with a yield
statement (or a generator expression instead of a function, if simple
enough). The class's 'next' method would loop through the generator
and return each value from it.

Let me try your example:

class kviter:
def __init__(self, d):
self.d = d
def __iter__(self):
def iter1(d, s=self):
for k in d:
# lambda not really needed here, but you wanted value to
# be callable instead of just an attribute
s.value = (lambda r=d[k]: r)
yield k
return iter1(self.d)

a = {1:5,2:6,3:7}
it = kviter(a)
for b in it:
print b, it.value()

>>> ## working on region in file /usr/tmp/python-15915wBw...
1 5
2 6
3 7

[Lonnie Princehouse]

AFAIK there's no way to have "yield" produce anything other than a
generator.
You could achieve the syntax you want with a decorator, although under
the hood it would still be iterating over a (key,value) tuples so
there's not really any point.

class GeneratorTupleWrapper:
def __init__(self, g):
self.g = g
def next(self):
key, self._value = g.next()
return key
def value(self):
return self._value

def magic_generator (func):
def wrapper(*a, **b):
return GeneratorTupleWrapper(func(*a, **b))

@magic_generator

def kviter(d):
for i in d:
yield i, d[i]

it = kviter(d)
for key in it:
if <something>:
foo(it.value())

Also: dict.iteritems() enumerates (key,value) tuples for dictionaries,
so kviter isn't necessary if d is a dictionary. I don't know the
internal mechanism used by dict for iteritems, but it's a fair bet that
it's more efficient than performing a lookup for every single key.

[Raymond Hettinger]

Andrew Koenig wrote:
> Can anyone think of an easy technique for creating an object that acts like
> a generator but has additional methods?

Perhaps the enable_attributes() recipe will help:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/164044

The recipe imbues generators with classlike attribute reads and writes.
It shouldn't be hard to adapt the technique for attaching methods.

Raymond

[Bengt Richter]

It does? (this just occurred to me, so there may be some gotcha ;-)

>>> class augiter(object):
... def __init__(self, d): self.d = d
... def __iter__(self):
... for k in self.d: self.value = self.d[k]; yield k
...
>>> it = augiter(dict(enumerate('abcd')))
>>> for i in it:
... if i%2: print i, it.value
...
1 b
3 d

You could of course store self._value and def value(self):return self._value to be closer to your syntax)

>So my question is: Can you think of an easy way to write something that
>looks like a generator (using yield), but can also incorporate methods other
>than next?
>

See above.

Regards,
Bengt Richter

[Bengt Richter]

Sorry to reply this way. I saw this on google, but neither this nor my previous post
has shown up yet on my news server. Wonder what all this delay is lately ;-/

>From: Paul Rubin <http://phr...@NOSPAM.invalid>
>Subject: Re: Augmented generators?
>Date: 10 Jan 2006 11:03:39 -0800
>Message-ID: <7xwth7n...@ruckus.brouhaha.com>

>"Andrew Koenig" <a...@acm.org> writes:
>> Can anyone think of an easy technique for creating an object that acts like
>> a generator but has additional methods?

>> For example, it might be nice to be able to iterate through an associative
>> container without having to index it for each element.

>Normally you'd define a class and give it __iter__ and next operations.

>I guess that can get messy. Does it qualify as easy for your purposes?

>> Of course I can write such a beast as a class, but that prevents me from

>> taking advantage of the yield statement in its implementation.

>You can make an internal function that's a generator with a yield

>statement (or a generator expression instead of a function, if simple
>enough). The class's 'next' method would loop through the generator
>and return each value from it.

>Let me try your example:

This suffers from the same problem as my first go (an exhausted iterator
is not supposed to restart).

>>> class kviter:
... def __init__(self, d):
... self.d = d
... def __iter__(self):
... def iter1(d, s=self):
... for k in d:
... # lambda not really needed here, but you wanted value to
... # be callable instead of just an attribute
... s.value = (lambda r=d[k]: r)
... yield k
... return iter1(self.d)
...

Using my example with your iterator:

>>> it = kviter(dict(enumerate('abcd')))
>>> for i in it:
... if i%2: print i, it.value()

...
1 b
3 d

Ok, but this shouldn't happen at this point:

>>> for i in it:
... if i%2: print i, it.value()

...
1 b
3 d

Whereas,

>>> class augiter(object):
... def __init__(self, d):

... self._it = self._gen(d)
... def __iter__(self): return self
... def _gen(self, d):
... for k, self._v in d.items(): yield k
... def next(self): return self._it.next()
... def value(self): return self._v

...
>>> it = augiter(dict(enumerate('abcd')))
>>> for i in it:

... if i%2: print i, it.value()

...
1 b
3 d
>>> for i in it:

... if i%2: print i, it.value()
...
>>>

Letting __init__ create the generator instance by calling a bound
method coded with yield makes integrating the latter style pretty easy,
even though you still need __iter__ and next methods.

Hm, you could factor that out into a base class though:
(then you just need the convention that a derived class must define at least
the _gen method. Then augment to taste)

>>> class AugiterBase(object):
... def __init__(self, *args, **kw):
... self._it = self._gen(*args, **kw)
... def __iter__(self): return self
... def next(self): return self._it.next()
...
>>> class DervAugit(AugiterBase):
... def _gen(self, d):
... for k, self._v in d.items(): yield k
... def value(self): return self._v
...
>>> it = DervAugit(dict(enumerate('abcd')))
>>> for i in it:
... if i%2: print i, it.value()

...
1 b
3 d

>>> for i in it:
... print i, it.value()
...
>>> it = DervAugit(dict(enumerate('abcd')))
>>> for i in it:
... print i, it.value()
...
0 a
1 b
2 c
3 d

Regards,
Bengt Richter