generator functions: why won't this work?
zill...@googlemail.com
I'm trying to understand generator functions and the yield keyword.
I'd like to understand why the following code isn't supposed to work.
(What I would have expected it to do is, for a variable number of
arguments composed of numbers, tuples of numbers, tuples of tuples,
etc., the function would give me the next number "in sequence")
####################################
def getNextScalar(*args):
for arg in args:
if ( isinstance(arg, tuple)):
getNextScalar(arg)
else:
yield arg
####################################
# here's an example that uses this function:
# creating a generator object:
g = getNextScalar(1, 2, (3,4))
g.next() # OK: returns 1
g.next() # OK: returns 2
g.next() # not OK: throws StopIteration error
####################################
I'm sure I'm making some unwarranted assumption somewhere, but I
haven't been able to figure it out yet (just started learning Python a
couple of days ago).
Any help will be appreciated :)
Akiel
Mel
`getNextScalar(arg)` doesn't yield anything (it creates, but doesn't
use, a new generator object,) so nothing comes out. Look at
>>> h = getNextScalar(1,2,(3,4),5)
>>> h.next()
1
>>> h.next()
2
>>> h.next()
5
Maybe you want
if isinstance (arg, tuple):
for s in getNextScalar (*arg):
yield s
Mel.
George Sakkis
You're pretty close, there's just one more thing left. The return
value of a generator function is an iterable, something you're
supposed to iterate over. In the 'if' clause you call recursively the
generator on arg but you don't use the result, you discard it as soon
as it is returned. What you have to do is iterate over the returned
iterable and yield its values:
# don't need parentheses around the if expression
if isinstance(arg, tuple):
for scalar in getNextScalar(arg):
yield scalar
Hope this helps,
George
Steve Holden
should create it to multiple arguments with a star. Neither do you do
anything with the iterator after you create it.
Try (untested)
####################################
def getNextScalar(*args):
for arg in args:
if ( isinstance(arg, tuple)):
for a in getNextScalar(*arg):
yield a
else:
yield arg
####################################
regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/
George Sakkis
Just after hitting send I noticed a second unrelated issue that has to
do with the generator's signature. Since you define it to take any
positional arguments instead of a single argument (as I mistakenly
assumed), the right way to call it recursively is expand the tuple arg
into positional arguments: getNextScalar(*arg) instead of
getNextScalar(arg):
# don't need parentheses around the if expression
if isinstance(arg, tuple):
for scalar in getNextScalar(*arg):
yield scalar
George
Gabriel Genellina
> I'm trying to understand generator functions and the yield keyword.
> I'd like to understand why the following code isn't supposed to work.
> (What I would have expected it to do is, for a variable number of
> arguments composed of numbers, tuples of numbers, tuples of tuples,
> etc., the function would give me the next number "in sequence")
> ####################################
> def getNextScalar(*args):
> for arg in args:
> if ( isinstance(arg, tuple)):
> getNextScalar(arg)
> else:
> yield arg
> ####################################
You're not the first one in getting confused. After all, this schema works
well for other recursive constructs.
Perhaps a progression of working code samples will help to understand what
happens here. The simplest thing would be to just print the items as
they're encountered, in a recursive call:
py> data = (1, 2, (3,4,(5,6),7))
py>
py> print "1) using print"
1) using print
py>
py> def getNextScalar(args):
... for arg in args:
... if isinstance(arg, tuple):
... getNextScalar(arg)
... else:
... print arg
...
py> getNextScalar(data)
1
2
3
4
5
6
7
Now one could try to collect the numbers in a list:
py> print "2) using extend"
2) using extend
py>
py> def getNextScalar(args):
... result = []
... for arg in args:
... if isinstance(arg, tuple):
... result.extend(getNextScalar(arg))
... else:
... result.append(arg)
... return result
...
py> getNextScalar(data)
[1, 2, 3, 4, 5, 6, 7]
Note that we use two different list methods: for individual items, we use
"append", but for tuples we use "extend" in the recursive call. If extend
weren't available, we could emulate it with append:
py> print "3) using append"
3) using append
py>
py> def getNextScalar(args):
... result = []
... for arg in args:
... if isinstance(arg, tuple):
... for item in getNextScalar(arg):
... result.append(item)
... else:
... result.append(arg)
... return result
...
py> getNextScalar(data)
[1, 2, 3, 4, 5, 6, 7]
See how we need an additional loop to iterate over the results that we get
from the recursive call.
Now instead of building an intermediate result list, we delegate such task
over the caller, and we use a generator that just yields items; this way,
we remove all references to the result list and all result.append calls
become yield statements. The inner loop has to remain the same. The yield
statement acts somewhat as an "append" over an outer list created by the
generator's caller.
py> print "4) using yield"
4) using yield
py>
py> def getNextScalar(args):
... for arg in args:
... if isinstance(arg, tuple):
... for item in getNextScalar(arg):
... yield item
... else:
... yield arg
...
py> getNextScalar(data)
<generator object at 0x00A3AE68>
py> list(getNextScalar(data))
[1, 2, 3, 4, 5, 6, 7]
I hope it's more clear now why you have to use yield on the recursive call
too.
<idea mode="raw">
Perhaps this:
yield *iterable
could be used as a shortcut for this:
for __temp in iterable: yield __temp
</idea>
--
Gabriel Genellina
zill...@googlemail.com
Thanks to everyone for your very helpful replies. I think I was trying
to use generator functions without first having really read about
iterators (something I still haven't done, although I've managed to
extrapolate some details based on your comments), and therefore really
"putting the cart before the horse". I was interpreting
getNextScalar(arg) on line 3 as a function call in the usual sense
rather than an object that was being created but never used.
I have a question about the other issue that was pointed out. With
reference to the following (corrected) code:
def getNextScalar(*args):
for arg in args:
if(isinstance(arg, tuple)):
for f in getNextScalar(*arg): # why not getNextScalar(arg)?
yield f
else:
yield arg
although I've verified that the line 4 needs to be "for f in
getNextScalar(*arg)" rather than "for f in getNextScalar(arg)" when
passing multiple arguments to the function, I'd just like to test my
understanding of this. Suppose I create the following generator
object:
g = getNextScalar(1, 2, (3, 4), 5)
when the iterator reaches the tuple argument (3, 4) then, according to
Steve and George, the * in *arg causes this tuple to be expanded into
positional arguments, and it makes sense to do it this way. But what
happens when getNextScalar(arg) is used instead? I presume that (3,4)
is passed as a single tuple argument, if(isinstance(arg, tuple)) is
true, and the tuple is passed again as an the argument to
getNextScalar()... ad infinitum. The alternative is to define a
function that takes a tuple, as Gabriel has done.
Regards,
AK
Steve Holden
existing functionality and recurse (at least to me). You should try
removing the * from the recursive call. I expect you would find that you
(eventually) experience a stack overflow due to the infinite nature of
the recursion.
Mel
I'd just like to test my
> understanding of this. Suppose I create the following generator
> object:
>
> g = getNextScalar(1, 2, (3, 4), 5)
>
> when the iterator reaches the tuple argument (3, 4) then, according to
> Steve and George, the * in *arg causes this tuple to be expanded into
> positional arguments, and it makes sense to do it this way. But what
> happens when getNextScalar(arg) is used instead?
Try it:
Python 2.5.1 (r251:54863, Mar 7 2008, 04:10:12)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def a (arg):
... print arg
...
>>> def astar (*arg):
... print arg
...
>>> a(3,4)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: a() takes exactly 1 argument (2 given)
>>> astar(3,4)
(3, 4)
>>> a((3,4))
(3, 4)
>>> astar((3,4))
((3, 4),)
>>>
Mel.
casti...@gmail.com
wrote:
> En Tue, 01 Apr 2008 23:56:50 -0300, <zillo...@googlemail.com> escribió:
>
> yield *iterable
>
> could be used as a shortcut for this:
>
> for __temp in iterable: yield __temp
How serious were you about that?
zill...@googlemail.com
Well, I understand that (unless I missed the point you're trying to
make). But with respect to the example I quoted:
def getNextScalar(*args):
for arg in args:
if(isinstance(arg, tuple)):
for f in getNextScalar(arg): # should use *arg
yield f
else:
yield arg
where the function is declared as def getNextScalar(*arg), but is
called using getNextScalar(arg), with arg being a tuple: here the
generator is being passed a single argument, so there's no TypeError
as in your example. However, it fails - as I understand it - because
the function keeps passing the same tuple (being unable to access the
elements inside it) and goes into an infinite loop:
>>> # works for this example, but not very useful:
>>> g = getNextScalar(1, 2, 3, 4)
>>> for i in g:
print i
1
2
3
4
# third argument is a tuple:
>>> g = getNextScalar(1, 2, (3, 4))
>>> for i in g:
print i
1
2
Traceback (most recent call last):
File "<pyshell#108>", line 1, in <module>
for i in g:
File "<pyshell#94>", line 4, in getNextScalar
for f in getNextScalar(arg):
File "<pyshell#94>", line 4, in getNextScalar
for f in getNextScalar(arg):
File "<pyshell#94>", line 4, in getNextScalar
for f in getNextScalar(arg):
...
AK
Gabriel Genellina
Not so much, I haven't thougth enough on it. Looks fine in principle, but
yield expressions may be a problem.
--
Gabriel Genellina
Mel
Yeah. I feel as though I haven't put the idea clearly. When a
function is defined as `def astar (a*)`, then the parameters that it's
called with, however many they are, are packed up into one tuple.
Illustrated by those print statements.
On the calling side, `a (*(3,4))` will unpack the (3,4) tuple and
treat the contents as ordinary positional parameters of a. The call
`a(*(3,4))` and the call `a(3,4)` are completely equivalent as far as
the function a can see.
This explanation is going nowhere fast. Anyway, if you define
getNextScalar to pack its positional arguments into a tuple that you
can iterate over, THEN if you want to feed it a tuple of values to
iterate over you have to star-unpack the tuple when you call. The
buggy behaviour is continually calling getNextScalar with a single
positional parameter (3,4), packing that single parameter into a
1-tuple ((3,4),) , iterating over the 1-tuple to extract (3,4) and
calling getNextScalar again -- ad dump.
Mel.
Robert Lehmann
Issue 2292: "Missing *-unpacking generalizations"
http://bugs.python.org/issue2292
Discussion on python-3000.devel:
http://thread.gmane.org/gmane.comp.python.python-3000.devel/12131
--
Robert "Stargaming" Lehmann
Arnaud Delobelle
wrote:
Funnily, there's a patch for py3k to make this work as in your idea.
It's currently being discussed on python-3000 :)
--
Arnaud