Odd closure issue for generators
Brian Quinlan
>>> c = (lambda : i for i in range(11, 16))
>>> for q in c:
... print(q())
...
11
12
13
14
15
>>> # This is expected
>>> c = (lambda : i for i in range(11, 16))
>>> d = list(c)
>>> for q in d:
... print(q())
...
15
15
15
15
15
>>> # I was very surprised
Looking at the implementation, I see why this happens:
>>> c = (lambda : i for i in range(11, 16))
>>> for q in c:
... print(id(q.__closure__[0]))
...
3847792
3847792
3847792
3847792
3847792
>>> # The same closure is used by every lambda
But it seems very odd to me and it can lead to some problems that are a
real pain in the ass to debug.
Cheers,
Brian
Gabriel Genellina
escribi�:
> This is from Python built from the py3k branch:
It's not new; same thing happens with 2.x
A closure captures (part of) the enclosing namespace, so names are
resolved in that environment even after the enclosing block has finished
execution.
As always, the name->value evaluation happens when it is required at
runtime, not earlier ("late binding"). So, in the generator expression
(lambda : i for i in range(11, 16)) the 'i' is searched in the enclosing
namespace when the lambda is evaluated, not when the lambda is created.
Your first example:
> [1] >>> c = (lambda : i for i in range(11, 16))
> [2] >>> for q in c:
> [3] ... print(q())
> ...
> 11
> 12
> 13
> 14
> 15
> >>> # This is expected
This code means:
[1] create a generator expression and name it `c`
[2] ask `c` for an iterator. A generator is its own iterator, so returns
itself.
Loop begins: Ask the iterator a new item (the first one, in fact). So the
generator executes one cycle, yields a lambda expression, and freezes.
Note that the current value of `i` (in that frozen environment) is 11.
Name the yielded object (the lambda) `q`.
[3] Call the q object. That means, execute the lambda. It returns the
current value of i, 11. Print it.
Back to [2]: ask the iterator a new item. The generator resumes, executes
another cycle, yields another lambda expression, and freezes. Now, i is 12
inside the frozen environment.
[3] execute the lambda -> 12
etc.
Your second example:
> [4] >>> c = (lambda : i for i in range(11, 16))
> [5] >>> d = list(c)
> [6] >>> for q in d:
> [7] ... print(q())
> ...
> 15
> 15
> 15
> 15
> 15
> >>> # I was very surprised
[4] creates a generator expression same as above.
[5] ask for an iterator (c itself). Do the iteration NOW until exhaustion,
and collect each yielded object into a list. Those objects will be
lambdas. The current (and final) value of i is 15, because the range()
iteration has finished.
[6] iterate over the list...
[7] ...and execute each lambda. At this time, `i` is always 15.
> Looking at the implementation, I see why this happens:
> >>> c = (lambda : i for i in range(11, 16))
> >>> for q in c:
> ... print(id(q.__closure__[0]))
> ...
> 3847792
> 3847792
> 3847792
> 3847792
> 3847792
> >>> # The same closure is used by every lambda
...because all of them refer to the same `i` name.
> But it seems very odd to me and it can lead to some problems that are a
> real pain in the ass to debug.
Yes, at least if one is not aware of the consequences. I think this (or a
simpler example) should be explained in the FAQ. The question comes in
this list again and again...
--
Gabriel Genellina
Carl Banks
It's really the only sane way to handle it, odd though it may seem in
this narrow case. In Python nested functions have to be able to
reference the current value of a variable because of use cases like
this:
def func():
def printx():
print x
x = 1
printx()
x = 2
printx()
Referencing a nonlocal variable always uses the current (or last)
value of that variable, not the value it had when the nested function
was defined.
The way to handle the issue you are seeing is to create a new scope
with a variable the remains at the value you want to close upon:
create_const_function(value):
def func():
return value
c = (create_const_function(i) for i in range(11, 16))
Or you can do it the slacker way and use a default argument:
c = (lambda i=i: i for i in range(11, 16))
Carl Banks
Scott David Daniels
> This is from Python built from the py3k branch:
> >>> c = (lambda : i for i in range(11, 16))
> >>> for q in c:
> ... print(q())
> ...
> 11
> 12
> 13
> 14
> 15
> >>> # This is expected
> >>> c = (lambda : i for i in range(11, 16))
> >>> d = list(c)
> >>> for q in d:
> ... print(q())
> ...
> 15
> 15
> 15
> 15
> 15
> >>> # I was very surprised
You are entitled to be surprised. Then figure out what is going on.
Hint: it is the moral equivalent of what is happening here:
>>> c = []
>>> for i in range(11, 16):
c.append(lambda: i)
>>> i = 'Surprise!'
>>> print([f() for f in c])
['Surprise!', 'Surprise!', 'Surprise!', 'Surprise!', 'Surprise!']
>>> i = 0
>>> print([f() for f in c])
[0, 0, 0, 0, 0]
The body of your lambda is an un-evaluated expression with a reference,
not an expression evaluated at the time of loading c. TO get what you
expected, try this:
>>> c = []
>>> for i in range(11, 16):
c.append(lambda i=i: i)
>>> i = 'Surprise!'
>>> print([f() for f in c])
[11, 12, 13, 14, 15]
When you evaluate a lambda expression, the default args are evaluated,
but the expression inside the lambda body is not. When you apply that
evaluated lambda expression, the expression inside the lambda body is
is evaluated and returned.
--Scott David Daniels
Scott....@Acm.Org
Brian Quinlan
> En Thu, 04 Jun 2009 18:40:07 -0300, Brian Quinlan <br...@sweetapp.com>
> escribi�:
>
>
>
> A closure captures (part of) the enclosing namespace, so names are
> resolved in that environment even after the enclosing block has finished
> execution.
> As always, the name->value evaluation happens when it is required at
> runtime, not earlier ("late binding"). So, in the generator expression
> (lambda : i for i in range(11, 16)) the 'i' is searched in the enclosing
> namespace when the lambda is evaluated, not when the lambda is created.
OK, I talked myself into agreeing that it is better for the generator
comprehension to share a context amongst every enclosed generator
expression (rather than having one context per generator expression) for
consistency reasons.
Thanks!
Cheers,
Brian
Michele Simionato
> It's really the only sane way to handle it, odd though it may seem in
> this narrow case. In Python nested functions have to be able to
> reference the current value of a variable because of use cases like
> this:
>
> def func():
> def printx():
> print x
> x = 1
> printx()
> x = 2
> printx()
>
> Referencing a nonlocal variable always uses the current (or last)
> value of that variable, not the value it had when the nested function
> was defined.
This is not a good argument. Counter example: Scheme works
exactly like Python
(define (func)
(define x 1)
(define (printx)
(display x) (newline))
(printx)
(set! x 2)
(printx)) ;; prints 1 and 2
but it is still possible to have a list comprehension working
as the OP wants:
(list-of (lambda () i) (i in (range 11 16)))
Actually, in Scheme one would have to fight to define
a list comprehension (more in general loops) working as
in Python: the natural definition works as the OP wants. See
http://www.artima.com/weblogs/viewpost.jsp?thread=251156 and the
comments below for the details.
Brian Quinlan
[snipped]
> When you evaluate a lambda expression, the default args are evaluated,
> but the expression inside the lambda body is not. When you apply that
> evaluated lambda expression, the expression inside the lambda body is
> is evaluated and returned.
But that's not really the issue. I knew that the lambda was not
evaluated but thought each generator expression got its own context
rather than sharing one.
Taken in isolation, having one context per expression has more
compelling semantics but it is inconsistent with the obvious
transliteration of the generator into a loop.
Cheers,
Brian
Dave Angel
> <snip>
>
> The way to handle the issue you are seeing is to create a new scope
> with a variable the remains at the value you want to close upon:
>
> create_const_function(value):
> def func():
> return value
>
> Or you can do it the slacker way and use a default argument:
>
>
>
> Carl Banks
>
>
I agree with most of what you say, but I think you missed the last line
of the function:
create_const_function(value):
def func():
return value
return func
Aahz
Michele Simionato <michele....@gmail.com> wrote:
>
>Actually, in Scheme one would have to fight to define
>a list comprehension (more in general loops) working as
>in Python: the natural definition works as the OP wants. See
>comments below for the details.
This URL isn't working for me, gives 500.
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
"Given that C++ has pointers and typecasts, it's really hard to have a
serious conversation about type safety with a C++ programmer and keep a
straight face. It's kind of like having a guy who juggles chainsaws
wearing body armor arguing with a guy who juggles rubber chickens wearing
a T-shirt about who's in more danger." --Roy Smith, c.l.py, 2004.05.23
Ned Deily
> > but the expression inside the lambda body is not. When you apply that
> > evaluated lambda expression, the expression inside the lambda body is
> > is evaluated and returned.
>
> evaluated but thought each generator expression got its own context
> rather than sharing one.
Each? Maybe that's a source of confusion. There is only one generator
expression in your example.
>>> c = (lambda : i for i in range(11, 16))
>>> c
<generator object <genexpr> at 0x114e90>
>>> d = list(c)
>>> d
[<function <lambda> at 0x119348>, <function <lambda> at 0x119390>,
<function <lambda> at 0x1193d8>, <function <lambda> at 0x119420>,
<function <lambda> at 0x119468>]
--
Ned Deily,
n...@acm.org
Lawrence D'Oliveiro
Quinlan wrote:
> >>> c = (lambda : i for i in range(11, 16))
> >>> d = list(c)
> >>> for q in d:
> ... print(q())
> ...
> 15
> 15
> 15
> 15
> 15
Try
>>> c = ((lambda i : lambda : i)(i) for i in range(11, 16))
>>> d = list(c)
>>> for q in d :
... print q()
Michele Simionato
> In article <05937a34-5490-4b31-9f07-a319b44dd...@r33g2000yqn.googlegroups.com>,
> Michele Simionato <michele.simion...@gmail.com> wrote:
>
>
>
> >Actually, in Scheme one would have to fight to define
> >a list comprehension (more in general loops) working as
> >in Python: the natural definition works as the OP wants. See
> >comments below for the details.
>
> This URL isn't working for me, gives 500.
This happens sometimes with Artima. Usually it is just a matter of
waiting a few minutes.
Michele Simionato
> In article <05937a34-5490-4b31-9f07-a319b44dd...@r33g2000yqn.googlegroups.com>,
>
>
>
> >Actually, in Scheme one would have to fight to define
> >a list comprehension (more in general loops) working as
> >in Python: the natural definition works as the OP wants. See
> >comments below for the details.
>
> This URL isn't working for me, gives 500.
Anyway, the point is that to explain Python behavior with closures in
list/generator comprehension it is not enough to invoke
late bindings (Scheme has late bindings too but list comprehension
works differently). The crux is in the behavior of the for loop:
in Python there is a single scope and the loop variable is
*mutated* at each iteration, whereas in Scheme (or Haskell or any
other functional language) a new scope is generated at each
iteration and there is actually a new loop variable at each iteration:
no mutation is involved. Common Lisp works like Python. It is a design
decision which at the end comes down to personal preference and
different
languages make different choices with no clear cut winner (I
personally
prefer the more functional way).
Lawrence D'Oliveiro
<78180b4c-68b2-4a0c...@c19g2000yqc.googlegroups.com>, Michele
Simionato wrote:
> The crux is in the behavior of the for loop:
> in Python there is a single scope and the loop variable is
> *mutated* at each iteration, whereas in Scheme (or Haskell or any
> other functional language) a new scope is generated at each
> iteration and there is actually a new loop variable at each iteration:
> no mutation is involved.
I think it's a bad design decision to have the loop index be a variable that
can be assigned to in the loop.
Gabriel Genellina
> In article
> <05937a34-5490-4b31...@r33g2000yqn.googlegroups.com>,
> Michele Simionato <michele....@gmail.com> wrote:
>>
>> Actually, in Scheme one would have to fight to define
>> a list comprehension (more in general loops) working as
>> in Python: the natural definition works as the OP wants. See
>> http://www.artima.com/weblogs/viewpost.jsp?thread=3D251156 >
Mmm, the URL ends with: thread, an equals sign, and the number 251156
If you see =3D -- that's the "=" encoded as quoted-printable...
--
Gabriel Genellina
Michele Simionato
wrote:
> Mmm, the URL ends with: thread, an equals sign, and the number 251156
> If you see =3D -- that's the "=" encoded as quoted-printable...
Actually this is the right URL:
Aahz
Brian Quinlan
> In article <4A28903B...@sweetapp.com>,
> Brian Quinlan <br...@sweetapp.com> wrote:
>> Scott David Daniels wrote:
>> [snipped]
>>> but the expression inside the lambda body is not. When you apply that
>>> evaluated lambda expression, the expression inside the lambda body is
>>> is evaluated and returned.
>> evaluated but thought each generator expression got its own context
>> rather than sharing one.
>
> Each? Maybe that's a source of confusion. There is only one generator
> expression in your example.
>
> <generator object <genexpr> at 0x114e90>
>>>> d = list(c)
>>>> d
> [<function <lambda> at 0x119348>, <function <lambda> at 0x119390>,
> <function <lambda> at 0x1193d8>, <function <lambda> at 0x119420>,
> <function <lambda> at 0x119468>]
>
Sorry, I wasn't as precise as I should have been.
If you consider this example:
(<expr> for x in y)
I thought that every time that <expr> was evaluated, it would be done in
a new closure with x bound to the value of x at the time that the
closure was created.
Instead, a new closure is created for the entire generator expression
and x is updated inside that closure.
Cheers,
Brian
Terry Reedy
>
> Sorry, I wasn't as precise as I should have been.
>
> If you consider this example:
> (<expr> for x in y)
>
> I thought that every time that <expr> was evaluated, it would be done in
> a new closure with x bound to the value of x at the time that the
> closure was created.
>
> Instead, a new closure is created for the entire generator expression
> and x is updated inside that closure.
Thanks you for explaining your confusion. Knowing what sort of
other-language-baggage people are being mislead by can only help in
explaining Python. But here is my question. In Python,
g = (<expr> for x in iterable)
is essentially an abbreviation for, and means the same as
def _(it):
for x in it:
yield <expr>
g = _(iterable)
del _
Are there language in which a similar construct has an essentially
different meaning?
Terry Jan Reedy
Michele Simionato
Yes, most functional languages have the concept of streams.
You can even define a stream-comprehension that looks like
Python generator comprehension but it is an essentially different
thing. See for instance
Terry Reedy
> Yes, most functional languages have the concept of streams.
> You can even define a stream-comprehension that looks like
> Python generator comprehension but it is an essentially different
> thing. See for instance
>
> http://www.artima.com/weblogs/viewpost.jsp?thread=251159
I read that. It seems that streams are virtual or lazy linked-lists(1).
I think, though, that comparing them to iterators is misleading. They
are iterables, but with a different iteration protocol. Python
iterables are generally reiterable just like streams.
chars = 'abc'
for c in chars: print(c,end=' ')
for c in chars: print(c,end=' ')
produces repeatable output just like your stream example. Python
*could* have given iterables .first and .rest methods instead of
.__iter__, but that works best for linked lists and awfully for arrays.
Anyway, I realize now that having generator comprehensions produce an
*iterator* rathar than an *iterable* or *generator function* is
something of an anomaly Set, dict, and list comprehensions in Python
produce iterables, of course, as does a stream comprehension in Scheme
and, I presume, comprehensions in other languages.
A generator expression could have been defined in Python to just produce
a generator function, without calling it, but the intent of the
abbreviation was for one-use situations. Multi-use gfs should be
defined with a def statement.
Terry Jan Reedy
(1) Calling the first and rest methods 'car' and 'cdr' convinces me that
schemers really do not want scheme to be popular, but prefer it to
remain a small cult language.
Aahz
Terry Reedy <tjr...@udel.edu> wrote:
>
>(1) Calling the first and rest methods 'car' and 'cdr' convinces me that
>schemers really do not want scheme to be popular, but prefer it to
>remain a small cult language.
What, you don't get a warm, fuzzy feeling from saying, "Today is the car
of the cdr of your life"?
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
"If you don't know what your program is supposed to do, you'd better not
start writing it." --Dijkstra
David Stanek
Getting around this can be pretty easy:
c = (lambda i=i: i for i in range(11, 16))
for q in (list(c)):
print(q())
--
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek