Confused about closures and scoping rules
Fernando Perez
consider the following small example:
"""
Small test to try to understand a strange subtlety with closures
"""
def outer(nmax):
aa = []
for n in range(nmax):
def a(y):
return (y,n)
print 'Closure and cell id:',id(a.func_closure),\
id(a.func_closure[0])
aa.append(a)
return aa
print 'Closure creation.'
nmax = 3
aa = outer(nmax)
print
print 'Closure use.'
for n in range(nmax):
print '%s:%s' % (n,aa[n]('hello'))
################## EOF #################
If I run this, I get:
planck[test]> python debug_closures.py
Closure creation.
Closure and cell id: 1075998828 1075618940
Closure and cell id: 1075999052 1075618940
Closure and cell id: 1075999084 1075618940
Closure use.
0:('hello', 2)
1:('hello', 2)
2:('hello', 2)
My confusion arises from the printout after 'closure use'. I was expecting that
each new function 'a' created inside the loop in 'outer' would capture the
value of n, therefore my expectation was to see a printout like:
0:('hello', 0)
1:('hello', 1)... etc.
However, what happens is a bit different. As can be seen from the printouts
of 'Closure and cell id', in each pass of the loop a new closure is created,
but it reuses the *same* cell object every time. For this reason, all the
closures end up sharing the scope with the values determined by the *last*
iteration of the loop.
This struck me as counterintuitive, but I couldn't find anything in the
official docs indicating what the expected behavior should be. Any
feedback/enlightenment would be welcome. This problem appeared deep inside a
complicated code and it took me almost two days to track down what was going
on...
Cheers,
f
Diez B. Roggisch
It's a FAQ. The reason is that the created closures don't capture the
_value_, but the _name_. Plus of course the locals()-dictionary outside
the function a to perform the lookup of that name. Which has the value
bound to it in the last iteration.
Common cure for this is to create an a-local name that shadows the outer
variable and is simultaneously bound to the desired value:
def outer(nmax):
aa = []
for n in range(nmax):
foo = 'bar'
def a(y,n=n):
bar = foo
return (y,n)
print 'Closure and cell id:',id(a.func_closure),\
id(a.func_closure[0])
aa.append(a)
return aa
print 'Closure creation.'
nmax = 3
aa = outer(nmax)
print
print 'Closure use.'
for n in range(nmax):
print '%s:%s' % (n,aa[n]('hello'))
Notice the foo/bar - that was necessary to actually create a closure at
all (to keep your printing working), as python statically checks if
there needs one to be.
Diez
Jean-Paul Calderone
> [snip]
>
>This struck me as counterintuitive, but I couldn't find anything in the
>official docs indicating what the expected behavior should be. Any
>feedback/enlightenment would be welcome. This problem appeared deep inside a
>complicated code and it took me almost two days to track down what was going
>on...
Lots of people ask about this. The behavior you observed is the expected
(by the implementors, anyway) behavior.
Jean-Paul
Chris Mellon
Are there languages where closures *don't* behave like this? A closure
that used a copy of the state rather than the actual state itself
doesn't seem as useful. For references sake, JavaScript (the only
language that a) has closures and b) I have a handy way to test with)
does the same thing.
Fernando Perez
> It's a FAQ. The reason is that the created closures don't capture the
> _value_, but the _name_. Plus of course the locals()-dictionary outside
> the function a to perform the lookup of that name. Which has the value
> bound to it in the last iteration.
>
> Common cure for this is to create an a-local name that shadows the outer
> variable and is simultaneously bound to the desired value:
Many thanks (also to JP) for the clear explanation. Greatly appreciated.
Cheers,
f
Michele Simionato
> On Nov 6, 2007 6:23 PM, Jean-Paul Calderone <exar...@divmod.com> wrote:
> > Lots of people ask about this. The behavior you observed is the expected
> > (by the implementors, anyway) behavior.
>
> Are there languages where closures *don't* behave like this? A closure
> that used a copy of the state rather than the actual state itself
> doesn't seem as useful. For references sake, JavaScript (the only
> language that a) has closures and b) I have a handy way to test with)
> does the same thing.
Closures in Haskell's list comprehensions work as Fernando (and many
others would expect). See for instance this post:
http://groups.google.com/group/comp.lang.python/browse_frm/thread/d691240a5cfebcdf/63234494ebbca54e?hl=en&lnk=gst&q=simionato+haskell#63234494ebbca54e
Michele Simionato
Bruno Desthuilliers
> On Nov 6, 7:37 pm, "Chris Mellon" <arka...@gmail.com> wrote:
>> On Nov 6, 2007 6:23 PM, Jean-Paul Calderone <exar...@divmod.com> wrote:
>>> Lots of people ask about this. The behavior you observed is the expected
>>> (by the implementors, anyway) behavior.
>> Are there languages where closures *don't* behave like this? A closure
>> that used a copy of the state rather than the actual state itself
>> doesn't seem as useful. For references sake, JavaScript (the only
>> language that a) has closures and b) I have a handy way to test with)
>> does the same thing.
>
> Closures in Haskell's list comprehensions work as Fernando (and many
> others would expect).
IIRC, Haskell's 'variables' are constant (ie: you cannot rebind nor
mutate them).
Rhamphoryncus
> On Nov 6, 2007 6:23 PM, Jean-Paul Calderone <exar...@divmod.com> wrote:
>
> > On Tue, 06 Nov 2007 17:07:47 -0700, Fernando Perez <fperez....@gmail.com> wrote:
> > > [snip]
>
> > >This struck me as counterintuitive, but I couldn't find anything in the
> > >official docs indicating what the expected behavior should be. Any
> > >feedback/enlightenment would be welcome. This problem appeared deep inside a
> > >complicated code and it took me almost two days to track down what was going
> > >on...
>
> > Lots of people ask about this. The behavior you observed is the expected
> > (by the implementors, anyway) behavior.
>
> Are there languages where closures *don't* behave like this? A closure
> that used a copy of the state rather than the actual state itself
> doesn't seem as useful. For references sake, JavaScript (the only
> language that a) has closures and b) I have a handy way to test with)
> does the same thing.
I've never needed to repeatedly modify closure variables. However, I
may not set them until after the function is defined. A simple
example is that of a recursive function:
def foo():
def bar():
bar() # useful work omitted
return bar
Obviously, bar can't be set until after the function object is
created.
Showing changes also matches the behaviour of globals, which is a good
thing IMO.
--
Adam Olsen, aka Rhamphoryncus
Jakub Hegenbart
> Are there languages where closures *don't* behave like this? A closure
> that used a copy of the state rather than the actual state itself
> doesn't seem as useful. For references sake, JavaScript (the only
> language that a) has closures and b) I have a handy way to test with)
> does the same thing.
The results in an equivalent code might depend on the semantics of the
looping construct used. For example, take Scheme (I'm using Gauche Scheme):
(define (outer-1 nmax)
(let ((aa '()))
(dotimes (n nmax)
(push! aa (lambda (y) (list "y:" y "n:" n))))
aa))
(define (outer-2 nmax)
(let ((aa '())
(n 0))
(until (= n nmax)
(push! aa (lambda (y) (list "y:" y "n:" n)))
(set! n (+ n 1)))
aa))
(print (map (lambda (f) (f 1)) (outer-1 5)))
(print (map (lambda (f) (f 1)) (outer-2 5)))
$ gosh closures.scm
((y: 1 n: 4) (y: 1 n: 3) (y: 1 n: 2) (y: 1 n: 1) (y: 1 n: 0))
((y: 1 n: 5) (y: 1 n: 5) (y: 1 n: 5) (y: 1 n: 5) (y: 1 n: 5))
In outer-1, the (dotimes ...) form expands into (do ...). R5RS defines
that a (do ...) loop is expected to _rebound_ all of its state variables
(here it is only n) in each iteration step. This means that each closure
created captures a different binding. Whereas in outer-2, I am updating
the binding destructively, so the value changes in the environment of
all the closures that have been already stored. Python seems to do the
latter. (I am not a pythonist right now, but I am learning... :))
Regards,
Jakub