closures and dynamic binding
Aaron "Castironpi" Brady
To me, this is a somewhat unintuitive behavior. I want to discuss the
parts of it I don't understand.
>>> f= [ None ]* 10
>>> for n in range( 10 ):
... f[ n ]= lambda: n
...
>>> f[0]()
9
>>> f[1]()
9
I guess I can accept this part so far, though it took a little getting
used to. I'm writing some code and found the following workaround,
but I don't think it should give different results. Maybe I'm not
understanding some of the details of closures.
>>> f= [ None ]* 10
>>> for n in range( 10 ):
... f[ n ]= (lambda n: ( lambda: n ) )( n )
...
>>> f[0]()
0
>>> f[1]()
1
Which is of course the desired effect. Why doesn't the second one
just look up what 'n' is when I call f[0], and return 9?
Marc 'BlackJack' Rintsch
> To me, this is a somewhat unintuitive behavior. I want to discuss the
> parts of it I don't understand.
>
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):
> ... f[ n ]= lambda: n
> ...
>>>> f[0]()
> 9
>>>> f[1]()
> 9
`n` is looked up at the time ``f[0]`` is called. At that time it is
bound to 9.
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):
> ... f[ n ]= (lambda n: ( lambda: n ) )( n ) ...
>>>> f[0]()
> 0
>>>> f[1]()
> 1
>
> Which is of course the desired effect. Why doesn't the second one just
> look up what 'n' is when I call f[0], and return 9?
It *does* look up `n` at the time you call ``f[0]`` but this time it's
the local `n` of the outer ``lambda`` function and that is bound to
whatever the function was called with. At the time it was called the
global `n` was bound to 0. Maybe it get's more clear if you don't name
it `n`:
In [167]: f = [None] * 10
In [168]: for n in xrange(10):
.....: f[n] = (lambda x: lambda: x)(n)
.....:
In [169]: f[0]()
Out[169]: 0
Ciao,
Marc 'BlackJack' Rintsch
Aaron "Castironpi" Brady
Hi Marc,
It's my understanding that 'n' gets a new value every time through the
loop. n= 0 on the first pass, n= 1 on the second pass, and so on. n=
9 by the end, and that's why `lambda: n` always returns 9. It queries
the variable 'n', and finds 9. (This got lengthy. I started thinking
aloud.)
In your version of the indirect example, it queries the variable 'x',
which it finds in a new distinct scope in each f element. In f[0], x=
0. In f[1], x= 1. There are 10 different 'x' variables throughout
the contents of f. The direct example does not do this allocation of
ten different 'x's.
It's sort of helping. I think I feel like the following is more like
what I'm looking for:
(Non-standard)
>>> f= [ None ]* 10
>>> for n in range( 10 ):
... f[ n ]= n
...
>>> f[0]
9
>>> f[1]
9
because when you access f[0], it looks up the variable 'n'. Obviously
not.
(Unproduced)
>>> f= [ None ]* 10
>>> for n in range( 10 ):
... f[ n ]= late( n ) #late/lazy
...
>>> f[0]
9
>>> f[1]
9
>>> f= [ None ]* 10
>>> for n in range( 10 ):
... f[ n ]= early( n ) #early/eager
...
>>> f[0]
0
>>> f[1]
1
For the functions, I want a function that returns 'n', either late or
early.
(Unproduced)
>>> for n in range( 10 ):
... f[ n ]= lambda: late( n )
>>> f[0]()
9
>>> for n in range( 10 ):
... f[ n ]= lambda: early( n )
>>> f[0]()
0
I don't think you could pull this off. 'late' and 'early' succeed
with quotes around n, early('n') and late('n'), in the direct
assignments, but the functions aren't so lucky. 'n' has gone on to a
better world by the time 'early' gets any control.
This might have some success.
(Unproduced)
>>> for n in range( 10 ):
... f[ n ]= late( lambda: n )
>>> f[0]()
9
>>> for n in range( 10 ):
... f[ n ]= early( lambda: n )
>>> f[0]()
0
Though it's beyond my foresight to tell if it's feasible as stated, if
you need quotes, how much introspection you would need, etc. Plus,
'late' and 'early' were accepting quoted parameters earlier. How
would they look inside a function? Could a simple decorator provide
the service?
On a tangent about mutables, it's not that numbers, strings, and
tuples are 'immutable' per se, it's just that they don't have any
methods which mutate them (or assignable properties). Lists and
dictionaries do. It's up to the user whether a custom class does.
Terry Reedy
> Hello all,
>
> To me, this is a somewhat unintuitive behavior. I want to discuss the
> parts of it I don't understand.
>
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):
> ... f[ n ]= lambda: n
This is equivalent to
for n in range(10):
def g(): return n
f[n] = g
which is equivalent to
def g(): return n
f = [g]*10
n = 9
>>>> f[0]()
> 9
>>>> f[1]()
> 9
which make this not so surprising as the original lambda version is to
some people.
> I guess I can accept this part so far, though it took a little getting
> used to. I'm writing some code and found the following workaround,
> but I don't think it should give different results. Maybe I'm not
> understanding some of the details of closures.
>
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):
> ... f[ n ]= (lambda n: ( lambda: n ) )( n )
This is equivalent to
for n in range(10):
def g(n):
def h:
return n
return h
f[n] = g(n)
Now, to avoid the needless confusion of 'n's, g is equivalent to
def g(x):
def h:
return x
return h
(One could do the same change in the lambdas, too, of course).
so that g(n)() == n, with n stored in each closure h...
> ...
>>>> f[0]()
> 0
>>>> f[1]()
> 1
to be regurgitated when each is called.
Terry Jan Reedy
Steven D'Aprano
That's an awfully complicated solution. A much easier way to get the
result you are after is to give each function its own local copy of n:
f[n] = lambda n=n: n
As for why the complicated version works, it may be clearer if you expand
it from a one-liner:
# expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
inner = lambda: n
outer = lambda n: inner
f[n] = outer(n)
outer(0) => inner with a local scope of n=0
outer(1) => inner with a local scope of n=1 etc.
Then, later, when you call inner() it grabs the local scope and returns
the number you expected.
--
Steven
Aaron "Castironpi" Brady
Steven,
I must have misunderstood. Here's my run of your code:
>>> inner = lambda: n
>>> outer = lambda n: inner
>>> outer(0)
<function <lambda> at 0x00A01170>
>>> a=outer(0)
>>> b=outer(1)
>>> a()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <lambda>
NameError: global name 'n' is not defined
Why doesn't 'inner' know it's been used in two different scopes, and
look up 'n' based on the one it's in?
Terry Reedy
> On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-
>> As for why the complicated version works, it may be clearer if you expand
>> it from a one-liner:
>>
>> # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
>>
>> inner = lambda: n
>> outer = lambda n: inner
>> f[n] = outer(n)
>>
>> outer(0) => inner with a local scope of n=0
>> outer(1) => inner with a local scope of n=1 etc.
For this to work, the 'expansion' has to be mental and not actual.
Which is to say, inner must be a text macro to be substituted back into
outer.
>> Then, later, when you call inner() it grabs the local scope and returns
>> the number you expected.
>
> I must have misunderstood. Here's my run of your code:
I cannot speak to what Steven meant, but
>>>> inner = lambda: n
when inner is actually compiled outside of outer, it is no longer a
closure over outer's 'n' and 'n' will be looked for in globals instead.
>>>> outer = lambda n: inner
>>>> outer(0)
> <function <lambda> at 0x00A01170>
>>>> a=outer(0)
>>>> b=outer(1)
>>>> a()
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "<stdin>", line 1, in <lambda>
> NameError: global name 'n' is not defined
>
> Why doesn't 'inner' know it's been used in two different scopes, and
> look up 'n' based on the one it's in?
That would be dynamic rather than lexical scoping.
Steven D'Aprano
> Aaron "Castironpi" Brady wrote:
>> On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-
>
>>> As for why the complicated version works, it may be clearer if you
>>> expand it from a one-liner:
>>>
>>> # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
>>>
>>> inner = lambda: n
>>> outer = lambda n: inner
>>> f[n] = outer(n)
>>>
>>> outer(0) => inner with a local scope of n=0 outer(1) => inner with a
>>> local scope of n=1 etc.
>
> For this to work, the 'expansion' has to be mental and not actual. Which
> is to say, inner must be a text macro to be substituted back into outer.
Er, yes, that's what I meant, sorry for not being more explicit. That's
why it wasn't a copy and paste of actual running code.
Or perhaps I just confused myself and was talking nonsense.
--
Steven
Aaron "Castironpi" Brady
I couldn't find how those apply on the wikipedia website. It says:
"dynamic scoping can be dangerous and almost no modern languages use
it", but it sounded like that was what closures use. Or maybe it was
what 'inner' in Steven's example would use. I'm confused.
Actually, I'll pick this apart a little bit. See above when I
suggested 'late' and 'early' functions which control (or simulate)
different bindings. I get the idea that 'late' bound functions would
use a dangerous "dynamic scope", but I could be wrong; that's just my
impression.
> >> inner = lambda: n
> >> outer = lambda n: inner
> >> f[n] = outer(n)
>
> >> outer(0) => inner with a local scope of n=0
> >> outer(1) => inner with a local scope of n=1 etc.
If you defined these as:
inner= late( lambda: n )
outer= lambda n: inner
You could get the right results. It's not even clear you need
quotes. Perhaps 'late' could carry the definition of 'n' with it when
it's returned from 'outer'.
In my proposal, it makes a copy of the "localest" namespace, at least
all the variables used below it, then returns its argument in an
original closure.
Terry Reedy
> On Sep 28, 4:47 pm, Terry Reedy <tjre...@udel.edu> wrote:
>> Aaron "Castironpi" Brady wrote:
>>>>>> inner = lambda: n
>> when inner is actually compiled outside of outer, it is no longer a
>> closure over outer's 'n' and 'n' will be looked for in globals instead.
>>
>>>>>> outer = lambda n: inner
>>>>>> outer(0)
>>> <function <lambda> at 0x00A01170>
>>>>>> a=outer(0)
>>>>>> b=outer(1)
>>>>>> a()
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in <module>
>>> File "<stdin>", line 1, in <lambda>
>>> NameError: global name 'n' is not defined
>>> Why doesn't 'inner' know it's been used in two different scopes, and
>>> look up 'n' based on the one it's in?
>> That would be dynamic rather than lexical scoping.
>
> I couldn't find how those apply on the wikipedia website. It says:
> "dynamic scoping can be dangerous and almost no modern languages use
> it", but it sounded like that was what closures use. Or maybe it was
> what 'inner' in Steven's example would use. I'm confused.
As I understand it, partly from postings here years ago...
Lexical: The namespace scope of 'n' in inner is determined by where
inner is located in the code -- where is is compiled. This is Python
(and nearly all modern languages). Even without closures, the global
scope of a function is the module it is defined in.
Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
determined by where inner is called from. This is what you seemed to be
suggesting -- look up 'n' based on the scope it is *used* in.
Even without closures, dynamic scoping would be if the global scope of a
function for each call were the module it is called in.
tjr
Lawrence D'Oliveiro
<3c75ce4b-bb58-4da4...@m45g2000hsb.googlegroups.com>,
Aaron "Castironpi" Brady wrote:
> [Wikipedia] says:
> "dynamic scoping can be dangerous and almost no modern languages use
> it", but it sounded like that was what closures use.
Closures will use whatever the language says they use. LISP used dynamic
binding, but this caused some interesting problems as mentioned above. Perl
allows both, depending on whether the local is declared with "local"
or "my". Python uses only lexical, though there's probably some way to get
dynamic behaviour if you want. :)
Paul Boddie
>
> As I understand it, partly from postings here years ago...
>
> Lexical: The namespace scope of 'n' in inner is determined by where
> inner is located in the code -- where is is compiled. This is Python
> (and nearly all modern languages). Even without closures, the global
> scope of a function is the module it is defined in.
This is how I understand it, too. The confusing part involves the
definition of any inner function and how any "external" names are
bound or defined. As we've seen...
def f(x):
def g():
return x
x += 1 # added for illustration
return g
...it might look at first glance like the function known as g (within
f) should return the initial value of x (as known within f), since
that was the value x had when g was defined. The following is an
example execution trace based on that mental model:
fn = f(1)
-> def f(1):
-> def g(): # g defined with x as 1
-> return x # would be 1
-> x += 1 # x becomes 2
-> return g
fn()
-> def g():
-> return x # would still be 1
However, as we know, this isn't the case in real Python since g isn't
initialised with the value of x at the time of its definition - it
instead maintains access to the namespace providing x. We must
therefore revise the example:
fn = f(1)
-> def f(1):
-> def g(): # g refers to x within f(1)
-> return x # would be the current value of x within f(1)
-> x += 1 # x becomes 2
-> return g
fn()
-> def g(): # g refers to x within f(1)
-> return x # would be the current value of x within f(1), which is
2
This is the dynamic aspect of closures: values aren't used to
initialise inner functions; names are looked up from their origin.
> Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
> determined by where inner is called from. This is what you seemed to be
> suggesting -- look up 'n' based on the scope it is *used* in.
Indeed. Dynamic scoping is confusing in that one has to set up an
appropriate "environment" for the closure to access so that references
to names can be meaningfully satisfied; obviously, this creates all
sorts of encapsulation problems. The confusing aspect of lexical
scoping, however, especially if non-local names can be changed, is
that the effects of closures are potentially distant - one doesn't
always invoke inner functions in the place where they were defined, as
we see above - and such effects may thus happen within "abandoned
namespaces" - that is, namespaces which can no longer be revisited and
used in their original context. So, in the above, once f has been
invoked, the namespace for that invocation effectively lives on, but
that namespace is not a general one for f - if we invoke f again, we
get another namespace as one should reasonably expect.
A somewhat separate issue is illustrated by the modification of x
within f. Although for most statements, we would expect the value of x
to evolve following from a top-to-bottom traversal of the code within
a unit, function definition statements do not behave like, say, "for",
"if" or "while" statements. Now although this should be obvious at the
module global level, I feel that it can be easy to overlook within a
function where one normally expects to find plain old control-flow
constructs, expressions, assignments and so on. It is pertinent to
note, with respect to the original inquiry, that lambda functions are
subject to the same caveats, and I believe that lexical scoping was
introduced precisely to make lambda functions less cumbersome to
employ, eliminating the need to explicitly initialise them using the
"identity" default parameters trick, but obviously introducing the
consequences and potential misunderstandings described above.
Paul
Michele Simionato
<castiro...@gmail.com> wrote:
> Hello all,
>
> To me, this is a somewhat unintuitive behavior. I want to discuss the
> parts of it I don't understand.
>
> >>> f= [ None ]* 10
> >>> for n in range( 10 ):
>
> ... f[ n ]= lambda: n
> ...>>> f[0]()
> 9
> >>> f[1]()
>
> 9
>
You may want to read this old thread that goes in detail explaining
scoping issues
with closures in for loops:
Terry Reedy
>> Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
>> determined by where inner is called from. This is what you seemed to be
>> suggesting -- look up 'n' based on the scope it is *used* in.
> A somewhat separate issue is illustrated by the modification of x
> within f. Although for most statements, we would expect the value of x
> to evolve following from a top-to-bottom traversal of the code within
> a unit, function definition statements do not behave like, say, "for",
> "if" or "while" statements. Now although this should be obvious at the
> module global level, I feel that it can be easy to overlook within a
> function where one normally expects to find plain old control-flow
> constructs, expressions, assignments and so on. It is pertinent to
> note, with respect to the original inquiry, that lambda functions are
> subject to the same caveats,
Please: Python does not have 'lambda functions'. Def statements and
lambda expressions both define instances of the function class. So this
amounts to saying "functions are subject to the same caveats as functions."
> and I believe that lexical scoping was introduced precisely to make
> lambda functions less cumbersome to
> employ, eliminating the need to explicitly initialise them using the
> "identity" default parameters trick, but obviously introducing the
> consequences and potential misunderstandings described above.
As I meant when I wrote "Even without closures, the global scope of a
function is the module it is defined in.", the concept of lexical
scoping applies even to non-nested top-level functions.
Yes, closures for nested functions were introduced in part to replace
the pseudo-default parameter trick (which does not require the confusion
of using the same name in both local namespaces). But this applies to
both syntactical forms of function definition and had nothing to do in
particular with easing the quite limited lambda form, which Guido does
not like and once thought to delete in 3.0.
Closures were also introduced because default parameters are limited to
capturing a value at the time of function definition instead of at
function invocation. Consider your example which I have moved down to here.
> This is how I understand it, too. The confusing part involves the
> definition of any inner function and how any "external" names are
> bound or defined. As we've seen...
>
> def f(x):
> def g():
> return x
> x += 1 # added for illustration
> return g
>
> ...it might look at first glance like the function known as g (within
> f) should return the initial value of x (as known within f), since
> that was the value x had when g was defined.
Only if one is confused about the difference between default parameter
expressions, which are evaluated when the function is defined, and body
expressions, which are evaluated when the function is called. If you
want g to use the 'initial' value that x has when g is defined, then say so.
def f(x):
def g(y=x) # or x=x if you really prefer
return y # or x, with the consequent possibility of confusion
x += 1
return g
If 'x' is fixed, this is a difference without effect. If 'x' is not, as
in this pair of examples, the difference is important.
The introduction of the 'nonlocal' keyword will make the latter case
more common and hence the second reason more important. Consider code like
def f(*args)
x = [0]
def g()
...
x[0] += 1
...
<code calling g some variable number of times>
return x[0], ...
This has x fixedly bound to a particular list. The default argument
trick would have worked here. In the future, this can and will be
written more naturally as
def f(*args)
x = 0
def g()
nonlocal x
...
x += 1
...
<code calling g some variable number of times>
return x, ...
With x being rebound, the default argument trick would *not* work. And,
of course, lambda expressions are not in the picture.
Terry Jan Reedy
Aaron "Castironpi" Brady
I'm thinking of a comparison to try to see an example.
I tried this which still surprised me at first.
>>> g= [0]
>>> f= lambda: g
>>> del g
>>> f()
NameError: global name 'g' is not defined
It took a little thinking. The namespace in 'f' is the global
namespace. They are one in the same dictionary. id( f.namespace ) ==
id( main.namespace ). f.namespace is main.namespace.
When f gets a parameter, its namespace is different by one (or two
when two parameters, etc).
>>> g=[0]
>>> f= lambda h: lambda: h
>>> i= f( g )
>>> del g
>>> i()
[0]
The statement 'del g' removes 'g' from the module/global namespace,
but it lives on in the namespace of 'f', or more accurately, 'i'.
>>> g=[0]
>>> f= lambda h: lambda: h
>>> i= f( g )
>>> j= f( g )
>>> del g
>>> i().append(1)
>>> j()
[0, 1]
Here, the namespaces of 'main', 'i', and 'j', all have references to
'g'. 'i' and 'j' still have them when 'main' loses its.
The lambda might be confusing. Take a simpler example of a namespace:
>>> g= [0]
>>> def h():
... class A:
... a= g
... return A
...
>>> i= h()
>>> del g
>>> i.a
[0]
Or even:
>>> g=[0]
>>> class A:
... a= g
...
>>> del g
>>> A.a
[0]
The class is executed immediately so 'A.a' gets a hold of 'g'.
Function definitions keep a reference to the -identity-, not value, of
the namespace they were defined in, and parameters are part of that
namespace.
>>> def h( a ):
... #new namespace in here, new on each call
... def i():
... return a
... return i
'i' is not defined in the global namespace, it's defined in a brand
new one that is created on every call of 'h'. 'a' is defined in that,
so that's what 'a' when 'i' is called refers to.
To be specific, the namespace 'h' defines only includes the names that
are different from its enclosing scope. Regardless of what's defined
below 'h', 'h' only defines one new variable, 'a'. Its value is
passed in at call-time. 'i' needs to know what that namespace is--
that's how closures work-- so it saves a reference. That saved
reference is to a namespace, not a variable though, which
distinguishes it from the 'class A: a= g' statement that's executes
immediately. There, 'a' stores the value of g. By contrast, 'i'
stores 'h's entire namespace.
In 'class A: a= g', the name 'a' is assigned to the contents of 'g'.
In 'def i(): return a', 'a' is the value of a look up in a namespace
by its name.
>>> def h( a ):
... #new namespace in here
... def i():
... return a
... return i
...
>>> g=[0]
>>> j= h(g)
>>> hex(id(g))
'0x9ff440'
>>> del g
>>> hex(id(j()))
'0x9ff440'
>>> j.func_closure
(<cell at 0x009FDF50: list object at 0x009FF440>,)
By the time 'g' is deleted, 'j' has already hijacked a reference to
it, which lives in the namespace of the 'i' it defined that time
through. 'j()', originally 'g', and 'j's namespace all refer to the
same object.
Variables are keys in a namespace, even if it's an "abandoned
namespace", adopting the term.
Paul Boddie
>
> Please: Python does not have 'lambda functions'. Def statements and
> lambda expressions both define instances of the function class. So this
> amounts to saying "functions are subject to the same caveats as functions."
I myself am aware of the nature of "lambda expressions", for want of a
better term, but it's important to emphasise their nature to anyone
reading who isn't fully aware of what they represent. My closing
paragraph touches on the issues of readability and programmer
expectation when I write that 'function definition statements do not
behave like, say, "for", "if" or "while" statements'. Although this
may seem obvious, a newcomer might overlook lambda expressions in this
regard.
Personally, I'm not a great enthusiast of closures, anyway. Perhaps I
spent too long writing Java to be able to look at them as being
anything other than a fairly imprecise way of encapsulating state in a
callable.
Paul
jhermann
unnecessary (and a hack). What you should do when you need early
binding is... early binding. ;)
Namely:
f = [lambda n=n: n for n in range(10)]
print f[0]()
print f[1]()
Note the "n=n", this prints 0 and 1 instead of 9/9.
Paul Boddie
>
> f = [lambda n=n: n for n in range(10)]
> print f[0]()
> print f[1]()
>
> Note the "n=n", this prints 0 and 1 instead of 9/9.
Yes, Terry mentioned this in his response to my first message. Not
with lambdas, however, but he did state that he didn't believe that
such a distinction needed to be made clear to everyone.
Paul
Aaron "Castironpi" Brady
Yes it was mentioned earlier. I think its similar. They both create
ten new namespaces. You could do something like this (I hit a bump
with eval and globals() when I tried it):
def early( string ):
return eval( string, current_namespace )
f = [early( 'lambda: n' ) for n for n in range(10)]
print f[0]()
print f[1]()
Furthermore, I don't think the binding semantics of the language are
completely static binding. What does that definition say about
mutating a value? I think it's ambiguous and there's no obvious use
case that favors either one.
greg
> I didn't see this mentioned in the thread yet: the double-lambda is
> unnecessary (and a hack).
Well, the alternative -- abusing default argument values --
is seen by many to be a hack as well, possibly a worse one.
It doesn't work in general, e.g. it fails if the function
needs to be called with a variable number of arguments.
The double lambda is conceptually more sound in some
ways, and can be made to work correctly in all cases.
The root of the problem actually has nothing to do with
lambdas or static vs. non-static scoping. It's the fact
that Python's for-loop doesn't create a new environment
for the loop variable each time around, but re-uses a
slot in the containing environment.
Contrast this with Scheme, where the equivalent of a
for-loop *does* create a new environment for each
value of the loop variable. Effectively it's using a
double lambda, except that one of the lambdas is
folded into the syntax of the loop, so you don't
notice it.
So if anything were to be done to the language to
fix this, it really should be focused on fixing the
semantics of the for-loop. Unfortunately, the
fact that the loop variable leaks out of the scope
of the loop is regarded as a feature, so anything
which changes that seems to be a non-starter.
--
Greg
Michele Simionato
> So if anything were to be done to the language to
> fix this, it really should be focused on fixing the
> semantics of the for-loop. Unfortunately, the
> fact that the loop variable leaks out of the scope
> of the loop is regarded as a feature, so anything
> which changes that seems to be a non-starter.
And Guido stated many times in the past that he is happy with the for
loop as it is,
so I don't think this will never change, even if the question keep
getting asked here
and there.
Notice that even generator expressions, where the loop variable does
not leak outside the loop,
have the same behavior.
The behavior of the list comprehension is a good test of how much
functional a language is;
Common Lisp and Python behaves in the same way (there is a single loop
variable which is
mutated at each iteration) wherea Scheme and Haskell introduce a new
binding at each iteration.
Hrvoje Niksic
> The root of the problem actually has nothing to do with lambdas or
> static vs. non-static scoping. It's the fact that Python's for-loop
> doesn't create a new environment for the loop variable each time
> around, but re-uses a slot in the containing environment.
>
> Contrast this with Scheme, where the equivalent of a for-loop *does*
> create a new environment for each value of the loop variable.
Note that Python's semantics of "for" regarding closures are not
unique to Python. Common Lisp behaves similar to Python in this
regard:
* (loop for i from 0 to 2 collect (lambda () i))
(#<CLOSURE... {A86F3CD}> #<CLOSURE... {A86F3E5}> #<CLOSURE... {A86F3FD}>)
* (mapcar #'funcall *)
(3 3 3)
Other looping constructs, such as "do", behave in equivalent fashion.
> So if anything were to be done to the language to fix this, it
> really should be focused on fixing the semantics of the
> for-loop. Unfortunately, the fact that the loop variable leaks out
> of the scope of the loop is regarded as a feature, so anything which
> changes that seems to be a non-starter.
I don't think it has anything to do with variable leaking out of the
loop. Common Lisp doesn't leak the loop variable, and it behaves the
same. It is more a result of the different views of what iteration
is. Common Lisp and Python define iteration in terms of repeating the
same instructions over and over, not different from what C does, in
which case it makes sense to reuse the same environment for all loop
passes. (Python's language ref defines that a standard "assignment"
is done for each new iteration.) In contrast, Scheme regards
iteration as a special case of recursion, and R5RS "do" prescribes
assigning loop variables to "fresh locations" to match what recursion
normally does. In most cases both definitions exhibit the same
behavior, but unfortunately not when closures are created inside the
loop.
Aaron "Castironpi" Brady
I agree that the default argument syntax is an abuse, but it
accomplishes exactly what I want: to create a copy of a namespace. I
don't think there's a way to create a closure in Python without
another function, so you might need new syntax if you wanted to.
Otherwise, using function syntax, I want a new namespace on each
iteration that nests inside the old one, except for one variable which
overrides the outer scope. I agree that a new variable isn't the
obviously correct meaning of a for loop, and functions are the same as
a new scope, just you have to call them, so why not use them as is?
(untested)
for i in range( 3 ):
def f( n ):
def g( ):
return n
return g
closures[ i ]= f( i )
Or:
(non-standard)
for i in range( 3 ):
closure f( i ):
def g( ):
return i
return g
closures[ i ]= f
Here the only difference is whether you call 'f' or not. 'closure'
would theoretically "call itself", and make a copy of its scope upon
execution of the definition, overriding the arguments. So, functions
are the same.
Terry Reedy
> jhermann wrote:
>
>> I didn't see this mentioned in the thread yet: the double-lambda is
>> unnecessary (and a hack).
>
> Well, the alternative -- abusing default argument values --
> is seen by many to be a hack as well, possibly a worse one.
I disagree. It is one way to evaluate an expression when a function is
compiled.
> It doesn't work in general, e.g. it fails if the function
> needs to be called with a variable number of arguments.
So? Many things do not work 'in general'. If one wants multiple
closures with a variable number of arguments, one should use a def
statement and some other binding method, such as given below
Here are four ways to get the list of closures desired:
All print 0 ... 9 with for f in lst: print(f()) #3.0
lst = []
for i in range(10):
lst.append(eval("lambda: %d" %i))
# use exec instead of eval with def statement instead of lambda expression
lst = []
def f(i): return lambda: i
for i in range(10):
lst.append(f(i))
#I would most likely use this, with a def instead of lambda inside f for
any real, non-trivial example.
def populate(n):
n -= 1
if n >= 0: return populate(n)+[lambda:n]
else: return []
lst = populate(10)
# body recursion
def populate(i,n,lst):
if i < n: return populate(i+1,n,lst+[lambda:i])
else: return lst
lst = populate(0,10,[])
# tail recursion
Terry Jan Reedy
Terry Reedy
> greg <gr...@cosc.canterbury.ac.nz> writes:
> I don't think it has anything to do with variable leaking out of the
> loop. Common Lisp doesn't leak the loop variable, and it behaves the
> same. It is more a result of the different views of what iteration
> is. Common Lisp and Python define iteration in terms of repeating the
> same instructions over and over, not different from what C does, in
> which case it makes sense to reuse the same environment for all loop
> passes. (Python's language ref defines that a standard "assignment"
> is done for each new iteration.) In contrast, Scheme regards
> iteration as a special case of recursion,
Whereas I regard iteration as 'within-namespace recursion' ;-)
which amounts to what Python, etc, do.
Aaron "Castironpi" Brady
Is there a way to get at the 'reduce' / 'setstate' mechanism that
pickle uses without going through the serialization? I.e. doing a
serial round trip?
In either case, 'copy' module or 'loads( dumps( obj ) )' gets you a
copy of the object, but 'def' is the only way to get its own namespace.
greg
> Common Lisp behaves similar to Python in this
> regard:
>
> * (loop for i from 0 to 2 collect (lambda () i))
I wouldn't call the CL loop macro the equivalent of
a for-loop. It's more like a while-loop.
The Lisp equivalent of a Python for-loop would be
to use one of the mapping functions. Then, since
the loop body is a lambda, you get a new scope for
each iteration automatically.
> I don't think it has anything to do with variable leaking out of the
> loop.
That's right, it doesn't. If I were designing a
for-loop from scratch, I wouldn't let it leak, but
that part seems to be necessary for backwards
compatibility.
> In contrast, Scheme regards
> iteration as a special case of recursion, and R5RS "do" prescribes
> assigning loop variables to "fresh locations" to match what recursion
> normally does.
I'd say it prescribes that because it's useful
behaviour, not because it has anything to do with
recursion.
--
Greg