I would expect that the example given is just sample code demonstrating
the problem. A slightly more realistic case might be something like
defining a bunch of callbacks for, say, a GUI application.

For example, the good ol' calculator app, where you have ten buttons for
digits 0-9. So you might give each button a callback function that
inserts its own digit into the text field:

buttons = [make_button(name=str(i)) for i in range(10)]

That works fine. So now you go to add a callback to each one:

buttons = [make_button(name=str(i), callback=lambda: FIELD.insert(i))
           for i in range(10)]

and lo and behold, the ten buttons named "0" through "9" all insert 9.
The reason is that the callback functions aren't given the value of i,
but only the name[1] "i". By the time the callbacks are actually used,
i == 9 and all the buttons share the same value.

[1] Technically closures may not store values by name, but the principle
is sound: the function, when called later, looks up the current value of
variable i, which is not necessarily the same as it was when the closure
was originally defined.

--
Steven

On Wed, Jul 11, 2012 at 9:59 PM, Steven D'Aprano

You know, partial does handle keyword arguments correctly:

funcs = [partial(lambda x, i: x ** i, i=i) for i in range(5)]

This might be bad practice though if there are other arguments to the
right of the keywords that the eventual caller might want to specify,
as those arguments would then effectively be keyword-only.  The error
message in that case is not particularly illuminating, either:

...   return (a, b, c)
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() got multiple values for keyword argument 'b'

It would be a bit clearer if it instead noted that 'c' were
keyword-only in the 'g' argspec.

Cheers,
Ian

On 12/07/2012 04:59, Steven D'Aprano wrote:

I don't think anyone has suggested this yet:

funcs = [(lambda i: lambda x: x**i)(i) for i in range(5)]

It's a bit more opaque than other solutions, but it fits in one line and
avoids defining functions with an implicit second argument, if that's
desirable for some reason.

--
I have made a thing that superficially resembles music:

http://soundcloud.com/eroneity/we-berated-our-own-crapiness

Oooh, nice! A factory function solution. Excellent!

--
Steven

On Jul 11, 11:41 am, Daniel Fetchinson <fetchin...@googlemail.com>
wrote:

Your expectations are reasonable.  Heres the equivalent in Haskell
from which python has taken comprehensions.
---------------------------------
GHCi, version 7.4.1: http://www.haskell.org/ghc/  :? for help

Prelude> let funcs = [ (\ x -> x ^ i)| i <- [0..4]]
Prelude> (funcs !! 0)(2)
1
Prelude> (funcs !! 1)(2)
2
Prelude> (funcs !! 2)(2)
4
Prelude>

You forget to finish that sentence.

"Your expectations are reasonable, for languages that don't have
variables which can actually vary."

*wink*

For purely functional languages like Haskell, the behaviour you show
below makes sense. Since Haskell doesn't allow variables to change their
value, once a closure sees i=1 (say), then it must *always* see i=1.

But that's not the case in Python, where the Haskell behaviour would be
unfortunate. Imagine if you did this:

VERBOSE = True

def function(arg):
    if VERBOSE:
        print("calling function with arg %r" % arg)
    process(arg)

You would expect the function to honour changes to the variable VERBOSE,
would you not? Using Python's scoping rules for closures, it does. Using
Haskell's rules, it wouldn't.

--
Steven

On Jul 13, 11:36 am, Steven D'Aprano <steve

Heres a more appropriate analog

------------------------------------------
VERBOSE = True

def function(arg):
    if VERBOSE:
       print("calling function with arg %r" % arg)
    process(arg)

def caller():
    VERBOSE = False
    function(1)

---------------------------------------------
Python semantics: function sees VERBOSE False
Haskell semantics: function sees VERBOSE True

To come back to the OPs question.

Variables can be assigned. Or they can be bound.
[C++ programmers will be familiar with the difference between
initialization and assignment]

List comprehensions are defined in terms of assignment to the local
variable rather than binding.
Hence the issue.

Below are two functions that simulate mapping (lambda x: x**i) where
the i's come from some given list

def binding_version(lst):
    if not lst: return []
    i = lst[0]
    return [(lambda x: x ** i)] + binding_version(lst[1:])

def assign_version(lst):
    ret = []
    for i in lst:
        ret.append(lambda x: x**i)
    return ret
----------------------

4

4

...     VERBOSE = False
...     function(1)
...     if VERBOSE:
...        print("calling function with arg %r" % arg)
...    
calling function with arg 1

I might be being OCD, but caller needs `global VERBOSE` for that to
work as you explain.

Ramit

Ramit Prasad | JPMorgan Chase Investment Bank | Currencies Technology
712 Main Street | Houston, TX 77002
work phone: 713 - 216 - 5423

--

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  

On Jul 13, 9:12 pm, "Prasad, Ramit" <ramit.pra...@jpmorgan.com> wrote:

Ok let me restate: if python were to work that way (without the
global) we could say either
a Python chooses to have dynamic scoping of variables
or
b There is a bug in python's scoping rules

I would guess that most younger folks (who've not seen lisp and apl)
would choose b

We can say the same analogously in the context of ZF expressions when
the i leaks as it does in the OPs example.

Another tack on the same question: python 3 cleaned up the variable
leakage from inside ZFs to outside.  It missed cleaning up the leakage
from one step to next.

tl;dr version: Beware of mixing up functional and imperative
programming
Double-beware when you are a language designer

Or c, there's a declaration at the top:

from __future__ import variable_declarations

Like braces, it's a MASSIVE improvement to the language, and it's part
of a huge conspiracy that it isn't there by default. But all it takes
is a little future statement and you're on Python 4000!

</troll>

ChrisA
(I shouldn't post at half past three in the morning. I get stupid... er.)

On 13/07/12 18:12:40, Prasad, Ramit wrote:

That would be quite different from what Rusi is after.

If you add `global VERBOSE` to `caller`, then there is only one
variable named `VERBOSE` and what `function` does, depends on
the most recent assignment to that variable.

If you remove your `global VERBOSE`, then there are two
variables by that name, one global and one local to `caller`.
In that case, there is the question of which one `function`
will use.

The function `function` refers to a variable `VERBOSE` that
isn't local.  In some programming langauages, the interpreter
would then scan the call stack at run-time, looking for a scope
where that name is defined.  It would find the local one in
`caller`.  This is known as "dynamic binding".

Other interpreters use the `VERBOSE` that was in scope at
the point in the program text where `function` was defined.
In this case, that would be the global one.  This is called
"lexical binding".

Some programming languages allow you to indicate on a per-
variable basis whether you want dynamic or lexical binding.

Python is firmly in the lexical camp.  Dynamic binding is not
available in Python, and never will be.

Hope this helps,

-- HansM

But that is not what Rusi writes.
"Python semantics: function sees VERBOSE False" <- function
will not see False without the use of global.

True and a good explanation, but not what I understood
Rusi to mean.

Ramit

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  

I don't believe that dynamic vs. lexical binding is what rusi was
attempting to describe.  If he was, then Python and Haskell would be a
bad comparison since both are lexical.  Rather, I think what he was
trying to show was capture by reference vs. capture by value in the
context of closures.  Python uses capture by reference, and so the
upvalue is the value of that reference at the time the closure is
called.  Haskell uses capture by value, and the upvalue is the value
at the time of definition.

I've also seen the distinction described as "early" vs. "late" binding
on this list, but I'm not sure how precise that is -- I believe that
terminology more accurately describes whether method and attribute
names are looked up at compile-time or at run-time, late binding being
the feature that makes duck typing possible.

Cheers,
Ian

On 13/07/12 20:54:02, Ian Kelly wrote:

I think that these terms describe the problem at the start of
this thread: the OP was expecting early binding.  However, Python
does late binding, or "acts funny" as it says in the subject line.

-- HansM

On Jul 13, 10:53 pm, Hans Mulder <han...@xs4all.nl> wrote:

Good point. See below.

Thats a good intention.  The question is whether it is uniformly
realized.

Consider the following

def foo(x):
    i = 100
    if x:
        j = [i for i in range(10)]
        return i
    else:
        return i

In python 2  two different 'i's could be returned. In python3 only one
i can be returned.
One could call it dynamic binding. Evidently Guido thinks it a bug
which is why he changed it.

The leakage of i in the OPs question is the same kind of bug.

tl;dr version:
This is 2012. Dynamic binding is considered a bug even by the lisp
community where it originated.
Lets at least call it a feature and a gotcha

A simpler example:

def foo():
    i = 100
    j = [i for i in range(10)]
    return i

In Python 3, foo() returns 100; in Python 2, it returns 9.

One could also call it a tractor, but it isn't either of those things.

The difference is whether or not list comprehensions create their own
scope. In Python 2, they don't; in Python 3, they do. Personally I don't
have an opinion as to which is better, but in neither case does this have
any thing to do with lexical versus dynamic binding.

It was a case that either list comps be changed to *not* expose their
loop variable, or generator expressions be changed *to* expose their loop
variable. The Python 2 situation where list comps and gen expressions
have opposite behaviour was unfortunate.

Not at all. The OP was *deliberately* creating a closure using i -- under
those circumstances, it would be a bug if i *didn't* leak.

The OP's gotcha was:

1) he expected the closure to use early binding, where i in each function
was bound to the value of i in the enclosing scope at the moment the
function was defined;

2) but Python actually uses late binding for closures, where the value of
i in each function is set to the value of i in the enclosing scope when
the function is called.

As far as I can tell, Python always uses late binding for scopes; the
only time it does early binding is for default values of function
parameters.

--
Steven

On Jul 14, 8:43 am, Steven D'Aprano <steve

You did not get the point.

Converting my example to your format:

def foo_steven(n):
    i = 100
    j = [i for i in range(n)]
    return i

$ python3
Python 3.2.3 (default, Jun 26 2012, 00:38:09)
[GCC 4.7.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.

...     i = 100
...     j = [i for i in range(n)]
...     return i
...
$ python
Python 2.7.3rc2 (default, Apr 22 2012, 22:35:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
...     i = 100
...     j = [i for i in range(n)]
...     return i
...

Python 2:
When n>0 comprehension scope i is returned
When n=0 function scope i is returned

Python 3: The return statement is lexically outside the comprehension
and so that outside-scope's i is returned in all cases.

I got the point. I just thought it was unnecessarily complex and that it
doesn't demonstrate what you think it does.

Yes, we know that in Python 3, list comprehensions create their own
scope, and the loop variable does not leak.

Yes, we know that in Python 2, list comprehensions don't create their own
scope, and consequently the list variable does leak.

Incorrect.

In Python 2, *there is no comprehension scope*. There is only local
scope. In Python 2, regardless of the value of n, the local variable i is
ALWAYS returned. It just happens that sometimes the local variable i is
modified by the list comprehension, and sometimes it isn't. In Python 2,
this is no more mysterious than this piece of code:

def example(n):
    i = 100
    for i in range(n):
        pass
    return i

py> example(0)
100
py> example(4)
3

If the loop doesn't execute, the loop variable isn't modified. In Python
2, it doesn't matter whether you use a for-loop or a list comprehension,
the loop variable is local to the function.

Correct. In Python 3, list comprehensions now match generator expressions
and introduce their own scope which does not effect the local function
scope.

Some history:

http://bugs.python.org/issue510384
http://bugs.python.org/issue1110705

--
Steven

Alister於 2012年7月12日星期四UTC+8下午5時44分15秒寫道：

Uhn, there are 5 objects in the list if not factored well to be executed in
the run time.

I don't remember whether it is Javascript or PHP that uses dynamic
binding, but whichever it is, it is generally considered to be a bad
idea, at least as the default or only behaviour.

Bash is another language with dynamic binding. Some very old versions of
Lisp use dynamic binding, because it was the easiest to implement. Most
modern languages use lexical (also known as static) binding, because it
is more sensible.

Here is an illustration of the difference: suppose we have two modules,
library.py and main.py:

# library.py
x = 23
def func(y):
    return x + y

# main.py
import library
x = 1000
print func(1)

If main.py prints 24 (and it does), then Python is using lexical scoping.
But if it prints 1001 (which it doesn't), then it is using dynamic
scoping. The obvious problem with dynamic binding is that the behaviour
of a function may vary depending on where you call it.

I don't think "by reference" versus "by value" are good terms to use
here, since they risk conflating the issue with "call by reference"
versus "call by value" semantics. I prefer "late" versus "early", as you
suggest below.

Not necessarily *compile* time, but the distinction is between when the
function is defined (which may at compile time, or it may be at run time)
versus when the function is called. I think that gets to the heart of the
issue, not whether the capture copies a value or a reference. At some
point, the capture must make use of the value: whether it does so via a
direct C-style memory location copy, or an object pointer, or some other
mechanism, is irrelevant. What matters is whether that value is grabbed
at the time the closure is created, or when the closure is called.

That is conflating two entirely distinct subjects. Python 1.5 had duck-
typing but no closures, so the one certainly does not depend on the other.

Duck-typing is more a philosophy than a language feature: the language
must be typed, and the programmer must prefer to program to a protocol or
a specification rather than to membership of a type.

--
Steven

On Sun, Jul 15, 2012 at 9:29 AM, Steven D'Aprano

I'd treat the def/lambda statement as "compile time" and the ()
operator as "run time".

ChrisA

But function definitions occur at run time, not compile time -- they are
executable statements, not instructions to the compiler to define a
function.

For example:

py> dis("def f(x): return x+1")  # Python 3.2
  1           0 LOAD_CONST               0 (<code object f at 0xb7b57de0,
                                            file "<dis>", line 1>)
              3 MAKE_FUNCTION            0
              6 STORE_NAME               0 (f)
              9 LOAD_CONST               1 (None)
             12 RETURN_VALUE

The code object is pre-compiled at compile time, but the function and
name-binding (the "def") doesn't occur until runtime.

At compile time, Python parses the source code and turns it into byte-
code. Class and function definitions are executed at run time, the same
as any other statement.

I'm not sure if this is a difference that makes a difference or not; I
think it is, but don't know enough about how closures and scoping rules
work in other languages to be sure that it does make a difference.

--
Steven

On Sun, Jul 15, 2012 at 6:32 PM, Steven D'Aprano

Between the parse step and the 'def' execution, a code object is
created. When you call it, that code object already exists. Nothing
else really matters, unless there's a bug in the Python optimizer or
something weird like that. The nearest thing Python _has_ to a
"compile time" is the execution of def.

ChrisA

On 7/15/2012 4:32 AM, Steven D'Aprano wrote:

The () operator is 'call time'. The main points are
a) the execution of def statements and lambda expressions and the
execution of calls happen at different run times.
b) default arg objects are calculated at def/lambda time.
c) names in def bodies and lambda expressions are resolved at call time.
and
d) people more often forget c) when thinking about lambda expressions
that for def statements.

--
Terry Jan Reedy