lambda in list comprehension acting funny

[Daniel Fetchinson]

funcs = [ lambda x: x**i for i in range( 5 ) ]
print funcs[0]( 2 )
print funcs[1]( 2 )
print funcs[2]( 2 )

This gives me

16
16
16

When I was excepting

1
2
4

Does anyone know why?

Cheers,
Daniel


--
Psss, psss, put it down! - http://www.cafepress.com/putitdown

[Daniel Fetchinson]

> funcs = [ lambda x: x**i for i in range( 5 ) ]
> print funcs[0]( 2 )
> print funcs[1]( 2 )
> print funcs[2]( 2 )
>
> This gives me
>
> 16
> 16
> 16
>
> When I was excepting
>
> 1
> 2
> 4
>
> Does anyone know why?

And more importantly, what's the simplest way to achieve the latter? :)

[Jurko Gospodnetić]

Hi.

>> funcs = [ lambda x: x**i for i in range( 5 ) ]
>> print funcs[0]( 2 )
>>

>> This gives me

>> 16
>>
>> When I was excepting
>> 1
>>

>> Does anyone know why?

Just the way Python lambda expressions bind their variable
references. Inner 'i' references the outer scope's 'i' variable and not
its value 'at the time the lambda got defined'.

> And more importantly, what's the simplest way to achieve the latter? :)

Try giving the lambda a default parameter (they get calculated and
have their value stored at the time the lambda is defined) like this:
funcs = [ lambda x, i=i: x**i for i in range( 5 ) ]


Hope this helps.

Best regards,
Jurko Gospodnetić

[Daniel Fetchinson]

Thanks a lot!
I worked around it by

def p(i):
return lambda x: x**i
funcs = [ p(i) for i in range(5) ]

But your variant is nicer (matter of taste of course).

Cheers,
Daniel

[Colin J. Williams]

On 11/07/2012 2:41 AM, Daniel Fetchinson wrote:
> funcs = [ lambda x: x**i for i in range( 5 ) ]
> print funcs[0]( 2 )
> print funcs[1]( 2 )
> print funcs[2]( 2 )
>
> This gives me
>
> 16
> 16
> 16
>
> When I was excepting
>
> 1
> 2
> 4
>
> Does anyone know why?
>
> Cheers,
> Daniel
>
>

I don't understand why you would expect 1, 2, 4.

Perhaps parentheses will help the order of evaluation:

funcs = [(lambda x: x**i) for i in range( 5 )]

This gives:
1
16
81

Colin W.

[Ian Kelly]

On Wed, Jul 11, 2012 at 4:28 AM, Colin J. Williams <c...@ncf.ca> wrote:
> I don't understand why you would expect 1, 2, 4.

Because:

funcs[0](2) == 2 ** 0 == 1
funcs[1](2) == 2 ** 1 == 2
funcs[2](2) == 2 ** 2 == 4

> Perhaps parentheses will help the order of evaluation:
>
> funcs = [(lambda x: x**i) for i in range( 5 )]
>
> This gives:
> 1
> 16
> 81

No, that gives 16, 16, 16 just like the original. I don't understand
why you would expect 1, 16, 81, unless you have misread the code.

[woooee]

You should not be using lambda in this case
.for x in [2, 3]:
. funcs = [x**ctr for ctr in range( 5 )]
. for p in range(5):
. print x, funcs[p]
. print

[John Ladasky]

Exactly. It's threads like these which remind me why I never use lambda. I would rather give a function an explicit name and adhere to the familiar Python syntax, despite the two extra lines of code. I don't even like the name "lambda". It doesn't tell you what it is (unless you're John McCarthy), a function that you won't re-use and so you don't really need to give it a persistent name.

I haven't seen any lambdas in any Python library code, or in any of the third-party modules I use (numpy, matplotlib, Biopython). Do they exist? Because I have not been forced to do so, I haven't retained a space in the top drawer of my programming brain for lambda.

I know the historical reason that Python ended up with lambda, it was requested by people in the Lisp community. While I appreciate some of the Lisp-like features which did find their way into Python (especially being able to treat code as data, and functions as objects), I've found that lambda does nothing for me.

[Hans Mulder]

The list is called "funcs" because it is meant to contain functions.
Your code does not put functions in the list, so it doesn't do what
he wants.

He could do:

funcs = []
for i in range(5):
def f(x):
return x**i
funcs.append(f)

print funcs[0]( 2 )
print funcs[1]( 2 )
print funcs[2]( 2 )

.... and it will print 16, 16, 16 for the same reason as the lambda
version. On the other hand, this does what he wants:

funcs = []
for i in range(5):
def f(x, i=i):
return x**i
funcs.append(f)

print funcs[0]( 2 )
print funcs[1]( 2 )
print funcs[2]( 2 )

The lambda keyword is a red herring. The question is really about
functions, and how they handle non-local variables. The good news
is that 'lambda' and 'def' work exactly the same in that regards.
So once you understand the finer points about 'def', you no longer
have a reason to avoid 'lambda'.

Hope this helps,

-- HansM

[Steven D'Aprano]

If you change the requirements, it's always easy to solve problems. But
it is the wrong problem that you have solved.

The problem we have been asked to solve is to create a sequence of
function objects, so that they can be called later, when needed, *not* to
pre-calculate the results.

In this case, the most obvious solution is to store a local variable in
each function object with the value you want.

funcs = [(lambda x, i=i: x**i) for i in range(5)]

creates a list of five functions:

lambda x, i=0: x**i
lambda x, i=1: x**i
lambda x, i=2: x**i
and so on.

In this case, each function has two local variables, x and i, with i
having a default value. The function parameter i is bound to the value of
i when the function was created.

Because parameter defaults are calculated once, when the function is
created, this causes the value of i to stick to the newly created
function, and we get the result we need.

What happens if you don't use a parameter with a default value?

funcs = [(lambda x: x**i) for i in range(5)]

In this case, each function body contains one local variable, x, and one
non-local or global variable, i.

Because i is a non-local, the function doesn't store a value for it.
Instead, the function stores a lump of data pointing to just enough of
the environment to fetch the current value of the non-local i when
needed. Since all five functions are in the same environment, they all
see the same value of i when you call them, regardless of what the value
of i was when they were created.

This is little different from doing this:

i = 1
def f1(x): return x**i

i = 2
def f2(x): return x**i

i = 3
def f3(x): return x**i

Is there any surprise that all three functions return the same value?
They all point to the same global variable i. I'm not sure what it is
about lambda that fools people into thinking that it is different (I've
even been fooled myself!) but it is not.


--
Steven

[Steven D'Aprano]

On Wed, 11 Jul 2012 13:21:34 -0700, John Ladasky wrote:

> Exactly. It's threads like these which remind me why I never use
> lambda. I would rather give a function an explicit name and adhere to
> the familiar Python syntax, despite the two extra lines of code.

lambda is familiar Python syntax, or at least it should be if you
consider yourself an expert Python programmer.

And besides, this is not a problem with lambda. You get the *exact* same
problem with "ordinary" function definitions. Of course you do -- lambdas
*are* ordinary functions, they just have a more compact syntax.

funcs = []
for i in range(5):

def pwr(x):
return x**i
print("Testing 3**%d:" % i, pwr(3))
funcs.append(pwr)

for j in range(5):
print("Expected", 3**j, "got", funcs[j](3))

gives you *exactly* the same issue as with the lambda. This is not a
problem with lambdas, this is a scoping issue.

> I don't even like the name "lambda". It doesn't tell you what it is

Whereas "def" does?

If you can learn that "def" defines a function (as opposed to a class, a
module, or anything else really) than it's not that hard to learn that
"lambda" also defines a function.

Although I do wonder whether people would be less scared of lambda, and
less likely to imagine that it creates something with strange mysterious
properties different from "regular functions", if the keyword was called
"function" instead of lambda?

> (unless you're John McCarthy), a function that you won't re-use and so
> you don't really need to give it a persistent name.

Whether or not a function has a persistent name has nothing to do with
reuse. Anonymous functions can be reused. The driving motivation for
lambdas is for callback functions, which are frequently reused, but you
don't need or want to fill your global namespace up with potentially
hundreds of callback functions.

Lambda syntax is only incidentally good for saving a line or two. If the
only benefit of lambda was to save a line of code, that would be stupid.
The real reason for lambda is to avoid polluting your namespace with
unnecessary names.

Python allows us to easily and simply use anonymous strings, anonymous
ints, anonymous objects of all types. Why should functions be treated as
second-class?

> I haven't seen any lambdas in any Python library code, or in any of the
> third-party modules I use (numpy, matplotlib, Biopython). Do they
> exist? Because I have not been forced to do so, I haven't retained a
> space in the top drawer of my programming brain for lambda.

There are over 800 uses of lambda in the standard library:

[steve@ando ~]$ cd /usr/local/lib/python3.2
[steve@ando python3.2]$ grep lambda *.py */*.py | wc -l
829


(although some of the above are comments, doctests, etc.)

See, for example, configparser, decimal, functools, gettext, inspect,
pydoc, symtable, uuid, and others. pydoc and idlelib in particular make
heavy use of lambda, and the test suite is overflowing with them -- over
600 uses in total.


--
Steven

[Daniel Fetchinson]

Thank you Steve!
Precise and clear, as always!

[88888 Dihedral]

On Thursday, July 12, 2012 12:34:33 AM UTC+8, Ian wrote:
> On Wed, Jul 11, 2012 at 4:28 AM, Colin J. Williams &lt;c...@ncf.ca&gt; wrote:
> &gt; I don&#39;t understand why you would expect 1, 2, 4.

>
> Because:
>
> funcs[0](2) == 2 ** 0 == 1
> funcs[1](2) == 2 ** 1 == 2
> funcs[2](2) == 2 ** 2 == 4
>

> &gt; Perhaps parentheses will help the order of evaluation:
> &gt;
> &gt; funcs = [(lambda x: x**i) for i in range( 5 )]
> &gt;
> &gt; This gives:
> &gt; 1
> &gt; 16
> &gt; 81
>
> No, that gives 16, 16, 16 just like the original. I don&#39;t understand

> why you would expect 1, 16, 81, unless you have misread the code.

I'll contribute my way of python programming:

def powerb(x, b): #
return x**b


One functor is enough!

[Steven D'Aprano]

On Wed, 11 Jul 2012 20:39:45 -0700, 88888 Dihedral wrote:

> I'll contribute my way of python programming:
>
> def powerb(x, b): #
> return x**b

Here's a shorter version:

py> pow
<built-in function pow>


> One functor is enough!

Nothing we have been discussing in this thread has been a functor, either
in the Haskell sense or the C++ sense.



--
Steven

[Steven D'Aprano]

On Wed, 11 Jul 2012 21:05:30 -0400, Dennis Lee Bieber wrote:

> {non sequitur: I still recall my archaic C++ class with the OOAD
> assignment of designing said calculator -- we never had to implement
> one, just design the basic classes/methods/attributes [on 3x5 cards] for
> a four-banger. I managed to persuade the team I was on that an RPN
> calculator would be simpler to (potentially) implement... And THEN
> persuaded them to NOT use the equivalent of an ALU class, but to put the
> math work into each operation button...

"ALU class"?

Googling gives me no clue.



--
Steven

[88888 Dihedral]

On Thursday, July 12, 2012 12:34:33 AM UTC+8, Ian wrote:
> On Wed, Jul 11, 2012 at 4:28 AM, Colin J. Williams &lt;c...@ncf.ca&gt; wrote:

> &gt; I don&#39;t understand why you would expect 1, 2, 4.

>
> Because:
>
> funcs[0](2) == 2 ** 0 == 1
> funcs[1](2) == 2 ** 1 == 2
> funcs[2](2) == 2 ** 2 == 4
>

> &gt; Perhaps parentheses will help the order of evaluation:
> &gt;
> &gt; funcs = [(lambda x: x**i) for i in range( 5 )]
> &gt;
> &gt; This gives:
> &gt; 1
> &gt; 16

> &gt; 81
>
> No, that gives 16, 16, 16 just like the original. I don&#39;t understand

> why you would expect 1, 16, 81, unless you have misread the code.

I'll contribute my way of python programming:

def powerb(x, b): #
return x**b

One functor is enough!

[Steven D'Aprano]

On Wed, 11 Jul 2012 08:41:57 +0200, Daniel Fetchinson wrote:

> funcs = [ lambda x: x**i for i in range( 5 ) ]

Here's another solution:

from functools import partial
funcs = [partial(lambda i, x: x**i, i) for i in range(5)]


Notice that the arguments i and x are defined in the opposite order.
That's because partial only applies positional arguments from the left.
If there was a "right partial" that applies from the right, we could use
the built-in instead:

funcs = [rpartial(pow, i) for i in range(5)]



--
Steven

[Terry Reedy]

4th hit: Arithmetic Logic Unit (hardware) -- Wikipedia
ALU class -- software version or simulation thereof.

The hardware version of Dennis's distributed design would be
interesting. Spreadsheets are based in part on the same idea of
distributed logic and arithmetic.

--
Terry Jan Reedy

[88888 Dihedral]

On Thursday, July 12, 2012 11:51:04 AM UTC+8, Steven D&#39;Aprano wrote:
> On Wed, 11 Jul 2012 20:39:45 -0700, 88888 Dihedral wrote:
>

> &gt; I&#39;ll contribute my way of python programming:
> &gt;
> &gt; def powerb(x, b): #
> &gt; return x**b
>
> Here&#39;s a shorter version:
>
> py&gt; pow
> &lt;built-in function pow&gt;
>
>
> &gt; One functor is enough!

>
> Nothing we have been discussing in this thread has been a functor, either
> in the Haskell sense or the C++ sense.
>
>
>
> --
> Steven

Well, I encountered this kind of problems before in OOP.

I have to make sure my functor to keep the state variable values
for different objects that call the same functor to behave correctly
in order to avoid passing extra parameters in various objects using the same functor.

[John O'Hagan]

On Wed, 11 Jul 2012 13:21:34 -0700 (PDT)
John Ladasky <john_l...@sbcglobal.net> wrote:

> Exactly. It's threads like these which remind me why I never use lambda. I
> would rather give a function an explicit name and adhere to the familiar
> Python syntax, despite the two extra lines of code. I don't even like the
> name "lambda". It doesn't tell you what it is (unless you're John McCarthy),
> a function that you won't re-use and so you don't really need to give it a
> persistent name.
>
> I haven't seen any lambdas in any Python library code, or in any of the
> third-party modules I use (numpy, matplotlib, Biopython). Do they exist?
> Because I have not been forced to do so, I haven't retained a space in the
> top drawer of my programming brain for lambda.

[...]

+1 about the name, -1 about the usefulness. For example it's a clear and
concise way to pass a key argument to min, max, sort and friends:

sort(seq, key=lambda x: x.a)

The alternative is to have trivial function definitions floating around the
namespace which readers of the code have to search for.

--
John

[Steven D'Aprano]

On Wed, 11 Jul 2012 22:04:51 -0700, 88888 Dihedral wrote:

> I have to make sure my functor to keep the state variable values for
> different objects that call the same functor to behave correctly in
> order to avoid passing extra parameters in various objects using the
> same functor.

Yo dawg, I heard you liked functors, so I put a functor in your functor
so you can train your markov chains with extra functor parameters to
functor objects that are functor factory decorator functors.



--
Steven

[Franck Ditter]

In article <mailman.2007.1341988...@python.org>,

Daniel Fetchinson <fetch...@googlemail.com> wrote:

> > funcs = [ lambda x: x**i for i in range( 5 ) ]
> > print funcs[0]( 2 )
> > print funcs[1]( 2 )
> > print funcs[2]( 2 )
> >
> > This gives me
> >
> > 16
> > 16
> > 16
> >
> > When I was excepting
> >
> > 1
> > 2
> > 4
> >
> > Does anyone know why?

In Python 3.x :

funcs = [lambda x: x**i for i in range(5)]

list(map(lambda f: f(2),funcs)) --> [16, 16, 16, 16, 16]

Ooops, cool semantics :-)

In Racket Scheme (http://racket-lang.org), they seem to know lambdas :

#lang racket

;;; quick and dirty list comprehension syntax as a macro, for fun :

(define-syntax-rule (list-of expr for x in L)
(map (lambda (x) expr) L))

(list-of (sqr x) for x in (range 5)) --> (0 1 4 9 16)

(define funcs (list-of (lambda (x) (expt x i)) for i in (range 5)))
(map (lambda (f) (f 2)) funcs) --> (1 2 4 8 16)

franck

[Mark Lawrence]

functory :) Ooh, that's bad even by my standards.

--
Cheers.

Mark Lawrence.

[Robert Kern]

On 7/11/12 9:21 PM, John Ladasky wrote:
> Exactly. It's threads like these which remind me why I never use lambda. I would rather give a function an explicit name and adhere to the familiar Python syntax, despite the two extra lines of code. I don't even like the name "lambda". It doesn't tell you what it is (unless you're John McCarthy), a function that you won't re-use and so you don't really need to give it a persistent name.
>
> I haven't seen any lambdas in any Python library code, or in any of the third-party modules I use (numpy, matplotlib, Biopython). Do they exist? Because I have not been forced to do so, I haven't retained a space in the top drawer of my programming brain for lambda.

I count 162 uses in the Python standard library, 69 uses in numpy, 108 in
matplotlib, and 238 uses in Biopython. Adding in the unit tests for each would
add significantly to those counts.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Alister]

On Wed, 11 Jul 2012 08:43:11 +0200, Daniel Fetchinson wrote:

>> funcs = [ lambda x: x**i for i in range( 5 ) ]
>> print funcs[0]( 2 )
>> print funcs[1]( 2 )
>> print funcs[2]( 2 )
>>
>> This gives me
>>
>> 16 16 16
>>
>> When I was excepting
>>
>> 1
>> 2
>> 4
>>
>> Does anyone know why?
>
> And more importantly, what's the simplest way to achieve the latter? :)

Having read Steve's explanation in the other thread (which I think has
finally flipped the light switch on lambda for me) it only requires a
minor change

funcs=[ lambda x,y=i:x**y for i in range(5) ]

although I cant actually think why this construct would be needed in
practice, how are you actually using it


--
* Simunye is so happy she has her mothers gene's
<Dellaran> you better give them back before she misses them!

[Steven D'Aprano]

On Thu, 12 Jul 2012 09:44:15 +0000, Alister wrote:

> On Wed, 11 Jul 2012 08:43:11 +0200, Daniel Fetchinson wrote:
>
>>> funcs = [ lambda x: x**i for i in range( 5 ) ]

[...]

> Having read Steve's explanation in the other thread (which I think has
> finally flipped the light switch on lambda for me) it only requires a
> minor change
>
> funcs=[ lambda x,y=i:x**y for i in range(5) ]
>
> although I cant actually think why this construct would be needed in
> practice, how are you actually using it

I would expect that the example given is just sample code demonstrating
the problem. A slightly more realistic case might be something like
defining a bunch of callbacks for, say, a GUI application.

For example, the good ol' calculator app, where you have ten buttons for
digits 0-9. So you might give each button a callback function that
inserts its own digit into the text field:

buttons = [make_button(name=str(i)) for i in range(10)]


That works fine. So now you go to add a callback to each one:

buttons = [make_button(name=str(i), callback=lambda: FIELD.insert(i))
for i in range(10)]


and lo and behold, the ten buttons named "0" through "9" all insert 9.
The reason is that the callback functions aren't given the value of i,
but only the name[1] "i". By the time the callbacks are actually used,
i == 9 and all the buttons share the same value.



[1] Technically closures may not store values by name, but the principle
is sound: the function, when called later, looks up the current value of
variable i, which is not necessarily the same as it was when the closure
was originally defined.


--
Steven

[Ian Kelly]

You know, partial does handle keyword arguments correctly:

funcs = [partial(lambda x, i: x ** i, i=i) for i in range(5)]

This might be bad practice though if there are other arguments to the
right of the keywords that the eventual caller might want to specify,
as those arguments would then effectively be keyword-only. The error
message in that case is not particularly illuminating, either:

>>> def f(a, b, c):
... return (a, b, c)
...
>>> g = partial(f, b=42)
>>> g(3, 14)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: f() got multiple values for keyword argument 'b'

It would be a bit clearer if it instead noted that 'c' were
keyword-only in the 'g' argspec.

Cheers,
Ian

[Rotwang]

I don't think anyone has suggested this yet:

funcs = [(lambda i: lambda x: x**i)(i) for i in range(5)]

It's a bit more opaque than other solutions, but it fits in one line and
avoids defining functions with an implicit second argument, if that's
desirable for some reason.


--
I have made a thing that superficially resembles music:

http://soundcloud.com/eroneity/we-berated-our-own-crapiness

[Steven D'Aprano]

On Thu, 12 Jul 2012 18:23:10 +0100, Rotwang wrote:

> I don't think anyone has suggested this yet:
>
> funcs = [(lambda i: lambda x: x**i)(i) for i in range(5)]

Oooh, nice! A factory function solution. Excellent!



--
Steven

[rusi]

On Jul 11, 11:41 am, Daniel Fetchinson <fetchin...@googlemail.com>

wrote:
> funcs = [ lambda x: x**i for i in range( 5 ) ]

> print funcs[0]( 2 )
> print funcs[1]( 2 )
> print funcs[2]( 2 )
>
> This gives me
>
> 16
> 16
> 16
>
> When I was excepting
>
> 1
> 2
> 4
>
> Does anyone know why?
>

> Cheers,
> Daniel

Your expectations are reasonable. Heres the equivalent in Haskell
from which python has taken comprehensions.
---------------------------------
GHCi, version 7.4.1: http://www.haskell.org/ghc/ :? for help

Prelude> let funcs = [ (\ x -> x ^ i)| i <- [0..4]]
Prelude> (funcs !! 0)(2)
1
Prelude> (funcs !! 1)(2)
2
Prelude> (funcs !! 2)(2)
4
Prelude>

[Steven D'Aprano]

On Thu, 12 Jul 2012 21:33:40 -0700, rusi wrote:

> On Jul 11, 11:41 am, Daniel Fetchinson <fetchin...@googlemail.com>
> wrote:
>> funcs = [ lambda x: x**i for i in range( 5 ) ] print funcs[0]( 2 )
>> print funcs[1]( 2 )
>> print funcs[2]( 2 )
>>
>> This gives me
>>
>> 16
>> 16
>> 16
>>
>> When I was excepting
>>
>> 1
>> 2
>> 4
>>
>> Does anyone know why?
>>
>> Cheers,
>> Daniel
>
> Your expectations are reasonable.

You forget to finish that sentence.

"Your expectations are reasonable, for languages that don't have
variables which can actually vary."

*wink*

For purely functional languages like Haskell, the behaviour you show
below makes sense. Since Haskell doesn't allow variables to change their
value, once a closure sees i=1 (say), then it must *always* see i=1.

But that's not the case in Python, where the Haskell behaviour would be
unfortunate. Imagine if you did this:

VERBOSE = True

def function(arg):
if VERBOSE:
print("calling function with arg %r" % arg)
process(arg)

You would expect the function to honour changes to the variable VERBOSE,
would you not? Using Python's scoping rules for closures, it does. Using
Haskell's rules, it wouldn't.

> Heres the equivalent in Haskell from which python has taken
> comprehensions.
> ---------------------------------
> GHCi, version 7.4.1: http://www.haskell.org/ghc/ :? for help
>
> Prelude> let funcs = [ (\ x -> x ^ i)| i <- [0..4]]
> Prelude> (funcs !! 0)(2)
> 1
> Prelude> (funcs !! 1)(2)
> 2
> Prelude> (funcs !! 2)(2)
> 4
> Prelude>

--
Steven

[rusi]

On Jul 13, 11:36 am, Steven D'Aprano <steve

Heres a more appropriate analog

------------------------------------------

VERBOSE = True

def function(arg):
if VERBOSE:
print("calling function with arg %r" % arg)
process(arg)

def caller():
VERBOSE = False
function(1)

---------------------------------------------
Python semantics: function sees VERBOSE False
Haskell semantics: function sees VERBOSE True

[rusi]

To come back to the OPs question.

Variables can be assigned. Or they can be bound.
[C++ programmers will be familiar with the difference between
initialization and assignment]

List comprehensions are defined in terms of assignment to the local
variable rather than binding.
Hence the issue.

Below are two functions that simulate mapping (lambda x: x**i) where
the i's come from some given list

def binding_version(lst):
if not lst: return []
i = lst[0]
return [(lambda x: x ** i)] + binding_version(lst[1:])

def assign_version(lst):
ret = []
for i in lst:
ret.append(lambda x: x**i)
return ret
----------------------

>>> fs= binding_version([0,1,2])
>>> fs[0](2)
1
>>> fs[1](2)
2
>>> fs[2](2)
4

>>> fs= assign_version([0,1,2])
>>> fs[0](2)
4
>>> fs[1](2)
4
>>> fs[2](2)
4

[Prasad, Ramit]

> VERBOSE = True
>
> def function(arg):
> if VERBOSE:
> print("calling function with arg %r" % arg)
> process(arg)
>
> def caller():
> VERBOSE = False
> function(1)
>
> ---------------------------------------------
> Python semantics: function sees VERBOSE False
> Haskell semantics: function sees VERBOSE True

>>> def caller():
... VERBOSE = False
... function(1)
>>> def function(arg):
... if VERBOSE:
... print("calling function with arg %r" % arg)
...
>>> VERBOSE = True
>>> caller()
calling function with arg 1

I might be being OCD, but caller needs `global VERBOSE` for that to
work as you explain.

Ramit


Ramit Prasad | JPMorgan Chase Investment Bank | Currencies Technology
712 Main Street | Houston, TX 77002
work phone: 713 - 216 - 5423

--

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.

[rusi]

On Jul 13, 9:12 pm, "Prasad, Ramit" <ramit.pra...@jpmorgan.com> wrote:
> > VERBOSE = True
>
> > def function(arg):
> >     if VERBOSE:
> >        print("calling function with arg %r" % arg)
> >     process(arg)
>
> > def caller():
> >     VERBOSE = False
> >     function(1)
>
> > ---------------------------------------------
> > Python semantics: function sees VERBOSE False
> > Haskell semantics: function sees VERBOSE True
> >>> def caller():
>
> ...     VERBOSE = False
> ...     function(1)>>> def function(arg):
>
> ...     if VERBOSE:
> ...        print("calling function with arg %r" % arg)
> ...    >>> VERBOSE = True
> >>> caller()
>
> calling function with arg 1
>
> I might be being OCD, but caller needs `global VERBOSE` for that to
> work as you explain.

Ok let me restate: if python were to work that way (without the
global) we could say either
a Python chooses to have dynamic scoping of variables
or
b There is a bug in python's scoping rules

I would guess that most younger folks (who've not seen lisp and apl)
would choose b

We can say the same analogously in the context of ZF expressions when
the i leaks as it does in the OPs example.

Another tack on the same question: python 3 cleaned up the variable
leakage from inside ZFs to outside. It missed cleaning up the leakage
from one step to next.

tl;dr version: Beware of mixing up functional and imperative
programming
Double-beware when you are a language designer

[Chris Angelico]

On Sat, Jul 14, 2012 at 2:46 AM, rusi <rusto...@gmail.com> wrote:
> Ok let me restate: if python were to work that way (without the
> global) we could say either
> a Python chooses to have dynamic scoping of variables
> or
> b There is a bug in python's scoping rules

Or c, there's a declaration at the top:

from __future__ import variable_declarations

Like braces, it's a MASSIVE improvement to the language, and it's part
of a huge conspiracy that it isn't there by default. But all it takes
is a little future statement and you're on Python 4000!

</troll>

ChrisA
(I shouldn't post at half past three in the morning. I get stupid... er.)

[Hans Mulder]

On 13/07/12 18:12:40, Prasad, Ramit wrote:
>> VERBOSE = True
>>
>> def function(arg):
>> if VERBOSE:
>> print("calling function with arg %r" % arg)
>> process(arg)
>>
>> def caller():
>> VERBOSE = False
>> function(1)
>>
>> ---------------------------------------------
>> Python semantics: function sees VERBOSE False
>> Haskell semantics: function sees VERBOSE True

>>>> def caller():
> ... VERBOSE = False
> ... function(1)
>>>> def function(arg):
> ... if VERBOSE:
> ... print("calling function with arg %r" % arg)
> ...
>>>> VERBOSE = True
>>>> caller()
> calling function with arg 1
>
> I might be being OCD, but caller needs `global VERBOSE` for that to
> work as you explain.

That would be quite different from what Rusi is after.

If you add `global VERBOSE` to `caller`, then there is only one
variable named `VERBOSE` and what `function` does, depends on
the most recent assignment to that variable.

If you remove your `global VERBOSE`, then there are two
variables by that name, one global and one local to `caller`.
In that case, there is the question of which one `function`
will use.

The function `function` refers to a variable `VERBOSE` that
isn't local. In some programming langauages, the interpreter
would then scan the call stack at run-time, looking for a scope
where that name is defined. It would find the local one in
`caller`. This is known as "dynamic binding".

Other interpreters use the `VERBOSE` that was in scope at
the point in the program text where `function` was defined.
In this case, that would be the global one. This is called
"lexical binding".

Some programming languages allow you to indicate on a per-
variable basis whether you want dynamic or lexical binding.

Python is firmly in the lexical camp. Dynamic binding is not
available in Python, and never will be.


Hope this helps,

-- HansM

[Prasad, Ramit]

But that is not what Rusi writes.
"Python semantics: function sees VERBOSE False" <- function
will not see False without the use of global.

> The function `function` refers to a variable `VERBOSE` that
> isn't local. In some programming langauages, the interpreter
> would then scan the call stack at run-time, looking for a scope
> where that name is defined. It would find the local one in
> `caller`. This is known as "dynamic binding".
>
> Other interpreters use the `VERBOSE` that was in scope at
> the point in the program text where `function` was defined.
> In this case, that would be the global one. This is called
> "lexical binding".
>
> Some programming languages allow you to indicate on a per-
> variable basis whether you want dynamic or lexical binding.
>
> Python is firmly in the lexical camp. Dynamic binding is not
> available in Python, and never will be.

True and a good explanation, but not what I understood
Rusi to mean.

Ramit

[Ian Kelly]

On Fri, Jul 13, 2012 at 11:53 AM, Hans Mulder <han...@xs4all.nl> wrote:
> The function `function` refers to a variable `VERBOSE` that
> isn't local. In some programming langauages, the interpreter
> would then scan the call stack at run-time, looking for a scope
> where that name is defined. It would find the local one in
> `caller`. This is known as "dynamic binding".
>
> Other interpreters use the `VERBOSE` that was in scope at
> the point in the program text where `function` was defined.
> In this case, that would be the global one. This is called
> "lexical binding".
>
> Some programming languages allow you to indicate on a per-
> variable basis whether you want dynamic or lexical binding.
>
> Python is firmly in the lexical camp. Dynamic binding is not
> available in Python, and never will be.

I don't believe that dynamic vs. lexical binding is what rusi was
attempting to describe. If he was, then Python and Haskell would be a
bad comparison since both are lexical. Rather, I think what he was
trying to show was capture by reference vs. capture by value in the
context of closures. Python uses capture by reference, and so the
upvalue is the value of that reference at the time the closure is
called. Haskell uses capture by value, and the upvalue is the value
at the time of definition.

I've also seen the distinction described as "early" vs. "late" binding
on this list, but I'm not sure how precise that is -- I believe that
terminology more accurately describes whether method and attribute
names are looked up at compile-time or at run-time, late binding being
the feature that makes duck typing possible.

Cheers,
Ian

[Hans Mulder]

On 13/07/12 20:54:02, Ian Kelly wrote:
> I've also seen the distinction described as "early" vs. "late" binding
> on this list, but I'm not sure how precise that is -- I believe that
> terminology more accurately describes whether method and attribute
> names are looked up at compile-time or at run-time, late binding being
> the feature that makes duck typing possible.

I think that these terms describe the problem at the start of
this thread: the OP was expecting early binding. However, Python
does late binding, or "acts funny" as it says in the subject line.


-- HansM

[rusi]

On Jul 13, 10:53 pm, Hans Mulder <han...@xs4all.nl> wrote:
> If you add `global VERBOSE` to `caller`, then there is only one
> variable named `VERBOSE` and what `function` does, depends on
> the most recent assignment to that variable.
>
> If you remove your `global VERBOSE`, then there are two
> variables by that name, one global and one local to `caller`.
> In that case, there is the question of which one `function`
> will use.

Good point. See below.

>
> The function `function` refers to a variable `VERBOSE` that
> isn't local.  In some programming langauages, the interpreter
> would then scan the call stack at run-time, looking for a scope
> where that name is defined.  It would find the local one in
> `caller`.  This is known as "dynamic binding".
>
> Other interpreters use the `VERBOSE` that was in scope at
> the point in the program text where `function` was defined.
> In this case, that would be the global one.  This is called
> "lexical binding".
>
> Some programming languages allow you to indicate on a per-
> variable basis whether you want dynamic or lexical binding.
>
> Python is firmly in the lexical camp.  Dynamic binding is not
> available in Python, and never will be.

Thats a good intention. The question is whether it is uniformly
realized.

Consider the following

def foo(x):
i = 100
if x:
j = [i for i in range(10)]
return i
else:
return i

In python 2 two different 'i's could be returned. In python3 only one
i can be returned.
One could call it dynamic binding. Evidently Guido thinks it a bug
which is why he changed it.

The leakage of i in the OPs question is the same kind of bug.

tl;dr version:
This is 2012. Dynamic binding is considered a bug even by the lisp
community where it originated.
Lets at least call it a feature and a gotcha

[Steven D'Aprano]

On Fri, 13 Jul 2012 19:31:24 -0700, rusi wrote:

> Consider the following
>
> def foo(x):
> i = 100
> if x:
> j = [i for i in range(10)]
> return i
> else:
> return i

A simpler example:

def foo():
i = 100

j = [i for i in range(10)]
return i

In Python 3, foo() returns 100; in Python 2, it returns 9.

> In python 2 two different 'i's could be returned. In python3 only one i
> can be returned.
> One could call it dynamic binding.

One could also call it a tractor, but it isn't either of those things.

The difference is whether or not list comprehensions create their own
scope. In Python 2, they don't; in Python 3, they do. Personally I don't
have an opinion as to which is better, but in neither case does this have
any thing to do with lexical versus dynamic binding.

> Evidently Guido thinks it a bug which is why he changed it.

It was a case that either list comps be changed to *not* expose their
loop variable, or generator expressions be changed *to* expose their loop
variable. The Python 2 situation where list comps and gen expressions
have opposite behaviour was unfortunate.

> The leakage of i in the OPs question is the same kind of bug.

Not at all. The OP was *deliberately* creating a closure using i -- under
those circumstances, it would be a bug if i *didn't* leak.

The OP's gotcha was:

1) he expected the closure to use early binding, where i in each function
was bound to the value of i in the enclosing scope at the moment the
function was defined;

2) but Python actually uses late binding for closures, where the value of
i in each function is set to the value of i in the enclosing scope when
the function is called.

As far as I can tell, Python always uses late binding for scopes; the
only time it does early binding is for default values of function
parameters.



--
Steven

[rusi]

On Jul 14, 8:43 am, Steven D'Aprano <steve

+comp.lang.pyt...@pearwood.info> wrote:
> On Fri, 13 Jul 2012 19:31:24 -0700, rusi wrote:
> > Consider the following
>
> > def foo(x):
> >     i = 100
> >     if x:
> >         j = [i for i in range(10)]
> >         return i
> >     else:
> >         return i
>
> A simpler example:
>
> def foo():
>     i = 100
>     j = [i for i in range(10)]
>     return i
>
> In Python 3, foo() returns 100; in Python 2, it returns 9.

You did not get the point.

Converting my example to your format:

def foo_steven(n):
i = 100
j = [i for i in range(n)]
return i

$ python3
Python 3.2.3 (default, Jun 26 2012, 00:38:09)
[GCC 4.7.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def foo_steven(n):
... i = 100
... j = [i for i in range(n)]
... return i
...
>>> foo_steven(0)
100
>>> foo_steven(4)
100
>>>
$ python
Python 2.7.3rc2 (default, Apr 22 2012, 22:35:38)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def foo_steven(n):
... i = 100
... j = [i for i in range(n)]
... return i
...
>>> foo_steven(0)
100
>>> foo_steven(3)
2
>>>

Python 2:
When n>0 comprehension scope i is returned
When n=0 function scope i is returned

Python 3: The return statement is lexically outside the comprehension
and so that outside-scope's i is returned in all cases.

[Steven D'Aprano]

On Fri, 13 Jul 2012 21:53:10 -0700, rusi wrote:

> On Jul 14, 8:43 am, Steven D'Aprano <steve
> +comp.lang.pyt...@pearwood.info> wrote:
>> On Fri, 13 Jul 2012 19:31:24 -0700, rusi wrote:
>> > Consider the following
>>
>> > def foo(x):
>> >     i = 100
>> >     if x:
>> >         j = [i for i in range(10)]
>> >         return i
>> >     else:
>> >         return i
>>
>> A simpler example:
>>
>> def foo():
>>     i = 100
>>     j = [i for i in range(10)]
>>     return i
>>
>> In Python 3, foo() returns 100; in Python 2, it returns 9.
>
> You did not get the point.

I got the point. I just thought it was unnecessarily complex and that it
doesn't demonstrate what you think it does.

> Converting my example to your format:
>
> def foo_steven(n):
> i = 100
> j = [i for i in range(n)]
> return i
>
> $ python3
> Python 3.2.3 (default, Jun 26 2012, 00:38:09) [GCC 4.7.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> def foo_steven(n):
> ... i = 100
> ... j = [i for i in range(n)]
> ... return i
> ...
>>>> foo_steven(0)
> 100
>>>> foo_steven(4)
> 100

Yes, we know that in Python 3, list comprehensions create their own
scope, and the loop variable does not leak.

> $ python
> Python 2.7.3rc2 (default, Apr 22 2012, 22:35:38) [GCC 4.6.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> def foo_steven(n):
> ... i = 100
> ... j = [i for i in range(n)]
> ... return i
> ...
>>>> foo_steven(0)
> 100
>>>> foo_steven(3)
> 2

Yes, we know that in Python 2, list comprehensions don't create their own
scope, and consequently the list variable does leak.

> Python 2:
> When n>0 comprehension scope i is returned
> When n=0 function scope i is returned

Incorrect.

In Python 2, *there is no comprehension scope*. There is only local
scope. In Python 2, regardless of the value of n, the local variable i is
ALWAYS returned. It just happens that sometimes the local variable i is
modified by the list comprehension, and sometimes it isn't. In Python 2,
this is no more mysterious than this piece of code:

def example(n):
i = 100
for i in range(n):
pass
return i

py> example(0)
100
py> example(4)
3

If the loop doesn't execute, the loop variable isn't modified. In Python
2, it doesn't matter whether you use a for-loop or a list comprehension,
the loop variable is local to the function.

> Python 3: The return statement is lexically outside the comprehension
> and so that outside-scope's i is returned in all cases.

Correct. In Python 3, list comprehensions now match generator expressions
and introduce their own scope which does not effect the local function
scope.

Some history:

http://bugs.python.org/issue510384
http://bugs.python.org/issue1110705




--
Steven

[88888 Dihedral]

Alister於 2012年7月12日星期四UTC+8下午5時44分15秒寫道：

> On Wed, 11 Jul 2012 08:43:11 +0200, Daniel Fetchinson wrote:
>

> &gt;&gt; funcs = [ lambda x: x**i for i in range( 5 ) ]
> &gt;&gt; print funcs[0]( 2 )
> &gt;&gt; print funcs[1]( 2 )
> &gt;&gt; print funcs[2]( 2 )
> &gt;&gt;
> &gt;&gt; This gives me
> &gt;&gt;
> &gt;&gt; 16 16 16
> &gt;&gt;
> &gt;&gt; When I was excepting
> &gt;&gt;
> &gt;&gt; 1
> &gt;&gt; 2
> &gt;&gt; 4
> &gt;&gt;
> &gt;&gt; Does anyone know why?
> &gt;
> &gt; And more importantly, what&#39;s the simplest way to achieve the latter? :)
>
> Having read Steve&#39;s explanation in the other thread (which I think has

> finally flipped the light switch on lambda for me) it only requires a
> minor change
>
> funcs=[ lambda x,y=i:x**y for i in range(5) ]
>
> although I cant actually think why this construct would be needed in
> practice, how are you actually using it
>
>
> --

> * Simunye is so happy she has her mothers gene&#39;s
> &lt;Dellaran&gt; you better give them back before she misses them!

Uhn, there are 5 objects in the list if not factored well to be executed in
the run time.

[Steven D'Aprano]

On Fri, 13 Jul 2012 12:54:02 -0600, Ian Kelly wrote:

> On Fri, Jul 13, 2012 at 11:53 AM, Hans Mulder <han...@xs4all.nl> wrote:
>> The function `function` refers to a variable `VERBOSE` that isn't
>> local. In some programming langauages, the interpreter would then scan
>> the call stack at run-time, looking for a scope where that name is
>> defined. It would find the local one in `caller`. This is known as
>> "dynamic binding".
>>
>> Other interpreters use the `VERBOSE` that was in scope at the point in
>> the program text where `function` was defined. In this case, that would
>> be the global one. This is called "lexical binding".
>>
>> Some programming languages allow you to indicate on a per- variable
>> basis whether you want dynamic or lexical binding.
>>
>> Python is firmly in the lexical camp. Dynamic binding is not available
>> in Python, and never will be.

I don't remember whether it is Javascript or PHP that uses dynamic
binding, but whichever it is, it is generally considered to be a bad
idea, at least as the default or only behaviour.

Bash is another language with dynamic binding. Some very old versions of
Lisp use dynamic binding, because it was the easiest to implement. Most
modern languages use lexical (also known as static) binding, because it
is more sensible.

Here is an illustration of the difference: suppose we have two modules,
library.py and main.py:

# library.py
x = 23
def func(y):
return x + y

# main.py
import library
x = 1000
print func(1)


If main.py prints 24 (and it does), then Python is using lexical scoping.
But if it prints 1001 (which it doesn't), then it is using dynamic
scoping. The obvious problem with dynamic binding is that the behaviour
of a function may vary depending on where you call it.

> I don't believe that dynamic vs. lexical binding is what rusi was
> attempting to describe. If he was, then Python and Haskell would be a
> bad comparison since both are lexical. Rather, I think what he was
> trying to show was capture by reference vs. capture by value in the
> context of closures. Python uses capture by reference, and so the
> upvalue is the value of that reference at the time the closure is
> called. Haskell uses capture by value, and the upvalue is the value at
> the time of definition.

I don't think "by reference" versus "by value" are good terms to use
here, since they risk conflating the issue with "call by reference"
versus "call by value" semantics. I prefer "late" versus "early", as you
suggest below.

> I've also seen the distinction described as "early" vs. "late" binding
> on this list, but I'm not sure how precise that is -- I believe that
> terminology more accurately describes whether method and attribute names
> are looked up at compile-time or at run-time,

Not necessarily *compile* time, but the distinction is between when the
function is defined (which may at compile time, or it may be at run time)
versus when the function is called. I think that gets to the heart of the
issue, not whether the capture copies a value or a reference. At some
point, the capture must make use of the value: whether it does so via a
direct C-style memory location copy, or an object pointer, or some other
mechanism, is irrelevant. What matters is whether that value is grabbed
at the time the closure is created, or when the closure is called.

> late binding being the
> feature that makes duck typing possible.

That is conflating two entirely distinct subjects. Python 1.5 had duck-
typing but no closures, so the one certainly does not depend on the other.

Duck-typing is more a philosophy than a language feature: the language
must be typed, and the programmer must prefer to program to a protocol or
a specification rather than to membership of a type.


--
Steven

[Chris Angelico]

On Sun, Jul 15, 2012 at 9:29 AM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
> Not necessarily *compile* time, but the distinction is between when the
> function is defined (which may at compile time, or it may be at run time)
> versus when the function is called.

I'd treat the def/lambda statement as "compile time" and the ()
operator as "run time".

ChrisA

[Steven D'Aprano]

But function definitions occur at run time, not compile time -- they are
executable statements, not instructions to the compiler to define a
function.

For example:

py> dis("def f(x): return x+1") # Python 3.2
1 0 LOAD_CONST 0 (<code object f at 0xb7b57de0,
file "<dis>", line 1>)
3 MAKE_FUNCTION 0
6 STORE_NAME 0 (f)
9 LOAD_CONST 1 (None)
12 RETURN_VALUE

The code object is pre-compiled at compile time, but the function and
name-binding (the "def") doesn't occur until runtime.

At compile time, Python parses the source code and turns it into byte-
code. Class and function definitions are executed at run time, the same
as any other statement.

I'm not sure if this is a difference that makes a difference or not; I
think it is, but don't know enough about how closures and scoping rules
work in other languages to be sure that it does make a difference.


--
Steven

[Chris Angelico]

On Sun, Jul 15, 2012 at 6:32 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
> At compile time, Python parses the source code and turns it into byte-
> code. Class and function definitions are executed at run time, the same
> as any other statement.

Between the parse step and the 'def' execution, a code object is
created. When you call it, that code object already exists. Nothing
else really matters, unless there's a bug in the Python optimizer or
something weird like that. The nearest thing Python _has_ to a
"compile time" is the execution of def.

ChrisA

[Terry Reedy]

On 7/15/2012 4:32 AM, Steven D'Aprano wrote:
> On Sun, 15 Jul 2012 10:49:48 +1000, Chris Angelico wrote:
>
>> On Sun, Jul 15, 2012 at 9:29 AM, Steven D'Aprano
>> <steve+comp....@pearwood.info> wrote:
>>> Not necessarily *compile* time, but the distinction is between when the
>>> function is defined (which may at compile time, or it may be at run
>>> time) versus when the function is called.
>>
>> I'd treat the def/lambda statement as "compile time" and the () operator
>> as "run time".
>
> But function definitions occur at run time, not compile time -- they are
> executable statements, not instructions to the compiler to define a
> function.

The () operator is 'call time'. The main points are
a) the execution of def statements and lambda expressions and the
execution of calls happen at different run times.
b) default arg objects are calculated at def/lambda time.
c) names in def bodies and lambda expressions are resolved at call time.
and
d) people more often forget c) when thinking about lambda expressions
that for def statements.

--
Terry Jan Reedy

[Hans Mulder]

"Compile time" is the phase when your Python code is turned into byte
code, or a SyntaxError is raised. In this phase, a "code object" is
created for the function.

"Function definition time" is when the "def" command is executed.
In this phase, default arguments are computed, and a "function object"
is created. Among the attributes of the function object are the code
object, and "cell" objects containing the bindings for its non-local
variables. These bindings are used to read the variable's current
value at the time the function uses the variable.


-- HansM

[Robert Miles]

On 7/11/2012 11:39 PM, Dennis Lee Bieber wrote:
> On 12 Jul 2012 03:53:03 GMT, Steven D'Aprano
> <steve+comp....@pearwood.info> declaimed the following in
> gmane.comp.python.general:
>
>
>> "ALU class"?
>>
>> Googling gives me no clue.
>
> Arithmetic/Logic Unit
>
> http://en.wikipedia.org/wiki/Arithmetic_logic_unit
> http://en.wikipedia.org/wiki/74181
> {diversion: http://en.wikipedia.org/wiki/VAX-11/780 -- incredible...
> that used to be considered a "super-mini" when I worked on them; and now
> would be shamed by most laptops except for the ability to support so
> many users concurrently (let me know when a Windows laptop supports 32
> VT-100 class connections <G>)}

Installing Cygwin (a Linux emulation) under Windows appears to add some
VT-100 support but without an easy way to find documentation on whether
it can support 32 of them or not.

I used to work on a VAX 11/780 and also a VAX 8600.

Cygwin has a version of Python available, in case you're interested.

Robert Miles