Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

868 views
Skip to first unread message

vasudevram

unread,
Mar 21, 2014, 4:42:53 PM3/21/14
to

Hi list,

Can anyone - maybe one of the Python language core team, or someone with knowledge of the internals of Python - can explain why this code works, and whether the different occurrences of the name x in the expression, are in different scopes or not? :

x = [[1,2], [3,4], [5,6]]
[x for x in x for x in x]

I saw this on a Hacker News thread about Python, and here is a post I wrote that gives more details about it, including relevant links, how I found that it can be extended to a triply-nested list, and my thoughts about the scope issue:

http://jugad2.blogspot.in/2014/03/flatten-list-of-lists-with-list.html

A few people commented about it, both on my blog, and on the Python Reddit where I also submitted my post, but I'm not sure I'm convinced of their reasoning or understand it, hence posting the question here.

Thanks,
Vasudev Ram
www.dancingbison.com
jugad2.blogspot.com

Rustom Mody

unread,
Mar 21, 2014, 4:54:00 PM3/21/14
to
Lets try without comprehending comprehensions :-)

>>> x=[[1,2],[3,4]]
>>> for x in x:
... for x in x:
... print x
...
1
2
3
4
>>>

vasudevram

unread,
Mar 21, 2014, 4:56:09 PM3/21/14
to
On Saturday, March 22, 2014 2:24:00 AM UTC+5:30, Rustom Mody wrote:
> Lets try without comprehending comprehensions :-)
> >>> x=[[1,2],[3,4]]
>
> >>> for x in x:
>
> ... for x in x:
>
> ... print x
>
> ...
>
> 1
>
> 2
>
> 3
>
> 4

Nice and all, thanks, but doesn't answer the question.

Rustom Mody

unread,
Mar 21, 2014, 5:09:19 PM3/21/14
to
Which is?



A 'for' introduces a scope:

>>> x = 42
>>> for x in [1,2,3]:
... print x
...
1
2
3

No sign of the 42 --v ie the outer x -- inside because of scope

And so we can do:

>>> x = [1,2,3]
>>> for x in x:
... print x
...
1
2
3

which implies that in a "for var in exp: ..."
the exp is evaluated in outer scope whereas the var has a new scope inside
the "..."

Now repeatedly apply that principle to the nested for.
Same principle for nested for in a comprehension.

Ian Kelly

unread,
Mar 21, 2014, 5:30:10 PM3/21/14
to Python
On Fri, Mar 21, 2014 at 3:09 PM, Rustom Mody <rusto...@gmail.com> wrote:
> A 'for' introduces a scope:

This is false.

>>>> x = 42
>>>> for x in [1,2,3]:
> ... print x
> ...
> 1
> 2
> 3
>
> No sign of the 42 --v ie the outer x -- inside because of scope

Try printing x again *after* the for loop:

>>> x = 42
>>> for x in [1,2,3]:
... print(x)
...
1
2
3
>>> print(x)
3

Notice that it's still 3. If the x in the for loop were really a
separately scoped variable, we would have expected to see 42 here. In
fact the x used in the for loop and the x used outside of it are the
same x variable.

The real reason that the OP's example works is because each value that
the x variable holds happens to only need to be read once. Start with
this code, and work through it line-by-line:

x = [[1, 2], [3, 4]]
for x in x:
for x in x:
print(x)

Initially, x is assigned the list. Then we enter the outer for loop.
The expression x is evaluated once to be iterated over. The first
value that the for loop iterator yields is [1, 2], which is then
assigned to x. Now we enter the inner for loop. The expression x is
again evaluated once to be iterated over. It first yields 1, which is
assigned to x, and then printed. The inner for loop iterator then
yields 2, which is also assigned to x and printed. The inner for loop
iterator then raises StopIteration, and the inner for loop terminates.
The outer for loop iterator then yields [3, 4], and this is assigned
to x. The inner for loop evaluates this expression and runs as
before. Afterward, the outer for loop iterator raises StopException,
and that for loop terminates. The final value of x here will be 4:

>>> x = [[1, 2], [3, 4]]
>>> for x in x:
... for x in x:
... print(x)
...
1
2
3
4
>>> print(x)
4

As I hope this makes clear, no nested scopes are needed to make this
happen. It works because nothing that is stored in x needs to remain
there after the next thing is stored in x.

Gregory Ewing

unread,
Mar 21, 2014, 5:34:48 PM3/21/14
to
Rustom Mody wrote:

> A 'for' introduces a scope:

No, it doesn't!

>>>>x = 42
>>>>for x in [1,2,3]:
> ... print x
> ...
> 1
> 2
> 3
>
> No sign of the 42 --v ie the outer x -- inside because of scope

You're right that there's no sign of the 42, but it's
*not* because of scope, as you'll see if you do one
more print:

>>> print x
3

> And so we can do:
>
>
>>>>x = [1,2,3]
>>>>for x in x:
> ... print x
> ...
> 1
> 2
> 3

Again, if you print x after the loop has finished,
you'll find that it's no longer bound to the original
list. This is because Python's for-loop does *not*
introduce a new scope -- there's only one x, and
the for-loop is sharing it with the rest of the code.

The real question is why the for-loop works in *spite*
of this fact.

The answer is that the for-loop machinery keeps an
internal reference to the list being iterated over,
so once the loop has started, it doesn't matter what
x is bound to any more.

--
Greg

Rustom Mody

unread,
Mar 21, 2014, 10:06:06 PM3/21/14
to
On Saturday, March 22, 2014 3:00:10 AM UTC+5:30, Ian wrote:
> On Fri, Mar 21, 2014 at 3:09 PM, Rustom Mody wrote:
> > A 'for' introduces a scope:

> This is false.


And

On Saturday, March 22, 2014 3:04:48 AM UTC+5:30, Gregory Ewing wrote:

> > A 'for' introduces a scope:

> No, it doesn't!


Ha -- funny that *I* missed that one. Thanks both Ian and Gregory

In fact one of my grumbles against python is that list comprehension's
are a poor imitation of haskell's comprehensions.

And then I promptly forgot about it!

Actually there are two leakages in python 2, one of which is corrected
in python 3.

One: a comprehension variable leaks outside after the comprehension
This is corrected in python3.
Two: A comprehension variable is not bound but reassigned across the
comprehension. This problem remains in python3 and causes weird behavior when
lambdas are put in a comprehension

>>> fl = [lambda y : x+y for x in [1,2,3]]
>>> [fl[i](2) for i in [0,1,2]]
[5, 5, 5]

The same in haskell:

Prelude> let fl = [\ y -> x + y | x <- [1,2,3]]
Prelude> [(fl!!i) 0 | i<- [0,1,2]]
[1,2,3]

Chris Angelico

unread,
Mar 21, 2014, 10:41:27 PM3/21/14
to pytho...@python.org
On Sat, Mar 22, 2014 at 1:06 PM, Rustom Mody <rusto...@gmail.com> wrote:
> Two: A comprehension variable is not bound but reassigned across the
> comprehension. This problem remains in python3 and causes weird behavior when
> lambdas are put in a comprehension
>
>>>> fl = [lambda y : x+y for x in [1,2,3]]
>>>> [fl[i](2) for i in [0,1,2]]
> [5, 5, 5]

To clarify, what you're saying here is that x in the first
comprehension's closures should be bound to separate values for x,
yes?

I'm not sure how that ought to be done. Having closures that can
reference and modify each other's variables is important.

def func_pair():
x = 0
def inc():
nonlocal x; x+=1
return x
def dec():
nonlocal x; x-=1
return x
return inc, dec

fooup, foodn = func_pair()
barup, bardn = func_pair()
>>> fooup(), fooup(), fooup(), foodn()
(1, 2, 3, 2)
>>> barup(), barup(), bardn(), bardn()
(1, 2, 1, 0)

Those functions are fundamentally linked. Very useful with callbacks.
A nice alternative to doing everything with bound methods.

So if that's not going to be broken, how is this fundamentally different?

def func_loop():
for x in 1,2,3:
yield (lambda: x)

one, two, three = func_loop()
one(), one(), two(), two(), three(), three()

This one does NOT work the way the names imply, and I can see that
you'd like to fix it. But I can't pinpoint a significant difference
between them. How do you distinguish?

ChrisA

Rustom Mody

unread,
Mar 22, 2014, 12:39:55 AM3/22/14
to
On Saturday, March 22, 2014 8:11:27 AM UTC+5:30, Chris Angelico wrote:
> On Sat, Mar 22, 2014 at 1:06 PM, Rustom Mody wrote:
> > Two: A comprehension variable is not bound but reassigned across the
> > comprehension. This problem remains in python3 and causes weird behavior when
> > lambdas are put in a comprehension
> >>>> fl = [lambda y : x+y for x in [1,2,3]]
> >>>> [fl[i](2) for i in [0,1,2]]
> > [5, 5, 5]

> To clarify, what you're saying here is that x in the first
> comprehension's closures should be bound to separate values for x,
> yes?

Yes

> I'm not sure how that ought to be done.

Thats an implementation question -- see below.

> Having closures that can
> reference and modify each other's variables is important.

Yes

> def func_pair():
> x = 0
> def inc():
> nonlocal x; x+=1
> return x
> def dec():
> nonlocal x; x-=1
> return x
> return inc, dec

> fooup, foodn = func_pair()
> barup, bardn = func_pair()
> >>> fooup(), fooup(), fooup(), foodn()
> (1, 2, 3, 2)
> >>> barup(), barup(), bardn(), bardn()
> (1, 2, 1, 0)

> Those functions are fundamentally linked. Very useful with callbacks.
> A nice alternative to doing everything with bound methods.

Yes

> So if that's not going to be broken, how is this fundamentally different?

> def func_loop():
> for x in 1,2,3:
> yield (lambda: x)

Thats using a for-loop
A 'for' in a comprehension carries a different intention, the matching names
being merely coincidental.

This 'pun' causes cognitive dissonance in all these questions, including
my gaffe above

> one, two, three = func_loop()
> one(), one(), two(), two(), three(), three()

> This one does NOT work the way the names imply, and I can see that
> you'd like to fix it. But I can't pinpoint a significant difference
> between them. How do you distinguish?

Using closures for carrying state is a different question

As for comprehensions, the appropriate *intention* would be like this:

Given

fl = [lambda y : x+y for x in [1,2,3]]

It means:

def rec(l):
if not l: return []
else:
x,ll = l[0],l[1:]
return [lambda y: x + y] + rec(ll)

followed by
fl = rec([1,2,3])

Naturally a reasonable *implementation* would carry this *intention* more
efficiently with standard techniques like
1. Using list extend/append methods
2. Using lambda y, x=x: x+y

Inside an implementation this is fine
Forcing such tricks as kosher on a programmer is not (IMHO)

[But then I find Lisp and much of basic haskell natural and most of C++ not,
so my views are likely prejudiced :-)

Steven D'Aprano

unread,
Mar 22, 2014, 12:47:36 AM3/22/14
to
On Fri, 21 Mar 2014 19:06:06 -0700, Rustom Mody wrote:

> Two: A comprehension variable is not bound but reassigned across the
> comprehension. This problem remains in python3 and causes weird behavior
> when lambdas are put in a comprehension

I don't know why you say the behaviour in Python is a problem. It's
somewhat unexpected if you don't think about what's going on, but it's
actually quite logical. On the contrary, I find the Haskell version weird.


>>>> fl = [lambda y : x+y for x in [1,2,3]]
>>>> [fl[i](2) for i in [0,1,2]]
> [5, 5, 5]

This makes perfect sense: by the time you call the functions, the name x
has been rebound to the value 3. If x were a global, or if comprehensions
leaked their variable, that would be obvious: you could print x and see
exactly what value it has. But visible or not, that's exactly what
happens. Since x is 3, you're adding 3+2 and should get 5 no matter which
function you call.

Unroll the first comprehension, and it's obvious what is going on:

def unrolled():
fl = []
x = 0
def lambda_(y):
return x + y
fl.append(lambda_)
x = 1
def lambda_(y):
return x + y
fl.append(lambda_)
x = 2
def lambda_(y):
return x + y
fl.append(lambda_)
return [f(2) for f in fl]

Don't be fooled by the coincidence that the three "lambda_" functions
have the same name and body. They could have different names and
different bodies, Either way, it is completely natural that they all
closures over the same x -- that's how you wrote them.


But the Haskell version, on the other hand, is just weird:

> The same in haskell:
>
> Prelude> let fl = [\ y -> x + y | x <- [1,2,3]]
> Prelude> [(fl!!i) 0 | i<- [0,1,2]]
> [1,2,3]

For this to be the case, the functions in fl have to somehow "look back
in time" to see the value of x, not as it is *now* when the function is
called, but how it *was* when it was created. That's very weird indeed.
If x were a global, it would be astonishing. The fact that x comes from a
closure instead makes it no less surprising. Unroll the loop, and the
magic is obvious.

Now I'm not sure precisely how Haskell implements this trick, but it
suggests to me that it creates a different closure each time around the
loop of the comprehension. That could end up being very expensive.



--
Steven D'Aprano
http://import-that.dreamwidth.org/

Chris Angelico

unread,
Mar 22, 2014, 12:51:13 AM3/22/14
to pytho...@python.org
On Sat, Mar 22, 2014 at 3:39 PM, Rustom Mody <rusto...@gmail.com> wrote:
>> So if that's not going to be broken, how is this fundamentally different?
>
>> def func_loop():
>> for x in 1,2,3:
>> yield (lambda: x)
>
> Thats using a for-loop
> A 'for' in a comprehension carries a different intention, the matching names
> being merely coincidental.

So what you're saying is that these two are fundamentally different:

def unrolled():
x = 1
yield (lambda: x)
x = 2
yield (lambda: x)
x = 3
yield (lambda: x)

def loop():
for x in 1,2,3:
yield (lambda: x)

In other words, a loop should be implemented as a separate binding
each time, not a rebinding. That's an interesting point, and it does
make some sense; effectively, what you want is for the body of a for
loop to be a new scope, a unique scope every iteration of the loop,
and one that automatically closes over all variables in the parent
scope (including for assignments) except for the loop
iteration/counter variable.

That does make some sense, but it doesn't really fit Python's concept.
It would, however, fit a more C-like language, where locals are
declared (in Python, you'd have to put a whole lot of implicit
'nonlocal' statements at the top of the loop).

ChrisAg

Mark H Harris

unread,
Mar 22, 2014, 12:51:38 AM3/22/14
to
On 3/21/14 11:39 PM, Rustom Mody wrote:
> Given

> fl = [lambda y : x+y for x in [1,2,3]]

> It means:

> def rec(l):
> if not l: return []
> else:
> x,ll = l[0],l[1:]
> return [lambda y: x + y] + rec(ll)
>
> followed by
> fl = rec([1,2,3])

> Naturally a reasonable *implementation* would carry this *intention*
{snip}
> [But then I find Lisp and much of basic haskell natural and most of C++ not,
> so my views are likely prejudiced :-)


This discussion (the entire thread) comes up again and again over the
years because python tries to be all things to all people, without much
reason behind the motivation. I'm speaking of Lambda (filter, map, reduce).

Python is not Lisp. (I love Lisp too). Python is not Haskell (I love
Haskell too).

Lambda is a problem, if only because it causes confusion. What's the
problem? Glad you asked. The constructs DO NOT work the way most people
would expect them to, having limited knowledge of python! I ran into
this thing about seven years ago (when I was studying Haskell, and
Scheme) and I wanted to see how "pure functional" python was (well, not
at all really).

I can see uses for python's lambda. But, honestly, I think python could
deprecate its use and in five years just remove it from the language;
along with filter, map, and reduce !

I'm just saying;

marcus

Chris Angelico

unread,
Mar 22, 2014, 1:05:29 AM3/22/14
to pytho...@python.org
On Sat, Mar 22, 2014 at 3:47 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
> Now I'm not sure precisely how Haskell implements this trick, but it
> suggests to me that it creates a different closure each time around the
> loop of the comprehension. That could end up being very expensive.

It needn't be terribly expensive, if you make a "scope stack" or
"scope chain". Imagine, if you will, a system of scopes like this:

current_scope = None
class Scope:
def __init__(self, *a, **kw):
self.names = dict(*a, **kw)
global current_scope
self.parent = current_scope
current_scope = self
def __getitem__(self, name):
try:
return self.names[name]
except KeyError:
if self.parent is None: raise
return self.parent[name]
def __setitem__(self, name, value):
self.names[name] = value
def exit_scope():
global current_scope
current_scope = current_scope.parent

Creating a new scope would be fairly cheap (and a lot more so if this
were implemented in C without the object overhead, of course). Lookups
would scan up through a series of namespaces, same as they currently
do (local, module, builtins), there'd just be more of them. And the
compiler could be smart enough to skip creating a scope if there are
no assignments. There'd need to be some handling in there for the
'nonlocal' keyword, but my expectation is that 'global' is handled by
retaining a reference to the current_scope at module level, and
starting the search there for a LOAD_GLOBAL.

Each iteration of the loop could begin with Scope() and end with
exit_scope(), and there you are, each iteration in its own closable
scope.

I'm not saying it would be a good thing to do - and it'd be a bad fit
for Python, I think, as I said in my other post - but it could be
done.

ChrisA

Rustom Mody

unread,
Mar 22, 2014, 1:26:26 AM3/22/14
to
On Saturday, March 22, 2014 10:21:13 AM UTC+5:30, Chris Angelico wrote:
> On Sat, Mar 22, 2014 at 3:39 PM, Rustom Mody wrote:
> >> So if that's not going to be broken, how is this fundamentally different?
> >> def func_loop():
> >> for x in 1,2,3:
> >> yield (lambda: x)
> > Thats using a for-loop
> > A 'for' in a comprehension carries a different intention, the matching names
> > being merely coincidental.

> So what you're saying is that these two are fundamentally different:

> def unrolled():
> x = 1
> yield (lambda: x)
> x = 2
> yield (lambda: x)
> x = 3
> yield (lambda: x)

> def loop():
> for x in 1,2,3:
> yield (lambda: x)

Well almost...

Except that the 'loop' I am talking of is one of

def loop():
return [yield (lambda: x) for x in [1,2,3]]
or
return (yield (lambda: x) for x in [1,2,3])
or just plain ol
(lambda x: for x in [1,2,3])

IOW loop is an imperative construct, comprehensions are declarative

Progressing through a loop in a sequential order is natural and to
be expected

Comprehensions being declarative, progressing through them
in some order is incidental.


> In other words, a loop should be implemented as a separate binding
> each time, not a rebinding. That's an interesting point, and it does
> make some sense; effectively, what you want is for the body of a for
> loop to be a new scope, a unique scope every iteration of the loop,
> and one that automatically closes over all variables in the parent
> scope (including for assignments) except for the loop
> iteration/counter variable.

> That does make some sense, but it doesn't really fit Python's concept.

Yes it does not fit in with imperative programming.

Its good to remember the history.
Haskell did not invent comprehensions

In programming, they started with Miranda where they were called
'ZF-expressions' after the Zermelo/Fraenkel of axiomatic set theory.

Because Russell's paradox had shaken the (intended) foundations of mathematics,
the way out (actually the one chosen by Zermelo and Fraenkel) was to
add a *comprehension axiom*

http://en.wikipedia.org/wiki/Zermelo%E2%80%93Fraenkel_set_theory#3._Axiom_schema_of_specification_.28also_called_the_axiom_schema_of_separation_or_of_restricted_comprehension.29

What this basically mandates is that set constructions like
{x | Pred(x) } are disqualified (called unrestricted comprehension)

Only
{x ∈ S | Pred(x) } is a valid.

IOW sets cannot be concocted out of the blue but only out of other sets.

Fast forward 50 years and what David Turner – inventor of Miranda –- realized
is that if the programming language is sufficiently mathematical – ie no
imperative constructs plus lazy evaluation, then comprehensions are
actually constructive enough to make make into programming constructs.

However as Mark Harris points out imperative and functional thinking styles
remain somewhat inconsistent with each other.

Rustom Mody

unread,
Mar 22, 2014, 1:48:06 AM3/22/14
to
On Saturday, March 22, 2014 10:21:13 AM UTC+5:30, Chris Angelico wrote:
> On Sat, Mar 22, 2014 at 3:39 PM, Rustom Mody wrote:
> >> So if that's not going to be broken, how is this fundamentally different?
> >> def func_loop():
> >> for x in 1,2,3:
> >> yield (lambda: x)
> > Thats using a for-loop
> > A 'for' in a comprehension carries a different intention, the matching names
> > being merely coincidental.

> So what you're saying is that these two are fundamentally different:



> def unrolled():
> x = 1
> yield (lambda: x)
> x = 2
> yield (lambda: x)
> x = 3
> yield (lambda: x)

> def loop():
> for x in 1,2,3:
> yield (lambda: x)

Well almost
Except that the 'loop' I am talking of is one of

def loop():
return [yield (lambda: x) for x in [1,2,3]]
or
return (yield (lambda: x) for x in [1,2,3])
or just plain ol
(lambda x: for x in [1,2,3])

IOW loops is an imperative construct, comprehensions are declarative

Progressing through a loop in a sequential order is natural and to
be expected

Comprehensions being declarative, progressing through them
in some order is incidental.



> In other words, a loop should be implemented as a separate binding
> each time, not a rebinding. That's an interesting point, and it does
> make some sense; effectively, what you want is for the body of a for
> loop to be a new scope, a unique scope every iteration of the loop,
> and one that automatically closes over all variables in the parent
> scope (including for assignments) except for the loop
> iteration/counter variable.

> That does make some sense, but it doesn't really fit Python's concept.

Yes it does not fit in with imperative programming.

Its good to remember the history.
Haskell did not invent comprehensions

In programming, they started with Miranda where they were called
'ZF-expressions' after the Zermelo/Fraenkel of axiomatic set theory.

Because Russell's paradox had shaken the (intended) foundations of mathematics,
the way out (actually the one chosen by Zermelo and Fraenkel) was to
add a *comprehension axiom*

http://en.wikipedia.org/wiki/Zermelo%E2%80%93Fraenkel_set_theory#3._Axiom_schema_of_specification_.28also_called_the_axiom_schema_of_separation_or_of_restricted_comprehension.29

What this basically mandates is that set constructions like
{x | Pred(x) } are disqualified (called unrestricted comprehension)

Only
{x ∈ S | Pred(x) } is a valid.

IOW sets cannot be concoted out of the blue but only out of other sets.

FF 50 years and what David Turner – inventor of Miranda –- realized is that
if the programming language is sufficiently mathematical – ie no
imperative constructs plus lazy evaluation, then comprehensions are
actually constructive enough to make make into programming constructs.

However as Mark Harris points out imperative and functional thinking styles
remain confusingly inconsistent.

Ian Kelly

unread,
Mar 22, 2014, 5:09:56 AM3/22/14
to Python
On Fri, Mar 21, 2014 at 8:06 PM, Rustom Mody <rusto...@gmail.com> wrote:
> Two: A comprehension variable is not bound but reassigned across the
> comprehension. This problem remains in python3 and causes weird behavior when
> lambdas are put in a comprehension

Because Python as a language only has the concept of assignment, not
binding. I think it would be weird and confusing if variables worked
this way in comprehensions and nowhere else.

> >>> fl = [lambda y : x+y for x in [1,2,3]]
> >>> [fl[i](2) for i in [0,1,2]]
> [5, 5, 5]

You can get the desired effect by adding a layer of indirection:

>>> fl = [(lambda x: lambda y: x+y)(x) for x in [1,2,3]]
>>> [f(2) for f in fl]
[3, 4, 5]

If that's too ugly then give the wrapper a proper name:

>>> def make_function(x):
... return lambda y: x+y
...
>>> fl = [make_function(x) for x in [1,2,3]]
>>> [f(2) for f in fl]
[3, 4, 5]

There is also the default argument trick:

>>> fl = [lambda y, *, x=x: x+y for x in [1,2,3]]
>>> [f(2) for f in fl]
[3, 4, 5]

Steven D'Aprano

unread,
Mar 22, 2014, 5:46:01 AM3/22/14
to
On Fri, 21 Mar 2014 23:51:38 -0500, Mark H Harris wrote:

> Lambda is a problem, if only because it causes confusion. What's the
> problem? Glad you asked. The constructs DO NOT work the way most people
> would expect them to, having limited knowledge of python!

Why is that a problem? Would you consider it a problem that people who
don't understand BASIC can't understand BASIC? ("I don't get the
difference between GOTO and GOSUB. Obviously GOSUB is a problem and
should be throw out.") Do you think that the fact that people who have
never used a TI-89 calculator before may have trouble used a TI-89
calculator means that TI-89 calculators are "a problem"? (I've taught
school children who didn't know how to use their calculator.)

The problem is ignorance, not the calculator, and not lambda.

You've told us that "most people" (which people? dentists? lawyers?
illiterate tribesmen from the Amazon? professional programmers?) are
surprised by lamba's behaviour, but you haven't actually told us what
behaviour is surprising, or what "most people" expect lambda to do.
Perhaps you ought to?


> I ran into
> this thing about seven years ago (when I was studying Haskell, and
> Scheme) and I wanted to see how "pure functional" python was (well, not
> at all really).

Python is not a pure functional language, but you can write functional
code in it. If you want a pure functional language, you know where to
find one.

(I wonder whether, say, Haskell has the people coming along and demanding
that it should become more like Python?)


> I can see uses for python's lambda. But, honestly, I think python could
> deprecate its use and in five years just remove it from the language;
> along with filter, map, and reduce !

Python could deprecate many things. It would make Python a worse language.

By the way, are you aware that lambda is *identical* to def except for
three things?

- lambda is an expression, not a statement like def;

- the body of a function created with lambda is limited to a
single expression, not a block;

- the function object created with lambda has a generic name.


So unless the surprising behaviour about lambda that you're about to tell
us about relates to one of those three things, I *guarantee* that
functions created with def have the same surprising behaviour.

Marko Rauhamaa

unread,
Mar 22, 2014, 6:24:47 AM3/22/14
to
Steven D'Aprano <steve+comp....@pearwood.info>:

> This makes perfect sense: by the time you call the functions, the name x
> has been rebound to the value 3.

> [...]

> Now I'm not sure precisely how Haskell implements this trick, but it
> suggests to me that it creates a different closure each time around
> the loop of the comprehension. That could end up being very expensive.

Haskell does not rebind variables. I guess Haskell simply physically
substitutes object references for each syntactic occurrence of a
variable.

If Python worked analogously, the following loop:

for i, x in enumerate([1, 2, 3]):
f[i] = lambda y: x + y

would unroll as follows:

f[0] = lambda y: 1 + y
f[1] = lambda y: 2 + y
f[2] = lambda y: 3 + y

That is actually how classic lambda calculus does it. Variables are
substituted, not "bound" or "assigned". They are syntactic slots. There
is no heap, there is no stack, there is no memory apart from the
expression itself.


Marko

Marko Rauhamaa

unread,
Mar 22, 2014, 6:30:28 AM3/22/14
to
Ian Kelly <ian.g...@gmail.com>:

> You can get the desired effect by adding a layer of indirection:
>
>>>> fl = [(lambda x: lambda y: x+y)(x) for x in [1,2,3]]

A trick to remember! Variable lifetime reduction by function invocation.


Marko

Mark Lawrence

unread,
Mar 22, 2014, 6:40:49 AM3/22/14
to pytho...@python.org
On 22/03/2014 02:06, Rustom Mody wrote:
>
> The same in haskell:
>
> Prelude> let fl = [\ y -> x + y | x <- [1,2,3]]
> Prelude> [(fl!!i) 0 | i<- [0,1,2]]
> [1,2,3]
>

My really big complaint about Python is that it's nothing like CORAL 66.
I think I'll raise this on python ideas in an attempt to get this
glaring deficiency corrected.

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com


Rustom Mody

unread,
Mar 22, 2014, 1:16:43 PM3/22/14
to
On Saturday, March 22, 2014 2:39:56 PM UTC+5:30, Ian wrote:
> On Fri, Mar 21, 2014 at 8:06 PM, Rustom Mody wrote:
> > Two: A comprehension variable is not bound but reassigned across the
> > comprehension. This problem remains in python3 and causes weird behavior when
> > lambdas are put in a comprehension

> Because Python as a language only has the concept of assignment, not
> binding. I think it would be weird and confusing if variables worked
> this way in comprehensions and nowhere else.

Bizarre viewpoint!

When you do this:

> There is also the default argument trick:

> >>> fl = [lambda y, *, x=x: x+y for x in [1,2,3]]
> >>> [f(2) for f in fl]
> [3, 4, 5]

how is that not-a-binding solution?

More generally, insofar as variable-scopes can be made and exited, there
is binding. Its just that imperative languages have
- assignment wherein the shape of the environment is preserved but
its content is changed
- there are binding-constructs -- functions, methods, classes etc etc
-- which leave extant bindings intact but create/remove new ones.

Ok, functional languages have only the latter.
But only the former?? Beyond assembly language I dont know what
that would/could be...

Mark Lawrence

unread,
Mar 22, 2014, 1:57:21 PM3/22/14
to pytho...@python.org
On 22/03/2014 09:09, Ian Kelly wrote:
> On Fri, Mar 21, 2014 at 8:06 PM, Rustom Mody <rusto...@gmail.com> wrote:
>> Two: A comprehension variable is not bound but reassigned across the
>> comprehension. This problem remains in python3 and causes weird behavior when
>> lambdas are put in a comprehension
>
> Because Python as a language only has the concept of assignment, not
> binding. I think it would be weird and confusing if variables worked
> this way in comprehensions and nowhere else.
>

My understanding has always been that an expression of the rhs is bound
to a name of the lhs. So is my understanding wrong, or is the above
wrong, or are we talking at cross purposes, or what?

Marko Rauhamaa

unread,
Mar 22, 2014, 2:40:44 PM3/22/14
to
Mark Lawrence <bream...@yahoo.co.uk>:

> On 22/03/2014 09:09, Ian Kelly wrote:
>> Because Python as a language only has the concept of assignment, not
>> binding. I think it would be weird and confusing if variables worked
>> this way in comprehensions and nowhere else.
>
> My understanding has always been that an expression of the rhs is
> bound to a name of the lhs. So is my understanding wrong, or is the
> above wrong, or are we talking at cross purposes, or what?

Hard to say without knowing more of your understanding.

Even Scheme doesn't have purely classical variable binding because
variables can be assigned to. You will notice that right away when you
try to implement a lisp dialect; a purely binding (substituting)
implementation fails with set!/setq because there is no variable to
assign a value to.

Python variables are named memory slots you can peek and poke into. Not
so in a purely functional language where variables are physically
removed from the code before the code is executed.

Example:

def redouble(x):
return x + x

redouble(17 + 4)

If Python were simply binding variables in the purely functional sense,
the interpreter would first evaluate 17 + 4 and then make a copy of the
"redouble" function substituting the result (21) for each syntactic
occurrence of x:

def __redouble__234001942():
return 21 + 21

The interpreter would then proceed to evaluate the variable-less
function instance.

If you leave out assignment, there is really no difference between the
two models. Only if you don't have assignment, you don't need to
complicate your computational model with memory. Instead, you deal with
things like continuations.

In fact, even "def" violates the classic functional paradigm. For
example, to calculate 10's factorial without assignments, you can do
this (even in Python):

(lambda n: (lambda fac: fac(fac, n)) \
(lambda fac, n: 1 if n < 2 else n * fac(fac, n - 1)))(10)


Marko

Rustom Mody

unread,
Mar 22, 2014, 2:42:42 PM3/22/14
to
The foll is fairly standard fare in denotational semantics -- please excuse
the length!

In order to understand (formally) the concept of 'variable'
we need to have at the least a concept of name(or identifier) -> value mapping.
This mapping is called an 'environment'

If we stop at that we get the 'simplest' (at least from a mathematical
pov!!) language -- λ calculus.

However programming languages also need to be implemented -- typically
on von Neumann machines. So we make 'Variable' a composition of two functions:
Env : Identifier -> Location
Store : Location -> Value

This seems fine and dandy until we realize that if the compositon is of
non one-one onto functions all kinds of troubles result; eg
-- Two locations going to one value -- aliasing
-- Store partial at a location -- uninitialized variable
etc etc the most innocuous looking…
-- assignment becomes a higher order function
because it converts a starting id -> -> value mapping to a new mapping


Seeing all these difficulties, some religious zealots (aka functional
programmers) decide that this composite mapping is the root of the problem
-- throw it out -- no aliasing, no assignment, no time. [Minor problem --
How to implement -- remains!]

The non-religious bigots (also called C programmers) see that managing
the Env at one time and the Store at a later time (the two times
are usually called compile-time and run-time but like bell-bottoms these
words are currently unfashionable!) can produce very effective engineering
expedience.

So strictly speaking whenever we have variables we have binding
[Probably mathematica is an exception... dunno for sure]

More loosely and conventionally if the mapping is a single direct one:
Env: Variable -> Value
as in λ calculus, functional languages etc, they are called binding-languages

To distinguish, the 'other' languages which need a
compose of Environment and Store are called variously:

- imperative language
- reference semantics
- conventional (imperative) variable (vs mathematical variable)
etc

IOW in most (normal) languages we have constructs to change the shape
of the environment and we have constructs to change the content of the
environment. The single pre-eminent way for the latter is assignment,
function is the typical way for the former.

[Unfortunately this is not universally true:
In C we have initialized variables that look like assignment but is not.
In python the exception is what Ian calls the default variable trick:
x=x would be a rather asinine assignment. But its not an assignment at all,
it just looks like one]

So no...

> My understanding has always been that an expression of the rhs is bound
> to a name of the lhs. So is my understanding wrong, or is the above
> wrong, or are we talking at cross purposes, or what?

assignment changes the content of the env, functions change the shape
-- which latter you may call binding.

vasudevram

unread,
Mar 22, 2014, 4:59:46 PM3/22/14
to

Thanks to all those who answered.

- Vasudev

Rhodri James

unread,
Mar 22, 2014, 8:32:22 PM3/22/14
to
On Sat, 22 Mar 2014 05:26:26 -0000, Rustom Mody <rusto...@gmail.com>
wrote:

> Well almost...
> Except that the 'loop' I am talking of is one of
> def loop():
> return [yield (lambda: x) for x in [1,2,3]]
> or
> return (yield (lambda: x) for x in [1,2,3])
> or just plain ol
> (lambda x: for x in [1,2,3])
> IOW loop is an imperative construct, comprehensions are declarative

I'm sorry, you've made a logical leap too far here. I understand loops
being imperative, but how are comprehensions declarative? What do they
declare that the loop equivalent doesn't.

You've made a great deal of the "for" in a comprehension not having the
same meaning as the "for" in a loop. That may well be true in the
equivalent Haskell constructs (I don't speak or write Haskell), but I
think you are wrong in Python. If so, please stop trying to write Haskell
in Python; you'll be as happy as the friend of mine I've finally persuaded
to stop writing Fortran in Python, I promise!

--
Rhodri James *-* Wildebeest Herder to the Masses

Ian Kelly

unread,
Mar 22, 2014, 10:46:28 PM3/22/14
to Python
On Sat, Mar 22, 2014 at 6:32 PM, Rhodri James <rho...@wildebst.org.uk> wrote:
> On Sat, 22 Mar 2014 05:26:26 -0000, Rustom Mody <rusto...@gmail.com>
> wrote:
>
>> Well almost...
>> Except that the 'loop' I am talking of is one of
>> def loop():
>> return [yield (lambda: x) for x in [1,2,3]]
>> or
>> return (yield (lambda: x) for x in [1,2,3])
>> or just plain ol
>> (lambda x: for x in [1,2,3])
>> IOW loop is an imperative construct, comprehensions are declarative
>
>
> I'm sorry, you've made a logical leap too far here. I understand loops
> being imperative, but how are comprehensions declarative? What do they
> declare that the loop equivalent doesn't.

I'm with Rustom on this point. A list comprehension is a syntax for
building a list by declaring a transformation from some other iterable
object. Forget comprehensions for a moment and think of literals.
Would you not consider this to be declarative?

x = [1, 2, 3]

A comprehension is syntactically similar to a literal, with just a
different type of construction in mind.

Where I disagree is on the question of whether Python should therefore
break its established closure rules for lambdas that are nested inside
comprehensions versus functions that are not. It breaks the
equivalence between comprehensions and loops, and to my mind it
introduces significant complexity for relatively little gain.

Rustom Mody

unread,
Mar 22, 2014, 11:16:47 PM3/22/14
to
On Sunday, March 23, 2014 8:16:28 AM UTC+5:30, Ian wrote:
> On Sat, Mar 22, 2014 at 6:32 PM, Rhodri James wrote:
> > wrote:
> >> Well almost...
> >> Except that the 'loop' I am talking of is one of
> >> def loop():
> >> return [yield (lambda: x) for x in [1,2,3]]
> >> or
> >> return (yield (lambda: x) for x in [1,2,3])
> >> or just plain ol
> >> (lambda x: for x in [1,2,3])
> >> IOW loop is an imperative construct, comprehensions are declarative
> > I'm sorry, you've made a logical leap too far here. I understand loops
> > being imperative, but how are comprehensions declarative? What do they
> > declare that the loop equivalent doesn't.

> I'm with Rustom on this point. A list comprehension is a syntax for
> building a list by declaring a transformation from some other iterable
> object. Forget comprehensions for a moment and think of literals.
> Would you not consider this to be declarative?

> x = [1, 2, 3]

> A comprehension is syntactically similar to a literal, with just a
> different type of construction in mind.

Aha! Very elegantly put!

> Where I disagree is on the question of whether Python should therefore
> break its established closure rules for lambdas that are nested inside
> comprehensions versus functions that are not.

No... see below

> It breaks the
> equivalence between comprehensions and loops, and to my mind it
> introduces significant complexity for relatively little gain.

[I am not completely sure whether the following can be proved/is true]

1. One can change lambda's closure rules which would amount to
"significant complexity for relatively little gain"

2. One can change comprehension rules to not reassign to the
running comprehension running varible but to rebind, using a recursive
function as the simulation of the comprehension rather than a for loop

3. 2 is semantically equivalent to 1
- trivially for normal (ie non-lambda containing) expressions
- and also for lambda containing expressions if your
default-argument trick is implemented by the python compiler
[This is the claim I am not completely sure of and would love to hear
of/if counter examples]

Assuming its true:

One *semantically specifies* a comprehension with a recursive function,
not a for loop

One *implements* a comprehension with the standard use of append
method inside a for as the expansion of a comprehension with the extra
caveat that interior lambdas are automatically wrapped inside a
default-argument binding for all outer comprehension variables.

A vanilla python programmer need not know anything about this any
more than a vanilla C programmer knows about
- strength reduction
- code hoisting
- loop unrolling
etc

that goes on inside an optimizing C compiler

Ian Kelly

unread,
Mar 22, 2014, 11:47:42 PM3/22/14
to Python
On Sat, Mar 22, 2014 at 9:16 PM, Rustom Mody <rusto...@gmail.com> wrote:
> [I am not completely sure whether the following can be proved/is true]
>
> 1. One can change lambda's closure rules which would amount to
> "significant complexity for relatively little gain"
>
> 2. One can change comprehension rules to not reassign to the
> running comprehension running varible but to rebind, using a recursive
> function as the simulation of the comprehension rather than a for loop
>
> 3. 2 is semantically equivalent to 1

Well, if you accept that 1) amounts to significant complexity for
relatively little gain, and if 2) is semantically equivalent to 1),
then it follows that 2) also amounts to significant complexity for
relatively little gain. My statement was in regard to the complexity
of the language, not the implementation of the language.

> - trivially for normal (ie non-lambda containing) expressions
> - and also for lambda containing expressions if your
> default-argument trick is implemented by the python compiler
> [This is the claim I am not completely sure of and would love to hear
> of/if counter examples]

The disadvantage of the default argument trick is that it does modify
the function's signature, by adding an extra argument with a default,
so they're not entirely equivalent.

> Assuming its true:
>
> One *semantically specifies* a comprehension with a recursive function,
> not a for loop

The problem with this is a cognitive one. The comprehension *looks
like* a for loop, not a recursive function. It is natural to reason
about it as if it were a for loop, not a recursive function. This is
that added complexity I was talking about.

> A vanilla python programmer need not know anything about this any
> more than a vanilla C programmer knows about
> - strength reduction
> - code hoisting
> - loop unrolling
> etc
>
> that goes on inside an optimizing C compiler

Until they go to unroll their comprehension into a for loop because
they need to add some imperative construct to it, and their code
breaks as a result. From there you'll get the programmers who will
add functions with side effects to their comprehensions because they
want the comprehension binding behavior while still performing
imperative tasks within the loop.

Rhodri James

unread,
Mar 23, 2014, 10:35:45 PM3/23/14
to
On Sun, 23 Mar 2014 02:46:28 -0000, Ian Kelly <ian.g...@gmail.com>
wrote:

> On Sat, Mar 22, 2014 at 6:32 PM, Rhodri James <rho...@wildebst.org.uk>
> wrote:
>> On Sat, 22 Mar 2014 05:26:26 -0000, Rustom Mody <rusto...@gmail.com>
>> wrote:
>>
>>> Well almost...
>>> Except that the 'loop' I am talking of is one of
>>> def loop():
>>> return [yield (lambda: x) for x in [1,2,3]]
>>> or
>>> return (yield (lambda: x) for x in [1,2,3])
>>> or just plain ol
>>> (lambda x: for x in [1,2,3])
>>> IOW loop is an imperative construct, comprehensions are declarative
>>
>>
>> I'm sorry, you've made a logical leap too far here. I understand loops
>> being imperative, but how are comprehensions declarative? What do they
>> declare that the loop equivalent doesn't.
> I'm with Rustom on this point. A list comprehension is a syntax for
> building a list by declaring a transformation from some other iterable
> object. Forget comprehensions for a moment and think of literals.
> Would you not consider this to be declarative?
>
> x = [1, 2, 3]

I'm not sure I would. I look at that line of code and think of it as
"Create a list...", very much in an imperative manner. Then again,
compared with C structs and typedefs and actual honest-to-God type
declarations, there's precious little in Python I would consider truly
declarative.

Chris Angelico

unread,
Mar 23, 2014, 11:27:32 PM3/23/14
to pytho...@python.org
On Mon, Mar 24, 2014 at 1:35 PM, Rhodri James <rho...@wildebst.org.uk> wrote:
>> Would you not consider this to be declarative?
>>
>> x = [1, 2, 3]
>
>
> I'm not sure I would. I look at that line of code and think of it as
> "Create a list...", very much in an imperative manner. Then again, compared
> with C structs and typedefs and actual honest-to-God type declarations,
> there's precious little in Python I would consider truly declarative.

I'm in the declarative group here. Yes, it has to be implemented as
creating a list and adding three elements to it, but conceptually, it
means "Bind x to a new list with these elements". And as long as
that's the end result, I don't care how it's done; the interpreter's
most welcome to have a "template list" that it copies, or maybe a
literal tuple that gets passed to the list() constructor, or
whatever's most efficient.

It gets a bit messier when there's stuff with side effects, though.
This has to be a bit more imperative:

x = [foo(y), bar(y), quux(y)]

That means "Call foo, bar, and quux, in that order, each with y as an
argument, and bind x to a new list with their return values". And if
Python had a simple notation for calling a series of functions, that
could be written something like this:

funcs = [foo, bar, quux]
x = funcs(y)

which is looking more declarative again. It's now "Take this list of
functions and call them all, and bind x to a list of their return
values". (This could be done, with a callable subclass of list. Call
it a sort of "implicit map" if you like.) Python doesn't have that
syntax, but it does have this:

x = [f(y) for f in funcs]

and I'd say that's reasonably declarative; it should be read like the
previous one: "Take this list of funcs, call each one, and bind x to a
list of their return values". And I could imagine a
parallel-processing version of a list comp that functions like
multiprocessing.Pool.map() and doesn't promise order, which would
*definitely* be declarative.

ChrisA

Chris Angelico

unread,
Mar 23, 2014, 11:32:13 PM3/23/14
to pytho...@python.org
On Mon, Mar 24, 2014 at 1:35 PM, Rhodri James <rho...@wildebst.org.uk> wrote:
> I'm not sure I would. I look at that line of code and think of it as
> "Create a list...", very much in an imperative manner. Then again, compared
> with C structs and typedefs and actual honest-to-God type declarations,
> there's precious little in Python I would consider truly declarative.

By the way: Python does have a difference between "declarative" and
"imperative".

def f(): # Imperative
global x # Declarative
x += 1 # Imperative

Declaratives control things, imperatives become byte code. Everything
in the byte code is imperative. "LOAD_GLOBAL" means "fetch this global
and put it on the stack". "INPLACE_ADD" means "iadd the top two stack
elements and push the result onto the stack". "STORE_GLOBAL" means
"pop the top stack element and store it in this global". Very very
imperative, and there's none of that created by the "global"
statement. So in that sense, yes, "x = [1, 2, 3]" is imperative; it
loads three constants, builds a list, and stores it. But digging into
the byte code isn't really helpful; it's much more useful to look at
the source code and how the programmer thinks about it.

ChrisA

Rustom Mody

unread,
Mar 24, 2014, 12:14:20 AM3/24/14
to
On Monday, March 24, 2014 8:57:32 AM UTC+5:30, Chris Angelico wrote:
You have described nicely a slippery slope!

The list
[(1, 1), (1, 2), (1, 3), (1, 4), (2, 1), (2, 2), (2, 3), (2, 4), (3, 1), (3, 2), (3, 3), (3, 4)]

looks neater written (and *thought of* ) as
[(x,y) for x in range(1,4) for y in range(1,5)]

Neat! So I play around... Change it to
[(x,y) for x in range(1,10000) for y in range(1,10000)]
and I dont have an answer but a thrashing machine!! (*)

IOW everything we write/read as programmers has a declarative and an
imperative side.
Which one one wants to focus on requires good sense and taste.

(*) Example also shows inter alia how some things -- range -- have gone from
imperative to declarative from python 2 to 3.
Some people have made languages whose main focus is generalizing this
idea to more dimensions and restrictions:
http://www.irisa.fr/cosi/Rajopadhye/dag-talk.ps

Chris Angelico

unread,
Mar 24, 2014, 1:04:51 AM3/24/14
to pytho...@python.org
On Mon, Mar 24, 2014 at 3:14 PM, Rustom Mody <rusto...@gmail.com> wrote:
> Neat! So I play around... Change it to
> [(x,y) for x in range(1,10000) for y in range(1,10000)]
> and I dont have an answer but a thrashing machine!! (*)

Yes, because you used square brackets, which means that the list has
to be fully realized. As you comment, range changed from returning a
list to returning an iterable, and this action is similarly cheap:

>>> ((x,y) for x in range(1,10000) for y in range(1,10000))
<generator object <genexpr> at 0x7f53ed61b360>

You can take a few elements from that cheaply:

>>> [next(_),next(_),next(_)]
[(1, 1), (1, 2), (1, 3)]

If you like thinking in "lazy lists", you can probably think just as
easily with generators; you can't pull up arbitrary elements from it,
or query its length, but for many purposes a generator will do.

ChrisA

Mark H Harris

unread,
Mar 24, 2014, 1:52:52 AM3/24/14
to
On 3/22/14 4:46 AM, Steven D'Aprano wrote:
> On Fri, 21 Mar 2014 23:51:38 -0500, Mark H Harris wrote:
>
>> Lambda is a problem, if only because it causes confusion. What's the
>> problem? Glad you asked. The constructs DO NOT work the way most people
>> would expect them to, having limited knowledge of python!

One of the best links for understanding what is wrong with lambda is
here, from Guido, (by the way I agree totally with his assessment, there
is no point really in writing it out again):

http://www.artima.com/weblogs/viewpost.jsp?thread=98196


> Why is that a problem? Would you consider it a problem that people who
> don't understand BASIC can't understand BASIC? ("I don't get the
> difference between GOTO and GOSUB. {snip}
>
> The problem is ignorance, not the calculator, and not lambda.

You're argument, based on analogy, is also a red herring, as well a
straw man. Lambda is a problem of confusion for scientists and other
mathematicians (amateur and otherwise) who may be confused because
python's lambda does not do precisely what they might expect from other
functional programming languages, nor does it match precisely with the
lambda calculus. Its also confusing to sophisticated users of all
stripes who may not be aware of "lambda" at all.

> You've told us that "most people" (which people? dentists? lawyers?
> illiterate tribesmen from the Amazon? professional programmers?) are
> surprised by lamba's behaviour, but you haven't actually told us what
> behaviour is surprising, or what "most people" expect lambda to do.
> Perhaps you ought to?

I love your rhetoric, actually, but red herrings all . . .

Here is a link that advocates lambda (map, filter, and reduce) as a
very powerful construct for doing various things: I agree:

http://www.secnetix.de/olli/Python/lambda_functions.hawk

Its a great link, but the important sentence for this discussion is
this unique quote, about "lambda,"

>"This is not exactly the same as lambda in functional
> programming languages, but it is a very powerful concept . . ."

People who understand lambda from functional languages must hassle
with the fact that lambda is different in python, and people who do not
understand functional programming languages (nor the lambda calculus)
are confused by the syntax, also just what it does.


> Python is not a pure functional language, but you can write functional
> code in it. If you want a pure functional language, you know where to
> find one.

Yes, that's obvious; but you're missing the point. Python is not a
functional language, and implying that it can be used as one is
misleading at best (maybe a lie at worst) just because it has a
construct for generating a dynamic anonymous function.

> (I wonder whether, say, Haskell has the people coming along and demanding
> that it should become more like Python?)

Another straw man.

Well, number one, I'm not demanding anything. Number two, everyone
who uses Haskell (for whatever reason) knows well from the start that
its a pure functional programming language. That is the advertisement,
and that is the expectation. No one expects anything different.

>> I can see uses for python's lambda. But, honestly, I think python could
>> deprecate its use and in five years just remove it from the language;
>> along with filter, map, and reduce !
>
> Python could deprecate many things. It would make Python a worse language.

How so? Read Guido's argument above. Another way to answer this
question is that I have been programming with Python for almost a decade
and I've not used lambda. In fact, I have gone out of my way to NOT use
lambda because I am fully aware that the BDFL hates it. Reduce is no
longer in the standard library (although you can import it) and there
are equally good ways to do what lambda was designed for without the
hassle nor confusion.

> By the way, are you aware that lambda is *identical* to def except for
> three things?
>
> - lambda is an expression, not a statement like def;
>
> - the body of a function created with lambda is limited to a
> single expression, not a block;
>
> - the function object created with lambda has a generic name.

Absolutely, thank you for pointing those out ! You forgot that
lambdas can be used in expressions while def functions may not... but
you really have proved my point. There is so little difference between
the two that (for the sake of clarity and reducing confusion) it might
be good to just remove the lambda thing altogether. Just say'in.

> So unless the surprising behaviour about lambda that you're about to tell
> us about relates to one of those three things, I *guarantee* that
> functions created with def have the same surprising behaviour.

No, the surprising behavior is that python lambda does not work
exactly the way that lambda works in functional programming languages
(see the two links provided) the confusion it creates for "normal" users
does not warrant its inclusion in the language.

Having said that, I must admit that I've "played" with lambda
extensively and find it interesting from a comp sci standpoint, but,
again, for normal users I don't think there is a really good value add
for having it in the language.

Ian Kelly

unread,
Mar 24, 2014, 5:03:12 AM3/24/14
to Python


On Mar 23, 2014 11:56 PM, "Mark H Harris" <harri...@gmail.com> wrote:
>
> On 3/22/14 4:46 AM, Steven D'Aprano wrote:
>>
>> On Fri, 21 Mar 2014 23:51:38 -0500, Mark H Harris wrote:
>>
>>> Lambda is a problem, if only because it causes confusion. What's the
>>> problem?  Glad you asked. The constructs DO NOT work the way most people
>>> would expect them to, having limited knowledge of python!
>
>
>    One of the best links for understanding what is wrong with lambda is here, from Guido, (by the way I agree totally with his assessment, there is no point really in writing it out again):
>
>   http://www.artima.com/weblogs/viewpost.jsp?thread=98196

That post doesn't point out anything "wrong" with lambda. The argument boils down to: 1) map and filter are not useful because we have comprehensions; 2) reduce is confusing; 3) if we remove those then lambda is not useful either.

> Lambda is a problem of confusion for scientists and other mathematicians (amateur and otherwise) who may be confused because python's lambda does not do precisely what they might expect from other functional programming languages, nor does it match precisely with the lambda calculus. Its also confusing to sophisticated users of all stripes who may not be aware of "lambda" at all.

The difference does not really lie in the lambda construct per se but in the binding style of closures. Functional languages tend to go one way here; imperative languages tend to go the other. Python being primarily an imperative language follows the imperative model. Anonymous functions in Python work the same way in this regard as anonymous functions in Lua or ECMAScript or Go -- those other languages merely avoid the cardinal sin of defining their anonymous functions using the keyword "lambda". The result may be more surprising to users accustomed to functional languages, but I claim that it is *less* surprising to users of other imperative languages.

>> Python is not a pure functional language, but you can write functional
>> code in it. If you want a pure functional language, you know where to
>> find one.
>
>
>    Yes, that's obvious; but you're missing the point. Python is not a functional language, and implying that it can be used as one is misleading at best (maybe a lie at worst) just because it has a construct for generating a dynamic anonymous function.

Oh, give me a break. If you find that you can't write functional code in Python just because closure bindings are slightly inconvenient, then you can't be trying very hard.

>    Well, number one, I'm not demanding anything. Number two, everyone who uses Haskell (for whatever reason) knows well from the start that its a pure functional programming language. That is the advertisement, and that is the expectation. No one expects anything different.

And I would hope that anybody who uses Python is likewise aware from the stay that it *isn't* a purely functional language.

>> Python could deprecate many things. It would make Python a worse language.
>
>
>    How so?  Read Guido's argument above. Another way to answer this question is that I have been programming with Python for almost a decade and I've not used lambda. In fact, I have gone out of my way to NOT use lambda because I am fully aware that the BDFL hates it.

If lambda were going to be deprecated and removed then it already would have happened in Python 3, because Guido tried to do precisely that. I'm not sure what the reasons were for keeping it in the end (according to PEP 3099 it was because nobody suggested a suitable replacement), but if he couldn't get rid of it then, he never will.

Steven D'Aprano

unread,
Mar 24, 2014, 5:49:35 AM3/24/14
to
On Mon, 24 Mar 2014 00:52:52 -0500, Mark H Harris wrote:

> On 3/22/14 4:46 AM, Steven D'Aprano wrote:
>> On Fri, 21 Mar 2014 23:51:38 -0500, Mark H Harris wrote:
>>
>>> Lambda is a problem, if only because it causes confusion. What's the
>>> problem? Glad you asked. The constructs DO NOT work the way most
>>> people would expect them to, having limited knowledge of python!
>
> One of the best links for understanding what is wrong with lambda is
> here, from Guido, (by the way I agree totally with his assessment, there
> is no point really in writing it out again):
>
> http://www.artima.com/weblogs/viewpost.jsp?thread=98196

It's only one paragraph. Here it is:

Why drop lambda? Most Python users are unfamiliar with Lisp or
Scheme, so the name is confusing; also, there is a widespread
misunderstanding that lambda can do things that a nested function
can't -- I still recall Laura Creighton's Aha!-erlebnis after I
showed her there was no difference! Even with a better name, I think
having the two choices side-by-side just requires programmers to
think about making a choice that's irrelevant for their program; not
having the choice streamlines the thought process. Also, once map(),
filter() and reduce() are gone, there aren't a whole lot of places
where you really need to write very short local functions; Tkinter
callbacks come to mind, but I find that more often than not the
callbacks should be methods of some state-carrying object anyway (the
exception being toy programs).


None of map(), filter() or reduce() are gone. reduce() has been pushed
out into a module, but map() and filter() have actually been enhanced.
Any suggestion that lambda is obsolete flounders on that fact.

There's no doubt that lambda is less-often useful than is the def
statement. But not only is it still useful, but there is a continual
stream of people asking for Python to make it *more useful* by allowing
the body of a lambda to be a full block, not just a single statement.
Guido has stated (although I cannot remember where, so cannot paste a
link) that he accepts that in principle, so long as somebody comes up
with acceptable syntax.

In Ruby, anonymous blocks are used *all the time*. Python is not Ruby,
and never will be, but a lightweight way to generate anonymous functions
in expressions is one difference between a modern, powerful language, and
an archaic or toy under-powered language. I'm too lazy to do this the
right way, but there are at least 99 mentions of "lambda" in the 3.3
standard library:

steve@runes:/usr/local/lib/python3.3$ grep lambda *.py | wc -l
99


It's used in calendar, cgitb, configparser, difflib, decimal, functools,
inspect, os, pydoc and others. Python is a programming language aimed at
programmers, not a toy. It should include tools that programmers use,
like threads, closures, first-class functions. And lambda.


>> Why is that a problem? Would you consider it a problem that people who
>> don't understand BASIC can't understand BASIC? ("I don't get the
>> difference between GOTO and GOSUB. {snip}
>>
>> The problem is ignorance, not the calculator, and not lambda.
>
> You're argument, based on analogy, is also a red herring, as well a
> straw man.

Please stop misusing "straw man". It has a specific meaning:

https://yourlogicalfallacyis.com/strawman


> Lambda is a problem of confusion for scientists and other
> mathematicians (amateur and otherwise) who may be confused because
> python's lambda does not do precisely what they might expect from other
> functional programming languages, nor does it match precisely with the
> lambda calculus.

Yes, you've already said that. And my response remains the same: users of
a language should be expected to learn the language, at least to the
point that they can "get by". One doesn't need ten thousand hours
experience and to be an expert. But nor should the confusion of somebody
who doesn't know the language count for much.

Python's ints don't work exactly the same as ints do in some other
languages, nor do Python's strings, nor is Python's object model
precisely the same as that of some other languages. I think that it would
be a ridiculous idea to discard int, str and the entire object system
just because some people are surprised that Python doesn't behave exactly
like some other language. I don't think lambda is any different.

The same argument applies to *any other language*. Ocaml is not precisely
the same as Haskell, Java not precisely the same as C++, Ruby is not
precisely the same as Javascript. Should Ruby discard their anonymous
blocks because they don't work "precisely" the same as Haskell? Or
perhaps Haskell should throw the towel in because it doesn't behave
"precisely" the same as Ruby?

Of course not -- new languages come into existence precisely so that they
will be different from what is already there. Who is to say that there
aren't people who like Python's lambda *because* it is different from the
anonymous functions in other languages? Or because it is less abstract
than the lambda calculus?

Other languages -- not even Lisp -- don't get to be the sole decider as
to what counts as an anonymous function. Python's lambda may be limited,
but it is a perfectly fine lambda for what it is intended to do.


> Its also confusing to sophisticated users of all
> stripes who may not be aware of "lambda" at all.

If they're not aware of lambda, how are they confused by it?


>> You've told us that "most people" (which people? dentists? lawyers?
>> illiterate tribesmen from the Amazon? professional programmers?) are
>> surprised by lamba's behaviour, but you haven't actually told us what
>> behaviour is surprising, or what "most people" expect lambda to do.
>> Perhaps you ought to?
>
> I love your rhetoric, actually, but red herrings all . . .

No, not really. I don't understand who your intended audience for Python
is. If I were to try to guess, I suspect that you want to dumb Python
down until it is like Dartmouth BASIC circa 1980 except without the line
numbers and with a slightly different syntax. Perhaps I'm wrong, I
certainly hope I'm wrong, but that's the impression I got from your
comments on the python-ideas list.

Python is a programming language. People who program in Python are
programmers. It doesn't matter whether they are school children, or
scientists, or farmhands, once they start writing programs, they are
programming, which makes them a programmer. They might not be paid for
it, but they are still programmers. Who should be using Python, if not
programmers?


> Here is a link that advocates lambda (map, filter, and reduce) as a
> very powerful construct for doing various things: I agree:
>
> http://www.secnetix.de/olli/Python/lambda_functions.hawk
>
> Its a great link, but the important sentence for this discussion is
> this unique quote, about "lambda,"
>
> >"This is not exactly the same as lambda in functional
>> programming languages, but it is a very powerful concept . . ."
>
> People who understand lambda from functional languages must hassle
> with the fact that lambda is different in python, and people who do not
> understand functional programming languages (nor the lambda calculus)
> are confused by the syntax, also just what it does.

You want to remove lambda because it causes confusion. Okay, for the sake
of the argument I will agree that lambda is confusing. Do you know what
else is confusing?

Threading is confusing. So is multiprocessing. So are databases. Unicode
is confusing. First-class functions are confusing. Recursion is
confusing. Closures are confusing. List comprehensions are confusing.
What the hell are trampolines? I'm certainly confused by them.

If we remove lambda because it is confusing, shouldn't we also remove all
these other confusing things? Where shall we stop?

Should we stop only when Python is a confusion-free, utterly useless toy
language? I don't think so. Or should we stop *before* going down that
path? You want to single out lambda. I don't think we should.


>> Python is not a pure functional language, but you can write functional
>> code in it. If you want a pure functional language, you know where to
>> find one.
>
> Yes, that's obvious; but you're missing the point. Python is not a
> functional language, and implying that it can be used as one is
> misleading at best (maybe a lie at worst) just because it has a
> construct for generating a dynamic anonymous function.

I didn't say that Python can be used as a functional language. I said you
can write functional code in Python. And that is so obviously true that I
cannot believe that you are disputing it.

Functions are first-class values. You can pass them around as arguments
to other functions. You can wrap them to make closures, or compose them.
You can return functions from other functions. You can, if you wish,
eschew holding state and write pure functions with no side-effects and no
state. You can even eschew local variables (well, mostly) inside your
functions and do (nearly) everything by composing other functions. You
can write pipelines of iterators, generators or co-routines.

It may be hard to write an *entire* application using nothing but
functional style in Python, but using functional style throughout the
application is so easy and simple that most people don't even realise
that they are writing in a functional style when they do so.


>> (I wonder whether, say, Haskell has the people coming along and
>> demanding that it should become more like Python?)
>
> Another straw man.
>
> Well, number one, I'm not demanding anything. Number two, everyone
> who uses Haskell (for whatever reason) knows well from the start that
> its a pure functional programming language. That is the advertisement,
> and that is the expectation. No one expects anything different.

And everyone who uses Python ought to know that Python is not. That is
the advertisement, and that ought to be the expectation.

Anonymous functions are not the sole preserve of purely functional
languages, any more than objects are the sole preserve of purely OOP
languages.


>>> I can see uses for python's lambda. But, honestly, I think python
>>> could deprecate its use and in five years just remove it from the
>>> language; along with filter, map, and reduce !
>>
>> Python could deprecate many things. It would make Python a worse
>> language.
>
> How so? Read Guido's argument above. Another way to answer this
> question is that I have been programming with Python for almost a decade
> and I've not used lambda. In fact, I have gone out of my way to NOT use
> lambda because I am fully aware that the BDFL hates it.

In other words, you have had situations where you *could have* used
lambda, perhaps even *should have* used lambda, but you intentionally
went *out of your way* (i.e. made the job harder than it needed to be) to
avoid it, just to slavishly ape the BDFL.

Thank you for just proving my point for me. Your code would have been
better, or at least easier, with lambda. Removing it would make the
language worse, not better.


> Reduce is no
> longer in the standard library (although you can import it)

You mean it is no longer in the builtin namespace. It is still in the
standard library.


> and there
> are equally good ways to do what lambda was designed for without the
> hassle nor confusion.

Incorrect. lambda was designed to allow you to create anonymous functions
in an expression, not as a statement. There is *no other way* to do that
in Python. Even if you come up with some weird hack involving exec or
FunctionType and code objects, it won't be *equally good*. It will be a
broken, third-rate imitation of lambda.


>> By the way, are you aware that lambda is *identical* to def except for
>> three things?
>>
>> - lambda is an expression, not a statement like def;
>>
>> - the body of a function created with lambda is limited to a
>> single expression, not a block;
>>
>> - the function object created with lambda has a generic name.
>
> Absolutely, thank you for pointing those out ! You forgot that
> lambdas can be used in expressions while def functions may not...

Umm, read my list again. It's the first item.


> but
> you really have proved my point. There is so little difference between
> the two that (for the sake of clarity and reducing confusion) it might
> be good to just remove the lambda thing altogether. Just say'in.

The *number* of differences may be small, but the *consequences* of those
differences are enormous.


>> So unless the surprising behaviour about lambda that you're about to
>> tell us about relates to one of those three things, I *guarantee* that
>> functions created with def have the same surprising behaviour.
>
> No, the surprising behavior is that python lambda does not work
> exactly the way that lambda works in functional programming languages

Then neither does def.

Would you like to discard def as well? No? Why not?


> (see the two links provided) the confusion it creates for "normal" users
> does not warrant its inclusion in the language.
>
> Having said that, I must admit that I've "played" with lambda
> extensively and find it interesting from a comp sci standpoint, but,
> again, for normal users I don't think there is a really good value add
> for having it in the language.




--
Steven

Marko Rauhamaa

unread,
Mar 24, 2014, 5:55:24 AM3/24/14
to
Ian Kelly <ian.g...@gmail.com>:

> If lambda were going to be deprecated and removed then it already
> would have happened in Python 3, because Guido tried to do precisely
> that. I'm not sure what the reasons were for keeping it in the end
> (according to PEP 3099 it was because nobody suggested a suitable
> replacement), but if he couldn't get rid of it then, he never will.

You never *need* (Python's) lambda for anything. Inner functions are
more capable and almost always more readable. It doesn't hurt to have
lambda, but I don't find any use for it, either.


Marko

Mark Lawrence

unread,
Mar 24, 2014, 5:58:58 AM3/24/14
to pytho...@python.org
On 24/03/2014 05:52, Mark H Harris wrote:
>
> How so? Read Guido's argument above. Another way to answer this
> question is that I have been programming with Python for almost a decade
> and I've not used lambda. In fact, I have gone out of my way to NOT use
> lambda because I am fully aware that the BDFL hates it. Reduce is no
> longer in the standard library (although you can import it) and there
> are equally good ways to do what lambda was designed for without the
> hassle nor confusion.
>

Where do you get reduce from if it's not in the standard library? As
for lambda I've no real interest in it, other than when copying examples
where it's used to (say) provide a key function.

Chris Angelico

unread,
Mar 24, 2014, 7:21:20 AM3/24/14
to pytho...@python.org
On Mon, Mar 24, 2014 at 8:49 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> I'm too lazy to do this the
> right way, but there are at least 99 mentions of "lambda" in the 3.3
> standard library:
>
> steve@runes:/usr/local/lib/python3.3$ grep lambda *.py | wc -l
> 99

I'm not too lazy to do it the right way, but I don't have 3.3 handy,
so I've done it on 3.4 instead. There are 77 instances of lambda nodes
in the files you list there - which are the ones that aren't in
packages. (Note that two instances of lambda on the same line would
count as one in Steven's figure, but as two in mine. Also, his counts
comments. Still, his way's a lot easier to calculate, and it's in the
right ball-park.) Including all subdirectories raises that figure to,
get this, 1230. That's actual uses of the lambda keyword as parsed by
Python. This does include the test suite, though. Removing all files
with "/test/" in the names cuts that figure to only 273. But that's
still two hundred and seventy-three places where the Python standard
library uses lambda - a respectable figure.

ChrisA

Chris Angelico

unread,
Mar 24, 2014, 7:49:38 AM3/24/14
to pytho...@python.org
They're often not more readable. A lot of people seem to equate
"verbose" with "readable", possibly by faulty extrapolation from
unreadably crunched code with one-letter variables and no line breaks.
But which of these is truly more readable?

squares = []
for n in range(30):
squares.append(n * n)

squares = [n * n for n in range(30)]

Similarly, there are plenty of cases where a nameless function is MUCH
clearer than breaking it out into a separate def and then using the
name once. Do you name the function for what it does internally?

def get_oneth_element_index(item):
return item[1].index
L.sort(key=get_oneth_element_index)

Or for how you're using it?

def keyfunc(item):
return item[1].index
L.sort(key=keyfunc)

Or do you just shortcut the whole thing by inlining it?

L.sort(key=lambda item:item[1].index)

Hey look, that's weakref.py line 941 right there. (It happens to be
the alphabetically last use of lambda in the stdlib. Made a good
example, although it'd be more common to have a space after that
colon.)

ChrisA

Marko Rauhamaa

unread,
Mar 24, 2014, 8:36:48 AM3/24/14
to
Chris Angelico <ros...@gmail.com>:

> Similarly, there are plenty of cases where a nameless function is MUCH
> clearer than breaking it out into a separate def and then using the
> name once. Do you name the function for what it does internally?
>
> def get_oneth_element_index(item):
> return item[1].index
> L.sort(key=get_oneth_element_index)
>
> Or for how you're using it?
>
> def keyfunc(item):
> return item[1].index
> L.sort(key=keyfunc)
>
> Or do you just shortcut the whole thing by inlining it?
>
> L.sort(key=lambda item:item[1].index)

I still prefer the "def" variant. It even allows you to clarify the
meaning of the tuple slot by using a nicer name.


Marko

Chris Angelico

unread,
Mar 24, 2014, 8:53:12 AM3/24/14
to pytho...@python.org
On Mon, Mar 24, 2014 at 11:36 PM, Marko Rauhamaa <ma...@pacujo.net> wrote:
>> def get_oneth_element_index(item):
>> return item[1].index
>> L.sort(key=get_oneth_element_index)
>>
>> Or do you just shortcut the whole thing by inlining it?
>>
>> L.sort(key=lambda item:item[1].index)
>
> I still prefer the "def" variant. It even allows you to clarify the
> meaning of the tuple slot by using a nicer name.

It's the index of element 1. What more do you need to know? Is it
actually any help to give that a name? All you gain is a chance for
the name, the purpose, and the functionality to come into
disagreement.

ChrisA

Steven D'Aprano

unread,
Mar 24, 2014, 10:04:59 AM3/24/14
to
On Mon, 24 Mar 2014 22:49:38 +1100, Chris Angelico wrote:

> On Mon, Mar 24, 2014 at 8:55 PM, Marko Rauhamaa <ma...@pacujo.net>
> wrote:

>> You never *need* (Python's) lambda for anything. Inner functions are
>> more capable and almost always more readable. It doesn't hurt to have
>> lambda, but I don't find any use for it, either.

Marko has missed an obvious use: lambda is the *only* solution when you
need a function embedded in an expression. You simply cannot do so
otherwise. It is truly astonishing that two people here who are such keen
supporters of functional programming, Marko and Mark Harris, are so
dismissive of lambda, and so forced to write declarative code using def.


> They're often not more readable. A lot of people seem to equate
> "verbose" with "readable", possibly by faulty extrapolation from
> unreadably crunched code with one-letter variables and no line breaks.
> But which of these is truly more readable?
>
> squares = []
> for n in range(30):
> squares.append(n * n)
>
> squares = [n * n for n in range(30)]

Readable for whom?

List comprehension syntax is often completely obscure to beginners. A
beginner would say that the explicit for-loop is more readable.

Actually, a *real* beginner, whose main programming experience before
Python was Pascal, would probably even say that the first example was an
unreadable mess. What's range(30)? What's this ".append" business? What
does [] mean? I know this because I was this beginner, once. The first
few times I tried reading Python code, I couldn't make head or tail of
it. "for" I recognised, because it was the same keyword as Pascal and
Hypertalk use. Pretty much everything else might as well have been
Bulgarian.

This was before the public internet, there was no Google, no tutorials I
could look up. It was only after a colleague convinced me that it would
be a good language to learn that I bought an actual dead tree book and
sat down and learned how to read Python.

I think that Python's reputation of being "executable pseudo-code" is
sometimes harmful. It fools people into thinking that unless Aunt Tilly
can understand a language feature, it's somehow a failure. Hence the
arguments by Mark against lambda.

http://www.catb.org/jargon/html/A/Aunt-Tillie.html

But nobody expects Aunt Tilly to read Scheme or C++ or Go without at
least a bit of learning -- or even Esperanto or French or Latin. You
still need to learn the syntax and the vocabulary, and have some idea of
the basic concepts. The same applies to Python. The learning curve may be
more gentle, but there is still a learning curve. And that is perfectly
fine.

So while I agree with you that, to a moderately fluent Python speaker,
not a beginner but not an expert either, the list comprehension is more
readable, for a beginner (one who has a basic Python vocab but isn't
fluent yet) the for-loop will probably be more readable.


> Similarly, there are plenty of cases where a nameless function is MUCH
> clearer than breaking it out into a separate def and then using the name
> once. Do you name the function for what it does internally?
>
> def get_oneth_element_index(item):
> return item[1].index
> L.sort(key=get_oneth_element_index)
>
> Or for how you're using it?
>
> def keyfunc(item):
> return item[1].index
> L.sort(key=keyfunc)

Why not both?! Don't forget to make it private so some other piece of
code doesn't use it. Or better still, delete it when done!

def _get_oneth_element_index_to_use_as_keyfunc_when_sorting_L(item):
return item[1].index

L.sort(key=_get_oneth_element_index_to_use_as_keyfunc_when_sorting_L)
del _get_oneth_element_index_to_use_as_keyfunc_when_sorting_L

Four lines of code to do what lambda lets you do in one. And people still
insist that lambda is pointless. Maybe they're being paid by the line.


> Or do you just shortcut the whole thing by inlining it?
>
> L.sort(key=lambda item:item[1].index)

Exactly.




--
Steven D'Aprano
http://import-that.dreamwidth.org/

Mark Lawrence

unread,
Mar 24, 2014, 10:21:02 AM3/24/14
to pytho...@python.org
Each to their own. Here give me the lambda version any day of the week.

Steven D'Aprano

unread,
Mar 24, 2014, 10:39:51 AM3/24/14
to
# Magic constants are wicked. Never use a constant without naming it.
ELEMENT_TO_USE_FOR_INDEXING_WHEN_SORTING_L = 1

# <summary>
# key func used when sorting L, returns item's 1th elem index method
# </summary>
# <function_name>
# _get_oneth_element_index_to_use_as_keyfunc_when_sorting_L
# </function_name>
# <author>Steven D'Aprano</author>
# <date_created>2014-03-25</date_created>
# <date_modified>2014-03-25</date_modified>
# <revision>1</revision>
# <param name="item">item to be sorted</param>
# <returns>index method of the oneth element</returns>
# <raises>NameError</raises> # FIXME can this fail any other way?
def _get_oneth_element_index_to_use_as_keyfunc_when_sorting_L(item):
"""Private key function for sorting list L.

Returns the index method of element 1 of the given item.

Example of use:

>>> item = (None, '')
>>> _get_oneth_element_index_to_use_as_keyfunc_when_sorting_L(item)
<built-in method index of str object at ...>

Relies on global constant ELEMENT_TO_USE_FOR_INDEXING_WHEN_SORTING_L.
May raise NameError if that constant is missing.

Warning: do not use this for anything else.
"""
return item[ELEMENT_TO_USE_FOR_INDEXING_WHEN_SORTING_L].index

L.sort(key=_get_oneth_element_index_to_use_as_keyfunc_when_sorting_L)
del _get_oneth_element_index_to_use_as_keyfunc_when_sorting_L
# Better to be safe than sorry.
del ELEMENT_TO_USE_FOR_INDEXING_WHEN_SORTING_L




Definitely being paid by the line :-)

Mark Lawrence

unread,
Mar 24, 2014, 11:22:50 AM3/24/14
to pytho...@python.org
One of the finest examples of extracting the urine I've ever read.
Please keep up the good work. Ah but wait, down voted on the grounds
that there are no unit tests and, far more importantly, it doesn't fit
on one line (unless you use gg that is). There is also no proof that
it's been committed to your source control system after going through
its code review.

Rustom Mody

unread,
Mar 24, 2014, 12:00:54 PM3/24/14
to
Ok so far

> So while I agree with you that, to a moderately fluent Python speaker,
> not a beginner but not an expert either, the list comprehension is more
> readable, for a beginner (one who has a basic Python vocab but isn't
> fluent yet) the for-loop will probably be more readable.

And now I wonder...
The case you are wanting to make and the case you are ending up making
seem to be opposite:

If it is so as you say that for "the beginner (one who has a basic
Python vocab but isn't fluent yet)"
the for-loop is more readable than the for-in-compr
it then suggests that those more experienced taking/helping the
beginners at 0 to the stage of "basic vocab-but-not-yet fluent" are
choosing a sub-optimal route.

Remember that you started by reminding that for an absolute beginner,
everything is 'Bulgarian' not just list comprehensions.

Then how come some things become accessible faster than others?
It may be that those things are actually easier (in some objective sense)
But it may also reveal a prejudice of the teachers.

As for the other argument -- python lambda is BAD -- Ive currently run out of pepper for sprinkling on these flames :-)
I'll just make a few points:

1. Miranda the predecessor of Haskell had no lambda.
Local defs + curried convention + operator sections was considered more than
sufficient for functional programming

2. Lisp was the originator of lambda (in programming languages) and it screwed up the semantics so badly that
a. It had to be corrected at high cost 25 years later -- aka scheme and common lisp
b. Some doyens of functional programming have suggested that lisp is a major setback for the acceptance of functional programming, eg
http://www.cs.kent.ac.uk/people/staff/dat/miranda/wadler87.pdf

3. One basic tenet of λ calculus
foo = λ x . exp
is equivalent to the more traditional
foo(x) = exp

is a little uh-uh in haskell thanks to the notorious monomorphism restriction
http://www.haskell.org/haskellwiki/Monomorphism_restriction

So if FP = λ calculus, Haskell does not quite make it!!

Ian Kelly

unread,
Mar 24, 2014, 1:24:00 PM3/24/14
to Python
So what? One might say the same thing about comprehensions -- loops

Mark H Harris

unread,
Mar 24, 2014, 2:58:16 PM3/24/14
to
On 3/24/14 4:58 AM, Mark Lawrence wrote:
> Where do you get reduce from if it's not in the standard library?

That was "a" proposal for 3000. Its there, but its not on the
built-ins; ie., you have to import it. The confusion: why reduce, why
not filter, nor map? {rhetorical}

> As for lambda I've no real interest in it, other than when copying examples
> where it's used to (say) provide a key function.
>

This is one of my main points to Steven. In my experience "most" people
do not intend to use lambda for anything; they are trying to sort this
or that and don't quite know how to get the key right and some helpful
somebody gives them a key=lambda yadda yadda . They use it, and it
works, but they are scratching their head saying to themselves, "what it
that, how does it work, how can I understand it and on and on".

That is what we mean by confusing. Or another really great example is
this thread. Somebody asks about a language feature and somebody else
helpfully answers the question by providing them with a similar lambda!!

Its the programmer's equivalent of explanation by reference to a more
complicated analogy; which leaves the OP left with, "Thanks for all the
responses".

marcus

PS You are absolutely right, all the expanding double spaces become
very annoying when viewed on Thunderbird; it is exasperating, genuinely.



Chris Angelico

unread,
Mar 24, 2014, 3:12:47 PM3/24/14
to pytho...@python.org
On Tue, Mar 25, 2014 at 1:04 AM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
>> But which of these is truly more readable?
>>
>> squares = []
>> for n in range(30):
>> squares.append(n * n)
>>
>> squares = [n * n for n in range(30)]
>
> Readable for whom?
>
> List comprehension syntax is often completely obscure to beginners. A
> beginner would say that the explicit for-loop is more readable.
>
> Actually, a *real* beginner, whose main programming experience before
> Python was Pascal, would probably even say that the first example was an
> unreadable mess. What's range(30)? What's this ".append" business? What
> does [] mean? I know this because I was this beginner, once. The first
> few times I tried reading Python code, I couldn't make head or tail of
> it. "for" I recognised, because it was the same keyword as Pascal and
> Hypertalk use. Pretty much everything else might as well have been
> Bulgarian.

Actually, that's a very good point. Python's for loop is more often
called a foreach loop in other languages, and Python completely lacks
any concept of a "classic" iteration-over-integer for loop. That is a
point of confusion. However, that's going to come up on both branches,
so it's not really a mark against either.

Incidentally, I've often modified my loop counter, in C or REXX or any
other language. About the only situation where I actually miss it in
Python, though, is iterating over a list and mutating the list on the
way through; and even that can often be done in other ways (maybe a
list comp, filtering out some of the elements?). It's amazing how
something can be so utterly fundamental (I mean, come ON! Who can
imagine a language with no equivalent of the basic "do i=1 to 10"
(REXX) or "for (int i=0;i<10;++i)" (C++) loop???) and yet so
dispensable.

ChrisA

Mark Lawrence

unread,
Mar 24, 2014, 3:13:13 PM3/24/14
to pytho...@python.org
On 24/03/2014 18:58, Mark H Harris wrote:
> On 3/24/14 4:58 AM, Mark Lawrence wrote:
>> Where do you get reduce from if it's not in the standard library?
>
> That was "a" proposal for 3000. Its there, but its not on the
> built-ins; ie., you have to import it. The confusion: why reduce, why
> not filter, nor map? {rhetorical}

So it is in the standard library then. And I'm not confused, seeing
this must have been decided years ago as Python 3 was released some five
years ago.

>
>> As for lambda I've no real interest in it, other than when copying
>> examples
>> where it's used to (say) provide a key function.
>>
>
> This is one of my main points to Steven. In my experience "most" people
> do not intend to use lambda for anything; they are trying to sort this
> or that and don't quite know how to get the key right and some helpful
> somebody gives them a key=lambda yadda yadda . They use it, and it
> works, but they are scratching their head saying to themselves, "what it
> that, how does it work, how can I understand it and on and on".

More fool them, I write Python as I let it take away the head
scratching, not add to it. If I wanted to start head scratching maybe
I'd go and investigate what line 247 of gcmodule.c does, but funnily
enough I've never been there, and don't intend starting now.

>
> That is what we mean by confusing. Or another really great example is
> this thread. Somebody asks about a language feature and somebody else
> helpfully answers the question by providing them with a similar lambda!!

One of the joys of this list from my POV, YMMV.

>
> Its the programmer's equivalent of explanation by reference to a more
> complicated analogy; which leaves the OP left with, "Thanks for all the
> responses".
>
> marcus
>
> PS You are absolutely right, all the expanding double spaces become
> very annoying when viewed on Thunderbird; it is exasperating, genuinely.
>

Yep, but like I said the situation has improved, partly thanks to the
guys who improved the words on the wiki showing how to successfuly use
gg. Thanks fellas :)

Ian Kelly

unread,
Mar 24, 2014, 3:12:54 PM3/24/14
to Python
On Mon, Mar 24, 2014 at 12:58 PM, Mark H Harris <harri...@gmail.com> wrote:
> That is what we mean by confusing. Or another really great example is this
> thread. Somebody asks about a language feature and somebody else helpfully
> answers the question by providing them with a similar lambda!!

That is not in fact how the topic of lambda arose in this thread.
Rustom Mody brought up the binding behavior in a tangent specifically
to complain about it, and that was the first mention of lambda in the
thread.

Chris Angelico

unread,
Mar 24, 2014, 3:22:28 PM3/24/14
to pytho...@python.org
On Tue, Mar 25, 2014 at 5:58 AM, Mark H Harris <harri...@gmail.com> wrote:
> Its there, but its not on the built-ins; ie., you have to import it. The
> confusion: why reduce, why not filter, nor map? {rhetorical}

In other languages with those three, and without list/array
comprehensions, I've used filter occasionally and map reasonably
often, but I don't remember the last time I used reduce. Actually,
Pike has special syntax that can take the place of map sometimes, so I
might use filter more often than map in Pike code, because these don't
need explicit map calls:

//Suppose that clients is an array of connected clients on some server
clients->sockets->write("System message: blah blah blah\n");

Indexing an array (the -> is like Python's . as Pike's . is resolved
at compile time) produces an array, effectively mapping the elements
through "lambda x: x->sockets" and ditto for "->write". Calling an
array calls all the non-empty elements in it, with the same
argument(s), and produces an array of return values. (In this case, I
don't care about the return values, which will simply be the number of
bytes written to each socket. If there's a problem, it'll throw an
exception.) Huh. Even with that, and the [*] automap syntax, and such,
I still use map far more often than filter... and filter orders of
magnitude more often than reduce.

Aside: You'll often hear people talking about "map-reduce" with big
data. Python supports that. Look!

>>> map.__reduce__
<method '__reduce__' of 'map' objects>

Oh wait, that's nothing to do with reduce()...

*ducks for cover*

ChrisA

Ian Kelly

unread,
Mar 24, 2014, 3:42:03 PM3/24/14
to Python
On Mon, Mar 24, 2014 at 1:12 PM, Chris Angelico <ros...@gmail.com> wrote:
> Incidentally, I've often modified my loop counter, in C or REXX or any
> other language. About the only situation where I actually miss it in
> Python, though, is iterating over a list and mutating the list on the
> way through; and even that can often be done in other ways (maybe a
> list comp, filtering out some of the elements?). It's amazing how
> something can be so utterly fundamental (I mean, come ON! Who can
> imagine a language with no equivalent of the basic "do i=1 to 10"
> (REXX) or "for (int i=0;i<10;++i)" (C++) loop???) and yet so
> dispensable.

I'm not sure "fundamental" is the right word. A for loop is just a
while loop with some syntactic sugar. For that matter, a while loop
is just a structured goto...

Chris Angelico

unread,
Mar 24, 2014, 3:45:53 PM3/24/14
to pytho...@python.org
On Tue, Mar 25, 2014 at 6:13 AM, Mark Lawrence <bream...@yahoo.co.uk> wrote:
>> That was "a" proposal for 3000. Its there, but its not on the
>> built-ins; ie., you have to import it. The confusion: why reduce, why
>> not filter, nor map? {rhetorical}
>
>
> So it is in the standard library then. And I'm not confused, seeing this
> must have been decided years ago as Python 3 was released some five years
> ago.

Terminology issue, is all. It's not in the builtins, but it is in the
standard library.

ChrisA

Mark H Harris

unread,
Mar 24, 2014, 3:47:11 PM3/24/14