Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

[vasudevram]

Hi list,

Can anyone - maybe one of the Python language core team, or someone with knowledge of the internals of Python - can explain why this code works, and whether the different occurrences of the name x in the expression, are in different scopes or not? :

x = [[1,2], [3,4], [5,6]]
[x for x in x for x in x]

I saw this on a Hacker News thread about Python, and here is a post I wrote that gives more details about it, including relevant links, how I found that it can be extended to a triply-nested list, and my thoughts about the scope issue:

http://jugad2.blogspot.in/2014/03/flatten-list-of-lists-with-list.html

A few people commented about it, both on my blog, and on the Python Reddit where I also submitted my post, but I'm not sure I'm convinced of their reasoning or understand it, hence posting the question here.

Thanks,
Vasudev Ram
www.dancingbison.com
jugad2.blogspot.com

[Rustom Mody]

Lets try without comprehending comprehensions :-)

>>> x=[[1,2],[3,4]]
>>> for x in x:
... for x in x:
... print x
...
1
2
3
4
>>>

[vasudevram]

On Saturday, March 22, 2014 2:24:00 AM UTC+5:30, Rustom Mody wrote:
> Lets try without comprehending comprehensions :-)
> >>> x=[[1,2],[3,4]]
>
> >>> for x in x:
>
> ... for x in x:
>
> ... print x
>
> ...
>
> 1
>
> 2
>
> 3
>
> 4

Nice and all, thanks, but doesn't answer the question.

[Rustom Mody]

Which is?



A 'for' introduces a scope:

>>> x = 42
>>> for x in [1,2,3]:

... print x
...
1
2
3

No sign of the 42 --v ie the outer x -- inside because of scope

And so we can do:

>>> x = [1,2,3]

>>> for x in x:
... print x
...
1
2
3

which implies that in a "for var in exp: ..."
the exp is evaluated in outer scope whereas the var has a new scope inside
the "..."

Now repeatedly apply that principle to the nested for.
Same principle for nested for in a comprehension.

[Ian Kelly]

On Fri, Mar 21, 2014 at 3:09 PM, Rustom Mody <rusto...@gmail.com> wrote:
> A 'for' introduces a scope:

This is false.

>>>> x = 42
>>>> for x in [1,2,3]:
> ... print x
> ...
> 1
> 2
> 3
>
> No sign of the 42 --v ie the outer x -- inside because of scope

Try printing x again *after* the for loop:

>>> x = 42
>>> for x in [1,2,3]:

... print(x)
...
1
2
3
>>> print(x)
3

Notice that it's still 3. If the x in the for loop were really a
separately scoped variable, we would have expected to see 42 here. In
fact the x used in the for loop and the x used outside of it are the
same x variable.

The real reason that the OP's example works is because each value that
the x variable holds happens to only need to be read once. Start with
this code, and work through it line-by-line:

x = [[1, 2], [3, 4]]
for x in x:
for x in x:
print(x)

Initially, x is assigned the list. Then we enter the outer for loop.
The expression x is evaluated once to be iterated over. The first
value that the for loop iterator yields is [1, 2], which is then
assigned to x. Now we enter the inner for loop. The expression x is
again evaluated once to be iterated over. It first yields 1, which is
assigned to x, and then printed. The inner for loop iterator then
yields 2, which is also assigned to x and printed. The inner for loop
iterator then raises StopIteration, and the inner for loop terminates.
The outer for loop iterator then yields [3, 4], and this is assigned
to x. The inner for loop evaluates this expression and runs as
before. Afterward, the outer for loop iterator raises StopException,
and that for loop terminates. The final value of x here will be 4:

>>> x = [[1, 2], [3, 4]]

>>> for x in x:
... for x in x:

... print(x)

...
1
2
3
4

>>> print(x)
4

As I hope this makes clear, no nested scopes are needed to make this
happen. It works because nothing that is stored in x needs to remain
there after the next thing is stored in x.

[Gregory Ewing]

Rustom Mody wrote:

> A 'for' introduces a scope:

No, it doesn't!

>>>>x = 42
>>>>for x in [1,2,3]:
> ... print x
> ...
> 1
> 2
> 3
>
> No sign of the 42 --v ie the outer x -- inside because of scope

You're right that there's no sign of the 42, but it's
*not* because of scope, as you'll see if you do one
more print:

>>> print x
3

> And so we can do:
>
>
>>>>x = [1,2,3]
>>>>for x in x:
> ... print x
> ...
> 1
> 2
> 3

Again, if you print x after the loop has finished,
you'll find that it's no longer bound to the original
list. This is because Python's for-loop does *not*
introduce a new scope -- there's only one x, and
the for-loop is sharing it with the rest of the code.

The real question is why the for-loop works in *spite*
of this fact.

The answer is that the for-loop machinery keeps an
internal reference to the list being iterated over,
so once the loop has started, it doesn't matter what
x is bound to any more.

--
Greg

[Rustom Mody]

On Saturday, March 22, 2014 3:00:10 AM UTC+5:30, Ian wrote:

> On Fri, Mar 21, 2014 at 3:09 PM, Rustom Mody wrote:
> > A 'for' introduces a scope:

> This is false.

And

On Saturday, March 22, 2014 3:04:48 AM UTC+5:30, Gregory Ewing wrote:

> > A 'for' introduces a scope:

> No, it doesn't!


Ha -- funny that *I* missed that one. Thanks both Ian and Gregory

In fact one of my grumbles against python is that list comprehension's
are a poor imitation of haskell's comprehensions.

And then I promptly forgot about it!

Actually there are two leakages in python 2, one of which is corrected
in python 3.

One: a comprehension variable leaks outside after the comprehension
This is corrected in python3.
Two: A comprehension variable is not bound but reassigned across the
comprehension. This problem remains in python3 and causes weird behavior when
lambdas are put in a comprehension

>>> fl = [lambda y : x+y for x in [1,2,3]]
>>> [fl[i](2) for i in [0,1,2]]
[5, 5, 5]

The same in haskell:

Prelude> let fl = [\ y -> x + y | x <- [1,2,3]]
Prelude> [(fl!!i) 0 | i<- [0,1,2]]
[1,2,3]

[Chris Angelico]

On Sat, Mar 22, 2014 at 1:06 PM, Rustom Mody <rusto...@gmail.com> wrote:
> Two: A comprehension variable is not bound but reassigned across the
> comprehension. This problem remains in python3 and causes weird behavior when
> lambdas are put in a comprehension
>
>>>> fl = [lambda y : x+y for x in [1,2,3]]
>>>> [fl[i](2) for i in [0,1,2]]
> [5, 5, 5]

To clarify, what you're saying here is that x in the first
comprehension's closures should be bound to separate values for x,
yes?

I'm not sure how that ought to be done. Having closures that can
reference and modify each other's variables is important.

def func_pair():
x = 0
def inc():
nonlocal x; x+=1
return x
def dec():
nonlocal x; x-=1
return x
return inc, dec

fooup, foodn = func_pair()
barup, bardn = func_pair()
>>> fooup(), fooup(), fooup(), foodn()
(1, 2, 3, 2)
>>> barup(), barup(), bardn(), bardn()
(1, 2, 1, 0)

Those functions are fundamentally linked. Very useful with callbacks.
A nice alternative to doing everything with bound methods.

So if that's not going to be broken, how is this fundamentally different?

def func_loop():
for x in 1,2,3:
yield (lambda: x)

one, two, three = func_loop()
one(), one(), two(), two(), three(), three()

This one does NOT work the way the names imply, and I can see that
you'd like to fix it. But I can't pinpoint a significant difference
between them. How do you distinguish?

ChrisA

[Rustom Mody]

On Saturday, March 22, 2014 8:11:27 AM UTC+5:30, Chris Angelico wrote:

> On Sat, Mar 22, 2014 at 1:06 PM, Rustom Mody wrote:
> > Two: A comprehension variable is not bound but reassigned across the
> > comprehension. This problem remains in python3 and causes weird behavior when
> > lambdas are put in a comprehension
> >>>> fl = [lambda y : x+y for x in [1,2,3]]
> >>>> [fl[i](2) for i in [0,1,2]]
> > [5, 5, 5]

> To clarify, what you're saying here is that x in the first
> comprehension's closures should be bound to separate values for x,
> yes?

Yes

> I'm not sure how that ought to be done.

Thats an implementation question -- see below.

> Having closures that can
> reference and modify each other's variables is important.

Yes

> def func_pair():
> x = 0
> def inc():
> nonlocal x; x+=1
> return x
> def dec():
> nonlocal x; x-=1
> return x
> return inc, dec

> fooup, foodn = func_pair()
> barup, bardn = func_pair()
> >>> fooup(), fooup(), fooup(), foodn()
> (1, 2, 3, 2)
> >>> barup(), barup(), bardn(), bardn()
> (1, 2, 1, 0)

> Those functions are fundamentally linked. Very useful with callbacks.
> A nice alternative to doing everything with bound methods.

Yes

> So if that's not going to be broken, how is this fundamentally different?

> def func_loop():
> for x in 1,2,3:
> yield (lambda: x)

Thats using a for-loop
A 'for' in a comprehension carries a different intention, the matching names
being merely coincidental.

This 'pun' causes cognitive dissonance in all these questions, including
my gaffe above

> one, two, three = func_loop()
> one(), one(), two(), two(), three(), three()

> This one does NOT work the way the names imply, and I can see that
> you'd like to fix it. But I can't pinpoint a significant difference
> between them. How do you distinguish?

Using closures for carrying state is a different question

As for comprehensions, the appropriate *intention* would be like this:

Given

fl = [lambda y : x+y for x in [1,2,3]]

It means:

def rec(l):
if not l: return []
else:
x,ll = l[0],l[1:]
return [lambda y: x + y] + rec(ll)

followed by
fl = rec([1,2,3])

Naturally a reasonable *implementation* would carry this *intention* more
efficiently with standard techniques like
1. Using list extend/append methods
2. Using lambda y, x=x: x+y

Inside an implementation this is fine
Forcing such tricks as kosher on a programmer is not (IMHO)

[But then I find Lisp and much of basic haskell natural and most of C++ not,
so my views are likely prejudiced :-)

[Steven D'Aprano]

On Fri, 21 Mar 2014 19:06:06 -0700, Rustom Mody wrote:

> Two: A comprehension variable is not bound but reassigned across the
> comprehension. This problem remains in python3 and causes weird behavior
> when lambdas are put in a comprehension

I don't know why you say the behaviour in Python is a problem. It's
somewhat unexpected if you don't think about what's going on, but it's
actually quite logical. On the contrary, I find the Haskell version weird.

>>>> fl = [lambda y : x+y for x in [1,2,3]]
>>>> [fl[i](2) for i in [0,1,2]]
> [5, 5, 5]

This makes perfect sense: by the time you call the functions, the name x
has been rebound to the value 3. If x were a global, or if comprehensions
leaked their variable, that would be obvious: you could print x and see
exactly what value it has. But visible or not, that's exactly what
happens. Since x is 3, you're adding 3+2 and should get 5 no matter which
function you call.

Unroll the first comprehension, and it's obvious what is going on:

def unrolled():
fl = []
x = 0
def lambda_(y):
return x + y
fl.append(lambda_)
x = 1
def lambda_(y):
return x + y
fl.append(lambda_)
x = 2
def lambda_(y):
return x + y
fl.append(lambda_)
return [f(2) for f in fl]

Don't be fooled by the coincidence that the three "lambda_" functions
have the same name and body. They could have different names and
different bodies, Either way, it is completely natural that they all
closures over the same x -- that's how you wrote them.


But the Haskell version, on the other hand, is just weird:

> The same in haskell:
>
> Prelude> let fl = [\ y -> x + y | x <- [1,2,3]]
> Prelude> [(fl!!i) 0 | i<- [0,1,2]]
> [1,2,3]

For this to be the case, the functions in fl have to somehow "look back
in time" to see the value of x, not as it is *now* when the function is
called, but how it *was* when it was created. That's very weird indeed.
If x were a global, it would be astonishing. The fact that x comes from a
closure instead makes it no less surprising. Unroll the loop, and the
magic is obvious.

Now I'm not sure precisely how Haskell implements this trick, but it
suggests to me that it creates a different closure each time around the
loop of the comprehension. That could end up being very expensive.



--
Steven D'Aprano
http://import-that.dreamwidth.org/

[Chris Angelico]

On Sat, Mar 22, 2014 at 3:39 PM, Rustom Mody <rusto...@gmail.com> wrote:
>> So if that's not going to be broken, how is this fundamentally different?
>
>> def func_loop():
>> for x in 1,2,3:
>> yield (lambda: x)
>
> Thats using a for-loop
> A 'for' in a comprehension carries a different intention, the matching names
> being merely coincidental.

So what you're saying is that these two are fundamentally different:

def unrolled():
x = 1
yield (lambda: x)
x = 2
yield (lambda: x)
x = 3
yield (lambda: x)

def loop():

for x in 1,2,3:
yield (lambda: x)

In other words, a loop should be implemented as a separate binding
each time, not a rebinding. That's an interesting point, and it does
make some sense; effectively, what you want is for the body of a for
loop to be a new scope, a unique scope every iteration of the loop,
and one that automatically closes over all variables in the parent
scope (including for assignments) except for the loop
iteration/counter variable.

That does make some sense, but it doesn't really fit Python's concept.
It would, however, fit a more C-like language, where locals are
declared (in Python, you'd have to put a whole lot of implicit
'nonlocal' statements at the top of the loop).

ChrisAg

[Mark H Harris]

On 3/21/14 11:39 PM, Rustom Mody wrote:
> Given

> fl = [lambda y : x+y for x in [1,2,3]]

> It means:

> def rec(l):
> if not l: return []
> else:
> x,ll = l[0],l[1:]
> return [lambda y: x + y] + rec(ll)
>
> followed by
> fl = rec([1,2,3])

> Naturally a reasonable *implementation* would carry this *intention*

{snip}

> [But then I find Lisp and much of basic haskell natural and most of C++ not,
> so my views are likely prejudiced :-)

This discussion (the entire thread) comes up again and again over the
years because python tries to be all things to all people, without much
reason behind the motivation. I'm speaking of Lambda (filter, map, reduce).

Python is not Lisp. (I love Lisp too). Python is not Haskell (I love
Haskell too).

Lambda is a problem, if only because it causes confusion. What's the
problem? Glad you asked. The constructs DO NOT work the way most people
would expect them to, having limited knowledge of python! I ran into
this thing about seven years ago (when I was studying Haskell, and
Scheme) and I wanted to see how "pure functional" python was (well, not
at all really).

I can see uses for python's lambda. But, honestly, I think python could
deprecate its use and in five years just remove it from the language;
along with filter, map, and reduce !

I'm just saying;

marcus

[Chris Angelico]

On Sat, Mar 22, 2014 at 3:47 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
> Now I'm not sure precisely how Haskell implements this trick, but it
> suggests to me that it creates a different closure each time around the
> loop of the comprehension. That could end up being very expensive.

It needn't be terribly expensive, if you make a "scope stack" or
"scope chain". Imagine, if you will, a system of scopes like this:

current_scope = None
class Scope:
def __init__(self, *a, **kw):
self.names = dict(*a, **kw)
global current_scope
self.parent = current_scope
current_scope = self
def __getitem__(self, name):
try:
return self.names[name]
except KeyError:
if self.parent is None: raise
return self.parent[name]
def __setitem__(self, name, value):
self.names[name] = value
def exit_scope():
global current_scope
current_scope = current_scope.parent

Creating a new scope would be fairly cheap (and a lot more so if this
were implemented in C without the object overhead, of course). Lookups
would scan up through a series of namespaces, same as they currently
do (local, module, builtins), there'd just be more of them. And the
compiler could be smart enough to skip creating a scope if there are
no assignments. There'd need to be some handling in there for the
'nonlocal' keyword, but my expectation is that 'global' is handled by
retaining a reference to the current_scope at module level, and
starting the search there for a LOAD_GLOBAL.

Each iteration of the loop could begin with Scope() and end with
exit_scope(), and there you are, each iteration in its own closable
scope.

I'm not saying it would be a good thing to do - and it'd be a bad fit
for Python, I think, as I said in my other post - but it could be
done.

ChrisA

[Rustom Mody]

On Saturday, March 22, 2014 10:21:13 AM UTC+5:30, Chris Angelico wrote:

> On Sat, Mar 22, 2014 at 3:39 PM, Rustom Mody wrote:
> >> So if that's not going to be broken, how is this fundamentally different?
> >> def func_loop():
> >> for x in 1,2,3:
> >> yield (lambda: x)
> > Thats using a for-loop
> > A 'for' in a comprehension carries a different intention, the matching names
> > being merely coincidental.

> So what you're saying is that these two are fundamentally different:

> def unrolled():
> x = 1
> yield (lambda: x)
> x = 2
> yield (lambda: x)
> x = 3
> yield (lambda: x)

> def loop():
> for x in 1,2,3:
> yield (lambda: x)

Well almost...

Except that the 'loop' I am talking of is one of

def loop():
return [yield (lambda: x) for x in [1,2,3]]
or
return (yield (lambda: x) for x in [1,2,3])
or just plain ol
(lambda x: for x in [1,2,3])

IOW loop is an imperative construct, comprehensions are declarative

Progressing through a loop in a sequential order is natural and to
be expected

Comprehensions being declarative, progressing through them
in some order is incidental.

> In other words, a loop should be implemented as a separate binding
> each time, not a rebinding. That's an interesting point, and it does
> make some sense; effectively, what you want is for the body of a for
> loop to be a new scope, a unique scope every iteration of the loop,
> and one that automatically closes over all variables in the parent
> scope (including for assignments) except for the loop
> iteration/counter variable.

> That does make some sense, but it doesn't really fit Python's concept.

Yes it does not fit in with imperative programming.

Its good to remember the history.
Haskell did not invent comprehensions

In programming, they started with Miranda where they were called
'ZF-expressions' after the Zermelo/Fraenkel of axiomatic set theory.

Because Russell's paradox had shaken the (intended) foundations of mathematics,
the way out (actually the one chosen by Zermelo and Fraenkel) was to
add a *comprehension axiom*

http://en.wikipedia.org/wiki/Zermelo%E2%80%93Fraenkel_set_theory#3._Axiom_schema_of_specification_.28also_called_the_axiom_schema_of_separation_or_of_restricted_comprehension.29

What this basically mandates is that set constructions like
{x | Pred(x) } are disqualified (called unrestricted comprehension)

Only
{x ∈ S | Pred(x) } is a valid.

IOW sets cannot be concocted out of the blue but only out of other sets.

Fast forward 50 years and what David Turner – inventor of Miranda –- realized
is that if the programming language is sufficiently mathematical – ie no
imperative constructs plus lazy evaluation, then comprehensions are
actually constructive enough to make make into programming constructs.

However as Mark Harris points out imperative and functional thinking styles
remain somewhat inconsistent with each other.

[Rustom Mody]

On Saturday, March 22, 2014 10:21:13 AM UTC+5:30, Chris Angelico wrote:

> On Sat, Mar 22, 2014 at 3:39 PM, Rustom Mody wrote:
> >> So if that's not going to be broken, how is this fundamentally different?
> >> def func_loop():
> >> for x in 1,2,3:
> >> yield (lambda: x)
> > Thats using a for-loop
> > A 'for' in a comprehension carries a different intention, the matching names
> > being merely coincidental.

> So what you're saying is that these two are fundamentally different:



> def unrolled():
> x = 1
> yield (lambda: x)
> x = 2
> yield (lambda: x)
> x = 3
> yield (lambda: x)

> def loop():
> for x in 1,2,3:
> yield (lambda: x)

Well almost

Except that the 'loop' I am talking of is one of

def loop():
return [yield (lambda: x) for x in [1,2,3]]
or
return (yield (lambda: x) for x in [1,2,3])
or just plain ol
(lambda x: for x in [1,2,3])

IOW loops is an imperative construct, comprehensions are declarative

Progressing through a loop in a sequential order is natural and to
be expected

Comprehensions being declarative, progressing through them
in some order is incidental.

> In other words, a loop should be implemented as a separate binding
> each time, not a rebinding. That's an interesting point, and it does
> make some sense; effectively, what you want is for the body of a for
> loop to be a new scope, a unique scope every iteration of the loop,
> and one that automatically closes over all variables in the parent
> scope (including for assignments) except for the loop
> iteration/counter variable.

> That does make some sense, but it doesn't really fit Python's concept.

Yes it does not fit in with imperative programming.

Its good to remember the history.
Haskell did not invent comprehensions

In programming, they started with Miranda where they were called
'ZF-expressions' after the Zermelo/Fraenkel of axiomatic set theory.

Because Russell's paradox had shaken the (intended) foundations of mathematics,
the way out (actually the one chosen by Zermelo and Fraenkel) was to
add a *comprehension axiom*

http://en.wikipedia.org/wiki/Zermelo%E2%80%93Fraenkel_set_theory#3._Axiom_schema_of_specification_.28also_called_the_axiom_schema_of_separation_or_of_restricted_comprehension.29

What this basically mandates is that set constructions like
{x | Pred(x) } are disqualified (called unrestricted comprehension)

Only
{x ∈ S | Pred(x) } is a valid.

IOW sets cannot be concoted out of the blue but only out of other sets.

FF 50 years and what David Turner – inventor of Miranda –- realized is that

if the programming language is sufficiently mathematical – ie no
imperative constructs plus lazy evaluation, then comprehensions are
actually constructive enough to make make into programming constructs.

However as Mark Harris points out imperative and functional thinking styles

remain confusingly inconsistent.

[Ian Kelly]

On Fri, Mar 21, 2014 at 8:06 PM, Rustom Mody <rusto...@gmail.com> wrote:
> Two: A comprehension variable is not bound but reassigned across the
> comprehension. This problem remains in python3 and causes weird behavior when
> lambdas are put in a comprehension

Because Python as a language only has the concept of assignment, not
binding. I think it would be weird and confusing if variables worked
this way in comprehensions and nowhere else.

> >>> fl = [lambda y : x+y for x in [1,2,3]]
> >>> [fl[i](2) for i in [0,1,2]]
> [5, 5, 5]

You can get the desired effect by adding a layer of indirection:

>>> fl = [(lambda x: lambda y: x+y)(x) for x in [1,2,3]]

>>> [f(2) for f in fl]

[3, 4, 5]

If that's too ugly then give the wrapper a proper name:

>>> def make_function(x):
... return lambda y: x+y
...
>>> fl = [make_function(x) for x in [1,2,3]]

>>> [f(2) for f in fl]

[3, 4, 5]

There is also the default argument trick:

>>> fl = [lambda y, *, x=x: x+y for x in [1,2,3]]

>>> [f(2) for f in fl]

[3, 4, 5]

[Steven D'Aprano]

On Fri, 21 Mar 2014 23:51:38 -0500, Mark H Harris wrote:

> Lambda is a problem, if only because it causes confusion. What's the
> problem? Glad you asked. The constructs DO NOT work the way most people
> would expect them to, having limited knowledge of python!

Why is that a problem? Would you consider it a problem that people who
don't understand BASIC can't understand BASIC? ("I don't get the
difference between GOTO and GOSUB. Obviously GOSUB is a problem and
should be throw out.") Do you think that the fact that people who have
never used a TI-89 calculator before may have trouble used a TI-89
calculator means that TI-89 calculators are "a problem"? (I've taught
school children who didn't know how to use their calculator.)

The problem is ignorance, not the calculator, and not lambda.

You've told us that "most people" (which people? dentists? lawyers?
illiterate tribesmen from the Amazon? professional programmers?) are
surprised by lamba's behaviour, but you haven't actually told us what
behaviour is surprising, or what "most people" expect lambda to do.
Perhaps you ought to?

> I ran into
> this thing about seven years ago (when I was studying Haskell, and
> Scheme) and I wanted to see how "pure functional" python was (well, not
> at all really).

Python is not a pure functional language, but you can write functional
code in it. If you want a pure functional language, you know where to
find one.

(I wonder whether, say, Haskell has the people coming along and demanding
that it should become more like Python?)

> I can see uses for python's lambda. But, honestly, I think python could
> deprecate its use and in five years just remove it from the language;
> along with filter, map, and reduce !

Python could deprecate many things. It would make Python a worse language.

By the way, are you aware that lambda is *identical* to def except for
three things?

- lambda is an expression, not a statement like def;

- the body of a function created with lambda is limited to a
single expression, not a block;

- the function object created with lambda has a generic name.


So unless the surprising behaviour about lambda that you're about to tell
us about relates to one of those three things, I *guarantee* that
functions created with def have the same surprising behaviour.

[Marko Rauhamaa]

Steven D'Aprano <steve+comp....@pearwood.info>:

> This makes perfect sense: by the time you call the functions, the name x
> has been rebound to the value 3.

> [...]

> Now I'm not sure precisely how Haskell implements this trick, but it
> suggests to me that it creates a different closure each time around
> the loop of the comprehension. That could end up being very expensive.

Haskell does not rebind variables. I guess Haskell simply physically
substitutes object references for each syntactic occurrence of a
variable.

If Python worked analogously, the following loop:

for i, x in enumerate([1, 2, 3]):
f[i] = lambda y: x + y

would unroll as follows:

f[0] = lambda y: 1 + y
f[1] = lambda y: 2 + y
f[2] = lambda y: 3 + y

That is actually how classic lambda calculus does it. Variables are
substituted, not "bound" or "assigned". They are syntactic slots. There
is no heap, there is no stack, there is no memory apart from the
expression itself.


Marko

[Marko Rauhamaa]

Ian Kelly <ian.g...@gmail.com>:

> You can get the desired effect by adding a layer of indirection:
>
>>>> fl = [(lambda x: lambda y: x+y)(x) for x in [1,2,3]]

A trick to remember! Variable lifetime reduction by function invocation.


Marko

[Mark Lawrence]

On 22/03/2014 02:06, Rustom Mody wrote:
>
> The same in haskell:
>
> Prelude> let fl = [\ y -> x + y | x <- [1,2,3]]
> Prelude> [(fl!!i) 0 | i<- [0,1,2]]
> [1,2,3]
>

My really big complaint about Python is that it's nothing like CORAL 66.
I think I'll raise this on python ideas in an attempt to get this
glaring deficiency corrected.

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com

[Rustom Mody]

On Saturday, March 22, 2014 2:39:56 PM UTC+5:30, Ian wrote:

> On Fri, Mar 21, 2014 at 8:06 PM, Rustom Mody wrote:
> > Two: A comprehension variable is not bound but reassigned across the
> > comprehension. This problem remains in python3 and causes weird behavior when
> > lambdas are put in a comprehension

> Because Python as a language only has the concept of assignment, not
> binding. I think it would be weird and confusing if variables worked
> this way in comprehensions and nowhere else.

Bizarre viewpoint!

When you do this:

> There is also the default argument trick:

> >>> fl = [lambda y, *, x=x: x+y for x in [1,2,3]]
> >>> [f(2) for f in fl]
> [3, 4, 5]

how is that not-a-binding solution?

More generally, insofar as variable-scopes can be made and exited, there
is binding. Its just that imperative languages have
- assignment wherein the shape of the environment is preserved but
its content is changed
- there are binding-constructs -- functions, methods, classes etc etc
-- which leave extant bindings intact but create/remove new ones.

Ok, functional languages have only the latter.
But only the former?? Beyond assembly language I dont know what
that would/could be...

[Mark Lawrence]

On 22/03/2014 09:09, Ian Kelly wrote:

> On Fri, Mar 21, 2014 at 8:06 PM, Rustom Mody <rusto...@gmail.com> wrote:
>> Two: A comprehension variable is not bound but reassigned across the
>> comprehension. This problem remains in python3 and causes weird behavior when
>> lambdas are put in a comprehension
>

> Because Python as a language only has the concept of assignment, not
> binding. I think it would be weird and confusing if variables worked
> this way in comprehensions and nowhere else.
>

My understanding has always been that an expression of the rhs is bound
to a name of the lhs. So is my understanding wrong, or is the above
wrong, or are we talking at cross purposes, or what?

[Marko Rauhamaa]

Mark Lawrence <bream...@yahoo.co.uk>:

> On 22/03/2014 09:09, Ian Kelly wrote:
>> Because Python as a language only has the concept of assignment, not
>> binding. I think it would be weird and confusing if variables worked
>> this way in comprehensions and nowhere else.
>
> My understanding has always been that an expression of the rhs is
> bound to a name of the lhs. So is my understanding wrong, or is the
> above wrong, or are we talking at cross purposes, or what?

Hard to say without knowing more of your understanding.

Even Scheme doesn't have purely classical variable binding because
variables can be assigned to. You will notice that right away when you
try to implement a lisp dialect; a purely binding (substituting)
implementation fails with set!/setq because there is no variable to
assign a value to.

Python variables are named memory slots you can peek and poke into. Not
so in a purely functional language where variables are physically
removed from the code before the code is executed.

Example:

def redouble(x):
return x + x

redouble(17 + 4)

If Python were simply binding variables in the purely functional sense,
the interpreter would first evaluate 17 + 4 and then make a copy of the
"redouble" function substituting the result (21) for each syntactic
occurrence of x:

def __redouble__234001942():
return 21 + 21

The interpreter would then proceed to evaluate the variable-less
function instance.

If you leave out assignment, there is really no difference between the
two models. Only if you don't have assignment, you don't need to
complicate your computational model with memory. Instead, you deal with
things like continuations.

In fact, even "def" violates the classic functional paradigm. For
example, to calculate 10's factorial without assignments, you can do
this (even in Python):

(lambda n: (lambda fac: fac(fac, n)) \
(lambda fac, n: 1 if n < 2 else n * fac(fac, n - 1)))(10)


Marko

[Rustom Mody]

The foll is fairly standard fare in denotational semantics -- please excuse
the length!

In order to understand (formally) the concept of 'variable'
we need to have at the least a concept of name(or identifier) -> value mapping.
This mapping is called an 'environment'

If we stop at that we get the 'simplest' (at least from a mathematical
pov!!) language -- λ calculus.

However programming languages also need to be implemented -- typically
on von Neumann machines. So we make 'Variable' a composition of two functions:
Env : Identifier -> Location
Store : Location -> Value

This seems fine and dandy until we realize that if the compositon is of
non one-one onto functions all kinds of troubles result; eg
-- Two locations going to one value -- aliasing
-- Store partial at a location -- uninitialized variable
etc etc the most innocuous looking…
-- assignment becomes a higher order function
because it converts a starting id -> -> value mapping to a new mapping


Seeing all these difficulties, some religious zealots (aka functional
programmers) decide that this composite mapping is the root of the problem
-- throw it out -- no aliasing, no assignment, no time. [Minor problem --
How to implement -- remains!]

The non-religious bigots (also called C programmers) see that managing
the Env at one time and the Store at a later time (the two times
are usually called compile-time and run-time but like bell-bottoms these
words are currently unfashionable!) can produce very effective engineering
expedience.

So strictly speaking whenever we have variables we have binding
[Probably mathematica is an exception... dunno for sure]

More loosely and conventionally if the mapping is a single direct one:
Env: Variable -> Value
as in λ calculus, functional languages etc, they are called binding-languages

To distinguish, the 'other' languages which need a
compose of Environment and Store are called variously:

- imperative language
- reference semantics
- conventional (imperative) variable (vs mathematical variable)
etc

IOW in most (normal) languages we have constructs to change the shape
of the environment and we have constructs to change the content of the
environment. The single pre-eminent way for the latter is assignment,
function is the typical way for the former.

[Unfortunately this is not universally true:
In C we have initialized variables that look like assignment but is not.
In python the exception is what Ian calls the default variable trick:
x=x would be a rather asinine assignment. But its not an assignment at all,
it just looks like one]

So no...

> My understanding has always been that an expression of the rhs is bound
> to a name of the lhs. So is my understanding wrong, or is the above
> wrong, or are we talking at cross purposes, or what?

assignment changes the content of the env, functions change the shape
-- which latter you may call binding.

[vasudevram]

Thanks to all those who answered.

- Vasudev

[Rhodri James]

On Sat, 22 Mar 2014 05:26:26 -0000, Rustom Mody <rusto...@gmail.com>
wrote:

> Well almost...
> Except that the 'loop' I am talking of is one of
> def loop():
> return [yield (lambda: x) for x in [1,2,3]]
> or
> return (yield (lambda: x) for x in [1,2,3])
> or just plain ol
> (lambda x: for x in [1,2,3])
> IOW loop is an imperative construct, comprehensions are declarative

I'm sorry, you've made a logical leap too far here. I understand loops
being imperative, but how are comprehensions declarative? What do they
declare that the loop equivalent doesn't.

You've made a great deal of the "for" in a comprehension not having the
same meaning as the "for" in a loop. That may well be true in the
equivalent Haskell constructs (I don't speak or write Haskell), but I
think you are wrong in Python. If so, please stop trying to write Haskell
in Python; you'll be as happy as the friend of mine I've finally persuaded
to stop writing Fortran in Python, I promise!

--
Rhodri James *-* Wildebeest Herder to the Masses

[Ian Kelly]

On Sat, Mar 22, 2014 at 6:32 PM, Rhodri James <rho...@wildebst.org.uk> wrote:
> On Sat, 22 Mar 2014 05:26:26 -0000, Rustom Mody <rusto...@gmail.com>
> wrote:
>
>> Well almost...
>> Except that the 'loop' I am talking of is one of
>> def loop():
>> return [yield (lambda: x) for x in [1,2,3]]
>> or
>> return (yield (lambda: x) for x in [1,2,3])
>> or just plain ol
>> (lambda x: for x in [1,2,3])
>> IOW loop is an imperative construct, comprehensions are declarative
>
>
> I'm sorry, you've made a logical leap too far here. I understand loops
> being imperative, but how are comprehensions declarative? What do they
> declare that the loop equivalent doesn't.

I'm with Rustom on this point. A list comprehension is a syntax for
building a list by declaring a transformation from some other iterable
object. Forget comprehensions for a moment and think of literals.
Would you not consider this to be declarative?

x = [1, 2, 3]

A comprehension is syntactically similar to a literal, with just a
different type of construction in mind.

Where I disagree is on the question of whether Python should therefore
break its established closure rules for lambdas that are nested inside
comprehensions versus functions that are not. It breaks the
equivalence between comprehensions and loops, and to my mind it
introduces significant complexity for relatively little gain.

[Rustom Mody]

On Sunday, March 23, 2014 8:16:28 AM UTC+5:30, Ian wrote:

> On Sat, Mar 22, 2014 at 6:32 PM, Rhodri James wrote:
> > wrote:
> >> Well almost...
> >> Except that the 'loop' I am talking of is one of
> >> def loop():
> >> return [yield (lambda: x) for x in [1,2,3]]
> >> or
> >> return (yield (lambda: x) for x in [1,2,3])
> >> or just plain ol
> >> (lambda x: for x in [1,2,3])
> >> IOW loop is an imperative construct, comprehensions are declarative
> > I'm sorry, you've made a logical leap too far here. I understand loops
> > being imperative, but how are comprehensions declarative? What do they
> > declare that the loop equivalent doesn't.

> I'm with Rustom on this point. A list comprehension is a syntax for
> building a list by declaring a transformation from some other iterable
> object. Forget comprehensions for a moment and think of literals.
> Would you not consider this to be declarative?

> x = [1, 2, 3]

> A comprehension is syntactically similar to a literal, with just a
> different type of construction in mind.

Aha! Very elegantly put!

> Where I disagree is on the question of whether Python should therefore
> break its established closure rules for lambdas that are nested inside
> comprehensions versus functions that are not.

No... see below

> It breaks the
> equivalence between comprehensions and loops, and to my mind it
> introduces significant complexity for relatively little gain.

[I am not completely sure whether the following can be proved/is true]

1. One can change lambda's closure rules which would amount to

"significant complexity for relatively little gain"

2. One can change comprehension rules to not reassign to the
running comprehension running varible but to rebind, using a recursive
function as the simulation of the comprehension rather than a for loop

3. 2 is semantically equivalent to 1
- trivially for normal (ie non-lambda containing) expressions
- and also for lambda containing expressions if your
default-argument trick is implemented by the python compiler
[This is the claim I am not completely sure of and would love to hear
of/if counter examples]

Assuming its true:

One *semantically specifies* a comprehension with a recursive function,
not a for loop

One *implements* a comprehension with the standard use of append
method inside a for as the expansion of a comprehension with the extra
caveat that interior lambdas are automatically wrapped inside a
default-argument binding for all outer comprehension variables.

A vanilla python programmer need not know anything about this any
more than a vanilla C programmer knows about
- strength reduction
- code hoisting
- loop unrolling
etc

that goes on inside an optimizing C compiler

[Ian Kelly]

On Sat, Mar 22, 2014 at 9:16 PM, Rustom Mody <rusto...@gmail.com> wrote:
> [I am not completely sure whether the following can be proved/is true]
>
> 1. One can change lambda's closure rules which would amount to
> "significant complexity for relatively little gain"
>
> 2. One can change comprehension rules to not reassign to the
> running comprehension running varible but to rebind, using a recursive
> function as the simulation of the comprehension rather than a for loop
>
> 3. 2 is semantically equivalent to 1

Well, if you accept that 1) amounts to significant complexity for
relatively little gain, and if 2) is semantically equivalent to 1),
then it follows that 2) also amounts to significant complexity for
relatively little gain. My statement was in regard to the complexity
of the language, not the implementation of the language.

> - trivially for normal (ie non-lambda containing) expressions
> - and also for lambda containing expressions if your
> default-argument trick is implemented by the python compiler
> [This is the claim I am not completely sure of and would love to hear
> of/if counter examples]

The disadvantage of the default argument trick is that it does modify
the function's signature, by adding an extra argument with a default,
so they're not entirely equivalent.

> Assuming its true:
>
> One *semantically specifies* a comprehension with a recursive function,
> not a for loop

The problem with this is a cognitive one. The comprehension *looks
like* a for loop, not a recursive function. It is natural to reason
about it as if it were a for loop, not a recursive function. This is
that added complexity I was talking about.

> A vanilla python programmer need not know anything about this any
> more than a vanilla C programmer knows about
> - strength reduction
> - code hoisting
> - loop unrolling
> etc
>
> that goes on inside an optimizing C compiler

Until they go to unroll their comprehension into a for loop because
they need to add some imperative construct to it, and their code
breaks as a result. From there you'll get the programmers who will
add functions with side effects to their comprehensions because they
want the comprehension binding behavior while still performing
imperative tasks within the loop.

[Rhodri James]

On Sun, 23 Mar 2014 02:46:28 -0000, Ian Kelly <ian.g...@gmail.com>
wrote:

> On Sat, Mar 22, 2014 at 6:32 PM, Rhodri James <rho...@wildebst.org.uk>
> wrote:
>> On Sat, 22 Mar 2014 05:26:26 -0000, Rustom Mody <rusto...@gmail.com>
>> wrote:
>>
>>> Well almost...
>>> Except that the 'loop' I am talking of is one of
>>> def loop():
>>> return [yield (lambda: x) for x in [1,2,3]]
>>> or
>>> return (yield (lambda: x) for x in [1,2,3])
>>> or just plain ol
>>> (lambda x: for x in [1,2,3])
>>> IOW loop is an imperative construct, comprehensions are declarative
>>
>>
>> I'm sorry, you've made a logical leap too far here. I understand loops
>> being imperative, but how are comprehensions declarative? What do they
>> declare that the loop equivalent doesn't.
> I'm with Rustom on this point. A list comprehension is a syntax for
> building a list by declaring a transformation from some other iterable
> object. Forget comprehensions for a moment and think of literals.
> Would you not consider this to be declarative?
>
> x = [1, 2, 3]

I'm not sure I would. I look at that line of code and think of it as
"Create a list...", very much in an imperative manner. Then again,
compared with C structs and typedefs and actual honest-to-God type
declarations, there's precious little in Python I would consider truly
declarative.

[Chris Angelico]

On Mon, Mar 24, 2014 at 1:35 PM, Rhodri James <rho...@wildebst.org.uk> wrote:
>> Would you not consider this to be declarative?
>>
>> x = [1, 2, 3]
>
>
> I'm not sure I would. I look at that line of code and think of it as
> "Create a list...", very much in an imperative manner. Then again, compared
> with C structs and typedefs and actual honest-to-God type declarations,
> there's precious little in Python I would consider truly declarative.

I'm in the declarative group here. Yes, it has to be implemented as
creating a list and adding three elements to it, but conceptually, it
means "Bind x to a new list with these elements". And as long as
that's the end result, I don't care how it's done; the interpreter's
most welcome to have a "template list" that it copies, or maybe a
literal tuple that gets passed to the list() constructor, or
whatever's most efficient.

It gets a bit messier when there's stuff with side effects, though.
This has to be a bit more imperative:

x = [foo(y), bar(y), quux(y)]

That means "Call foo, bar, and quux, in that order, each with y as an
argument, and bind x to a new list with their return values". And if
Python had a simple notation for calling a series of functions, that
could be written something like this:

funcs = [foo, bar, quux]
x = funcs(y)

which is looking more declarative again. It's now "Take this list of
functions and call them all, and bind x to a list of their return
values". (This could be done, with a callable subclass of list. Call
it a sort of "implicit map" if you like.) Python doesn't have that
syntax, but it does have this:

x = [f(y) for f in funcs]

and I'd say that's reasonably declarative; it should be read like the
previous one: "Take this list of funcs, call each one, and bind x to a
list of their return values". And I could imagine a
parallel-processing version of a list comp that functions like
multiprocessing.Pool.map() and doesn't promise order, which would
*definitely* be declarative.

ChrisA

[Chris Angelico]

On Mon, Mar 24, 2014 at 1:35 PM, Rhodri James <rho...@wildebst.org.uk> wrote:
> I'm not sure I would. I look at that line of code and think of it as
> "Create a list...", very much in an imperative manner. Then again, compared
> with C structs and typedefs and actual honest-to-God type declarations,
> there's precious little in Python I would consider truly declarative.

By the way: Python does have a difference between "declarative" and
"imperative".

def f(): # Imperative
global x # Declarative
x += 1 # Imperative

Declaratives control things, imperatives become byte code. Everything
in the byte code is imperative. "LOAD_GLOBAL" means "fetch this global
and put it on the stack". "INPLACE_ADD" means "iadd the top two stack
elements and push the result onto the stack". "STORE_GLOBAL" means
"pop the top stack element and store it in this global". Very very
imperative, and there's none of that created by the "global"
statement. So in that sense, yes, "x = [1, 2, 3]" is imperative; it
loads three constants, builds a list, and stores it. But digging into
the byte code isn't really helpful; it's much more useful to look at
the source code and how the programmer thinks about it.

ChrisA

[Rustom Mody]

On Monday, March 24, 2014 8:57:32 AM UTC+5:30, Chris Angelico wrote:

You have described nicely a slippery slope!

The list
[(1, 1), (1, 2), (1, 3), (1, 4), (2, 1), (2, 2), (2, 3), (2, 4), (3, 1), (3, 2), (3, 3), (3, 4)]

looks neater written (and *thought of* ) as
[(x,y) for x in range(1,4) for y in range(1,5)]

Neat! So I play around... Change it to
[(x,y) for x in range(1,10000) for y in range(1,10000)]
and I dont have an answer but a thrashing machine!! (*)

IOW everything we write/read as programmers has a declarative and an
imperative side.
Which one one wants to focus on requires good sense and taste.

(*) Example also shows inter alia how some things -- range -- have gone from
imperative to declarative from python 2 to 3.
Some people have made languages whose main focus is generalizing this
idea to more dimensions and restrictions:
http://www.irisa.fr/cosi/Rajopadhye/dag-talk.ps

[Chris Angelico]

On Mon, Mar 24, 2014 at 3:14 PM, Rustom Mody <rusto...@gmail.com> wrote:
> Neat! So I play around... Change it to
> [(x,y) for x in range(1,10000) for y in range(1,10000)]
> and I dont have an answer but a thrashing machine!! (*)

Yes, because you used square brackets, which means that the list has
to be fully realized. As you comment, range changed from returning a
list to returning an iterable, and this action is similarly cheap:

>>> ((x,y) for x in range(1,10000) for y in range(1,10000))
<generator object <genexpr> at 0x7f53ed61b360>

You can take a few elements from that cheaply:

>>> [next(_),next(_),next(_)]
[(1, 1), (1, 2), (1, 3)]

If you like thinking in "lazy lists", you can probably think just as
easily with generators; you can't pull up arbitrary elements from it,
or query its length, but for many purposes a generator will do.

ChrisA

[Mark H Harris]

On 3/22/14 4:46 AM, Steven D'Aprano wrote:
> On Fri, 21 Mar 2014 23:51:38 -0500, Mark H Harris wrote:
>
>> Lambda is a problem, if only because it causes confusion. What's the
>> problem? Glad you asked. The constructs DO NOT work the way most people
>> would expect them to, having limited knowledge of python!

One of the best links for understanding what is wrong with lambda is
here, from Guido, (by the way I agree totally with his assessment, there
is no point really in writing it out again):

http://www.artima.com/weblogs/viewpost.jsp?thread=98196

> Why is that a problem? Would you consider it a problem that people who
> don't understand BASIC can't understand BASIC? ("I don't get the

> difference between GOTO and GOSUB. {snip}

>
> The problem is ignorance, not the calculator, and not lambda.

You're argument, based on analogy, is also a red herring, as well a
straw man. Lambda is a problem of confusion for scientists and other
mathematicians (amateur and otherwise) who may be confused because
python's lambda does not do precisely what they might expect from other
functional programming languages, nor does it match precisely with the
lambda calculus. Its also confusing to sophisticated users of all
stripes who may not be aware of "lambda" at all.

> You've told us that "most people" (which people? dentists? lawyers?
> illiterate tribesmen from the Amazon? professional programmers?) are
> surprised by lamba's behaviour, but you haven't actually told us what
> behaviour is surprising, or what "most people" expect lambda to do.
> Perhaps you ought to?

I love your rhetoric, actually, but red herrings all . . .

Here is a link that advocates lambda (map, filter, and reduce) as a
very powerful construct for doing various things: I agree:

http://www.secnetix.de/olli/Python/lambda_functions.hawk

Its a great link, but the important sentence for this discussion is
this unique quote, about "lambda,"

>"This is not exactly the same as lambda in functional
> programming languages, but it is a very powerful concept . . ."

People who understand lambda from functional languages must hassle
with the fact that lambda is different in python, and people who do not
understand functional programming languages (nor the lambda calculus)
are confused by the syntax, also just what it does.

> Python is not a pure functional language, but you can write functional
> code in it. If you want a pure functional language, you know where to
> find one.

Yes, that's obvious; but you're missing the point. Python is not a
functional language, and implying that it can be used as one is
misleading at best (maybe a lie at worst) just because it has a
construct for generating a dynamic anonymous function.

> (I wonder whether, say, Haskell has the people coming along and demanding
> that it should become more like Python?)

Another straw man.

Well, number one, I'm not demanding anything. Number two, everyone
who uses Haskell (for whatever reason) knows well from the start that
its a pure functional programming language. That is the advertisement,
and that is the expectation. No one expects anything different.

>> I can see uses for python's lambda. But, honestly, I think python could
>> deprecate its use and in five years just remove it from the language;
>> along with filter, map, and reduce !
>
> Python could deprecate many things. It would make Python a worse language.

How so? Read Guido's argument above. Another way to answer this
question is that I have been programming with Python for almost a decade
and I've not used lambda. In fact, I have gone out of my way to NOT use
lambda because I am fully aware that the BDFL hates it. Reduce is no
longer in the standard library (although you can import it) and there
are equally good ways to do what lambda was designed for without the
hassle nor confusion.

> By the way, are you aware that lambda is *identical* to def except for
> three things?
>
> - lambda is an expression, not a statement like def;
>
> - the body of a function created with lambda is limited to a
> single expression, not a block;
>
> - the function object created with lambda has a generic name.

Absolutely, thank you for pointing those out ! You forgot that
lambdas can be used in expressions while def functions may not... but
you really have proved my point. There is so little difference between
the two that (for the sake of clarity and reducing confusion) it might
be good to just remove the lambda thing altogether. Just say'in.

> So unless the surprising behaviour about lambda that you're about to tell
> us about relates to one of those three things, I *guarantee* that
> functions created with def have the same surprising behaviour.

No, the surprising behavior is that python lambda does not work
exactly the way that lambda works in functional programming languages
(see the two links provided) the confusion it creates for "normal" users
does not warrant its inclusion in the language.

Having said that, I must admit that I've "played" with lambda
extensively and find it interesting from a comp sci standpoint, but,
again, for normal users I don't think there is a really good value add
for having it in the language.

[Ian Kelly]

On Mar 23, 2014 11:56 PM, "Mark H Harris" <harri...@gmail.com> wrote:
>
> On 3/22/14 4:46 AM, Steven D'Aprano wrote:
>>
>> On Fri, 21 Mar 2014 23:51:38 -0500, Mark H Harris wrote:
>>
>>> Lambda is a problem, if only because it causes confusion. What's the
>>> problem?  Glad you asked. The constructs DO NOT work the way most people
>>> would expect them to, having limited knowledge of python!
>
>
>    One of the best links for understanding what is wrong with lambda is here, from Guido, (by the way I agree totally with his assessment, there is no point really in writing it out again):
>
>   http://www.artima.com/weblogs/viewpost.jsp?thread=98196

That post doesn't point out anything "wrong" with lambda. The argument boils down to: 1) map and filter are not useful because we have comprehensions; 2) reduce is confusing; 3) if we remove those then lambda is not useful either.

> Lambda is a problem of confusion for scientists and other mathematicians (amateur and otherwise) who may be confused because python's lambda does not do precisely what they might expect from other functional programming languages, nor does it match precisely with the lambda calculus. Its also confusing to sophisticated users of all stripes who may not be aware of "lambda" at all.

The difference does not really lie in the lambda construct per se but in the binding style of closures. Functional languages tend to go one way here; imperative languages tend to go the other. Python being primarily an imperative language follows the imperative model. Anonymous functions in Python work the same way in this regard as anonymous functions in Lua or ECMAScript or Go -- those other languages merely avoid the cardinal sin of defining their anonymous functions using the keyword "lambda". The result may be more surprising to users accustomed to functional languages, but I claim that it is *less* surprising to users of other imperative languages.

>> Python is not a pure functional language, but you can write functional
>> code in it. If you want a pure functional language, you know where to
>> find one.
>
>
>    Yes, that's obvious; but you're missing the point. Python is not a functional language, and implying that it can be used as one is misleading at best (maybe a lie at worst) just because it has a construct for generating a dynamic anonymous function.

Oh, give me a break. If you find that you can't write functional code in Python just because closure bindings are slightly inconvenient, then you can't be trying very hard.

>    Well, number one, I'm not demanding anything. Number two, everyone who uses Haskell (for whatever reason) knows well from the start that its a pure functional programming language. That is the advertisement, and that is the expectation. No one expects anything different.

And I would hope that anybody who uses Python is likewise aware from the stay that it *isn't* a purely functional language.

>> Python could deprecate many things. It would make Python a worse language.
>
>
>    How so?  Read Guido's argument above. Another way to answer this question is that I have been programming with Python for almost a decade and I've not used lambda. In fact, I have gone out of my way to NOT use lambda because I am fully aware that the BDFL hates it.

If lambda were going to be deprecated and removed then it already would have happened in Python 3, because Guido tried to do precisely that. I'm not sure what the reasons were for keeping it in the end (according to PEP 3099 it was because nobody suggested a suitable replacement), but if he couldn't get rid of it then, he never will.

[Steven D'Aprano]

On Mon, 24 Mar 2014 00:52:52 -0500, Mark H Harris wrote:

> On 3/22/14 4:46 AM, Steven D'Aprano wrote:
>> On Fri, 21 Mar 2014 23:51:38 -0500, Mark H Harris wrote:
>>
>>> Lambda is a problem, if only because it causes confusion. What's the
>>> problem? Glad you asked. The constructs DO NOT work the way most
>>> people would expect them to, having limited knowledge of python!
>
> One of the best links for understanding what is wrong with lambda is
> here, from Guido, (by the way I agree totally with his assessment, there
> is no point really in writing it out again):
>
> http://www.artima.com/weblogs/viewpost.jsp?thread=98196

It's only one paragraph. Here it is:

Why drop lambda? Most Python users are unfamiliar with Lisp or
Scheme, so the name is confusing; also, there is a widespread
misunderstanding that lambda can do things that a nested function
can't -- I still recall Laura Creighton's Aha!-erlebnis after I
showed her there was no difference! Even with a better name, I think
having the two choices side-by-side just requires programmers to
think about making a choice that's irrelevant for their program; not
having the choice streamlines the thought process. Also, once map(),
filter() and reduce() are gone, there aren't a whole lot of places
where you really need to write very short local functions; Tkinter
callbacks come to mind, but I find that more often than not the
callbacks should be methods of some state-carrying object anyway (the
exception being toy programs).


None of map(), filter() or reduce() are gone. reduce() has been pushed
out into a module, but map() and filter() have actually been enhanced.
Any suggestion that lambda is obsolete flounders on that fact.

There's no doubt that lambda is less-often useful than is the def
statement. But not only is it still useful, but there is a continual
stream of people asking for Python to make it *more useful* by allowing
the body of a lambda to be a full block, not just a single statement.
Guido has stated (although I cannot remember where, so cannot paste a
link) that he accepts that in principle, so long as somebody comes up
with acceptable syntax.

In Ruby, anonymous blocks are used *all the time*. Python is not Ruby,
and never will be, but a lightweight way to generate anonymous functions
in expressions is one difference between a modern, powerful language, and
an archaic or toy under-powered language. I'm too lazy to do this the
right way, but there are at least 99 mentions of "lambda" in the 3.3
standard library:

steve@runes:/usr/local/lib/python3.3$ grep lambda *.py | wc -l
99


It's used in calendar, cgitb, configparser, difflib, decimal, functools,
inspect, os, pydoc and others. Python is a programming language aimed at
programmers, not a toy. It should include tools that programmers use,
like threads, closures, first-class functions. And lambda.

>> Why is that a problem? Would you consider it a problem that people who
>> don't understand BASIC can't understand BASIC? ("I don't get the
>> difference between GOTO and GOSUB. {snip}
>>
>> The problem is ignorance, not the calculator, and not lambda.
>
> You're argument, based on analogy, is also a red herring, as well a
> straw man.

Please stop misusing "straw man". It has a specific meaning:

https://yourlogicalfallacyis.com/strawman

> Lambda is a problem of confusion for scientists and other
> mathematicians (amateur and otherwise) who may be confused because
> python's lambda does not do precisely what they might expect from other
> functional programming languages, nor does it match precisely with the
> lambda calculus.

Yes, you've already said that. And my response remains the same: users of
a language should be expected to learn the language, at least to the
point that they can "get by". One doesn't need ten thousand hours
experience and to be an expert. But nor should the confusion of somebody
who doesn't know the language count for much.

Python's ints don't work exactly the same as ints do in some other
languages, nor do Python's strings, nor is Python's object model
precisely the same as that of some other languages. I think that it would
be a ridiculous idea to discard int, str and the entire object system
just because some people are surprised that Python doesn't behave exactly
like some other language. I don't think lambda is any different.

The same argument applies to *any other language*. Ocaml is not precisely
the same as Haskell, Java not precisely the same as C++, Ruby is not
precisely the same as Javascript. Should Ruby discard their anonymous
blocks because they don't work "precisely" the same as Haskell? Or
perhaps Haskell should throw the towel in because it doesn't behave
"precisely" the same as Ruby?

Of course not -- new languages come into existence precisely so that they
will be different from what is already there. Who is to say that there
aren't people who like Python's lambda *because* it is different from the
anonymous functions in other languages? Or because it is less abstract
than the lambda calculus?

Other languages -- not even Lisp -- don't get to be the sole decider as
to what counts as an anonymous function. Python's lambda may be limited,
but it is a perfectly fine lambda for what it is intended to do.

> Its also confusing to sophisticated users of all
> stripes who may not be aware of "lambda" at all.

If they're not aware of lambda, how are they confused by it?

>> You've told us that "most people" (which people? dentists? lawyers?
>> illiterate tribesmen from the Amazon? professional programmers?) are
>> surprised by lamba's behaviour, but you haven't actually told us what
>> behaviour is surprising, or what "most people" expect lambda to do.
>> Perhaps you ought to?
>
> I love your rhetoric, actually, but red herrings all . . .

No, not really. I don't understand who your intended audience for Python
is. If I were to try to guess, I suspect that you want to dumb Python
down until it is like Dartmouth BASIC circa 1980 except without the line
numbers and with a slightly different syntax. Perhaps I'm wrong, I
certainly hope I'm wrong, but that's the impression I got from your
comments on the python-ideas list.

Python is a programming language. People who program in Python are
programmers. It doesn't matter whether they are school children, or
scientists, or farmhands, once they start writing programs, they are
programming, which makes them a programmer. They might not be paid for
it, but they are still programmers. Who should be using Python, if not
programmers?

> Here is a link that advocates lambda (map, filter, and reduce) as a
> very powerful construct for doing various things: I agree:
>
> http://www.secnetix.de/olli/Python/lambda_functions.hawk
>
> Its a great link, but the important sentence for this discussion is
> this unique quote, about "lambda,"
>
> >"This is not exactly the same as lambda in functional
>> programming languages, but it is a very powerful concept . . ."
>
> People who understand lambda from functional languages must hassle
> with the fact that lambda is different in python, and people who do not
> understand functional programming languages (nor the lambda calculus)
> are confused by the syntax, also just what it does.

You want to remove lambda because it causes confusion. Okay, for the sake
of the argument I will agree that lambda is confusing. Do you know what
else is confusing?

Threading is confusing. So is multiprocessing. So are databases. Unicode
is confusing. First-class functions are confusing. Recursion is
confusing. Closures are confusing. List comprehensions are confusing.
What the hell are trampolines? I'm certainly confused by them.

If we remove lambda because it is confusing, shouldn't we also remove all
these other confusing things? Where shall we stop?

Should we stop only when Python is a confusion-free, utterly useless toy
language? I don't think so. Or should we stop *before* going down that
path? You want to single out lambda. I don't think we should.

>> Python is not a pure functional language, but you can write functional
>> code in it. If you want a pure functional language, you know where to
>> find one.
>
> Yes, that's obvious; but you're missing the point. Python is not a
> functional language, and implying that it can be used as one is
> misleading at best (maybe a lie at worst) just because it has a
> construct for generating a dynamic anonymous function.

I didn't say that Python can be used as a functional language. I said you
can write functional code in Python. And that is so obviously true that I
cannot believe that you are disputing it.

Functions are first-class values. You can pass them around as arguments
to other functions. You can wrap them to make closures, or compose them.
You can return functions from other functions. You can, if you wish,
eschew holding state and write pure functions with no side-effects and no
state. You can even eschew local variables (well, mostly) inside your
functions and do (nearly) everything by composing other functions. You
can write pipelines of iterators, generators or co-routines.

It may be hard to write an *entire* application using nothing but
functional style in Python, but using functional style throughout the
application is so easy and simple that most people don't even realise
that they are writing in a functional style when they do so.

>> (I wonder whether, say, Haskell has the people coming along and
>> demanding that it should become more like Python?)
>
> Another straw man.
>
> Well, number one, I'm not demanding anything. Number two, everyone
> who uses Haskell (for whatever reason) knows well from the start that
> its a pure functional programming language. That is the advertisement,
> and that is the expectation. No one expects anything different.

And everyone who uses Python ought to know that Python is not. That is
the advertisement, and that ought to be the expectation.

Anonymous functions are not the sole preserve of purely functional
languages, any more than objects are the sole preserve of purely OOP
languages.

>>> I can see uses for python's lambda. But, honestly, I think python
>>> could deprecate its use and in five years just remove it from the
>>> language; along with filter, map, and reduce !
>>
>> Python could deprecate many things. It would make Python a worse
>> language.
>
> How so? Read Guido's argument above. Another way to answer this
> question is that I have been programming with Python for almost a decade
> and I've not used lambda. In fact, I have gone out of my way to NOT use
> lambda because I am fully aware that the BDFL hates it.

In other words, you have had situations where you *could have* used
lambda, perhaps even *should have* used lambda, but you intentionally
went *out of your way* (i.e. made the job harder than it needed to be) to
avoid it, just to slavishly ape the BDFL.

Thank you for just proving my point for me. Your code would have been
better, or at least easier, with lambda. Removing it would make the
language worse, not better.

> Reduce is no
> longer in the standard library (although you can import it)

You mean it is no longer in the builtin namespace. It is still in the
standard library.

> and there
> are equally good ways to do what lambda was designed for without the
> hassle nor confusion.

Incorrect. lambda was designed to allow you to create anonymous functions
in an expression, not as a statement. There is *no other way* to do that
in Python. Even if you come up with some weird hack involving exec or
FunctionType and code objects, it won't be *equally good*. It will be a
broken, third-rate imitation of lambda.

>> By the way, are you aware that lambda is *identical* to def except for
>> three things?
>>
>> - lambda is an expression, not a statement like def;
>>
>> - the body of a function created with lambda is limited to a
>> single expression, not a block;
>>
>> - the function object created with lambda has a generic name.
>
> Absolutely, thank you for pointing those out ! You forgot that
> lambdas can be used in expressions while def functions may not...

Umm, read my list again. It's the first item.

> but
> you really have proved my point. There is so little difference between
> the two that (for the sake of clarity and reducing confusion) it might
> be good to just remove the lambda thing altogether. Just say'in.

The *number* of differences may be small, but the *consequences* of those
differences are enormous.

>> So unless the surprising behaviour about lambda that you're about to
>> tell us about relates to one of those three things, I *guarantee* that
>> functions created with def have the same surprising behaviour.
>
> No, the surprising behavior is that python lambda does not work
> exactly the way that lambda works in functional programming languages

Then neither does def.

Would you like to discard def as well? No? Why not?

> (see the two links provided) the confusion it creates for "normal" users
> does not warrant its inclusion in the language.
>
> Having said that, I must admit that I've "played" with lambda
> extensively and find it interesting from a comp sci standpoint, but,
> again, for normal users I don't think there is a really good value add
> for having it in the language.

--
Steven

[Marko Rauhamaa]

Ian Kelly <ian.g...@gmail.com>:

> If lambda were going to be deprecated and removed then it already
> would have happened in Python 3, because Guido tried to do precisely
> that. I'm not sure what the reasons were for keeping it in the end
> (according to PEP 3099 it was because nobody suggested a suitable
> replacement), but if he couldn't get rid of it then, he never will.

You never *need* (Python's) lambda for anything. Inner functions are
more capable and almost always more readable. It doesn't hurt to have
lambda, but I don't find any use for it, either.


Marko

[Mark Lawrence]

On 24/03/2014 05:52, Mark H Harris wrote:
>
> How so? Read Guido's argument above. Another way to answer this
> question is that I have been programming with Python for almost a decade
> and I've not used lambda. In fact, I have gone out of my way to NOT use
> lambda because I am fully aware that the BDFL hates it. Reduce is no
> longer in the standard library (although you can import it) and there
> are equally good ways to do what lambda was designed for without the
> hassle nor confusion.
>

Where do you get reduce from if it's not in the standard library? As
for lambda I've no real interest in it, other than when copying examples
where it's used to (say) provide a key function.

[Chris Angelico]

On Mon, Mar 24, 2014 at 8:49 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> I'm too lazy to do this the
> right way, but there are at least 99 mentions of "lambda" in the 3.3
> standard library:
>
> steve@runes:/usr/local/lib/python3.3$ grep lambda *.py | wc -l
> 99

I'm not too lazy to do it the right way, but I don't have 3.3 handy,
so I've done it on 3.4 instead. There are 77 instances of lambda nodes
in the files you list there - which are the ones that aren't in
packages. (Note that two instances of lambda on the same line would
count as one in Steven's figure, but as two in mine. Also, his counts
comments. Still, his way's a lot easier to calculate, and it's in the
right ball-park.) Including all subdirectories raises that figure to,
get this, 1230. That's actual uses of the lambda keyword as parsed by
Python. This does include the test suite, though. Removing all files
with "/test/" in the names cuts that figure to only 273. But that's
still two hundred and seventy-three places where the Python standard
library uses lambda - a respectable figure.

ChrisA

[Chris Angelico]

They're often not more readable. A lot of people seem to equate
"verbose" with "readable", possibly by faulty extrapolation from
unreadably crunched code with one-letter variables and no line breaks.
But which of these is truly more readable?

squares = []
for n in range(30):
squares.append(n * n)

squares = [n * n for n in range(30)]

Similarly, there are plenty of cases where a nameless function is MUCH
clearer than breaking it out into a separate def and then using the
name once. Do you name the function for what it does internally?

def get_oneth_element_index(item):
return item[1].index
L.sort(key=get_oneth_element_index)

Or for how you're using it?

def keyfunc(item):
return item[1].index
L.sort(key=keyfunc)

Or do you just shortcut the whole thing by inlining it?

L.sort(key=lambda item:item[1].index)

Hey look, that's weakref.py line 941 right there. (It happens to be
the alphabetically last use of lambda in the stdlib. Made a good
example, although it'd be more common to have a space after that
colon.)

ChrisA

[Marko Rauhamaa]

Chris Angelico <ros...@gmail.com>:

> Similarly, there are plenty of cases where a nameless function is MUCH
> clearer than breaking it out into a separate def and then using the
> name once. Do you name the function for what it does internally?
>
> def get_oneth_element_index(item):
> return item[1].index
> L.sort(key=get_oneth_element_index)
>
> Or for how you're using it?
>
> def keyfunc(item):
> return item[1].index
> L.sort(key=keyfunc)
>
> Or do you just shortcut the whole thing by inlining it?
>
> L.sort(key=lambda item:item[1].index)

I still prefer the "def" variant. It even allows you to clarify the
meaning of the tuple slot by using a nicer name.


Marko

[Chris Angelico]

On Mon, Mar 24, 2014 at 11:36 PM, Marko Rauhamaa <ma...@pacujo.net> wrote:
>> def get_oneth_element_index(item):
>> return item[1].index
>> L.sort(key=get_oneth_element_index)
>>

>> Or do you just shortcut the whole thing by inlining it?
>>
>> L.sort(key=lambda item:item[1].index)
>
> I still prefer the "def" variant. It even allows you to clarify the
> meaning of the tuple slot by using a nicer name.

It's the index of element 1. What more do you need to know? Is it
actually any help to give that a name? All you gain is a chance for
the name, the purpose, and the functionality to come into
disagreement.

ChrisA

[Steven D'Aprano]

On Mon, 24 Mar 2014 22:49:38 +1100, Chris Angelico wrote:

> On Mon, Mar 24, 2014 at 8:55 PM, Marko Rauhamaa <ma...@pacujo.net>
> wrote:

>> You never *need* (Python's) lambda for anything. Inner functions are
>> more capable and almost always more readable. It doesn't hurt to have
>> lambda, but I don't find any use for it, either.

Marko has missed an obvious use: lambda is the *only* solution when you
need a function embedded in an expression. You simply cannot do so
otherwise. It is truly astonishing that two people here who are such keen
supporters of functional programming, Marko and Mark Harris, are so
dismissive of lambda, and so forced to write declarative code using def.

> They're often not more readable. A lot of people seem to equate
> "verbose" with "readable", possibly by faulty extrapolation from
> unreadably crunched code with one-letter variables and no line breaks.
> But which of these is truly more readable?
>
> squares = []
> for n in range(30):
> squares.append(n * n)
>
> squares = [n * n for n in range(30)]

Readable for whom?

List comprehension syntax is often completely obscure to beginners. A
beginner would say that the explicit for-loop is more readable.

Actually, a *real* beginner, whose main programming experience before
Python was Pascal, would probably even say that the first example was an
unreadable mess. What's range(30)? What's this ".append" business? What
does [] mean? I know this because I was this beginner, once. The first
few times I tried reading Python code, I couldn't make head or tail of
it. "for" I recognised, because it was the same keyword as Pascal and
Hypertalk use. Pretty much everything else might as well have been
Bulgarian.

This was before the public internet, there was no Google, no tutorials I
could look up. It was only after a colleague convinced me that it would
be a good language to learn that I bought an actual dead tree book and
sat down and learned how to read Python.

I think that Python's reputation of being "executable pseudo-code" is
sometimes harmful. It fools people into thinking that unless Aunt Tilly
can understand a language feature, it's somehow a failure. Hence the
arguments by Mark against lambda.

http://www.catb.org/jargon/html/A/Aunt-Tillie.html

But nobody expects Aunt Tilly to read Scheme or C++ or Go without at
least a bit of learning -- or even Esperanto or French or Latin. You
still need to learn the syntax and the vocabulary, and have some idea of
the basic concepts. The same applies to Python. The learning curve may be
more gentle, but there is still a learning curve. And that is perfectly
fine.

So while I agree with you that, to a moderately fluent Python speaker,
not a beginner but not an expert either, the list comprehension is more
readable, for a beginner (one who has a basic Python vocab but isn't
fluent yet) the for-loop will probably be more readable.

> Similarly, there are plenty of cases where a nameless function is MUCH
> clearer than breaking it out into a separate def and then using the name
> once. Do you name the function for what it does internally?
>
> def get_oneth_element_index(item):
> return item[1].index
> L.sort(key=get_oneth_element_index)
>
> Or for how you're using it?
>
> def keyfunc(item):
> return item[1].index
> L.sort(key=keyfunc)

Why not both?! Don't forget to make it private so some other piece of
code doesn't use it. Or better still, delete it when done!

def _get_oneth_element_index_to_use_as_keyfunc_when_sorting_L(item):
return item[1].index

L.sort(key=_get_oneth_element_index_to_use_as_keyfunc_when_sorting_L)
del _get_oneth_element_index_to_use_as_keyfunc_when_sorting_L

Four lines of code to do what lambda lets you do in one. And people still
insist that lambda is pointless. Maybe they're being paid by the line.

> Or do you just shortcut the whole thing by inlining it?
>
> L.sort(key=lambda item:item[1].index)

Exactly.




--
Steven D'Aprano
http://import-that.dreamwidth.org/

[Mark Lawrence]

Each to their own. Here give me the lambda version any day of the week.

[Steven D'Aprano]

# Magic constants are wicked. Never use a constant without naming it.
ELEMENT_TO_USE_FOR_INDEXING_WHEN_SORTING_L = 1

# <summary>
# key func used when sorting L, returns item's 1th elem index method
# </summary>
# <function_name>
# _get_oneth_element_index_to_use_as_keyfunc_when_sorting_L
# </function_name>
# <author>Steven D'Aprano</author>
# <date_created>2014-03-25</date_created>
# <date_modified>2014-03-25</date_modified>
# <revision>1</revision>
# <param name="item">item to be sorted</param>
# <returns>index method of the oneth element</returns>
# <raises>NameError</raises> # FIXME can this fail any other way?
def _get_oneth_element_index_to_use_as_keyfunc_when_sorting_L(item):
"""Private key function for sorting list L.

Returns the index method of element 1 of the given item.

Example of use:

>>> item = (None, '')
>>> _get_oneth_element_index_to_use_as_keyfunc_when_sorting_L(item)
<built-in method index of str object at ...>

Relies on global constant ELEMENT_TO_USE_FOR_INDEXING_WHEN_SORTING_L.
May raise NameError if that constant is missing.

Warning: do not use this for anything else.
"""
return item[ELEMENT_TO_USE_FOR_INDEXING_WHEN_SORTING_L].index

L.sort(key=_get_oneth_element_index_to_use_as_keyfunc_when_sorting_L)
del _get_oneth_element_index_to_use_as_keyfunc_when_sorting_L
# Better to be safe than sorry.
del ELEMENT_TO_USE_FOR_INDEXING_WHEN_SORTING_L




Definitely being paid by the line :-)

[Mark Lawrence]

One of the finest examples of extracting the urine I've ever read.
Please keep up the good work. Ah but wait, down voted on the grounds
that there are no unit tests and, far more importantly, it doesn't fit
on one line (unless you use gg that is). There is also no proof that
it's been committed to your source control system after going through
its code review.

[Rustom Mody]

Ok so far

> So while I agree with you that, to a moderately fluent Python speaker,
> not a beginner but not an expert either, the list comprehension is more
> readable, for a beginner (one who has a basic Python vocab but isn't
> fluent yet) the for-loop will probably be more readable.

And now I wonder...
The case you are wanting to make and the case you are ending up making
seem to be opposite:

If it is so as you say that for "the beginner (one who has a basic

Python vocab but isn't fluent yet)"

the for-loop is more readable than the for-in-compr
it then suggests that those more experienced taking/helping the
beginners at 0 to the stage of "basic vocab-but-not-yet fluent" are
choosing a sub-optimal route.

Remember that you started by reminding that for an absolute beginner,
everything is 'Bulgarian' not just list comprehensions.

Then how come some things become accessible faster than others?
It may be that those things are actually easier (in some objective sense)
But it may also reveal a prejudice of the teachers.

As for the other argument -- python lambda is BAD -- Ive currently run out of pepper for sprinkling on these flames :-)
I'll just make a few points:

1. Miranda the predecessor of Haskell had no lambda.
Local defs + curried convention + operator sections was considered more than
sufficient for functional programming

2. Lisp was the originator of lambda (in programming languages) and it screwed up the semantics so badly that
a. It had to be corrected at high cost 25 years later -- aka scheme and common lisp
b. Some doyens of functional programming have suggested that lisp is a major setback for the acceptance of functional programming, eg
http://www.cs.kent.ac.uk/people/staff/dat/miranda/wadler87.pdf

3. One basic tenet of λ calculus
foo = λ x . exp
is equivalent to the more traditional
foo(x) = exp

is a little uh-uh in haskell thanks to the notorious monomorphism restriction
http://www.haskell.org/haskellwiki/Monomorphism_restriction

So if FP = λ calculus, Haskell does not quite make it!!

[Ian Kelly]

So what? One might say the same thing about comprehensions -- loops

[Mark H Harris]

On 3/24/14 4:58 AM, Mark Lawrence wrote:
> Where do you get reduce from if it's not in the standard library?

That was "a" proposal for 3000. Its there, but its not on the
built-ins; ie., you have to import it. The confusion: why reduce, why
not filter, nor map? {rhetorical}

> As for lambda I've no real interest in it, other than when copying examples
> where it's used to (say) provide a key function.
>

This is one of my main points to Steven. In my experience "most" people
do not intend to use lambda for anything; they are trying to sort this
or that and don't quite know how to get the key right and some helpful
somebody gives them a key=lambda yadda yadda . They use it, and it
works, but they are scratching their head saying to themselves, "what it
that, how does it work, how can I understand it and on and on".

That is what we mean by confusing. Or another really great example is
this thread. Somebody asks about a language feature and somebody else
helpfully answers the question by providing them with a similar lambda!!

Its the programmer's equivalent of explanation by reference to a more
complicated analogy; which leaves the OP left with, "Thanks for all the
responses".

marcus

PS You are absolutely right, all the expanding double spaces become
very annoying when viewed on Thunderbird; it is exasperating, genuinely.

[Chris Angelico]

On Tue, Mar 25, 2014 at 1:04 AM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
>> But which of these is truly more readable?
>>
>> squares = []
>> for n in range(30):
>> squares.append(n * n)
>>
>> squares = [n * n for n in range(30)]
>
> Readable for whom?
>
> List comprehension syntax is often completely obscure to beginners. A
> beginner would say that the explicit for-loop is more readable.
>
> Actually, a *real* beginner, whose main programming experience before
> Python was Pascal, would probably even say that the first example was an
> unreadable mess. What's range(30)? What's this ".append" business? What
> does [] mean? I know this because I was this beginner, once. The first
> few times I tried reading Python code, I couldn't make head or tail of
> it. "for" I recognised, because it was the same keyword as Pascal and
> Hypertalk use. Pretty much everything else might as well have been
> Bulgarian.

Actually, that's a very good point. Python's for loop is more often
called a foreach loop in other languages, and Python completely lacks
any concept of a "classic" iteration-over-integer for loop. That is a
point of confusion. However, that's going to come up on both branches,
so it's not really a mark against either.

Incidentally, I've often modified my loop counter, in C or REXX or any
other language. About the only situation where I actually miss it in
Python, though, is iterating over a list and mutating the list on the
way through; and even that can often be done in other ways (maybe a
list comp, filtering out some of the elements?). It's amazing how
something can be so utterly fundamental (I mean, come ON! Who can
imagine a language with no equivalent of the basic "do i=1 to 10"
(REXX) or "for (int i=0;i<10;++i)" (C++) loop???) and yet so
dispensable.

ChrisA

[Mark Lawrence]

On 24/03/2014 18:58, Mark H Harris wrote:
> On 3/24/14 4:58 AM, Mark Lawrence wrote:
>> Where do you get reduce from if it's not in the standard library?
>
> That was "a" proposal for 3000. Its there, but its not on the
> built-ins; ie., you have to import it. The confusion: why reduce, why
> not filter, nor map? {rhetorical}

So it is in the standard library then. And I'm not confused, seeing
this must have been decided years ago as Python 3 was released some five
years ago.

>
>> As for lambda I've no real interest in it, other than when copying
>> examples
>> where it's used to (say) provide a key function.
>>
>
> This is one of my main points to Steven. In my experience "most" people
> do not intend to use lambda for anything; they are trying to sort this
> or that and don't quite know how to get the key right and some helpful
> somebody gives them a key=lambda yadda yadda . They use it, and it
> works, but they are scratching their head saying to themselves, "what it
> that, how does it work, how can I understand it and on and on".

More fool them, I write Python as I let it take away the head
scratching, not add to it. If I wanted to start head scratching maybe
I'd go and investigate what line 247 of gcmodule.c does, but funnily
enough I've never been there, and don't intend starting now.

>
> That is what we mean by confusing. Or another really great example is
> this thread. Somebody asks about a language feature and somebody else
> helpfully answers the question by providing them with a similar lambda!!

One of the joys of this list from my POV, YMMV.

>
> Its the programmer's equivalent of explanation by reference to a more
> complicated analogy; which leaves the OP left with, "Thanks for all the
> responses".
>
> marcus
>
> PS You are absolutely right, all the expanding double spaces become
> very annoying when viewed on Thunderbird; it is exasperating, genuinely.
>

Yep, but like I said the situation has improved, partly thanks to the
guys who improved the words on the wiki showing how to successfuly use
gg. Thanks fellas :)

[Ian Kelly]

On Mon, Mar 24, 2014 at 12:58 PM, Mark H Harris <harri...@gmail.com> wrote:
> That is what we mean by confusing. Or another really great example is this
> thread. Somebody asks about a language feature and somebody else helpfully
> answers the question by providing them with a similar lambda!!

That is not in fact how the topic of lambda arose in this thread.
Rustom Mody brought up the binding behavior in a tangent specifically
to complain about it, and that was the first mention of lambda in the
thread.

[Chris Angelico]

On Tue, Mar 25, 2014 at 5:58 AM, Mark H Harris <harri...@gmail.com> wrote:
> Its there, but its not on the built-ins; ie., you have to import it. The
> confusion: why reduce, why not filter, nor map? {rhetorical}

In other languages with those three, and without list/array
comprehensions, I've used filter occasionally and map reasonably
often, but I don't remember the last time I used reduce. Actually,
Pike has special syntax that can take the place of map sometimes, so I
might use filter more often than map in Pike code, because these don't
need explicit map calls:

//Suppose that clients is an array of connected clients on some server
clients->sockets->write("System message: blah blah blah\n");

Indexing an array (the -> is like Python's . as Pike's . is resolved
at compile time) produces an array, effectively mapping the elements
through "lambda x: x->sockets" and ditto for "->write". Calling an
array calls all the non-empty elements in it, with the same
argument(s), and produces an array of return values. (In this case, I
don't care about the return values, which will simply be the number of
bytes written to each socket. If there's a problem, it'll throw an
exception.) Huh. Even with that, and the [*] automap syntax, and such,
I still use map far more often than filter... and filter orders of
magnitude more often than reduce.

Aside: You'll often hear people talking about "map-reduce" with big
data. Python supports that. Look!

>>> map.__reduce__
<method '__reduce__' of 'map' objects>

Oh wait, that's nothing to do with reduce()...

*ducks for cover*

ChrisA

[Ian Kelly]

On Mon, Mar 24, 2014 at 1:12 PM, Chris Angelico <ros...@gmail.com> wrote:
> Incidentally, I've often modified my loop counter, in C or REXX or any
> other language. About the only situation where I actually miss it in
> Python, though, is iterating over a list and mutating the list on the
> way through; and even that can often be done in other ways (maybe a
> list comp, filtering out some of the elements?). It's amazing how
> something can be so utterly fundamental (I mean, come ON! Who can
> imagine a language with no equivalent of the basic "do i=1 to 10"
> (REXX) or "for (int i=0;i<10;++i)" (C++) loop???) and yet so
> dispensable.

I'm not sure "fundamental" is the right word. A for loop is just a
while loop with some syntactic sugar. For that matter, a while loop
is just a structured goto...

[Chris Angelico]

On Tue, Mar 25, 2014 at 6:13 AM, Mark Lawrence <bream...@yahoo.co.uk> wrote:
>> That was "a" proposal for 3000. Its there, but its not on the
>> built-ins; ie., you have to import it. The confusion: why reduce, why
>> not filter, nor map? {rhetorical}
>
>

> So it is in the standard library then. And I'm not confused, seeing this
> must have been decided years ago as Python 3 was released some five years
> ago.

Terminology issue, is all. It's not in the builtins, but it is in the
standard library.

ChrisA

[Mark H Harris]

On 3/24/14 4:49 AM, Steven D'Aprano wrote:
> There's no doubt that lambda is less-often useful than is the def
> statement. But not only is it still useful, but there is a continual
> stream of people asking for Python to make it *more useful* by allowing
> the body of a lambda to be a full block, not just a single statement.
> Guido has stated (although I cannot remember where, so cannot paste a
> link) that he accepts that in principle, so long as somebody comes up
> with acceptable syntax.

Now, this will seem strange to you, but if lambda were expanded to
include an entire block, well now, I would know two things:
1) python has commitment to lambda and
2) lambda would become WAY more useful... ie., the only way to have a
"block expression" embedded in an expression ! I'd vote for that. But
as someone else noted, inner functions work as well (now) and are really
more readable, so ...

> It's used in calendar, cgitb, configparser, difflib, decimal, functools,
> inspect, os, pydoc and others. Python is a programming language aimed at
> programmers, not a toy. It should include tools that programmers use,
> like threads, closures, first-class functions. And lambda.

This is a valid point (its used a couple of hundred times). It could
be replaced in probably of week of constant programming effort. Neither
here nor there, because nobody at this point is demanding it.

>> Lambda is a problem of confusion for scientists and other

>> mathematicians (amateur and otherwise) {snip}
>{snip}

> But nor should the confusion of somebody
> who doesn't know the language count for much.

This is the main disagreement between the two of us on this topic.
It should count for TONS; whole metric TONs. Because someone who
understands python primarily, and after five years of coding (say me)
somebody tells (me) to use lambda in a key clause so that my sort will
work, or work better, or work "their" way; and I have no idea why I need
this nameless function (after I find out that "that" is what it is!)
Its confusing, and please contrast between complicated. I don't mind
complicated, or extensive, or even hard, but when something is
needlessly confusing, then why?

> Python's ints don't work exactly the same as ints do in some other
> languages, nor do Python's strings, nor is Python's object model
> precisely the same as that of some other languages. I think that it would
> be a ridiculous idea to discard int, str and the entire object system
> just because some people are surprised that Python doesn't behave exactly
> like some other language. I don't think lambda is any different.

Your logic is faulty here. Conceptually python's lambda doesn't
work as expected, from contexts that define lambda and where lambda in
functional programming is VERY important. For one thing, why would we
want to "spell out" the word "lambda". ( \x y -> x + y ) a b )
If we're going to use lambda, then use it.

> Other languages -- not even Lisp -- don't get to be the sole decider as
> to what counts as an anonymous function. Python's lambda may be limited,
> but it is a perfectly fine lambda for what it is intended to do.

Is that the same as George W. Bush's "decider" ?

I agree, and so does the link I referenced, for what it was designed
to do, its powerful; esp if one needs to embed a function in an
expression; that is what lambda is for.

>> Its also confusing to sophisticated users of all
>> stripes who may not be aware of "lambda" at all.
>
> If they're not aware of lambda, how are they confused by it?

See above. They are recommended its use (maybe on this list) and
they just don't get it; because its unnecessarily confusing.

> No, not really. I don't understand who your intended audience for Python
> is. If I were to try to guess, I suspect that you want to dumb Python
> down until it is like Dartmouth BASIC circa 1980 except without the line
> numbers and with a slightly different syntax. Perhaps I'm wrong, I
> certainly hope I'm wrong, but that's the impression I got from your
> comments on the python-ideas list.

Actually it was circa 1964. Actually, it was precisely 1964 (I was
there).

No, I'm not advocating for dumb-down simplicity. Aristotle said that
virtue is found at the mean. Either end of the spectrum is a problem
(overly complicated, towards overly simplified) those end-points we must
flee like the black plague. Normal people with reasonable training
(college education) should be able to use python easily without
confusion and without a computer science degree. Programming should be a
liberal art available to everyone in the set of all "normal" educated
people. Some developers (themselves perhaps with degrees Ph.D. in
mathematics or computer science) forget about the fact that "normal"
educated people want to leverage their computer (maybe with python) for
problem solution without having to become a professional computer
scientist. I am advocating for those people.

> You want to remove lambda because it causes confusion. Okay, for the sake
> of the argument I will agree that lambda is confusing. Do you know what
> else is confusing?

I don't want to remove anything at this point. I am only suggesting
that over time the python community might evaluate whether the lambda
(map, filter, reduce) thing is a language benefit. If so, manet in
aeternum. Elif, deprecate it and over time phase it out completely. Its
a community decision of course, and I'm NOT demanding anything.

> Threading is confusing. So is multiprocessing. So are databases. Unicode
> is confusing. First-class functions are confusing. Recursion is
> confusing. Closures are confusing. List comprehensions are confusing.
> What the hell are trampolines? I'm certainly confused by them.

Argument by analogy almost never works. Please don't confuse my word
"confusing" with your interpretation "complicated". Actually, I do not
find any of those above mentioned as "confusing," while I do admit they
are complicated--even extremely complicated. An intelligent person
educated in the liberal arts should be able to sit down over time and
pick-up the concepts in a new area--without--being confused by the
material or presentation. Lambda is confusing NOT because its
complicated (because its not) but because 1) it does not work like other
lambda concepts in functional programming, and 2) because the syntax
does not lend itself easily to immediate interpretation (by normal
people) nor does it lend itself easily to explanation even in the best
effort.

> If we remove lambda because it is confusing, shouldn't we also remove all
> these other confusing things? Where shall we stop?

No. see above.

> Should we stop only when Python is a confusion-free, utterly useless toy
> language? I don't think so. Or should we stop *before* going down that
> path? You want to single out lambda. I don't think we should.

No. see above.

> I didn't say that Python can be used as a functional language. I said you
> can write functional code in Python. And that is so obviously true that I
> cannot believe that you are disputing it.

I'm not disputing that, not in the least. But advertising that
python can be used to write functional code is misleading (particularly
when lambda is included ;map,filter, reduce) because those who KNOW
functional programming are going to be either 1) confused, or more
likely, 2) disappointed. Yes, I can make a nameless function
(expression) and pass it to another function as an argument. True
enough, and powerful enough, but very confusing when the other aspects
of functional programming with lambda do not hold precisely. Ok,
functional with caveats.

>> In fact, I have gone out of my way to NOT use
>> lambda because I am fully aware that the BDFL hates it.
>
> In other words, you have had situations where you *could have* used
> lambda, perhaps even *should have* used lambda, but you intentionally
> went *out of your way* (i.e. made the job harder than it needed to be) to
> avoid it, just to slavishly ape the BDFL.

No, no, no, not at all. I went out-of-my-way to understand how inner
functions could do everything lambda does. I use inner functions, not
lambda, and everything works well; not to ape the BDFL, but because it
does not appear that lambda is a stable concept, and because I want
other people "in the normal" category to be able to support or at least
read my code.

> Thank you for just proving my point for me. Your code would have been
> better, or at least easier, with lambda. Removing it would make the
> language worse, not better.

No. see above.

> lambda was designed to allow you to create anonymous functions
> in an expression, not as a statement. There is *no other way* to do that
> in Python.

Yes, and I agree, and if that were *necessary* we wouldn't be having
this conversation. The point is that there is *no* reason that you must
code your script that way.

> Would you like to discard def as well? No? Why not?

I am not advocating discarding anything. I am suggesting that
lambda is not necessary and might be deprecated without too much
trouble; if the community sees fit. That is all.

Since the name-space and arguably "def" itself is perhaps the most
powerful aspect of python generally (extensibility) than of course the
answer to your question is obvious, and discussing it would be silly.

def is fine, lambda not so much. BUT I'm not demanding anyone
discard it at the moment. Try to remember Aristotle's mean.


marcus

[Chris Angelico]

Of course, and function calls are just stack operations and gotos too.
That's not what makes it fundamental - I'm talking at a source code
level. Can you imagine a high level language without a simple notation
for variable assignment? Certainly not. [1] Variable assignment, name
binding, whatever you call it, is fundamental. Some kind of structured
looping is also pretty critical; you don't see languages that force
you to use bare goto everywhere and call themselves "high level". [2]
Every high level language also needs some way to iterate over numbers.
Most of them provide it as an intrinsic; Python happens to do it as a
foreach over an easily-constructed iterable.

ChrisA

[1] DeScribe Macro Language would borderline-fail this test if I
called it a HLL. It has "SET something TO somevalue". But DML is about
on the level of Python's "dis.dis" output - assembly language for a
byte-code interpreter.
[2] DML passes this test, even. It has a block IF/ELSE/END IF
statement, and a REPEAT/END REPEAT for looping, with the loop
condition being provided by a statement "EXIT WHEN condition" that's
like Python's "if condition: break". Mind you, it also provides
language-level support for message boxes, including prompt boxes (ask
the user to provide a string or integer), so it's a bit of an odd
duck.

[Mark H Harris]

On 3/24/14 4:03 AM, Ian Kelly wrote:
>
> The difference does not really lie in the lambda construct per se but in
> the binding style of closures. Functional languages tend to go one way

> here; imperative languages tend to go the other. {snip}

> The result may be more surprising to users accustomed to functional
> languages, but I claim that it is *less* surprising to users of other
> imperative languages.

Aside from the sin of spelling out "lambda,"
should be ( \x y -> x + y ) a b ) but, neither here nor there...

Yes, its about closures, totally; the most confusing aspect of
lambda in python is not only the syntax but the idea of scope and
closure (for that syntax). Everyone is confused by this initially, not
because its complicated, but because its confusing. An example:

>>>>
>>>> adders= list(range(4))
>>>> adders
> [0, 1, 2, 3]
>>>> for n in adders:
> adders[n]=lambda a: a+n
>
>
>>>> print(adders[1](3))
> 6
>>>>

The expected value as perceived by "normal" people is 4.

This comes up on the list over and again year after year in various
flavors, but always because lambda is unnecessarily confusing where it
comes to how does it function; and by that we mean simply, how does
scope and closure work in this context. Once the "normal" person is
introduced to the scope and closure idiosyncrasies of pythons lambda,
well then everything is much smoother. But how to fix? Consider:

>>>>
>>>> adders= list(range(4))
>>>> adders
> [0, 1, 2, 3]
>>>> for n in adders:
> adders[n] = (lambda b: lambda a: b + a)(n)
>
>
>>>> adders[1](3)
> 4
>>>> adders[2](3)
> 5
>>>>

Now, here is where I talk about confusion; explaining why the first
lambda above does not work because of scope and closure, and then even
worse, explaining why the second "double" lambda works in the lower
example!

Its a nightmare, really.

And you are correct, its really about closure.

marcus

[Marko Rauhamaa]

Mark H Harris <harri...@gmail.com>:

> Yes, its about closures, totally; the most confusing aspect of
> lambda in python is not only the syntax but the idea of scope and
> closure (for that syntax). Everyone is confused by this initially, not
> because its complicated, but because its confusing. An example:
>
>>>>> adders= list(range(4))

>>>>> for n in adders:
>> adders[n]=lambda a: a+n
>>>>> print(adders[1](3))
>> 6
>
> The expected value as perceived by "normal" people is 4.

1. No, I don't think that understanding is automatically natural.

2. It does not concern Python only. For example, what does this scheme
expression yield?

((let ((n 3))
(let ((f (lambda () n)))
(set! n 7)
f)))

Answer: 7

3. It doesn't concern lambda only. For example, rewrite your loop like
this:

for n in range(4):
def add(a):
return a + n
adders[n] = add

adders[1](3)
=> 6


Marko

[Steven D'Aprano]

On Tue, 25 Mar 2014 06:22:28 +1100, Chris Angelico wrote:

> Aside: You'll often hear people talking about "map-reduce" with big
> data. Python supports that. Look!
>
>>>> map.__reduce__
> <method '__reduce__' of 'map' objects>
>
> Oh wait, that's nothing to do with reduce()...
>
> *ducks for cover*

Ha ha, very funny :-P


http://code.activestate.com/recipes/577676-dirt-simple-mapreduce/

[Chris Angelico]

On Tue, Mar 25, 2014 at 8:43 AM, Mark H Harris <harri...@gmail.com> wrote:
>> adders[n] = (lambda b: lambda a: b + a)(n)
>

> Now, here is where I talk about confusion; explaining why the first
> lambda above does not work because of scope and closure, and then even
> worse, explaining why the second "double" lambda works in the lower example!

Easy fix. Use the "explicit capture" notation:

adders[n] = lambda a, n=n: a+n

And there you are, out of your difficulty at once!

ChrisA

[Chris Angelico]

On Tue, Mar 25, 2014 at 9:58 AM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
> On Tue, 25 Mar 2014 06:22:28 +1100, Chris Angelico wrote:
>
>> Aside: You'll often hear people talking about "map-reduce" with big
>> data. Python supports that. Look!
>>
>>>>> map.__reduce__
>> <method '__reduce__' of 'map' objects>
>>
>> Oh wait, that's nothing to do with reduce()...
>>
>> *ducks for cover*
>
> Ha ha, very funny :-P
>
>
> http://code.activestate.com/recipes/577676-dirt-simple-mapreduce/

That looks like a more serious map/reduce example. Mine came from a
double-take when I was looking at help(map) for some reason; there's a
__round__ magic method that helps define the round() function, there's
__abs__ for abs(), there's __str__ for str()... look, there's a
__reduce__ - it must be to help define reduce()! :)

ChrisA

[Mark H Harris]

On 3/24/14 6:01 PM, Chris Angelico wrote:

> Easy fix. Use the "explicit capture" notation:

> adders[n] = lambda a, n=n: a+n

> And there you are, out of your difficulty at once!

Yes, yes, yes, and example:

>>>> adders= list(range(4))
>>>> for n in adders:

> adders[n] = lambda a, n=n: a+n
>
>

>>>> adders[1](3)
>4
>>>> adders[2](3)
>5

But, and this is the big (WHY?) is that necessary? In other words,
its not about experts knowing how to make this work, its about "normal"
people not understanding in the first place why its a problem, and why
the various solutions work to fix it; even though "we" all know that
nothing is 'broken'.

And in reference to Marko's post, he's right, its not just python,
but the issue is not where else is capture and scope a problem for
confusion, the issue is whether python's lambda provides a significant
opportunity for confusion that in the larger scheme of things (no pun
intended) is not warranted.

marcus

[Mark H Harris]

On 3/24/14 5:43 PM, Marko Rauhamaa wrote:
> Mark H Harris<harri...@gmail.com>:

>> Yes, its about closures, totally; the most confusing aspect of
>> lambda in python is not only the syntax but the idea of scope and
>> closure (for that syntax). Everyone is confused by this initially, not
>> because its complicated, but because its confusing. An example:

>>>>>> adders= list(range(4))
>>>>>> for n in adders:
>>> adders[n]=lambda a: a+n
>>>>>> print(adders[1](3))
>>> 6

>> The expected value as perceived by "normal" people is 4.
>
> 1. No, I don't think that understanding is automatically natural.

It might not seem that way for an expert, but if you Google python
lambda function closure(s) you will notice that this is pretty much the
natural way of interpreting things.

Of course the problem is that the closure grabs the *last* number in
the list which is used for each of the adder[] functions created. So, in
other words, three (3) is the number added in each of the adder
functions. But here is the rub, it is not *ever* clear to people (even
experienced python coders, me for instance) that this is how it should
work. What is needed is the explicit closure "grab" recommended by
ChrisA. But for the normal, that is just as bad (conceptually) because
while it works it strays FAR away from expected lambda constructs known
to functional programmers, and it is difficult to explain to non
functional programmers... a proverbial catch 22.

marcus

[Chris Angelico]

On Tue, Mar 25, 2014 at 10:44 AM, Mark H Harris <harri...@gmail.com> wrote:
> On 3/24/14 6:01 PM, Chris Angelico wrote:
>
>> Easy fix. Use the "explicit capture" notation:
>> adders[n] = lambda a, n=n: a+n
>> And there you are, out of your difficulty at once!

> But, and this is the big (WHY?) is that necessary? In other words, its
> not about experts knowing how to make this work, its about "normal" people
> not understanding in the first place why its a problem, and why the various
> solutions work to fix it; even though "we" all know that nothing is
> 'broken'.

Why is it necessary? For the same reason that this works:

def func_pair():
x = 0
def inc():
nonlocal x; x+=1
return x
def dec():
nonlocal x; x-=1
return x
return inc, dec

fooup, foodn = func_pair()
barup, bardn = func_pair()
>>> fooup(), fooup(), fooup(), foodn()
(1, 2, 3, 2)
>>> barup(), barup(), bardn(), bardn()
(1, 2, 1, 0)

When you use the variable x in multiple places, it's the same variable
and it has a single value. If you don't want that, you have to make a
separate variable.

ChrisA

[Ian Kelly]

On Mon, Mar 24, 2014 at 3:43 PM, Mark H Harris <harri...@gmail.com> wrote:
> On 3/24/14 4:03 AM, Ian Kelly wrote:
>>
>>
>> The difference does not really lie in the lambda construct per se but in
>> the binding style of closures. Functional languages tend to go one way
>> here; imperative languages tend to go the other. {snip}
>
>
>> The result may be more surprising to users accustomed to functional
>> languages, but I claim that it is *less* surprising to users of other
>> imperative languages.
>
>
> Aside from the sin of spelling out "lambda,"
> should be ( \x y -> x + y ) a b ) but, neither here nor there...

Well no, it *should* be λx y . x + y but apparently some people don't
have that character on their keyboards, so it gets written as lambda
or \ instead. Personally I dislike the \ style; it doesn't really
resemble a λ that closely, and to me the backslash denotes escape
sequences and set differences. Nor is Python alone in spelling out
lambda: Scheme and Common Lisp spell it the same way. As far as I know
the \ for λ is unique to Haskell.

[Chris Angelico]

On Tue, Mar 25, 2014 at 10:56 AM, Mark H Harris <harri...@gmail.com> wrote:
> What is needed is the explicit closure "grab" recommended by ChrisA.

Which does work. You do know why, right?

ChrisA

[Mark H Harris]

Sure. ... but again, that's not the point. The point is NOT can you
explain why it works, the point is that as a lambda construct it is NOT
clear why it works, and because the construct does not match what lambda
users might expect (naturally) there are *constant* questions about it.

So, again, I'll restate that the community might consider (over time)
whether the confusion created by lambda in python is worth the time and
trouble to maintain the construct in the language. Is the value add
worth the cost of confusion. I don't think so; others are bound to disagree.

marcus

[Chris Angelico]

Pure functional programming, from what I understand, doesn't *have*
variables other than function arguments. So the way to implement "x =
1" is to call a subfunction with an argument of 1, which is referred
to as x. (Am I right so far?) In that case, the default argument trick
is exactly the right way to implement that in Python.

ChrisA

[Mark Lawrence]

I'd vote to have lambda taken out of the language if it meant avoiding
tedious threads like this one :(

[Mark H Harris]

On 3/24/14 7:32 PM, Mark Lawrence wrote:
>> marcus
>
> I'd vote to have lambda taken out of the language if it meant avoiding
> tedious threads like this one :(
>

Dude, you remind me of Eeyore; "days, weeks, months, who knows..."

Its just a conversation. Don't setup a polling booth yet. Its all in
fun and science.

marcus

[Terry Reedy]

On 3/24/2014 7:07 PM, Chris Angelico wrote:
> On Tue, Mar 25, 2014 at 9:58 AM, Steven D'Aprano
> <steve+comp....@pearwood.info> wrote:

>> On Tue, 25 Mar 2014 06:22:28 +1100, Chris Angelico wrote:
>>
>>> Aside: You'll often hear people talking about "map-reduce" with big
>>> data. Python supports that. Look!
>>>
>>>>>> map.__reduce__
>>> <method '__reduce__' of 'map' objects>
>>>
>>> Oh wait, that's nothing to do with reduce()...
>>>
>>> *ducks for cover*
>>
>> Ha ha, very funny :-P
>>
>>
>> http://code.activestate.com/recipes/577676-dirt-simple-mapreduce/
>

> That looks like a more serious map/reduce example. Mine came from a
> double-take when I was looking at help(map) for some reason; there's a
> __round__ magic method that helps define the round() function, there's
> __abs__ for abs(), there's __str__ for str()... look, there's a
> __reduce__ - it must be to help define reduce()! :)

That was my first think also. I believe __pickle__ or __unpickle__ would
have been more appropriate.

--
Terry Jan Reedy

[Terry Reedy]

On 3/24/2014 7:56 PM, Mark H Harris wrote:

> Of course the problem is that the closure

A function is not a closure unless defined within another function. In
the examples I remember, there was no nesting.

> grabs the *last* number in
> the list which is used for each of the adder[] functions created.

Wrong. Functions look up global and nonlocal names, such as n, when the
function is called.

>>> adders = list(range(4))

>>> for n in adders: adders[n] = lambda a: a+n

>>> n = 1000
>>> adders[1](3)
1003

Same result if the function *is* a closure, making n nonlocal rather
than global.

def f():
adders = list(range(4))

for n in adders: adders[n] = lambda a: a+n

n = 1000
return adders

print(f()[1](3))
>>>
1003

The only definition time grabbing is with default arg expressions.

This discussion is a bit funny in a way. Some people are puzzled that
default arg expressions 'grab' just once, as definition time, rather
than with each call. Others (I hope others) are puzzled that body
expressions 'grab' with each call, rather than just once, at definition
time. That seems to be particularly true when the body is in a lambda
expression rather than a def statement.

--
Terry Jan Reedy

[Terry Reedy]

On 3/24/2014 8:28 PM, Chris Angelico wrote:

> Pure functional programming, from what I understand, doesn't *have*
> variables other than function arguments.

function *parameters*, if in the 'variable = name' camp

> So the way to implement "x = 1" is to call a subfunction

with a parameter named 'x'

> with an argument of 1

Funny, I was just thinking about that last night. I really learned it
when trying to translate python code to the scheme dialect racket.

import math as m

s = m.sqrt(2)
a = m.sin(s) + m.cos(s)
del s # to be exactly equivalent to the below, but not needed

b = (lambda x: m.sin(x) + m.cos(x))(m.sqrt(2))

print(a,b)
>>>
1.14370964075811 1.14370964075811


--
Terry Jan Reedy

[Chris Angelico]

On Tue, Mar 25, 2014 at 12:31 PM, Terry Reedy <tjr...@udel.edu> wrote:
> On 3/24/2014 8:28 PM, Chris Angelico wrote:
>
>> Pure functional programming, from what I understand, doesn't *have*
>> variables other than function arguments.
>
>
> function *parameters*, if in the 'variable = name' camp
>
>
>> So the way to implement "x = 1" is to call a subfunction
>
>
> with a parameter named 'x'
>
>> with an argument of 1

Ah, yes, parameter. I tend to use the terms interchangeably, but they
do have formal definitions and distinctions. But otherwise, yes;
that's how I was seeing that.

ChrisA

[Steven D'Aprano]

On Mon, 24 Mar 2014 14:47:11 -0500, Mark H Harris wrote:

> On 3/24/14 4:49 AM, Steven D'Aprano wrote:
>> There's no doubt that lambda is less-often useful than is the def
>> statement. But not only is it still useful, but there is a continual
>> stream of people asking for Python to make it *more useful* by allowing
>> the body of a lambda to be a full block, not just a single statement.
>> Guido has stated (although I cannot remember where, so cannot paste a
>> link) that he accepts that in principle, so long as somebody comes up
>> with acceptable syntax.
>
> Now, this will seem strange to you, but if lambda were expanded to
> include an entire block, well now, I would know two things:
> 1) python has commitment to lambda

Python has a commitment to lambda. lambda is not going anywhere. There
was some idea chatter five or six years ago about dropping it from the
language. It didn't get dropped, so now it is as much a part of the
language as for-loops, print, classes and import.

> and 2) lambda would become WAY more useful...

Every one agrees that being able to include a full suite of statements in
an expression form would be useful. Unfortunately, due to the constraint
of Python's syntax, it seems to be impossible. People have looked for 20+
years, and nobody has come up with an acceptable syntax.


[...]

>>> Lambda is a problem of confusion for scientists and other
>>> mathematicians (amateur and otherwise) {snip}
>>{snip}
>> But nor should the confusion of somebody who doesn't know the language
>> count for much.
>
> This is the main disagreement between the two of us on this topic.
> It should count for TONS; whole metric TONs. Because someone who
> understands python primarily, and after five years of coding (say me)
> somebody tells (me) to use lambda in a key clause so that my sort will
> work, or work better, or work "their" way; and I have no idea why I need
> this nameless function (after I find out that "that" is what it is!)

Then you don't understand *sorting*. If you don't understand why you need
a key function, the problem is not the syntax used to create the key
function, but your understanding of sort. Time to remove sorting from
Python, yes? No?

It's not lambda that is giving you trouble. That's just syntax. There is
no conceptual difference between:

data.sort(key=lambda item: *magic goes on here*)

and

def key_func(item):
*magic goes on here*

data.sort(key=key_func)


If the first confuses you, so will the second. If you are capable of
learning what the first means, learning the second is no harder.

> Its confusing, and please contrast between complicated. I don't mind
> complicated, or extensive, or even hard, but when something is
> needlessly confusing, then why?

I don't know why you insist it is confusing. Maybe you're just no good at
seeing the correspondence between

lambda x: expression

and

def func(x):
return expression

>> Python's ints don't work exactly the same as ints do in some other
>> languages, nor do Python's strings, nor is Python's object model
>> precisely the same as that of some other languages. I think that it
>> would be a ridiculous idea to discard int, str and the entire object
>> system just because some people are surprised that Python doesn't
>> behave exactly like some other language. I don't think lambda is any
>> different.
>
> Your logic is faulty here. Conceptually python's lambda doesn't
> work as expected, from contexts that define lambda

Python defines lambda too. Why should Python have to bend over to suit
the definition used by other languages? They don't own the concept of
anonymous functions. Anonymous functions are found in all sorts of
languages, with different semantics and different syntax:

Pike: lambda(typ1 para1, typ2, para2, ...) { ... };
Perl: sub { my ($a, $b) = @_; ... }
Smalltalk: [:a :b| ... ]
Haskell: \a b -> ...
Erlang: fun(a, b) -> ... end
Dylan: method(a, b) ... end method


to mention just a few. Ruby even has at least three ways of spelling
lambda, one of which even uses the word lambda:

{|a, b| ... }
lambda {|a, b| ...}
proc {|a, b| ...}

> and where lambda in functional programming is VERY important.

And? Nobody is saying that they have to stop using the term lambda.
Certainly I'm not. I wouldn't be so rude.

> For one thing, why would we
> want to "spell out" the word "lambda". ( \x y -> x + y ) a b ) If
> we're going to use lambda, then use it.

So you don't like the syntax of Python's lambda. That's your prerogative.
But frankly, one thing I dislike about Haskell is that it is too terse.
Why would you use lambda without spelling out explicitly that you are
doing so?

>> Other languages -- not even Lisp -- don't get to be the sole decider as
>> to what counts as an anonymous function. Python's lambda may be
>> limited, but it is a perfectly fine lambda for what it is intended to
>> do.
>
> Is that the same as George W. Bush's "decider" ?

I have no idea why you are mocking my use of the word "decider". It's a
perfectly legitimate word, a standard term in English for someone who
decides. The Oxford dictionary dates it back to sometime between 1670 and
1699, so it is hardly a neologism or one of Junior Bush's malapropisms.

Let me put it another way. Functional programming languages do not get to
be the sole decision-maker of what counts as a legitimate syntax or
semantics for anonymous functions.

> I agree, and so does the link I referenced, for what it was designed
> to do, its powerful; esp if one needs to embed a function in an
> expression; that is what lambda is for.

Right.

>>> Its also confusing to sophisticated users of all stripes who may not
>>> be aware of "lambda" at all.
>>
>> If they're not aware of lambda, how are they confused by it?
>
> See above. They are recommended its use (maybe on this list) and
> they just don't get it; because its unnecessarily confusing.

Then they are aware of lambda.

Confusing in what way? Is it the keyword used? You don't need to know the
background of the term to learn to use it. If non-English speakers can
memorise that you use "class" to define classes, English speakers can
stop being so precious and memorise that "lambda" is used to define a
function in an expression.

'lambda' is just one of those jargon words, like 'tuple', 'iterate',
'boolean', 'closure', ... just learn it, and you'll be fine. If you want
to investigate the background of the word, go right ahead, but it won't
make you a better programmer to know that "boolean" is named after George
Boole, and I still have no idea what the origin of "tuple" is. (And after
15+ years I still want to spell it "turple".)


[...]

> Programming should be a
> liberal art available to everyone in the set of all "normal" educated
> people.

Why should it be? Programming is no more a liberal art than painting is
an offshoot of calculus.

Most "normal" (normal as defined by whom?) educated people have no
interest, no desire, and no aptitude for programming. Most *programmers*
have no aptitude for the rigor of programming -- the world is full of
lousy programmers churning out their 40 lines of buggy VB/PHP/Java/C code
per day but aren't any good at it:

http://www.yacoset.com/Home/signs-that-you-re-a-bad-programmer
http://haacked.com/archive/2007/02/27/Why_Cant_Programmers._Read.aspx/

Programming is a skill, like writing iambic pentameter. Should liberal
arts courses ban the use of iambic pentameter by poets because some
people find it confusing and can't count syllables or tell the difference
between stressed and unstressed? (I know I can't. My wife despairs that I
am so useless at anything like poetry.)

> Some developers (themselves perhaps with degrees Ph.D. in
> mathematics or computer science) forget about the fact that "normal"
> educated people want to leverage their computer (maybe with python) for
> problem solution without having to become a professional computer
> scientist. I am advocating for those people.

So you say. But I think you are actually being terribly condescending and
elitist at the same time. Consider:

(1) People who just want to get the job done, without learning a bunch of
theory, *won't care* how their sort key function is written. They're
looking for a recipe that they can copy and paste, and whether you write
it like this:

data.sort(key=lambda item: item[1])

or like this:

from operator import itemgetter
data.sort(key=itemgetter(1))

or like this:

def sort_key(item):
return item[1]
data.sort(key=sort_key)


*they won't care*. In fact, they'll probably prefer the first version,
with lambda, because it is fewer lines to copy.

And this is okay. Sometimes you've got more important things to do than
understand how every line of your code works, especially for throw-away
code, you just want to use it. It is very elitist to insist that features
that some people don't understand have to go. Do you understand how
import works? I bet you don't. It's so complicated that there are
probably only one or two core developers who really understand it in all
it's glory. You just use it: "import this", and it just works. lambda is
the same.

Sometimes programming is just a means to an end and the programmer just
wants an answer now.


(2) You're concluding that just because somebody is confused by lambda,
they would be better off if lambda was removed. That's horribly
patronising. Have you considered that maybe they will take that as an
opportunity to *learn something new*? That they'll learn how to use
lambda, they may even investigate the theory behind it?

I am especially aggravated by your attack on lambda because I was that
person. I had never heard of lambda calculus. When I first learned
Python, lambda's syntax troubled me for a while. I had to keep looking it
up, until suddenly the penny dropped that the argument list is exactly
the same as the argument list of def. Replace "def name" with "lambda",
drop the brackets, and use a single expression without "return", and
Joan's your Auntie.

But I learned it, and it got me interested in where the name came from,
and I discovered the lambda calculus and decided that while I have every
admiration for the people who fly that high into the rarefied air of
theory, that's not an area I have much interest in. In any case, I am a
better programmer -- or at least a more educated one -- precisely because
Python didn't dumb down, it didn't patronise me, it offered an
"advanced" (huh! not really) feature that I otherwise would never have
been exposed to.

Yes, Python could remove lambda, and take that opportunity away from
others. Python would be a lesser language for it, and Python programmers
will be lesser programmers for it.

Python programmers are some of most widely read programmers around,
compared to the average C or PHP or VB coder. That's because they are
exposed to so many different concepts in the language. Not everyone cares
to learn about them, and that's okay, but they aren't Python's core
audience.

>> You want to remove lambda because it causes confusion. Okay, for the
>> sake of the argument I will agree that lambda is confusing. Do you know
>> what else is confusing?
>
> I don't want to remove anything at this point. I am only suggesting
> that over time the python community might evaluate whether the lambda
> (map, filter, reduce) thing is a language benefit. If so, manet in
> aeternum. Elif, deprecate it and over time phase it out completely. Its
> a community decision of course, and I'm NOT demanding anything.

This discussion was held five years ago. And lambda is still here.

>> Threading is confusing. So is multiprocessing. So are databases.
>> Unicode is confusing. First-class functions are confusing. Recursion is
>> confusing. Closures are confusing. List comprehensions are confusing.
>> What the hell are trampolines? I'm certainly confused by them.
>
> Argument by analogy almost never works. Please don't confuse my word
> "confusing" with your interpretation "complicated". Actually, I do not
> find any of those above mentioned as "confusing,"

Good for you! I mean it, I'm not being sarcastic. But I *know* people
find all those things confusing, because (1) either I found, or still do,
find them confusing, or (2) because I've seen people confused by them.
When I say "confusing", I mean it. I don't mean complicated.

> while I do admit they
> are complicated--even extremely complicated. An intelligent person
> educated in the liberal arts should be able to sit down over time and
> pick-up the concepts in a new area--without--being confused by the
> material or presentation. Lambda is confusing NOT because its
> complicated (because its not) but because 1) it does not work like other
> lambda concepts in functional programming,

Do you really believe that people educated in the liberal arts are
exposed to the lambda calculus? Most MATHEMATICIANS aren't exposed to
lambda calculus.

> and 2) because the syntax
> does not lend itself easily to immediate interpretation (by normal
> people) nor does it lend itself easily to explanation even in the best
> effort.

I think that's just nonsense. Take a one-line function:

def func(args):
return stuff

Remove the "def func", the brackets, and the "return". Move it all to one
line, and stick the magic word "lambda" at the front:

lambda args: stuff


And you're done. That's all there is to it.

Your insistence that lambda is confusing is awfully condescending. People
are not as dumb as you insist, and they are perfectly capable of learning
lambda without a comp sci degree. Like any technical jargon, there is
vocabulary and meaning to learn, but the concept is no more difficult
than ordinary def functions.

[Mark H Harris]

On 3/22/14 3:59 PM, vasudevram wrote:
>
> Thanks to all those who answered.
>
> - Vasudev

I am wondering if the question was answered?


>>>>
>>>> x = [[1,2],[3,4],[5,6]]
>>>> import ast
>>>> ast.dump(ast.parse('[x for x in x for x in x]'))
>
> "Module(body=
>
> [Expr(value=ListComp(elt=Name(id='x', ctx=Load()),
>
> generators=
>
> [comprehension(target=Name(id='x', ctx=Store()), iter=Name(id='x', ctx=Load()), ifs=[]),
>
> comprehension(target=Name(id='x', ctx=Store()), iter=Name(id='x', ctx=Load()), ifs=[])]))])"
>>>>

This is really, I think, the answer to the OPs question. Knowing how
python is parsing the comprehension of comprehensions is important, yes?

Obviously each x is within a different scope, within a separate
iterator. This seems to unwind from right to left?

I didn't see any ast entries in the thread, so just wondering.

marcus

[Chris Angelico]

On Tue, Mar 25, 2014 at 12:45 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
> Programming is a skill, like writing iambic pentameter. Should liberal
> arts courses ban the use of iambic pentameter by poets because some
> people find it confusing and can't count syllables or tell the difference
> between stressed and unstressed? (I know I can't. My wife despairs that I
> am so useless at anything like poetry.)

Iambic pentameter is hard. I know, I tried writing eight lines of it
for my brother's wedding. (Okay, I was writing *acrostic* iambic
pentameter, putting his wife's new surname down the left edge of the
page, but still, it's pretty restrictive.) It's much more fun, I
reckon, to write paragraphs of text roughly eighty characters across.
Prosaic, I know....

ChrisA

[Mark Lawrence]

On 25/03/2014 01:45, Steven D'Aprano wrote:
>
> (1) People who just want to get the job done, without learning a bunch of
> theory, *won't care* how their sort key function is written. They're
> looking for a recipe that they can copy and paste, and whether you write
> it like this:
>
> data.sort(key=lambda item: item[1])
>
> or like this:
>
> from operator import itemgetter
> data.sort(key=itemgetter(1))
>
> or like this:
>
> def sort_key(item):
> return item[1]
> data.sort(key=sort_key)
>
>
> *they won't care*. In fact, they'll probably prefer the first version,
> with lambda, because it is fewer lines to copy.
>

I'm firmly in this camp, practicality beats purity and all that. I've
used the first and second versions shown above as they happened to be in
the recipes I was aquiring, I wouldn't contemplate the third. That's
just my mindset, which is what I love about Python, by pure luck it fits
me like made to measure clothing.

[Mark H Harris]

On 3/24/14 8:20 PM, Terry Reedy wrote:
> On 3/24/2014 7:56 PM, Mark H Harris wrote:

>> the list which is used for each of the adder[] functions created.
>
> Wrong. Functions look up global and nonlocal names, such as n, when the
> function is called.
>

hi Terry, yeah, I know; this is what's *wrong* all at once.

Functions "look up global and nonlocal names" such as n, is semantics
for "each created function 'grabs' the last number (n) in the list",
well, because that (n) is bound at call-time to (3).

Your version semantically is detailed and correct; my version
semantically is "how it is perceived" by the user, which is also
correct. Again, how the function gets the n (grab or lookup) is mute.
The user is often confused about how this happens. As you have shown,
even experts in this field disagree about how this is described, which
is also my secondary point--- the whole thing is VERY difficult to
explain to "normal" users. It often takes several sessions and goes on
and on, until Mark Lawrence calls it tedious.

marcus

[Rustom Mody]

On Tuesday, March 25, 2014 5:28:11 AM UTC+5:30, Ian wrote:

Yeah: Its 2014 (at least out here)...
About time we started using unicode in earnest dont you think??

Id like to see the following spellings corrected:
lambda to λ
in to ∈
(preferably with the 'in' predicate and the 'in' in 'for' disambiguated)
set([]) to ∅

And some parentheses disambiguation
Internal ambiguity: Is '(...)' a paren? a function? a tuple?
External ambiguity: {} in python vs in set theory

[And now I hand it over to our very dear resident troll to explain the glories
of the FSR]

[Mark H Harris]

On 3/24/14 10:00 PM, Rustom Mody wrote:
> About time we started using unicode in earnest dont you think??
>
> Id like to see the following spellings corrected:
> lambda to λ

great idea!

{snip}

>
> [And now I hand it over to our very dear resident troll to explain the glories
> of the FSR]

Now, that's not at all patronizing, ~is it?


!vdrt

marcus :)

[Chris Angelico]

On Tue, Mar 25, 2014 at 2:00 PM, Rustom Mody <rusto...@gmail.com> wrote:
> Yeah: Its 2014 (at least out here)...
> About time we started using unicode in earnest dont you think??

We do.

> Id like to see the following spellings corrected:
> lambda to λ
> in to ∈
> (preferably with the 'in' predicate and the 'in' in 'for' disambiguated)
> set([]) to ∅

The problems with these is not Unicode or a lack thereof, but keys. I
know how to type "lambda" on any keyboard I reach for; if it's a
full-sized QWERTY variant, I can type it without looking, and if it's
something else then I can peer at the thing and find the appropriate
five letters. (Phone keyboards are notoriously peer-worthy.) How do I
type λ? Do I have to memorize an alt-key sequence? Do I need to keep a
set of "language keywords" in a file somewhere so I can copy and
paste? Does my editor have to provide them?

What is really gained by using the short-hand? It becomes nigh
ungoogleable; yes, you can paste λ into Google and find out that it's
called lambda (and, if Python used that as a keyword, you could type
"λ python" into Google and get to the docs), but how do you figure out
which part of this to search for?

sockets.sort(key=λdata:data[1])

More likely you'd search for "sockets" or "sort" or maybe "key" or
"data", but you wouldn't expect to search for the symbol.

> And some parentheses disambiguation
> Internal ambiguity: Is '(...)' a paren? a function? a tuple?
> External ambiguity: {} in python vs in set theory

I don't know about the difference between {} in set theory and Python,
but the multiple uses of () actually boil down to two:

1) Grouping, which includes tuples; there's a special case whereby
grouping nothing makes a zero-item tuple, but everything else is just
the comma

2) Functions (both definition and call)

Disambiguating them might be of some small value, but since they're
the same in pretty much every language under the sun, it would feel
like syntactic salt: you have to use a different, and hard to type,
form of parenthesis for one (or even both!) of what ought to just be
simple brackets.

And what's the benefit? Shorter code, maybe. A few 2-6 letter
sequences that can become single characters. I can sympathize somewhat
with the set issue (because {} means an empty dict), although set() is
better than set([]); having a short form for that would be of some
advantage. But not if it's hard to type.

ChrisA

[Mark H Harris]

On 3/24/14 10:17 PM, Chris Angelico wrote:
> On Tue, Mar 25, 2014 at 2:00 PM, Rustom Mody<rusto...@gmail.com> wrote:
>> Yeah: Its 2014 (at least out here)...
>> About time we started using unicode in earnest dont you think??
>
> We do.
>
>> Id like to see the following spellings corrected:
>> lambda to λ
>> in to ∈
>> (preferably with the 'in' predicate and the 'in' in 'for' disambiguated)
>> set([]) to ∅
>
> The problems with these is not Unicode or a lack thereof, but keys. I
> know how to type "lambda" on any keyboard I reach for;

Yeah, its a fit. I found pi -> ∏

... and here's apple pi -> ∏

but, rats, can't find \ lambda

[Mark H Harris]

On 3/24/14 10:25 PM, Mark H Harris wrote:
>
> but, rats, can't find \ lambda

Ok, there we go -> λ


and ∈ and ∉ and ∀



no problem.

[Roy Smith]

In article <mailman.8483.1395717...@python.org>,

Chris Angelico <ros...@gmail.com> wrote:

> On Tue, Mar 25, 2014 at 2:00 PM, Rustom Mody <rusto...@gmail.com> wrote:
> > Yeah: Its 2014 (at least out here)...
> > About time we started using unicode in earnest dont you think??
>
> We do.
>
> > Id like to see the following spellings corrected:
> > lambda to λ
> > in to ∈
> > (preferably with the 'in' predicate and the 'in' in 'for' disambiguated)

> > set([]) to ?

>
> The problems with these is not Unicode or a lack thereof, but keys. I
> know how to type "lambda" on any keyboard I reach for; if it's a
> full-sized QWERTY variant, I can type it without looking, and if it's
> something else then I can peer at the thing and find the appropriate
> five letters. (Phone keyboards are notoriously peer-worthy.) How do I
> type λ? Do I have to memorize an alt-key sequence? Do I need to keep a
> set of "language keywords" in a file somewhere so I can copy and
> paste? Does my editor have to provide them?

I started programming on 029 keypunches and ASR-33 teletypes. If you asked me to
type most of the punctuation we take for granted today (not to mention lower case
letters), I would have looked at you as if you had asked me to type something in greek.

Hardware evolves. I assume that future generations of programmers will have input
devices better suited to unicode than the clumsy keyboards we use today.

[Rustom Mody]

On Tuesday, March 25, 2014 8:47:35 AM UTC+5:30, Chris Angelico wrote:

Of all your objections, the 'google-able' one is the most hard-hitting.
Yes its important and I have no answers.

What you are missing is that programmers spend
90% of their time reading code
10% writing code

You may well be in the super-whiz category (not being sarcastic here)
All that will change is upto 70-30. (ecause you rarely make a mistake)
You still have to read oodles of others' code

I find python (as haskell) sweet because they dont force me read
parsing-drudgery like '{}'.
Given that a compiler can parse whitespace (python), or '{}' (C) in microseconds
whereas I take seconds, this is really terrible economics

Increasing the lexical variety of the code is in the same direction
[There is no need implied that overdoing it and becoming APL is good :-) ]

[Rustom Mody]

On Tuesday, March 25, 2014 12:28:16 AM UTC+5:30, Mark H. Harris wrote:
> On 3/24/14 4:58 AM, Mark Lawrence wrote:
> > Where do you get reduce from if it's not in the standard library?

> That was "a" proposal for 3000. Its there, but its not on the
> built-ins; ie., you have to import it. The confusion: why reduce, why
> not filter, nor map? {rhetorical}

> > As for lambda I've no real interest in it, other than when copying examples
> > where it's used to (say) provide a key function.

> This is one of my main points to Steven. In my experience "most" people
> do not intend to use lambda for anything; they are trying to sort this
> or that and don't quite know how to get the key right and some helpful
> somebody gives them a key=lambda yadda yadda . They use it, and it
> works, but they are scratching their head saying to themselves, "what it
> that, how does it work, how can I understand it and on and on".

Your example backfires more than you perhaps realize
1. Most people who-have-not-heard-of (WHNHO) generators dont intend to use
generators. Somebody shows then a couple of uses and they then find them
useful in other contexts
2. Most programmers coming from C where vararg exists but is really
much too painful to use outside of printf, dont want to use default
arguments. Somebody shows them default arguments then they find nifty uses
for the same
3. Most people WHNHO special methods ...
4. Most people WHNHO slices ...

Just sayin...

> That is what we mean by confusing. Or another really great example is
> this thread. Somebody asks about a language feature and somebody else
> helpfully answers the question by providing them with a similar lambda!!

I am not in pro or anti lambda camp.
My grouses are 3, all re comprehensions

1. Technical: I consider the comprehension binding rules wrong -- not lambda
2. Methodological: Experienced people showing comprehensions
as 'nothing more than' a short-form for a for-loop are being less helpful
to the noob than they know.
3. Pedagogical: Comprehensions are hard, for loops are easy.

To convey comprehensions two contrasting perspectives are needed:
1. The for-loop view
2. The set-theory view

Give only one, and misunderstandings and confusions proliferate
Yes 'only one' can be either one. In the haskell world the pedagogical error
tends to be the opposite to the one out here:
Noobs write: [... x in l, y in m,...]
and wonder why the efficiency is vastly different from
[... y in m, x in l, ...]

Comes from emphasising the declarative (set-theory) view too much, the
imperative (for-loop) view too little

[Mark H Harris]

On 3/24/14 8:45 PM, Steven D'Aprano wrote:

> Your insistence that lambda is confusing is awfully condescending. People
> are not as dumb as you insist, and they are perfectly capable of learning
> lambda without a comp sci degree. Like any technical jargon, there is
> vocabulary and meaning to learn, but the concept is no more difficult
> than ordinary def functions.

This is an Ad Hominem. My opinion that lambda is confusing must not be
construed to mean condescension; not coming from my pen.

I do not insist that people are dumb, nor do I insist that people cannot
learn python without a comp sci degree. Pushing those words into my
mouth and then beating me up for saying them is, well, ad hominem.

What I am insisting is that *many* people, as point of fact, are
confused by the python lambda construct particularly when it is embedded
within a for x in and the lambda is supposed to capture the value x
(as in my previous examples). This say nothing of their intelligence and
says nothing about my supposed motive of condescension. (we my judge
actions, but not motives)

I am advocating for understanding, among all python users--- novice and
expert alike. Especially when I find so many experts who want to "know"
(like the OP on this thread) and other experts who (like ecumenical
councils) cannot agree (also noticed on this thread).

I am not seeking over simplification, and I am not seeking to limit the
expert in any way; just advocating for Aristotle's mean, whereat we
find virtue.

marcus

[Chris Angelico]

On Tue, Mar 25, 2014 at 2:29 PM, Roy Smith <r...@panix.com> wrote:
> I started programming on 029 keypunches and ASR-33 teletypes. If you asked me to
> type most of the punctuation we take for granted today (not to mention lower case
> letters), I would have looked at you as if you had asked me to type something in greek.
>
> Hardware evolves. I assume that future generations of programmers will have input
> devices better suited to unicode than the clumsy keyboards we use today.

And there you have a circular problem. Until the bulk of programmers
have access to such keyboards, programming languages shouldn't use
such symbols (because most of their users won't be able to type them);
and until programming languages make use of those symbols, there's
little reason to put them on keyboards.

Supporting both may look tempting, but you effectively create two ways
of spelling the exact same thing; it'd be like C's trigraphs. Do you
know what ??= is, without looking it up? [1]

ChrisA

[1] I will accept "An obfuscation tool" as a correct answer here.

[Rustom Mody]

On Tuesday, March 25, 2014 8:47:35 AM UTC+5:30, Chris Angelico wrote:

In set theory {} makes sets
In python {} makes dictionaries

> 1) Grouping, which includes tuples; there's a special case whereby
> grouping nothing makes a zero-item tuple, but everything else is just
> the comma

> 2) Functions (both definition and call)

> Disambiguating them might be of some small value, but since they're
> the same in pretty much every language under the sun, it would feel

What 'they'?? I dont get: If you are talking of disambiguating function definition and call -- yeah thats overkill

If you are talking of overlap between tuples parentheses and function (call)
well consider
f(x,y) vs f((x,y)) vs (x,y) vs ((x,y))
Paren vs tuples: why do we need to write (x,) not (x)

All this is because () is doing triple-duty

[Chris Angelico]

On Tue, Mar 25, 2014 at 2:43 PM, Rustom Mody <rusto...@gmail.com> wrote:
> What you are missing is that programmers spend
> 90% of their time reading code
> 10% writing code
>
> You may well be in the super-whiz category (not being sarcastic here)
> All that will change is upto 70-30. (ecause you rarely make a mistake)
> You still have to read oodles of others' code

No, I'm not missing that. But the human brain is a tokenizer, just as
Python is. Once you know what a token means, you comprehend it as that
token, and it takes up space in your mind as a single unit. There's
not a lot of readability difference between a one-symbol token and a
one-word token. Also, since the human brain works largely with words,
you're usually going to grok things based on how you would read them
aloud:

x = y + 1

eggs equals why plus one

They take up roughly the same amount of storage space. One of them,
being a more compact notation, lends itself well to a superstructure
of notation; compare:

x += 1

eggs plus-equals one

inc eggs

You can eyeball the first version and read it as the third, which is a
space saving in your brain. But it's not fundamentally different from
the second. So the saving from using a one-letter symbol that's read
"lambda" rather than the actual word "lambda" is extremely minimal.
Unless you can use it in a higher-level construct, which seems
unlikely in Python (maybe it's different in Haskell? Maybe you use
lambda more and actually do have those supernotations?), you won't
really gain anything.

ChrisA

[Mark H Harris]

On 3/24/14 10:51 PM, Chris Angelico wrote:
> Supporting both may look tempting, but you effectively create two ways
> of spelling the exact same thing; it'd be like C's trigraphs. Do you
> know what ??= is,

This was a fit for me, back in the day IBM (system36 & system38). When
we started supporting the C compiler (ha!) and non of our 5250 terminals
could provide the C punctuation we take for granted today--- so we
invented tri-graphs for { and } and others. It was a hoot.

I personally think the answer is extended key maps triggered by meta
keys shift ctrl opt alt command | which call up full alternate mappings
of Greek|Latin|Math|symbols &c which can be chosen by mouse|pointing
device.


The mac calls these keyboard viewer, and character viewer. In that way
the full unicode set can be available from a standard qwerty keyboard
without modifying the hardware right away.

marcus

[Rustom Mody]

I think Roy is right in saying that what looks unrealistic with one
technology looks natural with another (out phones are already handling
speech and handwriting)

And Chris is right in (rephrasing) we may have unicode-happy OSes and
languages. We cant reasonably have unicode-happy keyboards.
[What would a million-key keyboard look like? Lets leave the cost aside...]

Problem is that unicode deals with very different sides:
Universality of the math language
Locality of the zillions of human languages

If any character could be a variable, distinguishing the different things that
look like 'A' would be a nightmare

[Chris Angelico]

On Tue, Mar 25, 2014 at 2:56 PM, Rustom Mody <rusto...@gmail.com

>> I don't know about the difference between {} in set theory and Python,
>> but the multiple uses of () actually boil down to two:
>
> In set theory {} makes sets
> In python {} makes dictionaries

That's a backward-compatibility issue. Braces in Python meant a
dictionary before it acquired a set type (at least, so I learned in
history class - I wasn't using Python back then), so empty braces have
to continue to mean empty dictionary. I sympathize with the confusion,
but short of confusing everyone terribly by changing dicts to use
something other than braces (maybe borrow Pike's mapping notation, ([
])??), I don't really see a solution.

>> 1) Grouping, which includes tuples; there's a special case whereby
>> grouping nothing makes a zero-item tuple, but everything else is just
>> the comma
>
>> 2) Functions (both definition and call)
>
>> Disambiguating them might be of some small value, but since they're
>> the same in pretty much every language under the sun, it would feel
>
> What 'they'?? I dont get: If you are talking of disambiguating function definition and call -- yeah thats overkill

No no. Disambiguating grouping and fuction definition/call. There's no
reason to have definition and call of functions differ.

> If you are talking of overlap between tuples parentheses and function (call)
> well consider
> f(x,y) vs f((x,y)) vs (x,y) vs ((x,y))
> Paren vs tuples: why do we need to write (x,) not (x)
>
> All this is because () is doing triple-duty

Tuples don't use parentheses. You only confuse yourself when you
insist on that. The only case with parens that actually makes a tuple
is the special empty tuple; imagine if that were given a name instead,
like "Empty". That would solve that confusion.

There's another minorly special case, and that's that the trailing
comma is mandatory on a one-item tuple. In all others, it's optional,
but the one-item tuple would be ambiguous otherwise.

>>> len((1,2,3,))
3
>>> len((1,2,))
2
>>> len((1,))
1

(By the way, bringing in another discussion: The comma isn't part of
the element, nor is it exactly a separator, nor is it exactly a
terminator. Just like a newline in a text file.)

So the only ambiguity is between function calls and grouping. Tuples
are just part of grouping, in that you need to disambiguate a
one-tuple-arg function call from a multiple-arg function call - as in
the above len() calls. And that's where I think it would be highly
confusing to mandate something different. Suppose we used $( )$ for
function calls, and ^( )^ for grouping:

x = ^(1 + 2)^ * 3
y = range$(x)$

What have we gained by distinguishing them? Not a lot. If an open
parenthesis immediately follows another token, it's calling that
previous token; if it follows an operator, it's grouping. Seems pretty
simple to me.

(Cue the spate of emails pointing out something I've missed that
breaks that rule, in which case call it a rule of thumb.)

ChrisA

[Chris Angelico]

On Tue, Mar 25, 2014 at 2:59 PM, Mark H Harris <harri...@gmail.com> wrote:
> I personally think the answer is extended key maps triggered by meta keys
> shift ctrl opt alt command | which call up full alternate mappings of
> Greek|Latin|Math|symbols &c which can be chosen by mouse|pointing device.
>
>
> The mac calls these keyboard viewer, and character viewer. In that way the
> full unicode set can be available from a standard qwerty keyboard without
> modifying the hardware right away.

I can get up a character map on any platform fairly easily, and if
not, I can always Google the name of the character I want and copy and
paste from fileformat.info or some other handy site. It's not that
hard. But if I want to say "copyright", it's still quicker for me to
type nine letters than to hunt down U+00A9 © to paste in somewhere.
Even more so with lambda, which is a shorter word and a less common
symbol (having an easy way to type A9 isn't uncommon these days, but
not many give an easy way to type U+03BB). I'm much more comfortable
typing that out.

ChrisA

[Chris Angelico]

On Tue, Mar 25, 2014 at 3:08 PM, Rustom Mody <rusto...@gmail.com> wrote:
> And Chris is right in (rephrasing) we may have unicode-happy OSes and
> languages. We cant reasonably have unicode-happy keyboards.
> [What would a million-key keyboard look like? Lets leave the cost aside...]

Actually, it wouldn't be that bad. Unicode allows for only 1114112
characters (thanks to UTF-16), of which only the first three planes
have any actual characters on them (so, a maximum of about 200K
characters). All you'd need would be a system that organizes them
(using their hex codepoints isn't exactly useful), and you could type
any character with a maximum of, say, 6-8 keystrokes; Huffman coded,
of course, so the average would be 1.5 keystrokes per character
actually used. Add one meta key: Charset. Hold that and press L and
you get (say) λ, U+03BB; Charset+Shift+L is Λ, U+039B. Ctrl+Charset+L
might give you a Cyrillic л (U+043B), and Ctrl+Charset+Shift+L would
then be Л (U+041B), the upper-case version of that.

Emacs users would love it.

ChrisA

[Rustom Mody]

On Tuesday, March 25, 2014 9:59:48 AM UTC+5:30, Chris Angelico wrote:

Its already there -- and even easier
Switch to cyrillic-jis-russian (whatever that is!)
and I get л from k Л from K

Full layout

+----------------------------------------------------------------+
|１ ！|２ ＠|３ ＃|４ ” |５ ：|６ ，|７ ．|８ ＊|９ （|０ ）| − ＿|＝ ＋| ё Ё |
+----------------------------------------------------------------+
| й Й | ц Ц | у У | к К | е Е | н Н | г Г | ш Ш | щ Щ | з З | х Х | ъ Ъ |
+------------------------------------------------------------+
| ф Ф | ы Ы | в В | а А | п П | р Р | о О | л Л | д Д | ж Ж | э Э |＼ ｜|
+-----------------------------------------------------------+
| я Я | ч Ч | с С | м М | и И | т Т | ь Ь | б Б | ю Ю |／ ？|
+-------------------------------------------------+
+-----------------------------+
| space bar |
+-----------------------------+


The catch is in the claim to huffman coding
Huffman coding requires a statistical distribution
Whose shall we take

And now the discussion is political not technical

[Chris Angelico]

On Tue, Mar 25, 2014 at 4:00 PM, Rustom Mody <rusto...@gmail.com> wrote:
> Its already there -- and even easier
> Switch to cyrillic-jis-russian (whatever that is!)
> and I get л from k Л from K

How quickly can you switch, type one letter (to generate one Cyrillic
character), and switch back? If you can do that more quickly than
typing a word, then (for you!) it might be worth using those letters
as symbols.

ChrisA