map/filter/reduce/lambda opinions and background unscientific mini-survey
Tom Anderson
During our current discussion of the fate of functional constructs in
python, someone brought up Guido's bull on the matter:
http://www.artima.com/weblogs/viewpost.jsp?thread=98196
He says he's going to dispose of map, filter, reduce and lambda. He's
going to give us product, any and all, though, which is nice of him.
What really struck me, though, is the last line of the abstract:
"I expect tons of disagreement in the feedback, all from ex-Lisp-or-Scheme
folks. :-)"
I disagree strongly with Guido's proposals, and i am not an ex-Lisp,
-Scheme or -any-other-functional-language programmer; my only other real
language is Java. I wonder if i'm an outlier.
So, if you're a pythonista who loves map and lambda, and disagrees with
Guido, what's your background? Functional or not?
tom
--
Batman always wins
Ivan Van Laningham
Tom Anderson wrote:
>
> Comrades,
>
> "I expect tons of disagreement in the feedback, all from ex-Lisp-or-Scheme
> folks. :-)"
>
> I disagree strongly with Guido's proposals, and i am not an ex-Lisp,
> -Scheme or -any-other-functional-language programmer; my only other real
> language is Java. I wonder if i'm an outlier.
>
> So, if you're a pythonista who loves map and lambda, and disagrees with
> Guido, what's your background? Functional or not?
>
I'm a pythonista who doesn't love them. In fact, even though I've done
more than my fair share of lambda Tkinter programming using lambdas,
I've never been happy with lambda. And I've spent months inside of
Lisp/Emacs Lisp/Scheme, too (I have the world's second largest .emacs
file [my friend Andy Glew has the largest], even though I can't use it
on Windows;-). I find list comprehensions easier to understand than
map, and small named functions or, better yet, class methods, *tons*
easier to read/understand than lambda--there are too many restrictions
you have to remember.
Personally, I find that Lisp & its derivatives put your head in a very
weird place. Even weirder than PostScript/Forth/RPN, when you come
right down to it.
I won't miss them, but since I don't use them now, that doesn't mean a
whole lot.
Metta,
Ivan
----------------------------------------------
Ivan Van Laningham
God N Locomotive Works
http://www.andi-holmes.com/
http://www.foretec.com/python/workshops/1998-11/proceedings.html
Army Signal Corps: Cu Chi, Class of '70
Author: Teach Yourself Python in 24 Hours
François Pinard
> [Tom Anderson]
> > [Guido]
> > "I expect tons of disagreement in the feedback, all from ex-Lisp-or-Scheme
> > folks. :-)"
> > I disagree strongly with Guido's proposals, and i am not an ex-Lisp,
> > -Scheme or -any-other-functional-language programmer; my only other
> > real language is Java. I wonder if i'm an outlier.
> > So, if you're a pythonista who loves map and lambda, and disagrees
> > with Guido, what's your background? Functional or not?
> I'm a pythonista who doesn't love them.
Same here. `lambda' could go away. Yet `map' is sometimes useful...
> And I've spent months inside of Lisp/Emacs Lisp/Scheme [...]
I worked on Lisp / Scheme / Emacs-Lisp for many dozens of years.
Moreover, a few times for unusual machines, I implemented Lisps.
> (I have the world's second largest .emacs file [my friend Andy Glew
> has the largest], even though I can't use it on Windows;-).
You are a shameless lier! :-) It just _cannot_ beat the size of mine, at
least not so long ago when I still was an Emacs user. And despite its
size, my .emacs worked on a lot of systems, Windows included.
> Personally, I find that Lisp & its derivatives put your head in a very
> weird place.
Lisp / Scheme are very OK! Usable for a wide range of applications,
including system' -- with the proper choices, they can be fairly speedy
as well. Yet, for ubiquitous and day-to-day work, Python is nicer! :-)
--
François Pinard http://pinard.progiciels-bpi.ca
iK
same way. While that is great for corporate slaves it is terrible for the
creative programmer.
Python is quickly becoming the visual basic of the 21 century. If you want
to have fun while getting some work done you need to look elsewhere. It's a
shame...
George Sakkis
What do we have here, a perl troll ? Perhaps you need to post elsewhere
"to have fun".
George
mch...@gmail.com
> So, if you're a pythonista who loves map and lambda, and disagrees with
> Guido, what's your background? Functional or not?
I avoid map sometimes, because I find its syntax less readable
than list (and expression) comprehensions. But occasionally it
is the most readable way to do something, and I wouldn't want to lose
it.
Lambda serves a very specific purpose: declaring small, in-place
functions which are no bigger than a single expression. I do this often
enough that I DO want special syntax for it. But I'll admit that I
wish "lambda" were about 5 or 6 characters shorter and didn't have such
an obscure name.
I disagree (and I've mentioned it before) with Guido's plan to remove
these eventually. I'm perfectly satisfied with the alternate plan to
move the functions like map to a module (perhaps named "functional").
(That doesn't help with lambda, though since it requires syntactical
support.)
And my background is definitely NOT functional: I started with Basic,
then learned Pascal well, then _lots_ of other languages (including
Lisp) to an academic level. I've been using Java and Python heavily
now for about 8 or 9 years. I _DO_ however feel quite comfortable
using a functional approach *for certain problems*.
-- Michael Chermside
Devan L
with list comprehensions. reduce is redundant except when multiplying a
series; there's a sum function for a reason. Lambda looks cleaner in
some cases, but you don't gain any functionality.
What really struck me, though, is the last line of the abstract:
"I expect tons of disagreement in the feedback, all from
ex-Lisp-or-Scheme
folks. :-)"
Guido wrote somewhere that the original map, filter, and reduce came
from a lisp hacker who missed them.
Steven Bethard
> I avoid map sometimes, because I find its syntax less readable
> than list (and expression) comprehensions. But occasionally it
> is the most readable way to do something, and I wouldn't want to lose
> it.
I tend to avoid map as much as possible. The only places I'm still
tempted to use map is in cases like:
' '.join(map(str, objects))
But I'm slowly moving towards:
' '.join(str(o) for o in objects)
because it's easier to fix when I realize later that I should have written:
' '.join('x%sy' % o for o in objects)
In general, I don't think I'll really miss any of map, filter, reduce,
etc. My background's a lot of Java and and a little bit of LISP and ML.
I was never a fan of LISP, but I did like ML a lot. However, for
Python, I definitely find list comprehensions and generator expressions
easier to read, especially when I have to read back through code I wrote
a long time ago.
STeVe
Mike Meyer
> Seems like he wants python programmers to solve their problems all in the
> same way. While that is great for corporate slaves it is terrible for the
> creative programmer.
No, he wants Python to be Pythonic. TMTOWTDI is not Pythonic.
> Python is quickly becoming the visual basic of the 21 century. If you want
> to have fun while getting some work done you need to look elsewhere. It's a
> shame...
If you'd rather spend your time figuring out which of multiple ways to
do things is the best for the job at hand than producing code, there's
a language that makes TMTOWTDI a way of life.
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Erik Max Francis
> So, if you're a pythonista who loves map and lambda, and disagrees with
> Guido, what's your background? Functional or not?
I'm familiar with several function languages but haven't used them
extensively; I was primarily a C++ programmer before I found Python. I
definitely use lambda, map, filter, and reduce, and will miss them when
they're gone.
--
Erik Max Francis && m...@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
Heaven ne'er helps the man who will not act.
-- Sophocles
Ron Adam
> So, if you're a pythonista who loves map and lambda, and disagrees with
> Guido, what's your background? Functional or not?
I find map too limiting, so won't miss it. I'm +0 on removing lambda
only because I'm unsure that there's always a better alternative.
So what would be a good example of a lambda that couldn't be replaced?
Cheers,
Ron
BTW... I'm striving to be Pythonic. ;-)
Jp Calderone
>Tom Anderson wrote:
>
>> So, if you're a pythonista who loves map and lambda, and disagrees with
>> Guido, what's your background? Functional or not?
>
>I find map too limiting, so won't miss it. I'm +0 on removing lambda
>only because I'm unsure that there's always a better alternative.
>
>So what would be a good example of a lambda that couldn't be replaced?
lambda can always be replaced. Just like a for loop can always be replaced:
iterator = iter(<iterable>)
while True:
try:
<loop variable> = iterator.next()
except StopIteration:
break
else:
<loop body>
Let's get rid of for, too.
Jp
Sean McIlroy
Tom Anderson wrote:
<snip>
> So, if you're a pythonista who loves map and lambda, and disagrees with
> Guido, what's your background? Functional or not?
glad you asked. personally i don't know lisp (or scheme), but now i've
decided to learn it, because eventually it will no longer be possible
in python to pass functions as arguments or return them as values. the
education sig will have to change its motto to "computer programming
for every C-programmer". until then hangers-on like myself can use
home-grown substitutes for the functional constructs (examples below),
but in my opinion the best thing is to migrate as soon as possible. the
real programmers are squeezing us out. now is the time to abandon
python for an intelligent language (macros! real conditional evaluation
instead of the and/or kluge!)
def LISTCOMP(f,s,g):
reval = []
for x in s:
if g(x):
reval.append(f(x))
return reval
def LAMBDA(arguments,value):
symbols = arguments.split(',')
def reval(*args):
for i in range(len(args)):
locals()[symbols[i]] = args[i]
return eval(value)
return reval
def MAP(f,s):
return LISTCOMP(f,s,LAMBDA('x','True'))
def FILTER(f,s):
return type(s)(LISTCOMP(LAMBDA('x','x'),s,f))
def REDUCE(f,s,t):
if not s: return t
return f(s[0],REDUCE(f,s[1:],t))
Terry Hancock
> I find map too limiting, so won't miss it. I'm +0 on removing lambda
> only because I'm unsure that there's always a better alternative.
Seems like some new idioms would have to be coined, like:
def my_function(a1, a2):
def _(a,b): return a+b
call_a_lib_w_callback(callback=_)
doesn't seem too bad, and defeats the "wasting time thinking of names"
argument.
> So what would be a good example of a lambda that couldn't be replaced?
I suspect the hardest would be building a list of functions. Something
like:
powers = [lambda a, i=i: a**i for i in range(10)]
which you might be able to make like this:
powers = []
for i in range(10):
def _(a,i=i): return a**i
powers.append(_)
which works and is understandable, but a bit less concise.
The main obstacle to the lambda style here is that def statements
are not expressions. I think that's been proposed as an alternative,
too -- make def return a value so you could say:
powers = [def _(a,i=i): return a**i for i in range(10)]
Personally, I think this is understandable, and given that lambda
is to be pulled, a nice substitute (I would say it is easier to read
than the current lambda syntax, and easier for a newbie to
understand).
But it would probably encourage some bad habits, such as:
myfunc = def _(a,b):
print a,b
return a+b
which looks too much like Javascript, to me, where there are
about three different common idioms for defining a
function (IIRC). :-/
--
Terry Hancock ( hancock at anansispaceworks.com )
Anansi Spaceworks http://www.anansispaceworks.com
John Roth
news:Pine.LNX.4.62.05...@urchin.earth.li...
As far as biases is concerned: my background is assembler.
I've never learned Lisp (although I did learn Forth at one time.)
I think I could note that originally there were five of the things,
including apply, which was cleanly replaced by the * and **
notation in function definitions and calls.
Map, filter and reduce are three things one can do with lists
(or in fact more general sequences).
List comprehensions cleanly replace filter. They don't
quite replace map, and they definitely don't replace reduce.
Claiming that sum etc. do the same job is the whimper of
someone who doesn't want to openly disagree with Guido.
Lambda is a horse of a quite different color. There are times
when an inline definition is the best thing one can have for
notational expressiveness. While lambda has a lot of
problems, Guido seems to be resisting replacing it with
something that can actually do the job.
Having to define an external function or class doesn't
always improve expressiveness. Sometimes it reduces
it. To quote Sean O'Lochlain: "Sometimes the best
symbol for a sharp knife is a sharp knife." [1]
John Roth
[1] In one of Randall Garrett's Lord Darcy stories.
Peter Hansen
> personally i don't know lisp (or scheme), but now i've
> decided to learn it, because eventually it will no longer be possible
> in python to pass functions as arguments or return them as values. the
> education sig will have to change its motto to "computer programming
> for every C-programmer".
Huh? Where did that come from? Functions are objects in Python and
I've not heard the least discussion about this being changed, until now.
Sean, what gave you the impression this would change?
-Peter
Paul McGuire
grok "lambda" as a synonym for "expression to be evaluated later". I
then thought it was similar to Smalltalk's functional closures, since
you can define local arguments at the beginning of the block, and then
write the body of the block using those args. But then I saw that it
was only a subset of that capability, since one may only implement an
expression in a lambda, not a full code block.
Even with those limitations, I've found lambda to be a nice compact
form for specifying callbacks. It is especially helpful in pyparsing,
where I define a mechanism for the programmer to specify a parse action
to be performed, which can modify the matched tokens. Here is one that
is very compact:
quotedString.setParseResults( lambda s,loc,toks: toks[0][1:-1] )
Parse actions take 3 arguments: the original string being parsed, the
location of the beginning of the match, and a ParseResults object
containing the matched tokens (ParseResults objects can act as a list,
dict, or object with attributes). The purpose of this parse action is
to remove the opening and closing quotation marks from the matched
quoted string. One of the things I especially like about this
simplicity of parse actions is that there is no need for checking
whether toks is an empty list, or if the first and last characters are
quotation marks before stripping them - quotedStrings call their parse
actions *only* with a single element list, and only with the first
element containing a string with opening and closing quotes. Still, in
anticipation of the demise of lambda, and because this function is
frequently needed, I've added it as a built-in helper function to
pyparsing, called removeQuotes. But lambda is very simple and
immediate for defining such simple transforms, and there is no need to
go track down where a named function definition may be found. (Yes, I
*could* stop in my tracks just prior to this statement and define this
one-line function, but then this interrupts the flow of my grammar
definition.)
It seems to me that lambda's built-in limitation of *only* supporting
expressions, rather than complete code blocks, has led to people
applying their boundless creativity to trying to cram conditional logic
into bewildering and failure-prone boolean expressions, and it reminds
me of some of the C macro coding that I had to sift through about 20
years ago.
Not coincidentally, I think the lambda limitation is the true origin of
may of the requests we read on c.l.py for the proper syntax for
Python's version of the ternary ?: operator as in "how do I write
(x>10? a : b) in Python", which is invariably followed by a post such
as, "don't bother with that, just do "(x>10 and a or b)", which is then
usually followed with, "but watch out in case a evaluates to False..."
So personally, I like lambdas even if I am not found of the keyword
"lambda". Maybe we could replace "lambda" with "@" or "$"?
-- Paul
Sean McIlroy
<snip>
> Sean, what gave you the impression this would change?
just inductive reasoning. i've been wrong before (like anyone who makes
that claim), and i'm a former python enthusiast, so my judgement must
be colored to some extent by bitterness. maybe they have solid reasons
for scrapping the functional constructs. but to me it seems like
they're eliminating them just because they offend the sensibilities of
C-programmers. (i mean those stereotypical C-programmers, baffled by
recursion and the like, who don't want to be reproached with the fact
of their mathematical illiteracy.) if that's the case then list
comprehensions and/or "first class functions" are likely to be the next
target. even if they're not, it's pretty clear that python is leaving
its multiparadigmatic origins behind. "do it our way," the pundits are
effectively saying, "or get out". for my part, i'm getting out.
Chris Smith
Devan> None of them are really indispensible. Map and filter cab
Devan> be replaced with list comprehensions. reduce is redundant
Devan> except when multiplying a series; there's a sum function
Devan> for a reason. Lambda looks cleaner in some cases, but you
Devan> don't gain any functionality.
Devan> What really struck me, though, is the last line of the
Devan> abstract:
Devan> "I expect tons of disagreement in the feedback, all from
Devan> ex-Lisp-or-Scheme folks. :-)"
Devan> Guido wrote somewhere that the original map, filter, and
Devan> reduce came from a lisp hacker who missed them.
My question is, why not move them into, say, a "functional" library,
so that legacy code can be handled via an import, and those heads
preferring to think that way can be satisfied, and those little corner
cases not handled by the newer, sweller syntaxes can still be managed?
IOW, just ripping them out of the core and leaving everyone in the
lurch doesn't seem too pythonic to me.
Best,
Chris
Erik Max Francis
> if that's the case then list
> comprehensions and/or "first class functions" are likely to be the next
> target.
Slippery slope arguments are logical fallacies, you know.
--
Erik Max Francis && m...@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
Can I walk with you / Through your life
-- India Arie
Robert Kern
> My question is, why not move them into, say, a "functional" library,
> so that legacy code can be handled via an import, and those heads
> preferring to think that way can be satisfied, and those little corner
> cases not handled by the newer, sweller syntaxes can still be managed?
> IOW, just ripping them out of the core and leaving everyone in the
> lurch doesn't seem too pythonic to me.
Python 3000 will be breaking more backwards compatibility than
map/reduce/filter. That legacy code won't work anyways.
I predict, though, that one of the first 3rd party modules to come out
for Python 3000 will be such a library.
--
Robert Kern
rk...@ucsd.edu
"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter
Robert Kern
> Peter Hansen wrote:
> <snip>
>
>>Sean, what gave you the impression this would change?
>
> just inductive reasoning. i've been wrong before (like anyone who makes
> that claim), and i'm a former python enthusiast, so my judgement must
> be colored to some extent by bitterness. maybe they have solid reasons
> for scrapping the functional constructs. but to me it seems like
> they're eliminating them just because they offend the sensibilities of
> C-programmers.
This is incorrect.
> (i mean those stereotypical C-programmers, baffled by
> recursion and the like, who don't want to be reproached with the fact
> of their mathematical illiteracy.) if that's the case then list
> comprehensions and/or "first class functions" are likely to be the next
> target.
map and filter are being removed *because of* list comprehensions. Did
you even read Guido's articles about this issue? Your understanding of
why these changes are planned is incorrect; consequently your projection
based on that understanding is not on firm footing.
> even if they're not, it's pretty clear that python is leaving
> its multiparadigmatic origins behind. "do it our way," the pundits are
> effectively saying, "or get out". for my part, i'm getting out.
If that's what you want to do, no one is going to stop you. But please
do it quietly.
Mike Meyer
> Peter Hansen wrote:
> <snip>
>> Sean, what gave you the impression this would change?
> if that's the case then list comprehensions and/or "first class
> functions" are likely to be the next target.
The existence of list comprehensions are the reason that these
functions are going away, so they aren't likely to be next. It's all
part of "There should be one-- and preferably only one --obvious way
to do it."
Personally, I hope they wind up in a "functional" module, so you can add
"from functional import *" to the top of your scripts, and keep doing
exactly what you've been doing.
Ron Adam
> On Friday 01 July 2005 03:36 pm, Ron Adam wrote:
>
>>I find map too limiting, so won't miss it. I'm +0 on removing lambda
>>only because I'm unsure that there's always a better alternative.
>
>
> Seems like some new idioms would have to be coined, like:
>
> def my_function(a1, a2):
> def _(a,b): return a+b
> call_a_lib_w_callback(callback=_)
>
> doesn't seem too bad, and defeats the "wasting time thinking of names"
> argument.
Usually the time is regained later when you need to go back and figure
out what it is that the lambda is doing. Not so easy for beginners.
A hard to understand process that is easy to do, is not easier than an
easy to understand process that is a bit harder to do.
>>So what would be a good example of a lambda that couldn't be replaced?
>
> I suspect the hardest would be building a list of functions. Something
> like:
>
> powers = [lambda a, i=i: a**i for i in range(10)]
>
> which you might be able to make like this:
>
> powers = []
> for i in range(10):
> def _(a,i=i): return a**i
> powers.append(_)
>
> which works and is understandable, but a bit less concise.
This would be a more direct translation I think:
def put_func(i):
def power_of_i(a):
return a**i
return power_of_i
power = [put_func(i) for i in range(10)]
I think it's also clearer what it does. I had to type the lambda
version into the shell to be sure I understood it. I think that's what
we want to avoid.
> The main obstacle to the lambda style here is that def statements
> are not expressions. I think that's been proposed as an alternative,
> too -- make def return a value so you could say:
>
> powers = [def _(a,i=i): return a**i for i in range(10)]
Wouldn't it be:
powers = [(def _(a): return a**i) for i in range(10)]
The parens around the def make it clearer I think.
That would be pretty much just renaming lambda and changing a syntax a
tad. I get the feeling that the continual desire to change the syntax
of lambda is one of the reasons for getting rid of it. I'm not sure any
of the suggestions will fix that. Although I prefer this version over
the current lambda.
Instead of reusing 'def', resurrecting 'let' as a keyword might be an
option.
powers = [ (let a return a**i) for i in range(10) ]
I just now thought this up, but I like it a lot as an alternative syntax
to lambda. :-)
> Personally, I think this is understandable, and given that lambda
> is to be pulled, a nice substitute (I would say it is easier to read
> than the current lambda syntax, and easier for a newbie to
> understand).
>
> But it would probably encourage some bad habits, such as:
>
> myfunc = def _(a,b):
> print a,b
> return a+b
>
> which looks too much like Javascript, to me, where there are
> about three different common idioms for defining a
> function (IIRC). :-/
I don't think it would be used that way... Very often anyway.
Still none of these examples make for good use cases for keeping lambda.
And I think that's whats needed first, then the new syntax can be decided.
Ron
Donn Cave
...
| I disagree strongly with Guido's proposals, and i am not an ex-Lisp,
| -Scheme or -any-other-functional-language programmer; my only other real
| language is Java. I wonder if i'm an outlier.
|
| So, if you're a pythonista who loves map and lambda, and disagrees with
| Guido, what's your background? Functional or not?
Dysfunctional, I reckon.
I think I disagree with the question more than the answer.
First, map and lambda are two different things, and it's reasonable
to approve of one and abhor the other. Especially if you have a
background in a functional language where lambda works like it should.
On the other hand, the list comprehension gimmick that replaces some
of the "higher order functions" is borrowed from Haskell, as you probably
know, so it isn't exactly alien to functional programming. Prelude.hs
defines map: map f xs = [ f x | x <- xs ]
Secondly, if there's anything I detest about the Python development
model, it is the tendency to focus on gimmicks. For 2.X, elimination
of these features would be an atrocity, a gratuitous change that would
break programs - but I don't think anyone who counts has seriously
proposed to do that. With 3.X, we are talking about a different language.
May not ever even get off the ground, but if it does, it's supposed to
be distinctly different, and we need to know a lot more about it before
we can reasonably worry about trivial details like whether map is going
to be there.
I personally think real FP is seriously hot stuff, but I think Python
is a lousy way to do it, with or without map. I suppose there's a
remote possibility that 3.X will change all that. Or more likely,
there will by then be a really attractive FP language, maybe out
of the "links" initiative by Wadler et al.
Donn Cave, do...@drizzle.com
John Roth
news:mailman.1226.1120271...@python.org...
>
> map and filter are being removed *because of* list comprehensions. Did you
> even read Guido's articles about this issue? Your understanding of why
> these changes are planned is incorrect; consequently your projection based
> on that understanding is not on firm footing.
May I most respectfully point out that you've got it backwards.
Part of the justification for list comprehensions was that they could
be used to replace map and filter.
The jihad against the functional constructs has been going on for a
long time, and list comprehensions are only one piece of it.
John Roth
Tom Anderson
> Personally, I find that Lisp & its derivatives put your head in a very
> weird place. Even weirder than PostScript/Forth/RPN, when you come
> right down to it.
+1 QOTW!
tom
--
REMOVE AND DESTROY
Robert Kern
> "Robert Kern" <rk...@ucsd.edu> wrote in message
> news:mailman.1226.1120271...@python.org...
>
>>map and filter are being removed *because of* list comprehensions. Did you
>>even read Guido's articles about this issue? Your understanding of why
>>these changes are planned is incorrect; consequently your projection based
>>on that understanding is not on firm footing.
>
> May I most respectfully point out that you've got it backwards.
> Part of the justification for list comprehensions was that they could
> be used to replace map and filter.
That is essentially what I said. List comprehensions were made to
replace map and filter. Now that they are here, Guido wants to remove
map and filter to keep OOWTDI.
My response was incomplete, not incorrect.
Jamey Cribbs
> So, if you're a pythonista who loves map and lambda, and disagrees with
> Guido, what's your background? Functional or not?
I have no functional language background. Until recently, I had no use
for programming "expression to be evaluated later" or "deferred
expressions" or whatever else they are being called.
Where I came to see the awesomeness of "deferred expressions" was a few
months ago when I started a major rewrite of KirbyBase for Ruby. I
wanted to make the Ruby version of KirbyBase take advantage of the
strengths of the language. Another Ruby programmer, Hal Fulton, was
helping me by constantly pushing me to make KirbyBase more Ruby-ish.
One thing he kept pushing was to be able to specify select querys using
Ruby's "deferred expression" mechanism, code blocks (before anyone
starts yelling, I know that Ruby code blocks are *much* more than just
"deferred expressions"; I'm just using that descriptor here for the sake
of this discussion).
Code blocks allow you to wrap up any Ruby code and pass it to a method
and have it executed within that method. It is more powerful than
lambda, because you can have multiple statements in the code block and
you can do assignment within the code block. This allowed me to rewrite
KirbyBase so that you can do a select like this:
plane_tbl.select { |r| r.country == 'USA' and r.speed > 350 }
Now, this is cool, but you can do this using lambda in Python. Where
Ruby code blocks really shine is that you can also do this:
plane_tbl.update {|r| r.name == 'P-51'}.set {|r|
r.speed = 405
r.range = 1210
}
I have one code block that I pass to the update method which says
"Select all planes with name equal to P-51". Then, I pass a code block
to the set method which assigns new values to the speed and range fields
for those records (i.e. P-51) that were selected in the update method.
This is something you can't do with lambda.
Now, I think I can duplicate the same functionality of Ruby code blocks
by using Python functions, but it is not going to be as pretty.
So, even though lambda is not as powerful as Ruby code blocks, I was
still bummed to read that it is going away, because it is better than
nothing.
Hopefully, Guido will reconsider or, even better, give us something even
more powerful.
Jamey Cribbs
Mike Meyer
> Code blocks allow you to wrap up any Ruby code and pass it to a method
> and have it executed within that method. It is more powerful than
> lambda, because you can have multiple statements in the code block and
> you can do assignment within the code block.
Just FYI, the inability to have statements - even multiple statements
- in a lambda is what people are talking about when they talk about
the limitations of lambda. It's a common thing for someone with a
background that includes a proper lambda to trip over when they first
start programming in Python. It's not that uncommon for newcommers to
trip over it with "print".
Steven D'Aprano
> Lambda serves a very specific purpose: declaring small, in-place
> functions which are no bigger than a single expression. I do this often
> enough that I DO want special syntax for it. But I'll admit that I
> wish "lambda" were about 5 or 6 characters shorter
As in an empty string? :-)
> and didn't have such an obscure name.
Lambda is no more an obscure name than "function", "decorator", "closure",
"class", or "module". The first time you come across it, you don't know
what it means. Then you learn what it means, and then you know.
--
Steve
Steven D'Aprano
> "iK" <I...@sdsfsfd.com> writes:
>
>> Seems like he wants python programmers to solve their problems all in the
>> same way. While that is great for corporate slaves it is terrible for the
>> creative programmer.
>
> No, he wants Python to be Pythonic. TMTOWTDI is not Pythonic.
Too Many T--- Only Way To Do It?
There Might Tangle One Way To Do It?
T--- M--- Two Obvious Ways To Do It?
Nope, sorry, still not getting it.
>> Python is quickly becoming the visual basic of the 21 century. If you want
>> to have fun while getting some work done you need to look elsewhere. It's a
>> shame...
>
> If you'd rather spend your time figuring out which of multiple ways to
> do things is the best for the job at hand than producing code, there's
> a language that makes TMTOWTDI a way of life.
Figuring out which of multiple ways to do things is the best for the job
at hand _is_ part of producing code. There will always be multiple ways to
do the job. For starters, there is the choice, which language should I
use? Top-Down or Bottom-Up design? Test-driven or not? For loop or list
comprehension or generator? Procedural programming or object-oriented or a
mixture of both? Singleton or Borg design pattern? Test your data first or
deal with the exceptions when they happen? And so on.
Only One Obvious Way makes a nice slogan, but it is easy to turn a
flexible language like Python into a straight-jacket where there is Only
One Way To Do It Regards Of Whether It Is The Best For The Job On Hand Or
Not. Not such a short and concise slogan.
Now that Python has list comps, should for loops be removed from the
language? Why did Python bother introducing list comps when there is
nothing they can do that a for loop can't?
Functional programming using map etc does require a slightly different way
of thinking about programming than does procedural programming, just as
object-oriented needs a different way of thinking than spaghetti-coding
using GOTOs. Different ways of thinking about programming should be
encouraged, not discouraged. Even the much-maligned GOTO has its modern
usage case: the Exception.
If map/filter/reduce have to be removed from the built-ins, and I don't
think they should, I'd prefer for them to be moved into a module rather
than dropped altogether. Provided list comps are made as fast as map and
filter, then at the cost of readability they can be replaced by list
comps. But reduce can't be written as a list comp, only as a relatively
complex for loop at a HUGE loss of readability -- and I've never used
Lisp or Scheme in my life. I'm surely not the only one.
--
Steven
Steven D'Aprano
> Sean McIlroy wrote:
>
>> if that's the case then list
>> comprehensions and/or "first class functions" are likely to be the next
>> target.
>
> Slippery slope arguments are logical fallacies, you know.
Not if you are actually standing on a slippery slope. But seriously, no,
they aren't. The slippery slope argument is _not_ "X is happening now, so
Y will happen no matter what we do". That would be a fallacy.
The argument is actually "X is happening now. If X continues to happen
into the future, Y is the logical consequence of that process. If we wish
to avoid Y, we must stop X". And that is not a fallacy in general
(although of course it could be, if there is no causal relationship
between X and Y).
In this particular case, I suspect Sean is wrong. Guido seems to like list
comprehensions. Unless I'm mistaken (not for the first time) I think he
actually introduced them to the language. They won't be going anywhere
anytime soon.
--
Steven
Devan L
someone who doesn't want to openly disagree with Guido.
Could you give an example where sum cannot do the job(besides the
previously mentioned product situation?
Also, map is easily replaced.
map(f1, sequence) == [f1(element) for element in sequence]
Peter Hansen
> On Fri, 01 Jul 2005 13:42:10 -0400, Mike Meyer wrote:
>>No, he wants Python to be Pythonic. TMTOWTDI is not Pythonic.
>
> Too Many T--- Only Way To Do It?
>
> There Might Tangle One Way To Do It?
>
> T--- M--- Two Obvious Ways To Do It?
>
> Nope, sorry, still not getting it.
If you were serious, Google would be a real good friend here, since the
answer is in its first search result... without even having to click on
the link! Heck, it even points you to the web site: http://tmtowtdi.com :-)
-Peter
Steven D'Aprano
> Claiming that sum etc. do the same job is the whimper of
> someone who doesn't want to openly disagree with Guido.
>
> Could you give an example where sum cannot do the job(besides the
> previously mentioned product situation?
There is an infinite number of potential lambdas, and therefore an
infinite number of uses for reduce.
sum only handles a single case, lambda x,y: x+y
product adds a second case: lambda x,y: x*y
So sum and product together cover precisely 2/infinity, or zero percent,
of all possible uses of reduce.
> Also, map is easily replaced.
> map(f1, sequence) == [f1(element) for element in sequence]
Three mental tokens ( map, f1, sequence ) versus seven ( [], f1, element,
for, element, in, sequence ).
Also, map can take any number of sequences:
map(f1, seq1, seq2, seq3, seq4, ...)
--
Steven.
Christopher Subich
> comps. But reduce can't be written as a list comp, only as a relatively
> complex for loop at a HUGE loss of readability -- and I've never used
> Lisp or Scheme in my life. I'm surely not the only one.
See my reply to your other post for a more detailed explanation, but I
don't think that the for-loop solution is much less readable at all, and
the additional complexity involved is simply setting the initial value
and result for the accumulator. The for-loop solution is even more
flexible, because it can include anonymous code blocks and not just
expressions.
One caevat that I just noticed, though -- with the for-solution, you do
need to be careful about whether you're using a generator or list if you
do not set an explicit initial value (and instead use the first value of
'sequence' as the start). The difference is:
_accum = g.next()
for i in g: _accum = stuff(_accum,i)
versus
_accum = g[0]
for i in g[1:]: _accum = stuff(_accum,i)
The difference is because generators don't support subscripts, while
lists don't support .next() iteration. Unless I'm missing something in
the language (entirely possible), this suggests a missing feature for
same-syntax iteration over the two types.
Jp Calderone
>Steven D'Aprano wrote:
>> comps. But reduce can't be written as a list comp, only as a relatively
>> complex for loop at a HUGE loss of readability -- and I've never used
>> Lisp or Scheme in my life. I'm surely not the only one.
>
>See my reply to your other post for a more detailed explanation, but I
>don't think that the for-loop solution is much less readable at all, and
>the additional complexity involved is simply setting the initial value
>and result for the accumulator. The for-loop solution is even more
>flexible, because it can include anonymous code blocks and not just
>expressions.
>
>One caevat that I just noticed, though -- with the for-solution, you do
>need to be careful about whether you're using a generator or list if you
>do not set an explicit initial value (and instead use the first value of
>'sequence' as the start). The difference is:
>_accum = g.next()
>for i in g: _accum = stuff(_accum,i)
>
>versus
>_accum = g[0]
>for i in g[1:]: _accum = stuff(_accum,i)
In either case, you want to write:
i = iter(g)
_accum = i.next()
for elem in i:
_accum = stuff(_accum, elem)
You also want to catch the StopIteration from that explicit .next() call, but that's an unrelated matter.
Jp
Ron Adam
> On Sat, 02 Jul 2005 20:26:31 -0700, Devan L wrote:
>
>
>> Claiming that sum etc. do the same job is the whimper of
>>someone who doesn't want to openly disagree with Guido.
>>
>>Could you give an example where sum cannot do the job(besides the
>>previously mentioned product situation?
>
>
> There is an infinite number of potential lambdas, and therefore an
> infinite number of uses for reduce.
>
>
>
> sum only handles a single case, lambda x,y: x+y
>
> product adds a second case: lambda x,y: x*y
>
> So sum and product together cover precisely 2/infinity, or zero percent,
> of all possible uses of reduce.
But together, sum and product, probably cover about 90% of situations in
which you would use reduce. Getting a total (sum) from a list probably
covers 80% of the situations reduce would be used on it's own. (I can't
think of any real uses of product at the moment. It's late.)
I'm just estimating, but I think that is the gist of adding those two in
exchange for reduce. Not that they will replace all of reduce use
cases, but that sum and product cover most situations and can be
implemented more efficiently than using reduce or a for loop to do the
same thing. The other situations can easily be done using for loops, so
it's really not much of a loss.
Ron
Erik Max Francis
> But together, sum and product, probably cover about 90% of situations in
> which you would use reduce. Getting a total (sum) from a list probably
> covers 80% of the situations reduce would be used on it's own. (I can't
> think of any real uses of product at the moment. It's late.)
It's not uncommon in mathematics to do repeated products. If you're
familiar with the capital Greek letter sigma notation, if it's replaced
with a capital Greek letter pi, then it's an iterated product, rather
than an iterated sum:
http://mathworld.wolfram.com/Product.html
In general, pretty much _any_ operator can be replaced in this symbol to
indicate a repeated operation. Function composition, set unions and
intersections, logical conjunctions and disjunctions, direct sums and
products, the list goes on and on.
> I'm just estimating, but I think that is the gist of adding those two in
> exchange for reduce. Not that they will replace all of reduce use
> cases, but that sum and product cover most situations and can be
> implemented more efficiently than using reduce or a for loop to do the
> same thing. The other situations can easily be done using for loops, so
> it's really not much of a loss.
I really don't understand this reasoning. You essentially grant the
position that reduce has a purpose, but you still seem to approve
removing it. Let's grant your whole point and say that 90% of the use
cases for reduce are covered by sum and product, and the other 10% are
used by eggheads and are of almost no interest to programmers. But it
still serves a purpose, and a useful one. That it's not of immediate
use to anyone is an argument for moving it into a functional module
(something I would have no serious objection to, though I don't see its
necessity), not for removing it altogether! Why would you remove the
functionality that already exists _and is being used_ just because?
What harm does it do, vs. the benefit of leaving it in?
I'm not myself a huge functional programming guy, but I'm certainly not
in favor of the proposal to remove map, filter, reduce, and lambda. For
map and filter, I can at least see the argument, because they truly are
expressible with list comprehensions (which I do use myself, of course).
lambda I also don't buy, but at least there, yes, you can just define
a local function and use it locally, although I think expressively that
it skirts the line between expressivity and verbosity (if you know what
lambda is, straightforward use of lambda is not at all unclear, in fact
it's quite clear). So at least there's something to that, but I don't
follow it the whole way. But removing reduce is just removing
functionality for no other reason, it seems, than spite.
--
Erik Max Francis && m...@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
Who knows whether any of us will be around in 1972?
-- John F. Kennedy
egbert
>
> Also, map is easily replaced.
> map(f1, sequence) == [f1(element) for element in sequence]
>
map(f1,sequence1, sequence2)
especially if the sequences are of unequal length ?
I didn't see it mentioned yet as a candidate for limbo,
but the same question goes for:
zip(sequence1,sequence2)
--
Egbert Bouwman - Keizersgracht 197 II - 1016 DS Amsterdam - 020 6257991
========================================================================
Carl Banks
> "Robert Kern" <rk...@ucsd.edu> wrote in message
> news:mailman.1226.1120271...@python.org...
>
> >
> > map and filter are being removed *because of* list comprehensions. Did you
> > even read Guido's articles about this issue? Your understanding of why
> > these changes are planned is incorrect; consequently your projection based
> > on that understanding is not on firm footing.
>
> May I most respectfully point out that you've got it backwards.
> Part of the justification for list comprehensions was that they could
> be used to replace map and filter.
>
> The jihad against the functional constructs has been going on for a
> long time, and list comprehensions are only one piece of it.
Many people believe that the functional constructs in Python exist to
enhance Python's support of functional programming, but that's wrong.
They exist to enhance support of procedural programming.
In other words, the functional elements were added not because Python
embraced functional programming, but because discreet use of functional
code can make procedural programs simpler and more concise.
Listcomps et al. cannot do everything map, lambda, filter, and reduce
did. Listcomps are inferior for functional programming. But, you see,
functional is not the point. Streamlining procedural programs is the
point, and I'd say listcomps do that far better, and without all the
baroque syntax (from the procedural point of view).
Jihad? I'd say it's mostly just indifference to the functional
programming cause.
--
CARL BANKS
Scott David Daniels
> On Sat, Jul 02, 2005 at 08:26:31PM -0700, Devan L wrote:
>
>>Also, map is easily replaced.
>>map(f1, sequence) == [f1(element) for element in sequence]
>
> How do you replace
> map(f1,sequence1, sequence2)
> especially if the sequences are of unequal length ?
>
> I didn't see it mentioned yet as a candidate for limbo,
> but the same question goes for:
> zip(sequence1,sequence2)
OK, you guys are picking on what reduce "cannot" do.
The first is [f1(*args) for args in itertools.izip(iter1, iter2)]
How to _you_ use map to avoid making all the intermediate structures?
I never saw anything about making zip go away. It is easy to explain.
Now reduce maps to what I was taught to call "foldl."
How do you express "foldr?" How do you express:
_accum = initial()
for elem in iterable:
_accum = func(elem, _accum, expr)
...
If you want functional programming in python, you have at least three
big problems:
1) Python has side effect like mad, so order of evaluation matters.
I'd claim any useful language is like that (I/O to a printer is
kind of hard to do out-of-order), but I'd get sliced to death
by a bunch of bullies wielding Occam's razors.
2) Python's essential function call is not a single-argument
function which might be a tuple, it is a multi-argument
function which is not evaluated in the same way. The natural
duality of a function taking pairs to a function taking an arg
and returning a function taking an arg and returning a result
breaks down in the face of keyword args, and functions that
take an indeterminate number of arguments. Also, because of (1),
there is a big difference between a function taking no args and
its result.
3) Python doesn't have a full set of functional primitives.
Fold-right is one example, K-combinator is another, ....
Why pick on reduce as-is to keep? There is another slippery
slope argument going up the slope adding functional primitives.
--Scott David Daniels
Scott....@Acm.Org
Steven D'Aprano
> egbert wrote:
>> On Sat, Jul 02, 2005 at 08:26:31PM -0700, Devan L wrote:
>>
>>>Also, map is easily replaced.
>>>map(f1, sequence) == [f1(element) for element in sequence]
>>
>> How do you replace
>> map(f1,sequence1, sequence2)
>> especially if the sequences are of unequal length ?
>>
>> I didn't see it mentioned yet as a candidate for limbo,
>> but the same question goes for:
>> zip(sequence1,sequence2)
>
> OK, you guys are picking on what reduce "cannot" do.
> The first is [f1(*args) for args in itertools.izip(iter1, iter2)]
And now we get messier and messier... Compare these two idioms:
"Map function f1 to each pair of items from seq1 and seq2."
"Build a list comprehension by calling function f1 with the unpacked list
that you get from a list built by zipping seq1 and seq2 together in pairs."
Good thing that removing reduce is supposed to make code easier to
understand, right?
> How to _you_ use map to avoid making all the intermediate structures?
I don't understand the question. Presumably the sequences already exist.
That's not the point.
> I never saw anything about making zip go away. It is easy to explain.
I don't find map any less clear than zip.
Except for the arbitrary choice that zip truncates unequal sequences while
map doesn't, zip is completely redundant:
def my_zip(*seqs):
return map(lambda *t: t, *seqs)
Zip is just a special case of map. I find it disturbing that Guido is
happy to fill Python with special case built-ins like sum, zip and
(proposed) product while wanting to cut out more general purpose solutions.
[snip]
> If you want functional programming in python, you have at least
> three big problems:
>
> 1) Python has side effect like mad, so order of evaluation matters.
Not if you *just* use functional operations.
Not that I would ever do that. The point isn't to turn Python into a
purely functional language, but to give Python developers access to
functional tools for when it is appropriate to use them.
> 2) Python's essential function call is not a single-argument
> function which might be a tuple, it is a multi-argument function
> which is not evaluated in the same way.
And I'm sure that makes a difference to the functional programming
purists. But not to me.
> 3) Python doesn't have a full set of functional primitives.
> Fold-right is one example, K-combinator is another, .... Why pick on
> reduce as-is to keep? There is another slippery slope argument
> going up the slope adding functional primitives.
My car isn't amphibious, so I can't go everywhere with it. Should I throw
it away just because I can't drive under water?
No, of course not. Just because Python isn't a purely functional language
doesn't mean that we should reject what functional idioms (like list
comps, and zip, and reduce) it does have.
Personally, I'd like to learn more about about fold-right and
K-combinator, rather than dump reduce and map.
Frankly, I find this entire discussion very surreal. Reduce etc *work*,
right now. They have worked for years. If people don't like them, nobody
is forcing them to use them. Python is being pushed into directions which
are *far* harder to understand than map and reduce (currying, decorators,
etc) and people don't complain about those. And yet something as simple
and basic as map is supposed to give them trouble? These are the same
people who clamoured for zip, which is just a special case of map?
--
Steven.
Christopher Subich
> Listcomps et al. cannot do everything map, lambda, filter, and reduce
> did. Listcomps are inferior for functional programming. But, you see,
> functional is not the point. Streamlining procedural programs is the
> point, and I'd say listcomps do that far better, and without all the
> baroque syntax (from the procedural point of view).
I've heard this said a couple times now -- how can listcomps not
completely replace map and filter?
I'd think that:
mapped = [f(i) for i in seq]
filtered = [i for i in seq if f(i)]
The only map case that doesn't cleanly reduce is for multiple sequences
of different length -- map will extend to the longest one (padding the
others with None), while zip (izip) truncates sequences at the shortest.
This suggests an extension to (i)zip, possibly (i)lzip ['longest zip']
that does None padding in the same way that map does.
Reduce can be rewritten easily (if an initial value is supplied) as a
for loop:
_accum = initial
for j in seq: _accum=f(_accum,j)
result = _accum
(two lines if the result variable can also be used as the accumulator --
this would be undesirable of assigning to that can trigger, say, a
property function call)
Lambdas, I agree, can't be replaced easily, and they're the feature I'd
probably be least happy to see go, even though I haven't used them very
much.
Christopher Subich
> egbert wrote:
>> map(f1,sequence1, sequence2)
>> especially if the sequences are of unequal length ?
>>
>> I didn't see it mentioned yet as a candidate for limbo,
>> but the same question goes for:
>> zip(sequence1,sequence2)
>
> OK, you guys are picking on what reduce "cannot" do.
> The first is [f1(*args) for args in itertools.izip(iter1, iter2)]
> How to _you_ use map to avoid making all the intermediate structures?
Not quite -- zip an izip terminate at the shortest sequence, map extends
the shortest with Nones. This is resolvable by addition of an lzip (and
ilzip) function in Python 2.5 or something.
And egbert's Chicken Littling with the suggestion that 'zip' will be
removed.
Peter Hansen
> Frankly, I find this entire discussion very surreal. Reduce etc *work*,
> right now. They have worked for years. If people don't like them, nobody
> is forcing them to use them. Python is being pushed into directions which
> are *far* harder to understand than map and reduce (currying, decorators,
> etc) and people don't complain about those.
I find it surreal too, for a different reason.
Python *works*, right now. It has worked for years. If people don't
like the direction it's going, nobody is forcing them to upgrade to the
new version (which is not imminent anyway).
In the unlikely event that the latest and greatest Python in, what, five
years or more?, is so alien that one can't handle it, one has the right
to fork Python and maintain a tried-and-true-and-still-including-reduce-
-filter-and-map version of it, or even just to stick with the most
recent version which still has those features. And that's assuming it's
not acceptable (for whatever bizarre reason I can't imagine) to use the
inevitable third-party extension that will provide them anyway.
I wonder if some of those who seem most concerned are actually more
worried about losing the free support of a team of expert developers as
those developers evolve their vision of the language, than about losing
access to something as minor as reduce().
-Peter
Jp Calderone
This is a specious line of reasoning. Here's why:
Lots of people use Python, like Python, want to keep using Python. Moreover, they want Python to improve, rather than the reverse. Different people have different ideas about what "improve" means. Guido has his ideas, and since he's the BDFL, those are the ideas most likely to influence the direction of Python's development.
However, Guido isn't the only person with ideas, nor are his ideas the only ones that should be allowed to influence the direction of Python's development. Guido himself wouldn't even be silly enough to take this position. He knows he is not the ultimate source of wisdom in the world on all matters programming related.
So when people disagree with him, suggesting that they should leave the Python community is ridiculous. Just like Guido (and the overwhelming majority of the Python community - heck, maybe even all of it), these people are trying to improve the language.
Leaving the community isn't going to improve the language. Continuing to operate actively within it just might.
For my part, I lack the time and energy to participate in many of these discussions, but anyone who knows me knows I'm not silent because I see eye to eye with Guido on every issue :) I'm extremely greatful to the people who do give so much of their own time to try to further the Python language.
Suggesting people can "like it or lump it" is a disservice to everyone.
(Sorry to single you out Peter, I know you frequently contribute great content to these discussions too, and that there are plenty of other people who respond in the way you have in this message, but I had to pick /some/ post to reply to)
Jp
Ron Adam
> Ron Adam wrote:
>> I'm just estimating, but I think that is the gist of adding those two
>> in exchange for reduce. Not that they will replace all of reduce use
>> cases, but that sum and product cover most situations and can be
>> implemented more efficiently than using reduce or a for loop to do the
>> same thing. The other situations can easily be done using for loops,
>> so it's really not much of a loss.
>
> I really don't understand this reasoning. You essentially grant the
> position that reduce has a purpose, but you still seem to approve
> removing it. Let's grant your whole point and say that 90% of the use
> cases for reduce are covered by sum and product, and the other 10% are
> used by eggheads and are of almost no interest to programmers. But it
> still serves a purpose, and a useful one. That it's not of immediate
> use to anyone is an argument for moving it into a functional module
> (something I would have no serious objection to, though I don't see its
> necessity), not for removing it altogether! Why would you remove the
> functionality that already exists _and is being used_ just because? What
> harm does it do, vs. the benefit of leaving it in?
There are really two separate issues here.
First on removing reduce:
1. There is no reason why reduce can't be put in a functional module or
you can write the equivalent yourself. It's not that hard to do, so it
isn't that big of a deal to not have it as a built in.
2. Reduce calls a function on every item in the list, so it's
performance isn't much better than the equivalent code using a for-loop.
*** (note, that list.sort() has the same problem. I would support
replacing it with a sort that uses an optional 'order-list' as a sort
key. I think it's performance could be increased a great deal by
removing the function call reference. ***
Second, the addition of sum & product:
1. Sum, and less so Product, are fairly common operations so they have
plenty of use case arguments for including them.
2. They don't need to call a pre-defined function between every item, so
they can be completely handled internally by C code. They will be much
much faster than equivalent code using reduce or a for-loop. This
represents a speed increase for every program that totals or subtotals a
list, or finds a product of a set.
> But removing reduce is just removing
> functionality for no other reason, it seems, than spite.
No, not for spite. It's more a matter of increasing the over all
performance and usefulness of Python without making it more complicated.
In order to add new stuff that is better thought out, some things
will need to be removed or else the language will continue to grow and
be another visual basic.
Having sum and product built in has a clear advantage in both
performance and potential frequency of use, where as reduce doesn't have
the same performance advantage and most poeple don't use it anyway, so
why have it built in if sum and product are? Why not just code it as a
function and put it in your own module?
def reduce( f, seq):
x = 0
for y in seq:
x = f(x,y)
return x
But I suspect that most people would just do what I currently do and
write the for-loop to do what they want directly instead of using lambda
in reduce.
x = 1
for y in seq:
x = x**y
If performance is needed while using reduce with very large lists or
arrays, using the numeric module would be a much better solution.
http://www-128.ibm.com/developerworks/linux/library/l-cpnum.html
Cheers,
Ron
Carl Banks
Steven D'Aprano wrote:
> On Sun, 03 Jul 2005 08:14:28 -0700, Scott David Daniels wrote:
>
> > egbert wrote:
> >> On Sat, Jul 02, 2005 at 08:26:31PM -0700, Devan L wrote:
> >>
> >>>Also, map is easily replaced.
> >>>map(f1, sequence) == [f1(element) for element in sequence]
> >>
> >> How do you replace
> >> map(f1,sequence1, sequence2)
> >> especially if the sequences are of unequal length ?
> >>
> >> I didn't see it mentioned yet as a candidate for limbo,
> >> but the same question goes for:
> >> zip(sequence1,sequence2)
> >
> > OK, you guys are picking on what reduce "cannot" do.
> > The first is [f1(*args) for args in itertools.izip(iter1, iter2)]
>
> And now we get messier and messier... Compare these two idioms:
>
> "Map function f1 to each pair of items from seq1 and seq2."
>
> "Build a list comprehension by calling function f1 with the unpacked list
> that you get from a list built by zipping seq1 and seq2 together in pairs."
The shamelessness with which you inflated the verbosity of the latter
is hilarious.
> Good thing that removing reduce is supposed to make code easier to
> understand, right?
It was a bad example. I would say most people don't usually just call
a function in the list comp, because, frankly, they don't have to. A
realistic list comp would look something like this in a real program:
[ x**2 + y**2 for (x,y) in izip(xlist,ylist) ]
Now there's no longer much advantage in conciseness for the map version
(seeing that you'd have to define a function to pass to map), and this
is more readable.
--
CARL BANKS
Carl Banks
Christopher Subich wrote:
> Carl Banks wrote:
>
> > Listcomps et al. cannot do everything map, lambda, filter, and reduce
> > did. Listcomps are inferior for functional programming. But, you see,
> > functional is not the point. Streamlining procedural programs is the
> > point, and I'd say listcomps do that far better, and without all the
> > baroque syntax (from the procedural point of view).
>
> I've heard this said a couple times now -- how can listcomps not
> completely replace map and filter?
If you're doing heavy functional programming, listcomps are
tremendously unwieldy compared to map et al.
--
CARL BANKS
Scott David Daniels
> Suggesting people can "like it or lump it" is a disservice to everyone.
However, when people start with "why is this so" and rather than listen
for information, try to argue that they are right, that behavior is a
disservice to the group. Preparing a cogent argument to convince others
is one thing. Insisting on being convinced oneself is another. For the
latter, the only excuse I immediately forgive is for those who have
invested significant time and effort in providing the language, its
support tools, and/or its documents. For others it sounds a lot like
whining.
--Scott David Daniels
Scott....@Acm.Org
Steven D'Aprano
> First on removing reduce:
>
> 1. There is no reason why reduce can't be put in a functional module
Don't disagree with that.
> or
> you can write the equivalent yourself. It's not that hard to do, so it
> isn't that big of a deal to not have it as a built in.
Same goes for sum. Same goes for product, which doesn't have that many
common usages apart from calculating the geometric mean, and let's face
it, most developers don't even know what the geometric mean _is_.
If you look back at past discussions about sum, you will see that there is
plenty of disagreement about how it should work when given non-numeric
arguments, eg strings, lists, etc. So it isn't so clear what sum should do.
> 2. Reduce calls a function on every item in the list, so it's
> performance isn't much better than the equivalent code using a for-loop.
That is an optimization issue. Especially when used with the operator
module, reduce and map can be significantly faster than for loops.
> *** (note, that list.sort() has the same problem. I would support
> replacing it with a sort that uses an optional 'order-list' as a sort
> key. I think it's performance could be increased a great deal by
> removing the function call reference. ***
>
>
> Second, the addition of sum & product:
>
> 1. Sum, and less so Product, are fairly common operations so they have
> plenty of use case arguments for including them.
Disagree about product, although given that sum is in the language, it
doesn't hurt to put product as well for completion and those few usages.
> 2. They don't need to call a pre-defined function between every item, so
> they can be completely handled internally by C code. They will be much
> much faster than equivalent code using reduce or a for-loop. This
> represents a speed increase for every program that totals or subtotals a
> list, or finds a product of a set.
I don't object to adding sum and product to the language. I don't object
to adding zip. I don't object to list comps. Functional, er, functions
are a good thing. We should have more of them, not less.
>> But removing reduce is just removing
>> functionality for no other reason, it seems, than spite.
>
> No, not for spite. It's more a matter of increasing the over all
> performance and usefulness of Python without making it more complicated.
> In order to add new stuff that is better thought out, some things
> will need to be removed or else the language will continue to grow and
> be another visual basic.
Another slippery slope argument.
> Having sum and product built in has a clear advantage in both
> performance and potential frequency of use, where as reduce doesn't have
> the same performance advantage and most poeple don't use it anyway, so
> why have it built in if sum and product are?
Because it is already there.
> Why not just code it as a
> function and put it in your own module?
Yes, let's all re-invent the wheel in every module! Why bother having a
print statement, when it is so easy to write your own:
def myprint(obj):
sys.stdout.write(str(obj))
Best of all, you can customize print to do anything you like, _and_ it is
a function.
> def reduce( f, seq):
> x = 0
> for y in seq:
> x = f(x,y)
> return x
Because that is far less readable, and you take a performance hit.
> But I suspect that most people would just do what I currently do and
> write the for-loop to do what they want directly instead of using lambda
> in reduce.
That's your choice. I'm not suggesting we remove for loops and force you
to use reduce. Or even list comps.
--
Steven.
Christopher Subich
>
> Christopher Subich wrote:
>>I've heard this said a couple times now -- how can listcomps not
>>completely replace map and filter?
> If you're doing heavy functional programming, listcomps are
> tremendously unwieldy compared to map et al.
Interesting; could you post an example of this? Whenever I try to think
of that, I come up with unwieldly syntax for the functional case. In
purely functional code the results of map/filter/etc would probably be
directly used as arguments to other functions, which might make the
calls longer than I'd consider pretty. This is especially true with
lots of lambda-ing to declare temporary expressions.
Steven Bethard
> One caevat that I just noticed, though -- with the for-solution, you do
> need to be careful about whether you're using a generator or list if you
> do not set an explicit initial value (and instead use the first value of
> 'sequence' as the start). The difference is:
> _accum = g.next()
> for i in g: _accum = stuff(_accum,i)
>
> versus
> _accum = g[0]
> for i in g[1:]: _accum = stuff(_accum,i)
If you want to be general for all iterables (list, generators, etc), you
can write the code like:
itr = iter(g)
_accum = itr.next()
for i in itr:
_accum = stuff(_accum, i)
STeVe
Erik Max Francis
> Interesting; could you post an example of this? Whenever I try to think
> of that, I come up with unwieldly syntax for the functional case. In
> purely functional code the results of map/filter/etc would probably be
> directly used as arguments to other functions, which might make the
> calls longer than I'd consider pretty. This is especially true with
> lots of lambda-ing to declare temporary expressions.
I personally think that map looks clearer than a list comprehension for
a simple function call, e.g.
map(str, sequence)
vs.
[str(x) for x in sequence]
--
Erik Max Francis && m...@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
In Heaven all the interesting people are missing.
-- Friedrich Nietzsche
Mike Meyer
> I don't object to adding sum and product to the language. I don't object
> to adding zip. I don't object to list comps. Functional, er, functions
> are a good thing. We should have more of them, not less.
Yes, but where should they go? Adding functions in the standard
library is one thing. Adding builtins is another. Builtins make every
python process heavier. This may not matter on your desktop, but
Python gets used in embedded applications as well, and it does
there. Builtins also clutter the namespace. Nothing really wrong with
that, but it's unappealing.
I'd say that removing functions is a bad thing. On the other hand, I'd
say moving them from builtins to the standard library when Python has
functionality that covers most of the use cases for them is a good
thing.
The latter has occured for map, filter, and reduce. Lambda I'm not so
sure of, but it gets swept up with the same broom. Moving the first
three into a library module seems like a good idea. I'm not sure about
removing lambda. Removing map, filter and reduce remove most of my
use cases for it. But not all of them.
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Erik Max Francis
> I'd say that removing functions is a bad thing. On the other hand, I'd
> say moving them from builtins to the standard library when Python has
> functionality that covers most of the use cases for them is a good
> thing.
We all can pretty much guess that map, filter, and reduce will be
reimplemented in a functional module by a third party within mere
seconds of Python 3000 being released :-). So it's really just a
question of whether it will be let back in to the standard library as a
module (rather than builtins) or not. Even granting the reasons for
removing them as builtins, I really can't understand the motivation for
removing them entirely, not even as a standard library module.
--
Erik Max Francis && m...@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
Golf is a good walk spoiled.
-- Mark Twain
Ron Adam
> On Sun, 03 Jul 2005 19:31:02 +0000, Ron Adam wrote:
>
>
>>First on removing reduce:
>>
>>1. There is no reason why reduce can't be put in a functional module
>
>
> Don't disagree with that.
>
>
>>or
>>you can write the equivalent yourself. It's not that hard to do, so it
>>isn't that big of a deal to not have it as a built in.
>
>
Each item needs to stand on it's own. It's a much stronger argument for
removing something because something else fulfills it's need and is
easier or faster to use than just saying we need x because we have y.
In this case sum and product fulfill 90% (estimate of course) of reduces
use cases. It may actually be as high as 99% for all I know. Or it may
be less. Anyone care to try and put a real measurement on it?
which doesn't have that many
> common usages apart from calculating the geometric mean, and let's face
> it, most developers don't even know what the geometric mean _is_.
I'm neutral on adding product myself.
> If you look back at past discussions about sum, you will see that there is
> plenty of disagreement about how it should work when given non-numeric
> arguments, eg strings, lists, etc. So it isn't so clear what sum should do.
Testing shows sum() to be over twice as fast as either using reduce or a
for-loop. I think the disagreements will be sorted out.
>>2. Reduce calls a function on every item in the list, so it's
>>performance isn't much better than the equivalent code using a for-loop.
>
> That is an optimization issue. Especially when used with the operator
> module, reduce and map can be significantly faster than for loops.
I tried it... it made about a 1% improvement in the builtin reduce and
an equal improvement in the function that used the for loop.
The inline for loop also performed about the same.
See below..
>> *** (note, that list.sort() has the same problem. I would support
>>replacing it with a sort that uses an optional 'order-list' as a sort
>>key. I think it's performance could be increased a great deal by
>>removing the function call reference. ***
>>
>>
>>Second, the addition of sum & product:
>>
>>1. Sum, and less so Product, are fairly common operations so they have
>>plenty of use case arguments for including them.
>
> Disagree about product, although given that sum is in the language, it
> doesn't hurt to put product as well for completion and those few usages.
I'm not convinced about product either, but if I were to review my
statistics textbooks, I could probably find more uses for it. I suspect
that there may be a few common uses for it that are frequent enough to
make it worth adding. But it might be better in a module.
>>2. They don't need to call a pre-defined function between every item, so
>>they can be completely handled internally by C code. They will be much
>>much faster than equivalent code using reduce or a for-loop. This
>>represents a speed increase for every program that totals or subtotals a
>>list, or finds a product of a set.
>
> I don't object to adding sum and product to the language. I don't object
> to adding zip. I don't object to list comps. Functional, er, functions
> are a good thing. We should have more of them, not less.
Yes, we should have lots of functions to use, in the library, but not
necessarily in builtins.
>>>But removing reduce is just removing
>>>functionality for no other reason, it seems, than spite.
>>
>>No, not for spite. It's more a matter of increasing the over all
>>performance and usefulness of Python without making it more complicated.
>> In order to add new stuff that is better thought out, some things
>>will need to be removed or else the language will continue to grow and
>>be another visual basic.
>
> Another slippery slope argument.
Do you disagree or agree? Or are you undecided?
>>Having sum and product built in has a clear advantage in both
>>performance and potential frequency of use, where as reduce doesn't have
>>the same performance advantage and most poeple don't use it anyway, so
>>why have it built in if sum and product are?
>
> Because it is already there.
Hmm.. I know a few folks, Good people, but they keep everything to the
point of not being able to find anything because they have so much.
They can always think of reasons to keep things, "It's worth something",
"it means something to me", "I'm going to fix it", "I'm going to sell
it", "I might need it". etc..
"Because it is already there" sound like one of those type of reasons.
>>Why not just code it as a
>>function and put it in your own module?
>
> Yes, let's all re-invent the wheel in every module! Why bother having a
> print statement, when it is so easy to write your own:
>
> def myprint(obj):
> sys.stdout.write(str(obj))
Yes, Guido wants to make print a function in Python 3000. The good
thing about this is you can call your function just 'p' and save some
typing.
p("hello world")
Actually, I think i/o functions should be grouped in an interface
module. That way you choose the interface that best fits your need. It
may have a print if it's a console, or it may have a widget if it's a gui.
> Best of all, you can customize print to do anything you like, _and_ it is
> a function.
>
>
>> def reduce( f, seq):
>> x = 0
>> for y in seq:
>> x = f(x,y)
>> return x
>
>
> Because that is far less readable, and you take a performance hit.
They come out pretty close as far as I can tell.
def reduce_f( f, seq):
x = seq[0]
for y in seq[1:]:
x = f(x,y)
return x
import time
t = time.time()
r2 = reduce(lambda x,y: x*y, range(1,10000))
t2 = time.time()-t
print 'reduce builtin:', t2
t = time.time()
r1 = reduce_f(lambda x,y: x*y, range(1,10000))
t2 = time.time()-t
print 'reduce_f: ', t2
if r1!=r2: print "results not equal"
>>>
reduce builtin: 0.156000137329
reduce_f: 0.155999898911
>>>
reduce builtin: 0.15700006485
reduce_f: 0.155999898911
>>>
reduce builtin: 0.141000032425
reduce_f: 0.155999898911
>>But I suspect that most people would just do what I currently do and
>>write the for-loop to do what they want directly instead of using lambda
>>in reduce.
>
> That's your choice. I'm not suggesting we remove for loops and force you
> to use reduce. Or even list comps.
Just don't force me to use decorators! ;-)
Nah, they're ok too, but it did take me a little while to understand
their finer points.
Cheers,
Ron
Erik Max Francis
> Each item needs to stand on it's own. It's a much stronger argument for
> removing something because something else fulfills it's need and is
> easier or faster to use than just saying we need x because we have y.
>
> In this case sum and product fulfill 90% (estimate of course) of reduces
> use cases. It may actually be as high as 99% for all I know. Or it may
> be less. Anyone care to try and put a real measurement on it?
Well, reduce covers 100% of them, and it's one function, and it's
already there.
--
Erik Max Francis && m...@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
I'm not jumping in / A wave that just passes
-- Sandra St. Victor
Ron Adam
> Ron Adam wrote:
>
>> Each item needs to stand on it's own. It's a much stronger argument
>> for removing something because something else fulfills it's need and
>> is easier or faster to use than just saying we need x because we have y.
>>
>> In this case sum and product fulfill 90% (estimate of course) of
>> reduces use cases. It may actually be as high as 99% for all I know.
>> Or it may be less. Anyone care to try and put a real measurement on it?
>
>
> Well, reduce covers 100% of them, and it's one function, and it's
> already there.
So you are saying that anything that has a 1% use case should be
included as a builtin function?
I think I can find a few hundred other functions in the library that are
used more than ten times as often as reduce. Should those be builtins too?
This is a practical over purity issue, so what are the practical reasons
for keeping it. "It's already there" isn't a practical reason. And it
covers 100% of it's own potential use cases, is circular logic without a
real underlying basis.
Cheers,
Ron
Erik Max Francis
> So you are saying that anything that has a 1% use case should be
> included as a builtin function?
>
> I think I can find a few hundred other functions in the library that are
> used more than ten times as often as reduce. Should those be builtins too?
>
> This is a practical over purity issue, so what are the practical reasons
> for keeping it. "It's already there" isn't a practical reason. And it
> covers 100% of it's own potential use cases, is circular logic without a
> real underlying basis.
But the Python 3000 plan, at least what we've heard of it so far, isn't
to move it to a standard library module. It's to remove it altogether,
replacing it with sum and product. Since sum and product don't cover
all the uses cases for reduce, this is a case of taking one function
that handles all the required use cases and replacing it with _two_
functions that don't. Since it's doubling the footprint of the reduce
functionality, arguments about avoiding pollution are red herrings.
--
Erik Max Francis && m...@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
Every astronaut who goes up knows the risks he or she faces.
-- Sally Ride
Steven D'Aprano
> The shamelessness with which you inflated the verbosity of the latter
> is hilarious.
[snip]
> [ x**2 + y**2 for (x,y) in izip(xlist,ylist) ]
>
> Now there's no longer much advantage in conciseness for the map version
> (seeing that you'd have to define a function to pass to map), and this
> is more readable.
and then, five minutes later in another post, wrote:
> If you're doing heavy functional programming,
> listcomps are tremendously unwieldy compared to
> map et al.
Having a dollar each way I see :-)
--
Steven.
Steven D'Aprano
> Steven D'Aprano <st...@REMOVETHIScyber.com.au> writes:
>
>>I don't object to adding sum and product to the language. I don't object
>>to adding zip. I don't object to list comps. Functional, er, functions
>>are a good thing. We should have more of them, not less.
>
>
> Yes, but where should they go? Adding functions in the standard
> library is one thing. Adding builtins is another. Builtins make every
> python process heavier. This may not matter on your desktop, but
> Python gets used in embedded applications as well, and it does
> there. Builtins also clutter the namespace. Nothing really wrong with
> that, but it's unappealing.
>
> I'd say that removing functions is a bad thing. On the other hand, I'd
> say moving them from builtins to the standard library when Python has
> functionality that covers most of the use cases for them is a good
> thing.
>
> The latter has occured for map, filter, and reduce. Lambda I'm not so
> sure of, but it gets swept up with the same broom. Moving the first
> three into a library module seems like a good idea. I'm not sure about
> removing lambda. Removing map, filter and reduce remove most of my
> use cases for it. But not all of them.
Metoobe!!!
Practicality beats purity: I am perfectly happy to have
list comps in the language and fast, efficient
functional programming tools in a module. I'm even
happy to see sum and product and zip as builtins, even
though logically they belong with map and reduce in the
(hypothetical) functional module.
I know Python isn't, and never will be, a purely
functional language. But being able to use some
functional techniques is good, and shouldn't be lost.
--
Steven.
Robert Kern
> Ron Adam wrote:
>
>>So you are saying that anything that has a 1% use case should be
>>included as a builtin function?
>>
>>I think I can find a few hundred other functions in the library that are
>>used more than ten times as often as reduce. Should those be builtins too?
>>
>>This is a practical over purity issue, so what are the practical reasons
>>for keeping it. "It's already there" isn't a practical reason. And it
>>covers 100% of it's own potential use cases, is circular logic without a
>>real underlying basis.
>
> But the Python 3000 plan, at least what we've heard of it so far, isn't
> to move it to a standard library module. It's to remove it altogether,
> replacing it with sum and product. Since sum and product don't cover
> all the uses cases for reduce, this is a case of taking one function
> that handles all the required use cases and replacing it with _two_
> functions that don't. Since it's doubling the footprint of the reduce
> functionality, arguments about avoiding pollution are red herrings.
Four, in fact. sum(), product(), any(), and all().
The problem with this discussion is that no one is saying anything
new[1]. We've heard all of the arguments for and against removing these
functions. Over and over and over again. They're not changing anyone's
mind, not yours, not mine, and definitely not Guido's.
And it's not even like Python 3000 is around the corner or in any stage
of concrete planning. Adding our voices to the chorus *now* won't make a
bit of difference, nor should it. The size of the chorus doesn't matter;
Python isn't developed by votes. We tried that once before; it didn't
work so well.
When planning for Python 3000 really gets going, when Guido gets a
year's sabbatical to work on it, when we can see how
map/filter/reduce/lambda fit into the whole scheme of how this new
language works, *then* is the time to be having these discussions. Right
now, all we're doing is making each other bitter and angry for no good
reason.
[1] Okay, there was that guy who predicted that list comprehensions and
first-class functions were the next to go. That was new. But also wrong.
I think we can discount that.
--
Robert Kern
rk...@ucsd.edu
"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter
Carl Banks
Don't think so. The verbosity I spoke of was your describing the code
snippets in English, not the conciseness of the example. map and
friends are more concise than listcomps, I wasn't arguing that, except
that for the typical Pythonic use of listcomps it isn't much. One
listcomp or one call to map is not "heavily functional."
--
CARL BANKS
Carl Banks
I suspect you're misunderstanding what I mean by heavily functional.
You appear to see maps and listcomps merely as a shortcut for a for
loop. You're comparing the map shortcut and the listcomp shortcut and
seeing which one's less verbose. In a mostly procedural program which
uses functional constructs in isolation, listcomps are going to win
most of those battles.
Heavily functional programming is a different mindset altogether. In
heavily functional programming, things like maps and filters and
function applications are actually what you're thinking about. map
isn't an indirect way to do a for loop; it's a direct way to do a map.
When your mind is focused on "applying a function to each member of
this list and returning a list of the results" as opposed to
"convenient shortcut to a for loop", map is going to be far superior to
a listcomp. And when you're doing dozens and dozens of maps over a
large purely functional program, you don't want to write out a listcomp
every single time you want to do it.
--
CARL BANKS
Terry Hancock
> I personally think that map looks clearer than a list comprehension for
> a simple function call, e.g.
I have to disagree
> map(str, sequence)
This says "call a function 'map' on 'str' and 'sequence'"
Which, syntactically, is not terribly informative.
I have to remember:
* "str" is actually a callable
* "map" is a mathematical concept of linking one thing to another. What
things? "str to sequence"? No! Wrong guess. "str" is the "mapping function",
and the result is the thing sequence is to be linked to.
Now, sure, I know all this, and I learned what "map" did from the manual,
but it's not especially easy to remember.
This on the other hand,
> [str(x) for x in sequence]
is practically plain English:
"call the function "str" on x, for every x in sequence"
Other than chopping out a few words, and using the () operator instead
of "call", it's hard to imagine this being any closer to exactly what you
would say to describe the operation. And for most of us, English comes
easier than Computer Science jargon.
--
Terry Hancock ( hancock at anansispaceworks.com )
Anansi Spaceworks http://www.anansispaceworks.com
Tom Anderson
> Erik Max Francis wrote:
>> Ron Adam wrote:
>>
>>> So you are saying that anything that has a 1% use case should be included
>>> as a builtin function?
>>>
>>> I think I can find a few hundred other functions in the library that are
>>> used more than ten times as often as reduce. Should those be builtins
>>> too?
>>>
>>> This is a practical over purity issue, so what are the practical reasons
>>> for keeping it. "It's already there" isn't a practical reason. And it
>>> covers 100% of it's own potential use cases, is circular logic without a
>>> real underlying basis.
>>
>> But the Python 3000 plan, at least what we've heard of it so far, isn't
>> to move it to a standard library module. It's to remove it altogether,
>> replacing it with sum and product. Since sum and product don't cover
>> all the uses cases for reduce, this is a case of taking one function
>> that handles all the required use cases and replacing it with _two_
>> functions that don't. Since it's doubling the footprint of the reduce
>> functionality, arguments about avoiding pollution are red herrings.
>
> Four, in fact. sum(), product(), any(), and all().
I'll just chip in and say i'd quite like a flatten(), too; at the moment,
i have one like this:
def flatten(ll):
return reduce(lambda a, l: a.extend(l), ll, [])
A builtin, which had fast special-case code for then the argument is a
list of lists (rather than an iterable of iterables), would be nice, since
this is a reasonably big use of reduce for me.
How would one do that as a list comp, by the way? I'm really not very good
with them yet.
> [1] Okay, there was that guy who predicted that list comprehensions and
> first-class functions were the next to go. That was new. But also wrong.
> I think we can discount that.
True. Guido will only get rid of those after he's got rid of lowercase
letters in identifiers.
tom
--
A military-industrial illusion of democracy
Peter Hansen
> On Sunday 03 July 2005 07:05 pm, Erik Max Francis wrote:
>>I personally think that map looks clearer than a list comprehension for
>>a simple function call
> This on the other hand,
>> [str(x) for x in sequence]
> is practically plain English:
>
> "call the function "str" on x, for every x in sequence"
>
> Other than chopping out a few words, and using the () operator instead
> of "call", it's hard to imagine this being any closer to exactly what you
> would say to describe the operation. And for most of us, English comes
> easier than Computer Science jargon.
And with a better choice of names than "x", that line becomes even more
self-documenting.
[str(parrot) for parrot in sequence], for example, tells you much more
about what is going on than str(x) does.
Exactly what, I have no idea... but it says _so_ much more. ;-)
-Peter
George Sakkis
> I'll just chip in and say i'd quite like a flatten(), too; at the moment,
> i have one like this:
>
> def flatten(ll):
> return reduce(lambda a, l: a.extend(l), ll, [])
This doesn't work; a.extend() returns None, not the extended list a:
>>> seq = [[1,2],[3],[],[4,[5,6]]]
>>> flatten(seq)
AttributeError: 'NoneType' object has no attribute 'extend'
This works for 1-level flattening:
def flatten(ll):
return reduce(lambda a, l: a.extend(l) or a, ll, [])
>>> flatten(seq)
[1, 2, 3, 4, [5, 6]]
And finally for recursive flattening:
def flatten(seq):
return reduce(_accum, seq, [])
def _accum(seq, x):
if isinstance(x,list):
seq.extend(flatten(x))
else:
seq.append(x)
return seq
>>> flatten(seq)
[1, 2, 3, 4, 5, 6]
George
Christopher Subich
> [str(parrot) for parrot in sequence], for example, tells you much more
> about what is going on than str(x) does.
>
> Exactly what, I have no idea... but it says _so_ much more. ;-)
Yarr! Avast! Etc!
Christopher Subich
> I suspect you're misunderstanding what I mean by heavily functional.
> Heavily functional programming is a different mindset altogether. In
> heavily functional programming, things like maps and filters and
> function applications are actually what you're thinking about. map
> isn't an indirect way to do a for loop; it's a direct way to do a map.
That's true; I'm more comfortable with procedural programming in
general, but I had a few classes that used LISP and understand what
you're talking about.
That said, Python itself is mostly a procedural language, with the
functional tools really being bolted on[1]. When we're talking about
Py3K, I think we're really talking about a redesign and rethink of
pretty much the entire language -- with list and generator
comprehensions, for procedural programming the need for map and lambda
goes away. Reduce isn't directly replaced, of course, but a for-loop
implementation (for procedural programming) is clearer, more powerful,
more explicit, and possibly faster.
That said, I very much like the idea of putting map and filter in a
functional module. For applications like functional-style programming
where map/etc are clearer, that keeps them in the library for efficient
use, yet it leaves the native language with OO(g)WTDI [Only One (good)
Way to Do It].
[1] -- lambda excepted. I think it's kind of cute, in a baby-mammal
kind of way.
Steven Bethard
> Ron Adam wrote:
>
>> reduces use cases. It may actually be as high as 99% for all I know.
>> Or it may be less. Anyone care to try and put a real measurement on it?
>
> Well, reduce covers 100% of them, and it's one function, and it's
> already there.
And it's almost two times slower:
$ python -m timeit -s "x = xrange(1000)" "sum(x)"
10000 loops, best of 3: 92.5 usec per loop
$ python -m timeit -s "from operator import add; x = xrange(1000)"
"reduce(add, x)"
10000 loops, best of 3: 157 usec per loop
And that's only if I have the sense to import from operator:
$ python -m timeit -s "x = xrange(1000); add = lambda x, y: x + y"
"reduce(add, x)"
1000 loops, best of 3: 587 usec per loop
Note that the simple for-loop beats the case where you define your own
function because it doesn't have the repeated overhead of function calls
(which are expensive in Python):
$ python -m timeit -s "sum = 0; x = xrange(1000)" "for i in x: sum += i"
10000 loops, best of 3: 291 usec per loop
What would really help here is if you could identify the cases where you
think reduce is really a gain. A lot of them are actually not good
practice in Python because of the function-call overhead. However, I'm
willing to be convinced otherwise with a few good examples.
STeVe
Erik Max Francis
> And it's almost two times slower:
That's because you're not using operator.add.
--
Erik Max Francis && m...@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
Virtue has never been as respectable as money.
-- Mark Twain
Carl Banks
> That said, Python itself is mostly a procedural language, with the
> functional tools really being bolted on[1].
Yeah, that's pretty much what I said in the first place.
--
CARL BANKS
Ron Adam
> And finally for recursive flattening:
>
> def flatten(seq):
> return reduce(_accum, seq, [])
>
> def _accum(seq, x):
> if isinstance(x,list):
> seq.extend(flatten(x))
> else:
> seq.append(x)
> return seq
>
>
>>>>flatten(seq)
>
> [1, 2, 3, 4, 5, 6]
>
>
> George
>
How about this for a non recursive flatten.
def flatten(seq):
s = []
while seq:
while isinstance(seq[0],list):
seq = seq[0]+seq[1:]
s.append(seq.pop(0))
return s
seq = [[1,2],[3],[],[4,[5,6]]]
flatten(seq)
Ron
Steven Bethard
> Steven Bethard wrote:
>
>> And it's almost two times slower:
>
> That's because you're not using operator.add.
Huh? Please re-read my post. That's exactly what I used.
STeVe
mch...@gmail.com
> Lambda is no more an obscure name than "function", "decorator", "closure",
> "class", or "module". The first time you come across it, you don't know
> what it means. Then you learn what it means, and then you know.
I believe you've made two errors here. First of all, "lambda" is part
of the Python language, while "function", "decorator", "closure", and
"module" are not. The fact that some words which are NOT part of the
Python language are obscure has no bearing whatsoever on "lambda". The
obnubilation created by this comparison is an obscure word also, but,
it isn't relevent to the Python language.
The second error is that I believe most english speakers COULD provide
a definition for the fairly common words "function", "class", and
"decorator". The exact meaning of "class" might not be what they expect
at first, but exposure to any object oriented language would make the
concept quickly familiar. But "lambda" has a very clear meaning... it's
a letter of the greek alphabet. The connection between that letter and
anonymous functions is tenuous at best, and fails the test of making
Python read like "executable pseudocode".
-- Michael Chermside
Tom Anderson
> "Tom Anderson" <tw...@urchin.earth.li> wrote:
>
>> I'll just chip in and say i'd quite like a flatten(), too; at the moment,
>> i have one like this:
>>
>> def flatten(ll):
>> return reduce(lambda a, l: a.extend(l), ll, [])
>
> This doesn't work; a.extend() returns None, not the extended list a:
Ah, no, very good, i was hoping someone would notice that. Well done.
I think my lambda looks more like lambda a, b: a + b, but i realised the
other day that extend could make it more efficient, and didn't think it
through properly.
tom
--
The revolution will not be televised. The revolution will be live.
Steven D'Aprano
> Steven D'Aprano writes:
>> Lambda is no more an obscure name than "function", "decorator", "closure",
>> "class", or "module". The first time you come across it, you don't know
>> what it means. Then you learn what it means, and then you know.
>
> I believe you've made two errors here. First of all, "lambda" is part
> of the Python language, while "function", "decorator", "closure", and
> "module" are not.
Sorry, but you are mistaken. "lambda" is a _reserved_ word in the
Python language, while "function" etc are not, but they are certainly
part of the language. Try explaining what def and import do without
using the words "function" or "module". Maybe you can do it, using
circumlocutions, but it isn't easy, and costs clarity.
[snip]
> The second error is that I believe most english speakers COULD provide
> a definition for the fairly common words "function", "class", and
> "decorator". The exact meaning of "class" might not be what they expect
> at first,
Function, in the sense of a mathematical function, I agree. Class as in
the thing you go to at school and decorator as in the person who advises
you what colour curtains to have, certainly. But in the Python sense? No.
Especially not decorator, which I believe most _programmers_ would have
trouble explaining, let alone non-programmer English speakers. I know I do.
> but exposure to any object oriented language would make the
> concept quickly familiar.
Just as exposure to functional languages would make lambda very familiar.
> But "lambda" has a very clear meaning... it's
> a letter of the greek alphabet. The connection between that letter and
> anonymous functions is tenuous at best, and fails the test of making
> Python read like "executable pseudocode".
Think back to when you were a schoolboy at your first day of school.
Unless you had a very unusual upbringing, you probably had never heard the
word "function" before. There is nothing about the word "function" that
brings to mind "a mathematical entity which transforms a variable into a
different variable", let alone "a programming subroutine that returns a
result". (Or for that matter, "the purpose which a person or thing is
for", as in the function of a spanner is to tighten nuts on bolts.)
You had to learn that word, discover what it means, and then it becomes
familiar. You don't notice the process only because it happened so long
ago, at an age that your brain was operating in "language acquisition
mode" and picking up vocabulary at an incredible rate.
There is nothing about the word "string" that especially brings to mind
"an array of bytes representing characters". The analogy of "string of
characters" to "string of beads" breaks down as soon as you have multiple
lines of text. And as for float, that's what boats do, heaven only knows
what it has to do with numbers.
(Yes, I know what it has to do with numbers. But that is something I had
to learn, and even now I still have difficulty because I expect floats to
operate like mathematical real numbers, and they don't.)
And dare I say it, what do constricting snakes have to do with programming?
I won't say that the anonymous function meaning of lambda comes to my mind
before the Greek letter, but it isn't very far behind, and rapidly
catching up. (I use lambda a lot more than I speak Greek.) It wouldn't
surprise me if one day I think of Python programming before the Greek
letter, just as the world aleph brings to my mind the sense of infinity
before the sense of it being a Hebrew letter.
--
Steven.
Tom Anderson
> George Sakkis wrote:
>
>> And finally for recursive flattening:
>>
>> def flatten(seq):
>> return reduce(_accum, seq, [])
>>
>> def _accum(seq, x):
>> if isinstance(x,list):
>> seq.extend(flatten(x))
>> else:
>> seq.append(x)
>> return seq
>>
>>
>>>>> flatten(seq)
>>
>> [1, 2, 3, 4, 5, 6]
>
> How about this for a non recursive flatten.
>
> def flatten(seq):
> s = []
> while seq:
> while isinstance(seq[0],list):
> seq = seq[0]+seq[1:]
> s.append(seq.pop(0))
> return s
>
> seq = [[1,2],[3],[],[4,[5,6]]]
> flatten(seq)
The trouble with these is that they make a lot of temporary lists -
George's version does it with the recursive calls to flatten, and Ron's
with the slicing and concatenating. How about a version which never makes
new lists, only appends the base list? We can use recursion to root
through the lists ...
def isiterable(x):
return hasattr(x, "__iter__") # close enough for government work
def visit(fn, x): # perhaps better called applytoall
if (isiterable(x)):
for y in x: visit(fn, y)
else:
fn(x)
def flatten(seq):
a = []
def appendtoa(x):
a.append(x)
visit(appendtoa, seq)
return a
If you hate recursion, you can write a non-recursive version of visit;
you'll have to manage a stack of lists yourself, though. Something like:
def visit(fn, x):
if (not isiterable(x)): x = (x,)
stack = [None] # stack of iterators
cur = iter(x) # current iterator
while (cur != None):
try:
thing = cur.next()
if (isiterable(thing)):
stack.append(cur)
cur = iter(thing)
else:
fn(thing)
except StopIteration:
cur = stack.pop()
There might be a cleverer way to do this.
Terry Hancock
> Sorry, but you are mistaken. "lambda" is a _reserved_ word in the
> Python language, while "function" etc are not, but they are certainly
> part of the language. Try explaining what def and import do without
> using the words "function" or "module". Maybe you can do it, using
> circumlocutions, but it isn't easy, and costs clarity.
This is still a relevant distinction. One relevant point is that I am
perfectly free to use, say, the Spanish or Chinese word to describe
"module" or "function", but the keywords "def" and "import" will
remain the same.
> > The second error is that I believe most english speakers COULD provide
> > a definition for the fairly common words "function", "class", and
> > "decorator". The exact meaning of "class" might not be what they expect
> > at first,
>
> Function, in the sense of a mathematical function, I agree. Class as in
> the thing you go to at school and decorator as in the person who advises
> you what colour curtains to have, certainly. But in the Python sense? No.
> Especially not decorator, which I believe most _programmers_ would have
> trouble explaining, let alone non-programmer English speakers. I know I do.
The "Python sense" is not arbitrary. There are very direct visual or logical
(i.e. INTUITIVE) connections between these words' *English* meanings (not
just mathematical, either) and their meanings in Python.
A "Function" is (in English) (kdict-gcide*):
1.) The act of executing or performing any duty, office, or
calling; performance....
2.) The appropriate action of any special organ or
part of an animal or vegetable organism...
[...]
5.) (math) A quantity so connected with another quantity,
that if any alteration be made in the latter there will be
a consequent alteration in the former.
The programming use is probably *closer* to the English meaning
than the math jargon meaning (except in the sense of "functional
programming" which leans more on the math meaning.
Similarly, "decorate" is 'make more attractive by adding ornament, colour, etc.'
In Python, a "decorator" applies a wrapper to a function to provide it with
some additional functionality. The function definition is furthermore
"decorated" with the decorator declaration to give it more meaning, which
is arguably more aesthetically pleasing (otherwise, why not literally wrap
with a function after defining the function?). These meanings are very
connected with the English definition of the word.
"Class" can, of course, mean a room in which you teach classes, but again
Webster's will certainly provide meaning much closer to the programming
term:
1. A group of individuals ranked together as possessing
common characteristics...
[2 is the class of students sense]
3. A comprehensive division of animate or inanimate objects,
grouped together on account of their common
characteristics
4. A set; a kind or description, species or variety.
Meanings 1,3, & 4 are all arguably intimately connected to the OOP meaning,
especially meaning #3 which even mentions "objects". (And I won't bother
to point out that the English meaning of "object" is tied closely to what it
means in programming).
A similar argument can be made for "object", "module", "script", and
even "method" and "program".
Now, if you are armed ONLY with the English definition, you will possibly
run into some trouble, because the programming usage is a *specialization*
of the term -- we strictly take only *one* of the English definitions to apply,
and we narrow its meaning a bit. "Objects" in computer science never means
"the purpose of the program" nor does it ever refer to "a small brick", even
though the English word can mean both of those things. But it's not exactly
a shocker that a programming term is going to apply to things you can find
in a program, so I don't think we're stumping the newbie with such terms.
"lambda" has no such advantage. Here's the *entire* gcide definition:
Lambda \Lamb"da\, n. [NL., fr. Gr. la`mbda.]
1. The name of the Greek letter [Lambda], [lambda],
corresponding with the English letter L, l.
[1913 Webster]
2. (Anat.) The point of junction of the sagittal and lambdoid
sutures of the skull.
[1913 Webster]
3. (Phys.) A subatomic particle carrying no charge, having a
mass equal to 2183 times that of an electron; it decays
rapidly, typically forming a nucleon and a pion. --MW10
[PJC]
Lambda moth (Zool.), a moth so called from a mark on its
wings, resembling the Greek letter lambda ([Lambda]).
[1913 Webster]
> > but exposure to any object oriented language would make the
> > concept quickly familiar.
>
> Just as exposure to functional languages would make lambda very familiar.
Yeah, well, there *is* an entry in the "Online Dictionary of Computing":
LAMBDA
A version of typed lambda-calculus, used to describe
semantic domains.
["Outline of a Mathematical Theory of Computation",
D.S. Scott, TM PRG-2, PRG, Oxford U, 1971].
If even this means what "lambda" does in Python, I would be surprised,
certainly it doesn't mean a whole lot to me.
> Think back to when you were a schoolboy at your first day of school.
> Unless you had a very unusual upbringing, you probably had never heard the
> word "function" before.
Total BS. I knew the word "function" in it's English language sense, probably
by the time I was 6. I *know* my kids know it.
> There is nothing about the word "function" that
> brings to mind "a mathematical entity which transforms a variable into a
> different variable",
They'll get this in about the 6th grade, IIRC.
> let alone "a programming subroutine that returns a
> result".
It also *does something*, which is what my first understanding of a "function"
was. Side-effects would seem to be perfectly natural to anyone with an
English language background, I guess. ;-)
> (Or for that matter, "the purpose which a person or thing is
> for", as in the function of a spanner is to tighten nuts on bolts.)
You are really stretching if you think kids (let alone average adults) don't
know this meaning of the word "function".
> You had to learn that word, discover what it means, and then it becomes
> familiar. You don't notice the process only because it happened so long
> ago, at an age that your brain was operating in "language acquisition
> mode" and picking up vocabulary at an incredible rate.
If you're arguing that language is acquired rather than innate, you
are bludgeoning an obvious point. The point is that *jargon* should
ideally derive in a natural way from commonly-used language, if
we want it to be easy to acquire for people who don't learn programming
between the ages of 1 and 5 as we learn our native languages. Even
in the 21st century, I think this includes just about all of us. ;-)
> There is nothing about the word "string" that especially brings to mind
> "an array of bytes representing characters". The analogy of "string of
> characters" to "string of beads" breaks down as soon as you have multiple
> lines of text.
Ah, but that's useful. "Strings" AREN'T "multiple lines of text" in the
computer's memory, are they? '\n' is just another bead. The "multiple
lines" is a representation, or way of laying out the beads. Very useful
distinction, and immediately driven by the choice of analogy.
I'm going to use that, thanks. ;-)
> And as for float, that's what boats do, heaven only knows
> what it has to do with numbers.
Ah, yes. Here, it is clear that Fortran whips Python on readability, it
calls them "reals". The only problem is that real mathematicians probably
had conniption fits about the fact that "real" variables actually represent
"rational" numbers.
> (Yes, I know what it has to do with numbers. But that is something I had
> to learn, and even now I still have difficulty because I expect floats to
> operate like mathematical real numbers, and they don't.)
>
> And dare I say it, what do constricting snakes have to do with programming?
Nothing. Then again *we* know that "Python" hasn't anything to do
with snakes, either. ;-D
> I won't say that the anonymous function meaning of lambda comes to my mind
> before the Greek letter, but it isn't very far behind, and rapidly
> catching up. (I use lambda a lot more than I speak Greek.) It wouldn't
> surprise me if one day I think of Python programming before the Greek
> letter, just as the world aleph brings to my mind the sense of infinity
> before the sense of it being a Hebrew letter.
Then it is clearly *not you* who should be served by the naming scheme.
Anyone so deeply trained and experienced should be expected to adapt,
you have the wherewithall to do so. It is the new user for whom the clarity
of the jargon is so important.
Personally, I find the term "anonymous function" to be a whole lot clearer
than "lambda" or "lambda function". Indeed, if asked what "lambda" means,
my reply is it's a "stupid name for an anonymous function", and if my
listener is less savvy "for a function that doesn't have a name, because you
only use it once".
Having said that, I too will miss the *concept* of an anonymous function,
although I wouldn't mind at all if its name changed, or if it were somehow
integrated into the "def" keyword's usage. Using backticks or some other
syntax delimiter also sounds promising, although we're sort of running out
of them. ;-)
--
*An unfortunate acronym for "Gnu Collaborative International Dictionary
of English".
Christopher Subich
> concept quickly familiar. But "lambda" has a very clear meaning... it's
> a letter of the greek alphabet. The connection between that letter and
> anonymous functions is tenuous at best, and fails the test of making
> Python read like "executable pseudocode".
But 'lambda' does have a very clear meaning in the realm of functional
programming, and it means precisely (mostly) what it means in Python: an
anonymous function.
It might not be the -best- possible name, but anyone who's had a
computer science education should have had a class that introduced basic
functional programming topics (even if only for academic interest), and
so they should be familiar with the keyword name.
If not, then it's just a magic word. Kind of like 'def'.
mch...@gmail.com
high-school. I provided support for the English teachers who taught
*all* students -- but they taught things like the use of a word
processor or the internet, and never covered the meaning of "lambda". I
taught a computer applications course which was taken by only small
fraction of the students (<10%) but there I taught things like the use
of photo-editing software, creating web sites, and the use of simple
databases; I never covered the meaning of "lambda". I also taught the
programming class (taken by only a dozen or so students per graduating
class) -- students learned basic concepts like variables, looping, up
through fancier bits like a couple different sorting algorithms. But I
didn't cover the meaning of "lambda". And I also taught the "AP"
computer course (taken by an average of just 4 students per year!), in
which I explained things like object oriented programming and recursion
and managed to get the students to the level where they could work
together as a group to write a moderately complex program, like a
simple video game. And I didn't teach the meaning of "lambda", nor was
it covered by the "AP" exam, which is supposed to be equivalent to a
single college-level course in computer programming.
So I'd say that it's a pretty obscure name that most people wouldn't
know.
And besides, "def" isn't a "magic" word... it's an abreviation for
"define"... I hope that any student who didn't understand a word as
common as "define" wouldn't have graduated from our school.
-- Michael Chermside
Peter Hansen
[snip description of experience teaching high school students]
> So I'd say that it's a pretty obscure name that most people wouldn't
> know.
It would be hard to argue against that statement; certainly "lambda" in
this context (or probably any) is not a word "most people" would know.
On the other hand, the name itself is probably not very important. I
still remember my first encounter with "lambda" in Python very clearly.
I saw the word, thought "huh? what the heck is that?", then read a
sentence about it that included some comment about its background in
other fields.
"Oh," I said, "pre-existing usage. Whatever." I proceeded to read
about what it did and how to use it.
The name was irrelevant. If the text had said "anonymous functions are
created using the keyword 'tribble' (named for a similar feature in a
fictional Klingon programming language)", I wouldn't have felt any
differently about it. So it makes some sense to a few trekkers... big
furry deal.
What bothered me was the syntax. Arguments without parentheses? What
possessed anyone to put something so inconsistent in the language? No
statements? Dang, that will limit my interest in using them. Oh well,
what's page four of the tutorial got for me next? It shouldn't take
anyone more than ten seconds to integrate "lambda" into their brain and
carry on with useful work.
Really, the name is such a trivial, unimportant part of this whole thing
that it's hardly worth discussing. The syntax is more important, and
the limitations are of definite interest. Not the name.
-Peter
Grant Edwards
> Up until a few years ago, I ran the computer science department at a
> high-school. I provided support for the English teachers who taught
> *all* students -- but they taught things like the use of a word
> processor or the internet,
That's not computer science.
> and never covered the meaning of "lambda". I taught a computer
> applications course which was taken by only small fraction of
> the students (<10%) but there I taught things like the use of
> photo-editing software, creating web sites, and the use of
> simple databases;
That's not computer science.
> I never covered the meaning of "lambda". I also taught the
> programming class (taken by only a dozen or so students per
> graduating class) -- students learned basic concepts like
> variables, looping, up through fancier bits like a couple
> different sorting algorithms.
Now you're getting a little closer to computer science.
It sounds like you ran a computer user training department. I
don't think it could be called computer science.
> But I didn't cover the meaning of "lambda". And I also taught
> the "AP" computer course (taken by an average of just 4
> students per year!), in which I explained things like object
> oriented programming and recursion and managed to get the
> students to the level where they could work together as a
> group to write a moderately complex program, like a simple
> video game. And I didn't teach the meaning of "lambda", nor
> was it covered by the "AP" exam, which is supposed to be
> equivalent to a single college-level course in computer
> programming.
Computer programming isn't the same thing as computer science.
It's just one of the tools used to do computer science.
> So I'd say that it's a pretty obscure name that most people
> wouldn't know.
I can't believe that anybody with any computer science
background doesn't know it.
> And besides, "def" isn't a "magic" word... it's an abreviation
> for "define"... I hope that any student who didn't understand
> a word as common as "define" wouldn't have graduated from our
> school.
Lamda isn't a magic word either. It comes from lambda
calculus.
--
Grant Edwards grante Yow! Is this where people
at are HOT and NICE and they
visi.com give you TOAST for FREE??
Devan L
if not hasattr(iterable, '__iter__'):
return [iterable]
return sum([flatten(element) for element in iterable],[])
Recursion makes things so much shorter.
Sybren Stuvel
> It sounds like you ran a computer user training department. I don't
> think it could be called computer science.
Being a computer science student at the University of Amsterdam, I can
tell you that it definitely should not be called "computer science".
> I can't believe that anybody with any computer science background
> doesn't know it.
I agree.
Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa
George Sakkis
The last line can faster and more compact by:
from itertools import imap
def flatten(iterable):
if not hasattr(iterable, '__iter__'):
return [iterable]
return sum(imap(flatten,iterable),[])
George
Tom Anderson
> Having said that, I too will miss the *concept* of an anonymous
> function, although I wouldn't mind at all if its name changed, or if it
> were somehow integrated into the "def" keyword's usage. Using backticks
> or some other syntax delimiter also sounds promising, although we're
> sort of running out of them. ;-)
I understand that the backslash is popular in some ivory-tower functional
languages. Currently, a backslash can be used for explicit line joining,
and is illegal elsewhere on a line outside a string literal, so i think
it's available for this. It would be utterly unpythonic to use puntuation
instead of a keyword, and it would make no sense to novices, but it would
scare the crap out of C programmers, which has to be worth something.
tom
--
[Philosophy] is kind of like being driven behind the sofa by Dr Who - scary, but still entertaining. -- Itchyfidget
Ivan Van Laningham
Tom Anderson wrote:
>
> I understand that the backslash is popular in some ivory-tower functional
> languages. Currently, a backslash can be used for explicit line joining,
> and is illegal elsewhere on a line outside a string literal, so i think
> it's available for this. It would be utterly unpythonic to use puntuation
> instead of a keyword, and it would make no sense to novices, but it would
> scare the crap out of C programmers, which has to be worth something.
>
Oh, I don't think so. Don't forget that Perl was written by impatient C
programmers. I think scaring C programmers, like giving engineers too
much information, is really hard to do. Live by the sword, die by the
sword.
Metta,
<int *(*(*foo)())()>-ly y'rs,
Ivan
----------------------------------------------
Ivan Van Laningham
God N Locomotive Works
http://www.andi-holmes.com/
http://www.foretec.com/python/workshops/1998-11/proceedings.html
Army Signal Corps: Cu Chi, Class of '70
Author: Teach Yourself Python in 24 Hours
Ron Adam
> The trouble with these is that they make a lot of temporary lists -
> George's version does it with the recursive calls to flatten, and Ron's
> with the slicing and concatenating. How about a version which never
> makes new lists, only appends the base list? We can use recursion to
> root through the lists ...
Ok... How about a non-recursive flatten in place? ;-)
def flatten(seq):
i = 0
while i!=len(seq):
while isinstance(seq[i],list):
seq.__setslice__(i,i+1,seq[i])
i+=1
return seq
seq = [[1,2],[3],[],[4,[5,6]]]
print flatten(seq)
I think I'll be using the __setslice__ method more often.
And the test:
#--------------------------------
# Georges recursive flatten
init_a = """
def flatten(seq):
return reduce(_accum, seq, [])
def _accum(seq, x):
if isinstance(x,list):
seq.extend(flatten(x))
else:
seq.append(x)
return seq
seq = [[1,2],[3],[],[4,[5,6]]]
"""
# Ron's non-recursive
init_b = """
def flatten(seq):
s = []
while seq:
while isinstance(seq[0],list):
seq = seq[0]+seq[1:]
s.append(seq.pop(0))
return s
seq = [[1,2],[3],[],[4,[5,6]]]
"""
# Tom's recursive, no list copies made
init_c = """
def isiterable(x):
return hasattr(x, "__iter__") # close enough for government work
def visit(fn, x): # perhaps better called applytoall
if (isiterable(x)):
for y in x: visit(fn, y)
else:
fn(x)
def flatten(seq):
a = []
def appendtoa(x):
a.append(x)
visit(appendtoa, seq)
return a
seq = [[1,2],[3],[],[4,[5,6]]]
"""
# Devan' smallest recursive
init_d = """
def flatten(iterable):
if not hasattr(iterable, '__iter__'):
return [iterable]
return sum([flatten(element) for element in iterable],[])
seq = [[1,2],[3],[],[4,[5,6]]]
"""
# Ron's non-recursive flatten in place! Much faster too!
init_e = """
def flatten(seq):
i = 0
while i!=len(seq):
while isinstance(seq[i],list):
seq.__setslice__(i,i+1,seq[i])
i+=1
return seq
seq = [[1,2],[3],[],[4,[5,6]]]
"""
import timeit
t = timeit.Timer("flatten(seq)",init_a)
print 'recursive flatten:',t.timeit()
import timeit
t = timeit.Timer("flatten(seq)",init_b)
print 'flatten in place-non recursive:',t.timeit()
import timeit
t = timeit.Timer("flatten(seq)",init_c)
print 'recursive-no copies:',t.timeit()
import timeit
t = timeit.Timer("flatten(seq)",init_d)
print 'smallest recursive:',t.timeit()
import timeit
t = timeit.Timer("flatten(seq)",init_e)
print 'non-recursive flatten in place without copies:',t.timeit()
#--------------------------------------------
The results on Python 2.3.5: (maybe someone can try it on 2.4)
recursive flatten: 23.6332723852
flatten in place-non recursive: 22.1817641628
recursive-no copies: 30.909762833
smallest recursive: 35.2678756658
non-recursive flatten in place without copies: 7.8551944451
A 300% improvement!!!
This shows the value of avoiding copies, recursion, and extra function
calls.
Cheers,
Ron Adam
Ron Adam
> Ok... How about a non-recursive flatten in place? ;-)
>
> def flatten(seq):
> i = 0
> while i!=len(seq):
> while isinstance(seq[i],list):
> seq.__setslice__(i,i+1,seq[i])
> i+=1
> return seq
>
> seq = [[1,2],[3],[],[4,[5,6]]]
> print flatten(seq)
>
> I think I'll be using the __setslice__ method more often.
This is probably the more correct way to do it. :-)
def flatten(seq):
i = 0
while i!=len(seq):
while isinstance(seq[i],list):
seq[i:i+1]=seq[i]
i+=1
return seq
Steven D'Aprano
> On Tuesday 05 July 2005 08:17 am, Steven D'Aprano wrote:
>> Sorry, but you are mistaken. "lambda" is a _reserved_ word in the
>> Python language, while "function" etc are not, but they are certainly
>> part of the language. Try explaining what def and import do without
>> using the words "function" or "module". Maybe you can do it, using
>> circumlocutions, but it isn't easy, and costs clarity.
>
> This is still a relevant distinction. One relevant point is that I am
> perfectly free to use, say, the Spanish or Chinese word to describe
> "module" or "function", but the keywords "def" and "import" will
> remain the same.
No, that isn't relevant. That's just localisation. I'm also free to fork
Python and change the keywords.
[snip]
> The "Python sense" is not arbitrary. There are very direct visual or
> logical (i.e. INTUITIVE) connections between these words' *English*
> meanings (not just mathematical, either) and their meanings in Python.
I had NEVER even heard the word "tuple" before learning Python. I spent
weeks mispelling it as "turple", and I finally had to look it up in a
dictionary to see if it was a real English word. Out of the four English
dictionaries in my house, none of them have the word.
Let's dump tuple from the language too, its just "a stupid name for a list
that can't change". Agreed?
[snip]
> Similarly, "decorate" is 'make more attractive by adding ornament,
> colour, etc.' In Python, a "decorator" applies a wrapper to a function
> to provide it with some additional functionality.
Which is certainly NOT decoration, since the point of decoration is that
it is not functional! Plain bathroom tiles are functional. Decorating them
with leaves or flowers or patterns is not functional.
[snip]
> Now, if you are armed ONLY with the English definition, you will
> possibly run into some trouble, because the programming usage is a
> *specialization* of the term -- we strictly take only *one* of the
> English definitions to apply, and we narrow its meaning a bit.
Yes. Every specialization has its own jargon, and computer programming is
no different. One of the things new programmers have to learn is the
jargon.
[snip]
> "lambda" has no such advantage. Here's the *entire* gcide definition:
Nonsense. All that means is that lambda in the programming sense is too
specialized to make it into most ordinary dictionaries. Just like tuple.
[snip]
>> Think back to when you were a schoolboy at your first day of school.
>> Unless you had a very unusual upbringing, you probably had never heard
>> the word "function" before.
>
> Total BS. I knew the word "function" in it's English language sense,
> probably by the time I was 6. I *know* my kids know it.
Okay, maybe my memories of being five are faulty.
But if not six, then five, or four, or three, or two. At _some_time_
function was entirely unknown to you, and you had to just learn it.
[snip]
>> You had to learn that word, discover what it means, and then it becomes
>> familiar. You don't notice the process only because it happened so long
>> ago, at an age that your brain was operating in "language acquisition
>> mode" and picking up vocabulary at an incredible rate.
>
> If you're arguing that language is acquired rather than innate, you are
> bludgeoning an obvious point. The point is that *jargon* should ideally
> derive in a natural way from commonly-used language,
That's a bonus, sure. It is an important rule to follow, but not at the
expense of distorting the language:
t = immutable_list(L)
map(anonymous_function x: x+1, L)
versus:
t = tuple(L)
map(lambda x: x+1, L)
> if we want it to be
> easy to acquire for people who don't learn programming between the ages
> of 1 and 5 as we learn our native languages. Even in the 21st century,
> I think this includes just about all of us. ;-)
If they can't memorize one or two things, they aren't going to be much
good at programming no matter how easy the language is to use.
>> There is nothing about the word "string" that especially brings to mind
>> "an array of bytes representing characters". The analogy of "string of
>> characters" to "string of beads" breaks down as soon as you have
>> multiple lines of text.
>
> Ah, but that's useful. "Strings" AREN'T "multiple lines of text" in the
> computer's memory, are they? '\n' is just another bead. The "multiple
> lines" is a representation, or way of laying out the beads. Very useful
> distinction, and immediately driven by the choice of analogy.
Who cares about the implementation details of how the bytes are laid out
in the computer's memory? Unless you are programming in a low level
language like C or assembly, this is entirely irrelevant. You have
characters laid out along lines, and lines laid out in a second dimension.
The natural way to work with text is something like this:
for line in text:
for word in line: # or character
do_something()
[snip]
>> I won't say that the anonymous function meaning of lambda comes to my
>> mind before the Greek letter, but it isn't very far behind, and rapidly
>> catching up. (I use lambda a lot more than I speak Greek.) It wouldn't
>> surprise me if one day I think of Python programming before the Greek
>> letter, just as the world aleph brings to my mind the sense of infinity
>> before the sense of it being a Hebrew letter.
>
> Then it is clearly *not you* who should be served by the naming scheme.
> Anyone so deeply trained and experienced should be expected to adapt,
> you have the wherewithall to do so. It is the new user for whom the
> clarity of the jargon is so important.
>
> Personally, I find the term "anonymous function" to be a whole lot
> clearer than "lambda" or "lambda function". Indeed, if asked what
> "lambda" means, my reply is it's a "stupid name for an anonymous
> function", and if my listener is less savvy "for a function that doesn't
> have a name, because you only use it once".
And any half-way experienced Python programmer will say, "What are you
talking about?"
add_one = lambda x: x+1
Yes, I know that the name "add_one" is not the same as the name for a
function when you use def, but the function still has a name. The
difference might be an important difference, but it is not one you care
about unless you are doing introspection.
> Having said that, I too will miss the *concept* of an anonymous
> function, although I wouldn't mind at all if its name changed, or if it
> were somehow integrated into the "def" keyword's usage.
Def would be short for ... defend? defile? defer? defame? default? deflect?
There's always *something* to learn. Why def instead of define? Because
"easy to write" beats "instantly obvious to a beginner", if the word is
used all the time and is easy to memorize.
> Using backticks
> or some other syntax delimiter also sounds promising,
Ew, perl :-(
> although we're sort of running out of them. ;-)
--
Steven.
Steven D'Aprano
> And besides, "def" isn't a "magic" word... it's an abreviation for
> "define"...
Really? I thought it was an abbreviation for "definition". As in,
"definition of MyFunc is..."
> I hope that any student who didn't understand a word as
> common as "define" wouldn't have graduated from our school.
How about tuple?
--
Steven.
Tom Anderson
> Tom Anderson wrote:
>
>> The trouble with these is that they make a lot of temporary lists -
>> George's version does it with the recursive calls to flatten, and Ron's
>> with the slicing and concatenating. How about a version which never
>> makes new lists, only appends the base list? We can use recursion to
>> root through the lists ...
>
> Ok... How about a non-recursive flatten in place? ;-)
How about, er, oh, i give up.
> def flatten(seq):
> i = 0
> while i!=len(seq):
> while isinstance(seq[i],list):
> seq.__setslice__(i,i+1,seq[i])
> i+=1
> return seq
>
> seq = [[1,2],[3],[],[4,[5,6]]]
> print flatten(seq)
>
> I think I'll be using the __setslice__ method more often.
Um, can't you just do slice assignment? Make that line:
seq[i:i+1] = seq[i]
> And the test:
Stupendous and timely work!
> The results on Python 2.3.5: (maybe someone can try it on 2.4)
>
> recursive flatten: 23.6332723852
> flatten in place-non recursive: 22.1817641628
> recursive-no copies: 30.909762833
> smallest recursive: 35.2678756658
> non-recursive flatten in place without copies: 7.8551944451
GAAAAH!
> A 300% improvement!!!
>
> This shows the value of avoiding copies, recursion, and extra function
> calls.
Specifically, it shows the value of avoiding extra function calls, since
my zerocopy version is slower than the copy-happy single-function
versions.
Also, there are some differences between the functions which constitute
potential hillocks on the playing field - i test flattenability by looking
for an __iter__ method, whereas other implementations mostly ask
"instanceof list?" (less general, and less in the spirit of duck typing,
IMNERHO). I'd argue that my decomposition into functions is this sort of
difference, too - a matter of style (good style!), not algorithm. So,
levelling those differences, and throwing in my non-recursive zerocopy
foolery, here's my take on it ...
# ----
# here be a quick reimplementation of timeit to time function objects
# no exec for me no siree bob
# all you need to know is that timeit(fn) gives you time taken to run fn
import sys
import time
TIMERS = {
"win32": time.clock
}
timer = TIMERS.get(sys.platform, time.time)
def timeit(fn, n=None):
if (n == None):
t = 0.1
n = 1
while (t < 1.0):
n = max(int((n * min((1.0 / t), 10))), (n + 1))
t = _timeit(fn, n)
else:
t = _timeit(fn, n)
return t / n
def _timeit(fn, n):
it = xrange(n)
t0 = timer()
for i in it:
fn()
t1 = timer()
return float((t1 - t0))
# there is real code now
# i've rewritten the functions to use uniform variable names
# and to use isiterable
def isiterable(obj):
return hasattr(obj, "__iter__")
def georges_recursive_flatten(seq):
return reduce(_accum, seq, [])
def _accum(a, item):
if isiterable(item):
a.extend(georges_recursive_flatten(item))
else:
a.append(item)
return a
def rons_nonrecursive_flatten(seq):
a = []
while seq:
while isiterable(seq[0]):
seq = seq[0] + seq[1:]
a.append(seq.pop(0))
return a
def toms_recursive_zerocopy_flatten(seq, a=[]):
if (isiterable(seq)):
for item in seq:
toms_recursive_zerocopy_flatten(item, a)
else:
a.append(seq)
return a
def toms_iterative_zerocopy_flatten(seq):
stack = [None]
cur = iter(seq)
a = []
while (cur != None):
try:
item = cur.next()
if (isiterable(item)):
stack.append(cur)
cur = iter(item)
else:
a.append(item)
except StopIteration:
cur = stack.pop()
return a
def devans_smallest_recursive_flatten(seq):
if (isiterable(seq)):
return sum([devans_smallest_recursive_flatten(item) for item in seq], [])
else:
return [seq]
def rons_nonrecursive_inplace_flatten(seq):
i = 0
while (i != len(seq)):
while (isiterable(seq[i])):
seq[i:(i + 1)] = seq[i] # setslice takes iterators!
i = i + 1
return seq
flattens = [
georges_recursive_flatten,
rons_nonrecursive_flatten,
toms_recursive_zerocopy_flatten,
toms_iterative_zerocopy_flatten,
devans_smallest_recursive_flatten,
rons_nonrecursive_inplace_flatten
]
seq = [[1,2],[3],[],[4,[5,6]]]
def timeflatten(flatten):
return timeit(lambda: flatten(seq))
def funcname(fn):
return fn.func_name
print zip(map(funcname, flattens), map(timeflatten, flattens))
# ----
The output (in python 2.3 on a 1.5 GHz G4 pbook with OS X 10.3) is:
[
('georges_recursive_flatten', 0.00015331475192276888),
('rons_nonrecursive_flatten', 0.00015447115513356376),
('toms_recursive_zerocopy_flatten', 0.00012239551614106925),
('toms_iterative_zerocopy_flatten', 0.00035910996630353429),
('devans_smallest_recursive_flatten', 0.00019606360118084218),
('rons_nonrecursive_inplace_flatten', 5.8524826144294404e-05)
]
So, my zerocopy flatten is, after all, faster than George, you and Devan's
copy-happy versions, although not by much, my iterative version is much,
much slower, and the winner is still your in-place flatten, which beats my
not-too-bad 122 microseconds with a scorching 58!
I guess setslice is a lot faster than i thought. How are python lists
implemented? Presumably not as straightforward arrays, where inserting a
bunch of items at the head would mean moving everything else in the list
along?
We really ought to do this benchmark with a bigger list as input - a few
thousand elements, at least. But that would mean writing a function to
generate random nested lists, and that would mean specifying parameters
for the geometry of its nestedness, and that would mean exploring the
dependence of the performance of each flatten on each parameter, and that
would mean staying up until one, so i'm not going to do that.
Steven Bethard
> mch...@gmail.com wrote:
>> So I'd say that it's a pretty obscure name that most people
>> wouldn't know.
>
> I can't believe that anybody with any computer science
> background doesn't know it.
Perhaps this reflects on the quality of education in the United States
;) but I managed to get a BS in Computer Science at the University of
Arizona without ever seeing the word lambda (in the programming
languages sense).
However, I only took one class in programming languages, and it was more
of a survey class than a theory class. When I did take a theory class,
here at University of Colorado at Boulder, they did, of course, cover
lambda calculus. But there was at least a year or two in which I would
have considered myself somebody "with any computer science background"
who wasn't familiar with lambda.
OTOH, I fully agree with Peter Hansen: "Really, the name is such a
trivial, unimportant part of this whole thing that it's hardly worth
discussing."
STeVe
George Sakkis
> On Tue, 05 Jul 2005 09:46:41 -0500, Terry Hancock wrote:
[snip]
> > Having said that, I too will miss the *concept* of an anonymous
> > function, although I wouldn't mind at all if its name changed, or if it
> > were somehow integrated into the "def" keyword's usage.
>
> Def would be short for ... defend? defile? defer? defame? default? deflect?
>
> There's always *something* to learn. Why def instead of define? Because
> "easy to write" beats "instantly obvious to a beginner", if the word is
> used all the time and is easy to memorize.
Still it's hard to explain why four specific python keywords - def,
del, exec and elif - were chosen to be abbreviated, while all the rest
are full words (http://docs.python.org/ref/keywords.html). "Ease of
typing" is a joke for an excuse; "finally" is longer than
"define","delete" and equally long to "execute" and "else if" (the
latter has the even more serious flaw of going against TOOWTDI and in
this case practicallity hardly beats purity IMO). In any case, python
was never about minimizing keystrokes; theres another language that
strives for this <wink>.
So, who would object the full-word versions for python 3K ?
def -> define
del -> delete
exec -> execute
elif -> else if
George