[Python-ideas] Proposal for function expressions
Chris Perkins
definition expression, similar (superficially) to Ruby's blocks.
The big problem with making "def" an expression is the indentation of
the function body. You can't just embed an indented block of code into
the middle of an expression. The solution I propose (inspired by Ruby)
is to allow a block of code to be appended to the end of an expression
in certain, limited circumstances. The block is simply syntactic sugar
for a local anonymous function definition which is passed as an
argument to the function call that it follows.
First, a simple example to give the broad strokes:
foo() do:
BODY
is equivalent to:
def ANON():
BODY
foo(ANON)
where the name ANON is a placeholder for illustration - it is never
actually bound in the local namespace.
Specifically, here is what I propose:
* A new syntactic construct, the "block", which consists of:
* the keyword "do" (alternative: "def", repurposed)
* an optional argument list
* a colon
* an indented block
* A block is only allowed immediately following a call (or indexing
expression? see below), on the same line as the closing right brace
* The block defines an anonymous local function which is passed as an
argument to the call that it follows.
* The block is passed as the last argument to the function by default,
but this can be changed by putting a placeholder, the character "&",
into the parameter list of function. This is best illustrated by
example:
# 1) In the following, foo recieves TWO arguments: 23 and a callable,
in that order.
foo(23) do:
pass
# 2) Here, foo also recieves TWO arguments: a callable and 23, in that
order.
foo(&, 23) do:
pass
Why "&"? No particular reason - just because Ruby uses "&" for
something similar (but not the same).
Why do we need the "&" feature? Consider this example:
map(&, my_list) do(item):
return do_something_to(item)
This also works for keyword arguments:
foo(a=3, b=&, c=7) do():
whatever()
To make this syntax work, we need several restrictions on where a
block is allowed. Intuitively, the rules are:
* If the line following "do" would normally be indented, then a block
is not allowed.
* If the line following the "do" would be one on which leading
indentation is insignificant, a block is not allowed.
To clarify, the first rule means that this is not allowed:
if foo() do:
# are we in the body of the block, or of the if?
The second rule means that this is not allowed:
bar(23, foo() do:
body_of_block()
) # closing brace of call to bar
Here are some properties of blocks that may not be immediately
obvious:
* Blocks are a feature of the call site, and do not affect function
definitions. In other words, there is no such thing as a "function
expecting a block", as there is in Ruby. From the callee's point of
view, he has simply been passed a callable as an argument.
* Blocks can be used with any existing callable that expects to be
passed a function as one of its arguments.
OK, let's move on to the fun part: Motivating Examples.
###################
# Networking
dfr = twisted.whatever(...)
dfr.addCallback() do(result):
handle(result)
dfr.addErrback() do(err):
handle_err(err)
###################
# GUI (for some hypothetical library)
b = my_widget.add_button('Go')
b.on('click') do(evt):
if evt.ctrl_key:
do_something()
else:
do_other_stuff()
###################
# Routing HTTP requests
map.match('/') do(req):
return render_home(req)
map.match('/about') do(req):
return render_about(req)
map.match(/(?P<controller>\w+)\/$/) do(req, **kw):
return handlers[kw['controller']](req)
###################
# Threads
thread.start_new() do:
do_some_work()
###################
# Reduce the temptation to cram too much into a lambda.
# Sort a list of filenames that look like this:
# Xyz-1.1.3.tgz, acbd-4.7.tgz, Xyz-1.2.5.tgz, ...
my_list.sort(key=&, reverse=True) do(item):
name, _ = os.path.splitext(item)
root, _, version = name.partition('-')
parts = version.split('.')
return (root.lower(),) + tuple(map(int, parts))
stuff = filter(&, my_list) do(item):
# some code here...
return pred(item)
stuff = re.sub(rx, &, s) do(m):
t = m.group(0)
if cond(t): return t.upper()
else if othercond(t): return t.lower()
else: return ''
###################
# Faking switch
switch(expr) do(case):
case(int) do:
stuff()
case(str, bytes) do:
other_stuff()
switch(value) do(case):
case[0:10] do(val):
handle_small(val)
case[10:100] do(val):
handle_medium(val)
case.default() do(val):
handle_big(val)
###################
# Overloaded functions (adapted from PEP 3124)
@overload
def flatten(when):
when(basestring) do(ob):
yield ob
when(Iterable) do(ob):
for o in ob:
for ob in flatten(o):
yield ob
when.otherwise() do(ob):
yield ob
# later:
flatten.when(file) do(f):
for line in f:
yield line
###################
# Sort-of a "with(lock)" block, but running the body in a thread.
pool.run_with(my_lock) do:
# lock is held here
work()
other_work() # in parallel
###################
Well, that should give you the general idea.
Some final random points:
* As proposed, blocks are a new kind of beast in Python - an indented
block that is part of an expression, rather than being a statement. I
think that by constraining a block to be the last thing in an
expression, this works out OK, but I may be missing something
* Since writing this, I have started to lean towards "def" as the
keyword, rather than "do". I used "do" in the examples mostly to make
them look like Ruby, but I suspect that making Python "look like Ruby"
is not a design goal for most people :)
* Terminology: "block"? "def expression"? "anonymous def"? "lambda++"?
* Idea: Could we relax the rules for where a block is allowed,
removing the constraint that it follows a call or indexing expression?
Then we could do, for example, this:
def f():
return def(args):
pass
* Would a proliferation of nameless functions be a nightmare for
debugging?
Implementation is here: http://bitbucket.org/grammati/python-trunk/
Patch is here: http://bugs.python.org/issue6469
To be honest, I'm not entirely sure if the patch is in the right
format - it comes from hg, not svn.
Chris Perkins
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas
Terry Reedy
> I have a proposal for a language feature - a limited form of function-
> definition expression, similar (superficially) to Ruby's blocks.
>
> The big problem with making "def" an expression is the indentation of
> the function body. You can't just embed an indented block of code into
> the middle of an expression. The solution I propose (inspired by Ruby)
> is to allow a block of code to be appended to the end of an expression
> in certain, limited circumstances. The block is simply syntactic sugar
> for a local anonymous function definition which is passed as an
> argument to the function call that it follows.
>
> First, a simple example to give the broad strokes:
> foo() do:
> BODY
>
> is equivalent to:
>
> def ANON():
> BODY
> foo(ANON)
Add del anon and the equivalence is almost exact.
Call all one-off functions '_' and there is no need to ever delete the
name and you have
def _(): body
foo(_)
which is nearly identical to your proposed foo(&).
In any case, ideas similar to this have been proposed and discussed ad
nausem and rejected. Guide decided for Python 3 to neiher delete
function expressions, which he seriously considered, nor to expand them.
Feel free to peruse the archives and PEPs.
Terry Jan Reedy
Jim Jewett
> First, a simple example to give the broad strokes:
> foo() do:
> BODY
> is equivalent to:
> def ANON():
> BODY
> foo(ANON)
So the named and anoymous functions don't share scope in any way?
Then what is the advantage? Is putting the call ahead of the def that
valuable for making the code clear?
> Implementation is here: http://bitbucket.org/grammati/python-trunk/
> Patch is here: http://bugs.python.org/issue6469
> To be honest, I'm not entirely sure if the patch is in the right
> format - it comes from hg, not svn.
If using hg for patches is a problem, then the conversion from svn to
hg is in for some rough sledding.
-jJ
Chris Perkins
> So the named and anoymous functions don't share scope in any way?
>
> Then what is the advantage? Is putting the call ahead of the def that
> valuable for making the code clear?
Yes, exactly - or at least, I think it is. I have found that putting
the call before the def adds a surprising amount readability. I came
to this conclusion from doing primarily JavaScript for the last couple
of years - there, you have a choice of whether to predefine a local
function, or to put one inline. eg:
var callback = function(result) {
// 10 lines of code...
};
make_ajax_call(url, callback);
vs.
make_ajax_call(url, function(){
// 10 lines of code...
});
I have come to hate the former style, and find myself frequently
refactoring it into the latter, because the main thrust of what this
code does is summed up by "make_ajax_call(url, ..)", so that is what I
want to see first - not way down at the end, like an afterthought.
In other words, I think code that puts things in the opposite order
from the way you think of them is harder to read.
Another example: if I'm thinking that what I want to do is "substitute
some stuff in a string", then I want to start by typing/reading
"re.sub(...)", and not "def some_made_up_name(...):"
re.sub(pat, &, s) do(m):
# several lines of code
vs.
def temp(m):
# several lines of code
re.sub(pat, temp, s)
The latter relegates the key line of code to the very last line,
making it harder to see at a glance what it does.
So yes, the point of this really is just to allow you to write code
"in the right order".
I guess I should have made all this clearer in my original email :)
Chris Perkins
Jan Kaliszewski
It's my first post in the list, so let me introduce myself. My name is Jan
Kaliszewski, I am a composer (studied in Frederic Chopin Academy of Music
in Warsaw) and a programmer (currently working in Record System company on
Anakonda -- ERP system being programmed in Python).
Comming to the matter...
13-07-2009 Chris Perkins <chrispe...@gmail.com>:
> In other words, I think code that puts things in the opposite order
> from the way you think of them is harder to read.
>
> Another example: if I'm thinking that what I want to do is "substitute
> some stuff in a string", then I want to start by typing/reading
> "re.sub(...)", and not "def some_made_up_name(...):"
>
> re.sub(pat, &, s) do(m):
> # several lines of code
>
> vs.
>
> def temp(m):
> # several lines of code
> re.sub(pat, temp, s)
>
> The latter relegates the key line of code to the very last line,
> making it harder to see at a glance what it does.
>
> So yes, the point of this really is just to allow you to write code
> "in the right order".
I like the idea of '&' (or another character/word in that role) -- but
maybe it'd be enough to extend an existing feature: decorators with it,
and to add possibility to use that special character/word as a function
name after 'def'?
E.g.:
###################
# Networking
dfr = twisted.whatever(...)
# @dfr.addCallback(&) could be equivalent
@dfr.addCallback
def &(result):
handle(result)
@dfr.addErrback
def &(err):
handle_err(err)
###################
# GUI (for some hypothetical library)
@my_widget.add_button('Go').on('click', &)
def &(evt):
if evt.ctrl_key:
do_something()
else:
do_other_stuff()
# or, if we want to keep the result:
@my_widget.add_button('Go').on('click', &)
def button_clicked(evt):
if evt.ctrl_key:
do_something()
else:
do_other_stuff()
###################
# Threads
@thread.start_new:
def &():
do_some_work()
Best regards,
Jan Kaliszewski
Oleg Broytmann
> var callback = function(result) {
> // 10 lines of code...
> };
> make_ajax_call(url, callback);
>
> vs.
>
> make_ajax_call(url, function(){
> // 10 lines of code...
> });
For me, the latter is unreadable (it's hard to see where is the inner
block, and where is the outer; where are the make_ajax_call's arguments and
where are the function's arguments and body) even if the function is 1-2
line(s) long; if the function's body is longer the style becomes completely
unacceptable for my not so sharp eyes and brain.
Oleg.
--
Oleg Broytmann http://phd.pp.ru/ p...@phd.pp.ru
Programmers don't die, they just GOSUB without RETURN.
Mathias Panzenböck
+0.5
-panzi
Nick Coghlan
> Chris Perkins wrote:
>> First, a simple example to give the broad strokes:
>> foo() do:
>> BODY
>>
>> is equivalent to:
>>
>> def ANON():
>> BODY
>> foo(ANON)
>
> Add del anon and the equivalence is almost exact.
> Call all one-off functions '_' and there is no need to ever delete the
> name and you have
>
> def _(): body
> foo(_)
>
> which is nearly identical to your proposed foo(&).
You can even use decorator notation to move the call before the function
body:
@foo
def thunk():
BODY
(Note that most code block ideas haven't actually made it to the PEP
stage - the only one along those lines that I can see in PEP 0 is PEP
359's "make" statement, which had more to do with metaclasses than code
blocks. That said, I recall assorted code block based ideas being thrown
around in the PEP 340/342/343/346 discussions that eventually lead to
the introduction of PEP 343's with statement and PEP 342's generator
enhancements)
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
David Stanek
>
> var callback = function(result) {
> // 10 lines of code...
> };
> make_ajax_call(url, callback);
>
> vs.
>
> make_ajax_call(url, function(){
> // 10 lines of code...
> });
>
This is common in Javascript, but still makes me cry. I would rather
see something like this:
function main() {
make_ajax_call(url, ajax_call);
}
function ajax_call() {
// 10 lines of code
}
This way ajax_call can be in a different file and can be used in other
code without duplication.
> I have come to hate the former style, and find myself frequently
> refactoring it into the latter, because the main thrust of what this
> code does is summed up by "make_ajax_call(url, ..)", so that is what I
> want to see first - not way down at the end, like an afterthought.
I think you are refactoring in reverse.
--
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
Carl Johnson
"Proposal for Ruby-style anonymous block functions (that don't kill
the indention)":
http://mail.python.org/pipermail/python-ideas/2008-November/002340.html
The real difference is that your proposal uses & (which is a bad idea,
since & already has a meaning: __and__) and in
foo() do:
BODY
it implicitly injects the BODY function into the args of foo, which
violates Explicit Is Better Than Implicit.
I agree though that sometimes rather than doing
def spam():
BODY
result = foo(spam)
you would rather do
result = foo(spam*) *To Be Defined on the next line, OK!
def spam():
BODY
because it reads better. As I understand it, this was the reason the
decorator syntax was invented. Basically,
@classmethod
def foo():
BODY
reads better than
def foo():
BODY
foo = classmethod(foo)
so decorators are just syntatic sugar to improve readability.
So, I agree with the basic motivation of your proposal, but I think
the specifics of it aren't as good as my old one (although, I *would*
think that, wouldn't I?) and from experience, I expect the BDFL to
come and give you -2 in a couple days if the thread keeps up…
In terms of refining my old proposal, it strikes me that ! isn't used
in Python currently. If we have to use a sigil of some sort, we could
try:
new_list = sorted(old_list, key=!keyfunc)
def keyfunc(item):
normalize data…
or something to that effect with the general semantic being that !
means "here comes a variable name, I will define this variable on the
next line, please wait for it!"
2-cents-ly-yrs,
-- Carl Johnson
Chris Perkins
Johnson<cmjohnson....@gmail.com> wrote:
> This idea seems to be almost identical to my thread from November,
> "Proposal for Ruby-style anonymous block functions (that don't kill
> the indention)":
> http://mail.python.org/pipermail/python-ideas/2008-November/002340.html
Cool! I had not seen that thread, but the fact that you and I have
proposed nearly the same thing independently increases the odds that
I'm not just nuts. :)
> The real difference is that your proposal uses & (which is a bad idea,
> since & already has a meaning: __and__) and in
>
> foo() do:
> BODY
>
> it implicitly injects the BODY function into the args of foo, which
> violates Explicit Is Better Than Implicit.
I considered the use of a "magic" placeholder character to be an
unfortunate but necessary evil - the alternative seemed to be to
restrict the block/def-expression to becoming the last argument to the
function call that it follows, and that was just too restrictive. I
have no particular attachment to "&" - but note that there is no
ambiguity with the bitwise and operator, as far as I can tell. Eg.
this works:
>>> def f(a): return a()
...
>>> 3 & f(&) do: return 5
1
(not that you would write real code like that...)
I prefer not repeating the function name, I prefer putting the "do:"
(or "def:") on the same line (makes it clearer that the indented block
is effectively part of the expression that it follows), and I like the
fact that the function has no name - it tells you immediately that
"this function is only used within this one call - don't go looking
for uses of it later on, because they won't be there".
Other than that, it seems we are in perfect agreement on the feature.
It also seems that your earlier proposal met with the same rabid
ambivalence that mine has so far :)
A couple of interesting quotes from the discussion of your proposal:
Carl: "this proposal is not about adding power to Python, just making
things more readable"
Exactly!
Greg: "You could just allow any expression-statement to be followed by
a block, and sort out whether it makes sense later on."
That's exactly how I ended up implementing it.
Carl, if you still have any interest in this feature, you should try
out the patch - I think you'll be the first. :) Also I'll be happy to
give you commit privileges to the hg repo where I did the
implementation, in case you want to try to come up with some middle
ground between my proposal and yours:
http://bitbucket.org/grammati/python-trunk/
Failing that, it looks like our proposals will die the same
not-so-slow death for lack of interest. Oh well.
Chris Perkins
Paul Moore
> Failing that, it looks like our proposals will die the same
> not-so-slow death for lack of interest. Oh well.
OK, so I'm not accused of "lack of interest"... :-)
First of all, I have never felt the need for a construct like this. So
it adds nothing but complexity to the language for me. But assuming
that in the future, I do find a need for this, or there's a large
group of other users who would find this useful, please can you
address the following:
Your
fn() do (...):
stmt1
stmt2
is equivalent to
@fn
def _(...):
stmt1
stmt2
Why is your proposal better (ignoring "I like it better", as
subjective preferences other than Guido's don't get the language
changed)?
For the case of function calls with arguments, why is
fn(a, b, c) do(...):
stmt1
stmt2
better than
@functools.partial(fn, a, b, c)
def _(...):
stmt1
stmt2
(note that I could make my version look a bit nicer by a "from
functools import partial", so you're still not allowed to argue
aesthetics :-))?
That leaves your general cases where you stick an "&" somewhere in the
call as a placeholder. Only a couple of your motivating examples used
that form, and to be honest I didn't find them very convincing. By the
time you're getting that complex, I really do think you're trying to
cram too much into one statement (and I know that's a subjective
statement, sorry :-))
The same applies to the "x = fn(...) do (...)" cases that you include
in the example, but don't explicitly state in your specification. I
don't see any objective reason to think your form is clearer - yes,
"put the assignment, which is the main thing, up front" is an argument
in your favour, but "don't put too many things together just to avoid
thinking of a name" is an equally valid against.
OK, I hope that helps. I'm still against the idea, but I hope this
gives you some idea why - it's not simply indifference.
A construct like this needs to bring something better than mere
improved readability to the language, in my view.
Paul.
MRAB
Another possibility is to permit a local name (it's not an anonymous
function any more!) so that you can have multiple functions:
3 & f(foo) def foo:
return 5
3 & f(foo, bar) def foo, bar:
return 5
and:
return 6
Anyway, how would you handle condition expressions, as used in 'if' and
'while' statements? /They/ also expect a terminating colon and an
indented block.
Chris Perkins
>
> Anyway, how would you handle condition expressions, as used in 'if' and
> 'while' statements? /They/ also expect a terminating colon and an
> indented block.
Python 2.7a0 (trunk, Jul 10 2009, 16:43:32) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> def f(a): return a()
...
>>> if f() do: return True:
File "<stdin>", line 1
if f() do: return True:
^
SyntaxError: invalid syntax
>>> while (f() do: return 1):
File "<stdin>", line 1
while (f() do: return 1):
^
SyntaxError: invalid syntax
>>>
Yeah, so the error message could be clearer :) But I think I have
covered all the bases. I think. That's why I'm hoping at least a few
people will apply the patch and find the edge cases that I have
inevitably missed.
To answer your question more specifically: the modification to the
grammar allows a block following a "simple_stmt" only. Here's the
relevant bit:
stmt: simple_stmt | compound_stmt
# additional restrictions enforced by the interpreter - only certain
# types of small_stmt are allowed to have a block attached:
simple_stmt: small_stmt (';' small_stmt)* (([';'] NEWLINE) | block)
block: 'do' [parameters] ':' suite
The cases you are concerned about are "compount_stmt"s, so a block is
not allowed.
Chris Perkins
Chris Perkins
> 2009/7/14 Chris Perkins <chrispe...@gmail.com>:
>> Failing that, it looks like our proposals will die the same
>> not-so-slow death for lack of interest. Oh well.
>
> OK, so I'm not accused of "lack of interest"... :-)
>
> First of all, I have never felt the need for a construct like this. So
> it adds nothing but complexity to the language for me. But assuming
> that in the future, I do find a need for this, or there's a large
> group of other users who would find this useful, please can you
> address the following:
>
> Your
>
> fn() do (...):
> stmt1
> stmt2
>
> is equivalent to
>
> @fn
> def _(...):
> stmt1
> stmt2
>
> Why is your proposal better (ignoring "I like it better", as
> subjective preferences other than Guido's don't get the language
> changed)?
I agree that decorators can handle some of the use cases - I
anticipated that argument, and it is indeed a valid one. Nevertheless,
I think that's a bit a perversion of the the purpose of decorators.
And as you point out below, decorators are significantly more limited
in what they can do. And besides that, "I like it better" :)
> For the case of function calls with arguments, why is
>
> fn(a, b, c) do(...):
> stmt1
> stmt2
>
> better than
>
> @functools.partial(fn, a, b, c)
> def _(...):
> stmt1
> stmt2
>
> (note that I could make my version look a bit nicer by a "from
> functools import partial", so you're still not allowed to argue
> aesthetics :-))?
Yes, again, that does work, but it's getting uglier (yes, I know,
subjectiveness again), and it forces the callable to be the last
argument, which is limiting.
> That leaves your general cases where you stick an "&" somewhere in the
> call as a placeholder. Only a couple of your motivating examples used
> that form, and to be honest I didn't find them very convincing. By the
> time you're getting that complex, I really do think you're trying to
> cram too much into one statement (and I know that's a subjective
> statement, sorry :-))
Indeed, like any language construct it could be used for evil. I
prefer to think of it as enabling the programmer to write clearer code
in some cases - trying to prevent him from writing unclear code is, in
my opinion, a lost cause. Some programmers will find a way to write
unclear code no matter what tools you give them.
> The same applies to the "x = fn(...) do (...)" cases that you include
> in the example, but don't explicitly state in your specification. I
> don't see any objective reason to think your form is clearer - yes,
> "put the assignment, which is the main thing, up front" is an argument
> in your favour, but "don't put too many things together just to avoid
> thinking of a name" is an equally valid against.
For me, avoiding thinking of a name was never the primary motivation -
putting things in the "right order" is. But I do believe that there
are situations where the context that a block of code sits in tells
far more about its purpose that a name ever could - in those
circumstances, a forced function name can be useless at best, and
distracting clutter at worst. For example:
y = sorted(x, key=&) do(item):
name = item.split('-')[1]
return name.upper()
I think that forcing that two-line block of code to have a name would
serve no purpose - it's abundantly clear what it does. I also think
that the current state of affairs encourages poorer readability - I
mean, admit it, today you would probably write that snippet like this:
y = sorted(x, key=lambda item: item.split('-')[0].upper())
which to me loses clarity both by being crammed into one line, and by
not being able to create the local variable "name", which tells you
quite a bit about why the the item is being chopped up the way it is.
> OK, I hope that helps. I'm still against the idea, but I hope this
> gives you some idea why - it's not simply indifference.
It does help - thank you for your comments, Paul.
> A construct like this needs to bring something better than mere
> improved readability to the language, in my view.
Aha! You said "improved readability"! So I'm going to conveniently
forget everything else you said and write your name in the "pro"
column. :)
Seriously, I guess this is where we differ - I think that improved
readability _is_ a sufficient goal unto itself. The only question is
whether other people (OK, one other person in particular) thinks it
improves readability, and if so, then by how much.
Chris Perkins
Paul Moore
> For me, avoiding thinking of a name was never the primary motivation -
> putting things in the "right order" is. But I do believe that there
> are situations where the context that a block of code sits in tells
> far more about its purpose that a name ever could - in those
> circumstances, a forced function name can be useless at best, and
> distracting clutter at worst. For example:
>
> y = sorted(x, key=&) do(item):
> name = item.split('-')[1]
> return name.upper()
>
> I think that forcing that two-line block of code to have a name would
> serve no purpose - it's abundantly clear what it does.
That's a very telling statement. To me, that code is utterly baffling.
(Seriously! Without analysing it line by line, it means nothing to
me). By contrast.
def uc_name(item):
name = item.split('-')[1]
return name.upper()
y = sorted(x, key=uc_name)
is a lot clearer to me. And it would be even better if I knew what was
going on with the splitting of items on dashes, and taking the second
bit, was all about. Then, I could give the function a more meaningful
name.
[[An admission here - when I first wrote that example, I spelled the
function name differently in the 2 places. So my "see how much better
my way is" example was wrong - in a way yours couldn't be! In my
defence, the error would have come up as soon as I ran the code, and I
could say that if I had thought of a better name it wouldn't have
happened - but nevertheless. In the spirit of friendly debate, I'll
give you that argument for your case free of charge :-)]]
My point is that *technically* I can see what the 2-line block of code
does - but without giving it a name, I don't have a clue what it's
*for*.
But I don't imagine we're ever going to agree on this. Let it stand
that (without implying any criticism - people differ) your idea of
readable code is confusing and difficult to follow for me. Hence my
distrust of readability arguments :-)
>> A construct like this needs to bring something better than mere
>> improved readability to the language, in my view.
>
> Aha! You said "improved readability"! So I'm going to conveniently
> forget everything else you said and write your name in the "pro"
> column. :)
Good try, but you don't win that easily :-)
I don't think this construct improves readability. Personally, I think
it *harms* readability (although I concede that like anything, it
could be used tastefully). What I was saying was that your *only*
argument was readability, and I don't think that is enough of an
argument in itself.
> Seriously, I guess this is where we differ - I think that improved
> readability _is_ a sufficient goal unto itself. The only question is
> whether other people (OK, one other person in particular) thinks it
> improves readability, and if so, then by how much.
It has to improve readability, uncontroversially, in enough cases to
justify the cost in terms extra of language complexity (both for
implementation, and definition/teaching).
To demonstrate the benefits, a good place to look is often the
standard library. If your proposed change can be demonstrated to
improve the code shipped with Python in the stdlib, in a number of
places, people will start to listen.
Also, apart from the basic cost of any change (the fundamental barrier
to entry, if you like) the cost of your change is higher because there
are obscure corner cases that aren't easy to explain - witness
questions here about interaction of this form with if statements, etc.
This is a unique construct in that it's the only indentation-sensitive
multi-line element of an expression. People don't have an intuition
about how it "should" work, because there's nothing similar to
extrapolate from. Ultimately, that's why this idea (in one form or
another) has been around for so long and never been accepted - nobody
has ever managed to come up with an idea that prompted the "oh, yes,
obviously!" reaction in people needed to confirm that it's the "right"
solution.
I'd like to think that one day, someone will come up with the ideal
answer which will finally address all these "extended lambda"/"code
block" needs. But I don't think it's happened yet.
Sorry :-)
Paul.
Steven D'Aprano
> For me, avoiding thinking of a name was never the primary motivation
> - putting things in the "right order" is. But I do believe that there
> are situations where the context that a block of code sits in tells
> far more about its purpose that a name ever could - in those
> circumstances, a forced function name can be useless at best, and
> distracting clutter at worst. For example:
>
> y = sorted(x, key=&) do(item):
> name = item.split('-')[1]
> return name.upper()
>
> I think that forcing that two-line block of code to have a name would
> serve no purpose - it's abundantly clear what it does.
I'm afraid I'm with Paul Moore on this one -- it's not clear to me
either.
The key=& seems like a mysterious Perlism -- what's the & operator doing
in there? (Admittedly, it *might* be nice to be able to write things
like reduce(&, alist) instead of reduce(operator.and_, alist). But
that's another story.) "do(item):" confuses me, because it looks like
the start of a do loop, but it isn't.
None of this is to say that I couldn't learn the proposed syntax, but in
my opinion it doesn't mesh well with existing Python constructs.
Did I understand correctly that you prohibited using `if`, `while` and
other block statements inside the do block? If so, then to my mind the
proposal is useless -- it's neither an expression, like the body of
lambda, nor a block, like the body of a function, but a freakish
chimera.
> I also think
> that the current state of affairs encourages poorer readability - I
> mean, admit it, today you would probably write that snippet like
> this:
>
> y = sorted(x, key=lambda item: item.split('-')[0].upper())
>
> which to me loses clarity both by being crammed into one line, and by
> not being able to create the local variable "name", which tells you
> quite a bit about why the the item is being chopped up the way it is.
If I were doing that operation more than once, I'd make a function:
def get_canonical_name(item):
return item.split('-')[0].upper()
and I'd give it a docstring and doctests. The function
get_canonical_name tells you *much* more about why the item is
processed the way it is than a mere local variable "name".
You'll note that I wouldn't bother using a local -- this is short
enough, and simple enough, that I don't need it. But if I only did it
once, then I see nothing wrong with the lambda version. If you're
worried about it being "crammed" into one line:
y = sorted(x, # use the canonical name as the sort key
key=lambda item: item.split('-')[0].upper()
)
works for me.
> Seriously, I guess this is where we differ - I think that improved
> readability _is_ a sufficient goal unto itself. The only question is
> whether other people (OK, one other person in particular) thinks it
> improves readability, and if so, then by how much.
Personally, the only time I've ever missed the ability to create
multi-line lambdas was when I had a series of code blocks like this:
try:
BLOCK
except exception, e:
process_error(e)
where the BLOCK was different in each one, and I really wanted to factor
out the common code into a function "process" and pass each BLOCK as an
argument:
while condition:
process(BLOCK1)
process(BLOCK2)
if something:
process(BLOCK3)
sort of thing.
To my mind, multi-line lambdas are really only useful for the ability to
factor out common code, not to avoid defining a function or spreading
lambdas out over multiple lines.
--
Steven D'Aprano
Carl Johnson
ambiguous between "insert block here" and "and these two things
together." I'd need to think about it some more to be sure. One
advantage of using & as a sigil is that it jives with what Rubyists
are used to. That said, I have to insist that what you've proposed as
f() do:
BLOCK
be written instead as
f(&) do:
BLOCK
Just magically inserting the block as the last argument to a call is
something I hate about Ruby, and I think their experience has shown
that it's a mistake, since it hurts flexibility. EIBTI, etc.
To those claiming this can all be done with decorators, see another
one of my old threads: "Allow Lambda Decorators"
http://mail.python.org/pipermail/python-ideas/2009-February/002787.html
It is true that instead of using
new_list = sorted(old_list, key=&) do item:
name = item.split("-")[0]
return name.lower()
one might do
@list_sorted(oldlist)
def new_list(item):
name = item.split("-")[0]
return name.lower()
with list_sorted defined as
def list_sorted(seq):
def inner(f):
return sorted(seq, key=f)
return inner
But the problem with this is that the reader of your code will see the
"@" and think "here comes the definition of a callable." It's only
after a lot of thinking that the reader will realize that new_list is
a list, not a callable. So, in terms of readability, using decorators,
though possible, is probably a mistake.
-- Carl Johnson
Carl Johnson
> The key=& seems like a mysterious Perlism -- what's the & operator doing
> in there? (Admittedly, it *might* be nice to be able to write things
> like reduce(&, alist) instead of reduce(operator.and_, alist). But
> that's another story.) "do(item):" confuses me, because it looks like
> the start of a do loop, but it isn't.
The thing about this debate that confuses me is why was the @
decorator ever approved in the first place? It's line-noise, and
worse, it's line-noise with a meaning completely different from any
other language, ever. On top of that if you do help(), you get nothing
[Ed: in Python <2.6. Good job, whoever fixed this!], and Googling is
worthless. When I first learned Python, I was very confused by
decorators--in part because I didn't know "decorator" was the term to
Google for when trying to figure out the meaning of "@"!
And yet, @ was approved, in spite of its worthlessness (it literally
adds no power to the language, just sugar) and strangeness because of
the perceived benefits to readability. And in my opinion, there are
similar cases here, where readability is improved by having the
definition of a function come after the expression to which the
function will be passed. Yet, there's a lot of resistance to this, and
I'm not entirely sure why. If it's just a problem with using "&" and
"do" other line-noise and keywords can be proposed to substitute for
it. $ has no meaning in Python, for example. If it's the lack of
function names and docstrings, there's no reason these
lambda-like-thingies can't have support for names and docstrings added
(as in my old proposal, for example). But I feel like there's some
deeper reason people don't think this is a readability win. What is
it?
-- Carl
Mike Meyer
Carl Johnson <cmjohnson....@gmail.com> wrote:
> Steven D'Aprano wrote:
>
> > The key=& seems like a mysterious Perlism -- what's the & operator doing
> > in there? (Admittedly, it *might* be nice to be able to write things
> > like reduce(&, alist) instead of reduce(operator.and_, alist). But
> > that's another story.) "do(item):" confuses me, because it looks like
> > the start of a do loop, but it isn't.
>
> The thing about this debate that confuses me is why was the @
> decorator ever approved in the first place? It's line-noise, and
> worse, it's line-noise with a meaning completely different from any
> other language, ever.
You just stumbled on it. What other language, ever, had something like
decorators? It's a magic syntax that performs magic on the following
function definition. An unusual symbol in an unusual location is a clue
that something unusual is going on.
For a more thorough discussion of the reason for the adopted syntax, read
the PEP: http://www.python.org/dev/peps/pep-0318/.
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Jim Jewett
> Steven D'Aprano wrote:
> The thing about this debate that confuses me is why was the @
> decorator ever approved in the first place?
It wasn't, for quite a while. I believe it was proposed for 2.2. I'm
sure it was proposed for 2.3. Applying it to classes as well as
functions was in the working patch -- but waited until another version
just in case.
> And yet, @ was approved, in spite of its worthlessness (it literally
> adds no power to the language, just sugar) and strangeness because of
> the perceived benefits to readability.
I have a bit of an unusual perspective; I ended up writing some drafts
of the PEP, but not because I wanted decorators. Rather, I wanted to
avoid a really bad syntax. (@ wasn't my favorite, but it is way
better than several of the earlier proposals -- particularly in terms
of what it doesn't assume about the future.)
Almost any syntax is a big improvement *for*the*relevant*use*cases*.
The concern is what it would do to python in general, and which future
alternatives it would cut off.
> And in my opinion, there are
> similar cases here, where readability is improved by having the
> definition of a function come after the expression to which the
> function will be passed.
Some of the examples looked like the order was an improvement.
But are there huge areas where this is needed all over the place?
(Extension frameworks such as Cocoa were mentioned for decorators.)
Is it useful across several types of programming? (I see it as
obvious for callbacks, but maybe not so clearcut elsewhere.)
What is the cost of not doing it? With decorators, that cost was
often delaying the wrapper for 20-30 lines, and then having to repeat
the function name three times, and then using a code style that does
confuse some people.
In this case, I think the cost of not doing it is more like "Think up
a name for the 2-3 line function, and then move the definition up
ahead of the usage." Moving the definition is annoying, but not so
horrible with small functions, and there is no need to repeat names.
(I suppose you could argue that there is one repetition, if the
function will never be used elsewhere.)
What would the proposed syntax cost? In the case of @, that cost was
reduced to burning a punctuation character, and explaining that
functions can be annotated -- and the annotation is more than a
comment.
For the currently proposed syntax, it confuses several issues with
suites (blocks). It sounds like the parser can probably handle it,
but it isn't as clear that humans can. It might well preclude later
block extensions, including more powerful versions, such as thunks or
macros. (And it still burns a punctuation character anyhow.)
> Yet, there's a lot of resistance to this, and
> I'm not entirely sure why. If it's just a problem with using "&" and
> "do" other line-noise and keywords can be proposed to substitute for
> it.
Yes. Or "using:", of course. The catch is that those are all so
generic that this is only one of a zillion possible meanings -- and
for most people, not the most obvious. A more specific keyword might
have a better chance. (Not a *good* chance -- there is quite a high
bar for keywords -- but still *better*.)
> $ has no meaning in Python, for example.
I think it starts to with string templates.
(And I think "!" is used by some popular extensions, such as scipy,
and I think @ interfered with the leo editor, and ...)
I'm not saying it is ambiguous to the parser, but the universe of
unused ascii-punctuation isn't really all that huge.
> If it's the lack of
> function names and docstrings, there's no reason these
> lambda-like-thingies can't have support for names and docstrings added
> (as in my old proposal, for example).
That wouldn't help unless people actually used the names and
docstrings -- which again brings up the question about why existing
use of functions as first class objects isn't enough.
> But I feel like there's some
> deeper reason people don't think this is a readability win. What is
> it?
Cost-benefit. It is harder to see the costs, because they're spread
out thinly all over the language, while the benefits are concentrated.
But those costs are important.
In this case, it looks like the benefits are smaller than for
decorators, and the costs (at least with the proposed syntax) are
higher. And decorators were not a shoe-in.
-jJ
Stephen J. Turnbull
> The thing about this debate that confuses me is why was the @
> decorator ever approved in the first place?
Decorators serve the same kind of purpose that define-syntax does in
Scheme, and are similarly hygenic (which is a technical term in Scheme
that more or less means "the rope is too short to get all the way
around your neck"). They are *not* worthless to me; they allow
boilerplate to be given a name, and to be applied to transform the
behavior of a function at definition time rather than invocation time.
By giving them a syntactic implementation, it becomes possible and
encouraged to associate them with the declaration of a function,
rather than placing them at some arbitrary point later in the file.
(I can just see somebody inventing a style where all the decorators
are defined and applied at the very end of the file, for example.)
Agreed, that said, "@" doesn't evoke any association with the concept
of decoration, but that's a problem of documentation. So I guess you
had to be there to understand it....
> And yet, @ was approved, in spite of its worthlessness (it literally
> adds no power to the language, just sugar)
But that is true of *all* operators. Lisp does just fine with no
operators at all.
> and strangeness because of the perceived benefits to readability.
> And in my opinion, there are similar cases here, where
> readability is improved by having the definition of a function come
> after the expression to which the function will be passed.
Right. That's your opinion, and as you've seen the majority of those
posting to this thread disagree that the examples given are more
readable. Even more distressing, all though he hasn't posted, GvR
almost certainly would be included in them.
On the other hand, most Pythonistas agree that
@compute_some_boilerplate(variant)
def foo ():
# 20 lines of code
is more readable than
def foo ():
# 20 lines of code
foo = compute_some_boilerplate(foo, variant)
> But I feel like there's some deeper reason people don't think this
> is a readability win. What is it?
Some people feel that giving names to a cognitive chunk is an
essential part of the process of chunking. They are made
uncomfortable, and often confused, by code which is named only by its
own source (aka "blocks"). Others don't feel that way. The former
oppose this kind of proposal strongly, the latter support it more or
less actively.
Guido apparently is in the former camp. But I think the deciding
factor most likely is that it is the opinion of the BDFL that the harm
done by allowing blocks to the cognition of the "please give it a
name" group is much greater than the convenience to the "I don't need
a name when I've got the code in front of me" group.
George Sakkis
> 2009/7/14 Chris Perkins <chrispe...@gmail.com>:
>>
>> y = sorted(x, key=&) do(item):
>> name = item.split('-')[1]
>> return name.upper()
>>
>> I think that forcing that two-line block of code to have a name would
>> serve no purpose - it's abundantly clear what it does.
>
> That's a very telling statement. To me, that code is utterly baffling.
> (Seriously! Without analysing it line by line, it means nothing to
> me).
Seconded, there's too much going on ('&', what's "do", a function ?,
what's "item" ?, etc.). FWIW the only readable suggestion for
multiline lambdas with almost zero learning curve I have seen is the
one implemented in Boo, anonymous def and regular indentation:
y = sorted(x, key=def (item):
name = item.split('-')[1]
return name.upper()
)
There must be somewhere posted the reason this was rejected but my
google-fu is failing me.
George
[1] http://boo.codehaus.org/Closures
Chris Rebert
> On Tue, Jul 14, 2009 at 3:16 PM, Paul Moore<p.f....@gmail.com> wrote:
>
>> 2009/7/14 Chris Perkins <chrispe...@gmail.com>:
>>>
>>> y = sorted(x, key=&) do(item):
>>> name = item.split('-')[1]
>>> return name.upper()
>>>
>>> I think that forcing that two-line block of code to have a name would
>>> serve no purpose - it's abundantly clear what it does.
>>
>> That's a very telling statement. To me, that code is utterly baffling.
>> (Seriously! Without analysing it line by line, it means nothing to
>> me).
>
> Seconded, there's too much going on ('&', what's "do", a function ?,
> what's "item" ?, etc.). FWIW the only readable suggestion for
> multiline lambdas with almost zero learning curve I have seen is the
> one implemented in Boo, anonymous def and regular indentation:
>
> y = sorted(x, key=def (item):
> name = item.split('-')[1]
> return name.upper()
> )
>
> There must be somewhere posted the reason this was rejected but my
> google-fu is failing me.
IIRC, it screws with the essential statement-expression dichotomy of Python:
http://unlimitednovelty.com/2009/03/indentation-sensitivity-post-mortem.html
Cheers,
Chris
--
http://blog.rebertia.com
Mike Meyer
George Sakkis <george...@gmail.com> wrote:
> Seconded, there's too much going on ('&', what's "do", a function ?,
> what's "item" ?, etc.). FWIW the only readable suggestion for
> multiline lambdas with almost zero learning curve I have seen is the
> one implemented in Boo, anonymous def and regular indentation:
>
> y = sorted(x, key=def (item):
> name = item.split('-')[1]
> return name.upper()
> )
>
> There must be somewhere posted the reason this was rejected but my
> google-fu is failing me.
It gets ugly when carried to - well, extremes is the word I want to
use, but is wanting two or three functional arguments inlined really
"extreme"? Likewise, what happens if you want to inline a functional
argument in an already inlined function?
The current proposal seems to avoid that - by limiting the number of
function parameters to 1, which is a pretty serious shortcoming.
Pretty much all such proposals fail on one of these two issues.
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
George Sakkis
> On Tue, Jul 14, 2009 at 8:50 PM, George Sakkis<george...@gmail.com> wrote:
>> On Tue, Jul 14, 2009 at 3:16 PM, Paul Moore<p.f....@gmail.com> wrote:
>>
>>> 2009/7/14 Chris Perkins <chrispe...@gmail.com>:
>>>>
>>>> y = sorted(x, key=&) do(item):
>>>> name = item.split('-')[1]
>>>> return name.upper()
>>>>
>>>> I think that forcing that two-line block of code to have a name would
>>>> serve no purpose - it's abundantly clear what it does.
>>>
>>> That's a very telling statement. To me, that code is utterly baffling.
>>> (Seriously! Without analysing it line by line, it means nothing to
>>> me).
>>
>> Seconded, there's too much going on ('&', what's "do", a function ?,
>> what's "item" ?, etc.). FWIW the only readable suggestion for
>> multiline lambdas with almost zero learning curve I have seen is the
>> one implemented in Boo, anonymous def and regular indentation:
>>
>> y = sorted(x, key=def (item):
>> name = item.split('-')[1]
>> return name.upper()
>> )
>>
>> There must be somewhere posted the reason this was rejected but my
>> google-fu is failing me.
>
> IIRC, it screws with the essential statement-expression dichotomy of Python:
> http://unlimitednovelty.com/2009/03/indentation-sensitivity-post-mortem.html
Thanks, this was it. The important part is Guido's quote "any solution
that embeds an indentation-based block in the middle of an expression
is unacceptable", so it's basically a design judgement, not an
implementation constraint.
George
Carl Johnson
> Seconded, there's too much going on ('&', what's "do", a function ?,
> what's "item" ?, etc.). FWIW the only readable suggestion for
> multiline lambdas with almost zero learning curve I have seen is the
> one implemented in Boo, anonymous def and regular indentation:
>
> y = sorted(x, key=def (item):
> name = item.split('-')[1]
> return name.upper()
> )
FWIW, I find that completely ugly and non-Python-esque (unpythonic
being perhaps too strong of criticism). :-D
Clearly this is an issue which has a lot of good arguments on both
sides, and since the final metric has to be "readability of code" it's
going to hard to make a knock-down argument for one side or the other.
I'm fine with the status quo for now: in Guido's gut we trust.
My suggestion for future would be language extenders is that they try
to solve a broader problem than just the "how do I make callbacks more
convenient" problem. Just solving that one problem by itself seems
unlikely to make it into the language, since it is pure "sugar" and
Pythoneers have a well justified wariness toward anonymous functions
(why not just name them and be done with it?). Perhaps someone should
take a look at the original "with" proposal, or some of the various
"do" and "block" proposals. One idea I think about sometimes is having
a more convenient way to do something along the lines of
metaclass.__prepare__ but for functions…
-- Carl Johnson
Greg Ewing
> My suggestion for future would be language extenders is that they try
> to solve a broader problem than just the "how do I make callbacks more
> convenient" problem.
One such extension might be a "where" block. Applied
to the current problem:
foo(f) where:
def f(x):
...
Benefits include:
* It's pretty much self-explanatory, as it builds on
the existing syntax for defining functions, and
"where" is used in a similar way in mathematics and
some existing programming languages.
* You get to give the function a meaningful name if
you want.
* It's not restricted to a single function argument:
def foo(f, g) where:
def f(x):
...
def g(y):
...
* It needn't be restricted to functions:
foo(x, y) where:
x = a * 42
y = b / 17
Drawbacks include:
* Execution happens out of order, although that's an
inherent feature of the original proposal as well.
There are precedents in the language already, such
as list comprehensions and conditional expressions.
* Allowing arbitrary statements in the block could possibly
be considered detrimental to code readability. To mitigate
that, the block contents could perhaps be restricted to
binding constructs only.
--
Greg
Paul Moore
> My suggestion for future would be language extenders is that they try
> to solve a broader problem than just the "how do I make callbacks more
> convenient" problem. Just solving that one problem by itself seems
> unlikely to make it into the language, since it is pure "sugar" and
> Pythoneers have a well justified wariness toward anonymous functions
> (why not just name them and be done with it?). Perhaps someone should
> take a look at the original "with" proposal, or some of the various
> "do" and "block" proposals. One idea I think about sometimes is having
> a more convenient way to do something along the lines of
> metaclass.__prepare__ but for functions…
I agree entirely. There's certainly *something* here worthy of
consideration, otherwise it wouldn't keep coming up. But there's
probably a more general construct (yes, I know, "fully general
anonymous functions" - maybe there's something *else*...?) lurking
underneath all of the proposals, and it's worth waiting until someone
discovers it.
The important principle that George quoted:
2009/7/15 George Sakkis <george...@gmail.com>:
> Thanks, this was it. The important part is Guido's quote "any solution
> that embeds an indentation-based block in the middle of an expression
> is unacceptable", so it's basically a design judgement, not an
> implementation constraint.
is key here - keep blocks and expressions distinct. Future proposals
should keep that in mind.
Maybe there's value in having a "language design principles" PEP where
pronouncements like this could be recorded, for the benefit of people
proposing new features?
Paul.
Carl Johnson
Nitpicking:
> * It's not restricted to a single function argument:
>
> def foo(f, g) where:
> def f(x):
> ...
> def g(y):
> ...
>
Surely, you meant
foo(f, g) where:
def f(x):
...
def g(y):
...
Ending a statement that ends in a ":" normally with a "where" is too
ambiguous and ought to be prohibited.
I need to think about this proposal some more, but it would help solve
the issue brought up on the list recently that [f(x) for f in fs if
f(x)] is inefficient and [i for i in (f(x) for f in fs) if i] is ugly.
new_list = [fx for f in fs if fx] where:
fx = f(x)
The "where" on the end of the clause does catch your eye and make you
think "OK, be aware, something is undefined on this line but will be
defined on the next."
OTOH, in
new_list = [i for i in xs
for j in ys if fx] where:
fx = f(x)
or
new_list = [i for i in [j for j in ys if fx?] if fx?] where:
fx = f(x)
Does the "where" get re-used with each loop of the inner list
comprehension or just the outer one?
What about
total = sum(count for item in l) where:
count = item.size if item.size > 0 else 1
It seems like for that case, we definitely want the "where" to apply
every time the generator expression loops though, but… Is that
feasible?
Hmm.
-- Carl Johnson
Paul Moore
> It seems like for that case, we definitely want the "where" to apply
> every time the generator expression loops though, but… Is that
> feasible?
To make the form follow the intent, the where block would need to be
inside the [...] - which clearly violates the "no indented block
structure within expressions" design rule.
My initial instinct was that the original form violated that rule, too
- but if you interpret the where block as a statement postfix, you're
(almost) OK. The (almost) bit is because the where block has to be
limited to simple (single-line) statements.
But with those restrictions, the semantics become
stmt where:
a
b
c
=>
a
b
c
stmt
with the proviso that variables defined in a,b,c are only available
within a, b, c and stmt.
But of course that immediately prompts the question:
x = 1
y = x where:
x = 2
print x
What does that print?
x = []
y = 1 where:
x.append(1)
print x
What about that?
x = []
y = x
z = 1 where:
x.append(1)
print x, y, x is y
And what about that???
And before someone points out Greg's proposal to limit the construct
to "binding statements", please define (clearly) what is a "binding
statement". And then explain how that helps if I replace x.append(1)
with _ = x.append(1) in the above...
I sort of like the idea, but it needs much better defined syntax and
semantics if it's going to work.
Paul.
Weeble
> From: Greg Ewing <greg....@canterbury.ac.nz>
> To: python...@python.org
> Subject: [Python-ideas] Where-statement (Proposal for function
> expressions)
> Message-ID: <4A5D8B63...@canterbury.ac.nz>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> Carl Johnson wrote:
>
>> My suggestion for future would be language extenders is that they try
>> to solve a broader problem than just the "how do I make callbacks more
>> convenient" problem.
>
> to the current problem:
>
> foo(f) where:
> def f(x):
> ...
I was just thinking of something similar:
in:
foo(f,a)
let:
def f(x):
...
a = ...
(Keywords open to debate.) It's a little verbose, but I dislike it
much less than all the proposals that introduce special sigils or do
weird things to the syntax of the code. Things I like about it:
there's a clear warning (the "in:") right at the start that the
statement (or block?) is special and relies on code that comes after;
the syntax isn't complex; both lets you provide a name and prevents it
leaking into places you don't want it; and the contents of both the
"in:" block and the "let:" block are normal Python. I'd expect the
behaviour to be something like this: execute the "let:" block in its
own nested scope, then execute the "in:" block in the regular local
scope but with the symbols defined by the "let:" block available. Does
that make some sort of sense?
Most of the things I like are achieved by Greg's "where:" block, the
only thing I don't like about it is tacking a keyword onto the end of
a statement, which feels a bit awkward to me. That may just be
aesthetics. I certainly like the "where:" idea a lot better than the
other proposals. It seems to achieve the most utility with the least
confusing syntax.
Chris Perkins
>
> The important principle that George quoted:
>
> 2009/7/15 George Sakkis <george...@gmail.com>:
>> Thanks, this was it. The important part is Guido's quote "any solution
>> that embeds an indentation-based block in the middle of an expression
>> is unacceptable", so it's basically a design judgement, not an
>> implementation constraint.
>
> is key here - keep blocks and expressions distinct. Future proposals
> should keep that in mind.
I think you may be misinterpreting - note Guido's use of the word
"middle". That is precisely why I proposed allowing blocks only at
the _end_ of expressions, and not in the middle. OTOH, I may just be
projecting the interpretation of that statement that suits me best. :)
To answer some of the other points that have been raised:
Steven:
>Did I understand correctly that you prohibited using `if`, `while` and
>other block statements inside the do block?
No, you misunderstood. What you are thinking of is the fact that the
test expression in an if block, or a while block, cannot have an
appended block. A block is really just syntactic sugar for a def - it
lacks only a name, and the ability to be decorated. It can have a
docstring, it can be a generator, etc.
Steven:
>Personally, the only time I've ever missed the ability to create
>multi-line lambdas was when I had a series of code blocks like this:
>
>try:
> BLOCK
>except exception, e:
> process_error(e)
>
>where the BLOCK was different in each one, and I really wanted to factor
>out the common code into a function "process" and pass each BLOCK as an
>argument:
>
>while condition:
> process(BLOCK1)
>process(BLOCK2)
>if something:
> process(BLOCK3)
>
>sort of thing.
Do you mean something like this?
def safely(body):
try:
body()
except exception, e:
process_error(e)
while condition:
safely() do:
BLOCK1
safely() do:
BLOCK2
if something:
safely() do:
BLOCK3
I don't claim that a pattern like that will work in all cases, because
the body of the blocks have their own local scope. If you need to set
a variable in the enclosing scope, it might not do what you want (but
we do have "nonlocal" now, so that helps).
Carl:
>That said, I have to insist that what you've proposed as
>
>f() do:
> BLOCK
>
>be written instead as
>
>f(&) do:
> BLOCK
>
>Just magically inserting the block as the last argument to a call is
>something I hate about Ruby
Sure, that's OK with me. I also expected the argument that the empty
parentheses after "do" should not be optional, for symmetry with def,
and that's fine with me to. I was just trying to keep the syntax as
"lightweight" as possible for the simplest cases.
Stephen:
>Some people feel that giving names to a cognitive chunk is an
>essential part of the process of chunking. They are made
>uncomfortable, and often confused, by code which is named only by its
>own source (aka "blocks"). Others don't feel that way. The former
>oppose this kind of proposal strongly, the latter support it more or
>less actively.
Agreed, and I do have a slight nagging worry that blocks could be
abused, in that people could use them in cases where a named function
would be better (and I agree that there will always be many such
cases). But then I remember that "we're all adults here", and that in
practice people can just as easily, and often do, give useless names
to small functions, and then I don't worry about it so much anymore.
Carl:
>If it's the lack of function names and docstrings
Python 2.7a0 (trunk, Jul 12 2009, 07:54:12) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> def f(b): return b.__doc__
...
>>> f() do:
... "Look, ma, I'm a docstring!"
...
"Look, ma, I'm a docstring!"
>>>
Overall, I think we're at the point where most of us agree that this
is a very subjective matter, and it comes down to taste. Guido has a
well-recorded history of saying (paraphrasing) "just write a def with
a name and quit whining!", so I suppose the odds for this proposal are
not great. But I have certainly enjoyed the discussion, so thanks to
everyone who has participated.
Chris Perkins
Daniel Stutzbach
total = sum(count for item in l) where:
count = item.size if item.size > 0 else 1
It seems like for that case, we definitely want the "where" to apply
every time the generator expression loops though, but… Is that
feasible?
total = sum(f(item) for item in lst) where:
def f(item):
return item.size if item.size > 0 else 1
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC
Daniel Stutzbach
One such extension might be a "where" block. Applied
to the current problem:
foo(f) where:
def f(x):
...
I like this proposal much more than any of the previous proposals. It has some of the flavor of Lisp's "let", but puts the important line first instead of at the end. In a nutshell, it says "define these symbols for this one line only". On the downside, for the common case of wanting to define a single function, the body of the function must be indented twice.
I suggest the following grammar and meaning, which would be referred to by the assignment, yield, return, and expression statements:
where_expression ::= expression_list "where" ":" suite | expression_list
It evaluates as follows.
[return|yield|x=] expression_list where:
suite
is roughly equivalent to:
def where_expression():
suite
return expression_list
[return|yield|x=] expression_list()
How about the following as additional syntactic sugar for the common one-function case?
x = blah(f) where def f(item):
body_of_f
Gerald Britton
foo(x,y) = myfunc(bar) where:
myfunc, bar = f(x), g(y) where:
f,g = func1, func2
assuming func1 and func2 were previously defined, otherwise another
"where" clause could follow. That way, introducing the "where" clause
effectively causes foo to be defined as a function. Applying that to
the previous example:
l = [i for i in myiterator if fi] where:
f = f(i)
if "myiterator" was previously defined, or even:
l = [i for i in myiterator if f(i)] where:
myiterator, f = (i for i in xrange(10)), bool
if "myiterator" was not previously defined.
> _______________________________________________
> Python-ideas mailing list
> Python...@python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>
--
Gerald Britton
Nick Coghlan
> (note that I could make my version look a bit nicer by a "from
> functools import partial", so you're still not allowed to argue
> aesthetics :-))?
Make that a "from functools import partial as do" and you get:
@do(fn)
def _(...):
stmt1
stmt2
@do(fn, a, b, c)
def _(...):
stmt1
stmt2
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
---------------------------------------------------------------
Aahz
>
> Maybe there's value in having a "language design principles" PEP where
> pronouncements like this could be recorded, for the benefit of people
> proposing new features?
Sure, although PEP 20 seems like an obvious spot.
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
"If you think it's expensive to hire a professional to do the job, wait
until you hire an amateur." --Red Adair
Jan Kaliszewski
15-07-2009, 09:55 Greg Ewing <greg....@canterbury.ac.nz> wrote:
> One such extension might be a "where" block. Applied
> to the current problem:
>
> foo(f) where:
> def f(x):
> ...
At first glance, +1 from me.
15-07-2009, 12:25 Paul Moore <p.f....@gmail.com> wrote:
> inside the [...] - which clearly violates the "no indented block
> structure within expressions" design rule.
>
> My initial instinct was that the original form violated that rule, too
> - but if you interpret the where block as a statement postfix, you're
> (almost) OK. The (almost) bit is because the where block has to be
> limited to simple (single-line) statements.
> stmt where:
> a
> b
> c
> =>
>
> a
> b
> c
> stmt
>
> with the proviso that variables defined in a,b,c are only available
> within a, b, c and stmt.
I don't like the direction of limiting content of the block to set of
singular simple statements. I suppose where-block should simply create a
separate nested scope.
So...
> x = 1
> y = x where:
> x = 2
> print x
>
> What does that print?
I tkink it should print 1 -- because, if where-block creates a separate
nested scope, its local variables shouldn't leak.
But there is another question: what is the value of y: 1 or 2? Or, more
genarally: should be the line ended with 'where' included *in* that nested
scope or not? I suppose it should, so the final value of y should be 2.
Another example:
a = 7
b = 8
x = 'X'
z = 'zzzzz'
foo(a, b, c("It's..."), x) where:
# a = input() in Python 3.x :)
a = raw_input(z) # -> zzzzz
try:
a = int(a)
except ValueError:
a = 0
def b(bar):
'very complex function'
print(a, bar)
else:
def b(bar):
'also very complex function'
print(bar, a, bar)
def c(*args, **kwargs):
'also also very complex function'
print(a, b) # -> 7 8
Best regards,
--
Jan Kaliszewski <z...@chopin.edu.pl>
Jan Kaliszewski
> I don't like the direction of limiting content of the block to set of
> singular simple statements. I suppose where-block should simply create a
> separate nested scope.
...Because IMHO an indented block suggests that the content is a sequence
of statements creating a piece of code without unnatural limitations (like
in class or function body).
*j
Jae Kwon
namespace management. i'd like temporary names to be discarded
automatically.
where:
def foo(x,y,z):
return x+y+z
do:
bar = foo(1,2,3)
print foo(1,2,3) // foo is undefined!
print bar // 6
you could create a local namespace w/ just the 'where' block.
where:
x = 1
y = x
print x // x is undefined!
Perhaps we could bind the variables and pass them around.
where my_locals:
x = 1
y = 2
print my_locals() // {'x': 1, 'y': 2}
print x // x is undefined!
- Jae
Chris Rebert
> On Wed, Jul 15, 2009, Paul Moore wrote:
>>
>> Maybe there's value in having a "language design principles" PEP where
>> pronouncements like this could be recorded, for the benefit of people
>> proposing new features?
>
> Sure, although PEP 20 seems like an obvious spot.
Also, PEP 3099, although it's more about prohibitions than principles:
http://www.python.org/dev/peps/pep-3099/
Cheers,
Chris
--
http://blog.rebertia.com
Scott David Daniels
...
> from functools import partial as do
>
^^^ The moral equivalent of add_callback(lambda x: function(x)) ^^^
> def _(...):
> stmt1
> ...
--Scott David Daniels
Scott....@Acm.Org
Carl Johnson
> Unlikely, but this is:
>
> total =3D sum(f(item) for item in lst) where:
> def f(item):
> return item.size if item.size > 0 else 1
Yes, that seems much more sensible than trying to repeat the where's
inside of list comp or gen exp.
Daniel Stutzbach wrote:
> How about the following as additional syntactic sugar for the common one-=
function case?
>
> x =3D blah(f) where def f(item):
> body_of_f
I like it.
Gerald Britton wrote:
> Why not just use the Haskell approach?
>
> foo(x,y) =3D myfunc(bar) where:
> =A0 =A0 =A0 =A0 =A0 myfunc, bar =3D f(x), g(y) where:
> =A0 =A0 =A0 =A0 =A0 =A0 =A0f,g =3D func1, func2
Because I've been looking at that for a couple minutes and still have
no idea what it is supposed to mean. :-( List comprehensions and
pattern matching are the only Haskell features that make any sense to
me.
Jan Kaliszewski wrote:
> I don't like the direction of limiting content of the block to set of sin=
gular simple statements. I suppose where-block should simply create a separ=
ate nested scope.
Yes. It strikes me that one of the problems with Ruby is that there
are different scoping rules for blocks, lambda, methods, procs, and
whatever else=85 It would be much easier for users if "where" blocks had
the exact same scoping rules as functions. Daniel Stutzbach's
suggested equivalence strikes me as exactly correct:
[return|yield|x=3D] expression_list where:
suite
is roughly equivalent to:
def where_expression():
suite
return expression_list
[return|yield|x=3D] where_expression()
-- Carl Johnson
Gerald Britton
myfunc = lambda x:f(x)
bar = lambda y:g (y)
where:
f = lambda x:getattr(x, 'func')
g = lambda y:str(y)
would translate to:
def foo(x,y):
return getattr(x, 'func')str(y)
Basically Haskell reads like math:
circle_area = pi * radius**2 where pi is 3.14159 and radius is the
radius of the circle
You unpack it as you go. You could do the same this way:
f = lambda x:getattr(x,'func')
g = lambda y:str(y)
bar = lambda y: g(y)
myfunc = lambda x: f(x)
foo(x,y) = lambda x,y: myfunc(x)(bar(y))
though this is just a toy example to give the general flavor.
--
Gerald Britton
Greg Ewing
> Surely, you meant
>
> foo(f, g) where:
> def f(x):
> ...
> def g(y):
> ...
Yes, that's what I meant.
> new_list = [fx for f in fs if fx] where:
> fx = f(x)
No, there would have to be a separate, analogous
extension to list comprehensions:
new_list = [fx for f in fs if fx where fx = f(x)]
or maybe
new_list = [fx where fx = f(x) for f in fs if fx]
--
Greg
Piet van Oostrum
>GB> foo(x,y) = myfunc(bar) where:
>GB> myfunc = lambda x:f(x)
>GB> bar = lambda y:g (y)
>GB> where:
>GB> f = lambda x:getattr(x, 'func')
>GB> g = lambda y:str(y)
>GB> would translate to:
>GB> def foo(x,y):
>GB> return getattr(x, 'func')str(y)
I don't think so. In the original code x and y after the where are only
used as bound variables in the lambda's so they are not the x and y of
the foo def. You could replace all of them by p and q without changing
the meaning.
If you replace all lambda x: fun(x) by the equivalent fun then you get
this:
def foo(x,y):
return getattr(str, 'func')
x and y are not used.
>GB> Basically Haskell reads like math:
>GB> circle_area = pi * radius**2 where pi is 3.14159 and radius is the
>GB> radius of the circle
>GB> You unpack it as you go. You could do the same this way:
>GB> f = lambda x:getattr(x,'func')
>GB> g = lambda y:str(y)
>GB> bar = lambda y: g(y)
>GB> myfunc = lambda x: f(x)
>GB> foo(x,y) = lambda x,y: myfunc(x)(bar(y))
Huh?
How do you come at this step? IMHO myfunc(bar) is not the same as
lambda x,y: myfunc(x)(bar(y)).
It would be the same as (lambda x: myfunc(x))(lambda y: bar(y))
--
Piet van Oostrum <pi...@cs.uu.nl>
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: pi...@vanoostrum.org
Gerald Britton
> How do you come at this step? IMHO myfunc(bar) is not the same as
> lambda x,y: myfunc(x)(bar(y)).
> It would be the same as (lambda x: myfunc(x))(lambda y: bar(y))
Possibly, I wrote it up in a hurry. Perhaps this one is clearer:
foo = lambda (x,y): myfunc(bar) where:
myfunc = getattr(x, 'func')
bar = str(y)
Should be the same as:
foo = lambda (x,y): getattr(x, 'func')(str(y))
The "where" clause is just syntactic sugar but can make some things
easier to follow. Even in the toy example above it is tough to
(visually) parse:
getattr(x, 'func')(str(y))
and some feel that breaking it down bit by bit is easier. So the
example could be written:
def foo(x,y):
myfunc = getattr(x, 'func')
bar = str(y)
return myfunc(bar)
It really comes down to a matter of taste. Should the programmer be
forced to define things in execution order (like the def foo, above)
or should there be freedom to define things when you absolutely need
to (the example with the "where" clause)?
Another example:
area(x) = pi * r ** 2 where:
import math
pi = math.pi
r = circle.get_radius()
where:
circle = Circle(object)
where:
class Circle(radius):
def __init__(self, radius):
self.radius = radius
def get_radius(self):
return self.radius
I know, it starts to look absurd after a while. However, I would use
it if available to make long expressions more readable. For example,
say I have a database class with many functions with descriptive names
that are a bit long. Let's set it up this way:
class MyDataBase(DB):
def __init__(self, path):
self.path = path
...
def get_records_with_key(self, key):
rec = db.get_by_key(key)
while rec:
yield rec
rec = db.get_next_by_key(key)
Now, in my code I might have a snippet:
employees = MyDataBase('employees.db')
....
smiths = [record for record in db.get_records_with_key('smith')]
If I has a "where" clause, I'd prefer to write this:
smiths = [record for record in stuff] where:
stuff = db.get_records_with_key('smith')
Of course there are other ways to write this in standard Python, but I
don't think that's the point of this thread. I think that adding a
"where" clause would make it possible to write deep list
comprehensions and other long expressions without lots of parentheses.
I suspect that I'm not alone in messing up either the placement or
number of parentheses or both.
Here's another one. Remember that a generator expression sometimes
needs parentheses and sometimes not? For example:
from itertools import chain
c = chain(i for i in [2,3,5,7,11]) # no parentheses around the generator
d = chain( (i for i in [2,3,5]) , (j for j in [7,11] ) # needs parentheses
It's easy to mess up the extra parentheses (I sure do (and did, above!))
This might be nicer:
d = chain(g1, g2) where:
g1 = (i for i in [2,3,5])
g1 = (j for j in [7,11])
Even though there are just as many parentheses, they're broken out and
easier to spot. Sure, you could define g1 and g2 ahead of time, but
if the "where" clause creates a new namespace like a function def in a
function, then g1 and g2 are just placeholders in the expression and
don't pollute the variable namespace of the calling context.
Anyway, I don't feel strongly either way. I see the benefits of
"where" clauses and I'd use them if they were available, but I can
certainly do without.
Chris Perkins
>
> d = chain(g1, g2) where:
> g1 = (i for i in [2,3,5])
> g1 = (j for j in [7,11])
>
> Even though there are just as many parentheses, they're broken out and
> easier to spot. Sure, you could define g1 and g2 ahead of time, but
> if the "where" clause creates a new namespace like a function def in a
> function, then g1 and g2 are just placeholders in the expression and
> don't pollute the variable namespace of the calling context.
This idea is really growing on me.
Here's another example - how do you calculate sample variance? You're
probably thinking "Um, well, I think it's the square root of the sum
of the squares minus the square of the sums". Great, that's a high-ish
level, intuitive description of the algorithm. The precise details of
how to go about calculating that are not important yet. So write it
that way:
def variance(data):
return sqrt(sum_of_squares - square_of_sums) where:
... elided ...
There - we've written the core of the algorithm at the same level of
abstraction as we think about it. The code matches our mental model.
Only then do we go about filling in the boring details.
def variance(data):
return sqrt(sum_of_squares - square_of_sums) where:
def sqrt(v): return v ** 0.5
sum_of_squares = sum(v**2 for v in data)
square_of_sums = sum(data) ** 2
The advantage is that now readers of the code, having developed some
intuition about what "where blocks" mean, can see at a glance that the
code at the first level of indentation is the important stuff, and the
indented block is the implementation details; the stuff that's at a
lower level of abstraction.
I think this construct would be a great way to enable SLAP - the Same
Level of Abstraction Principle (which I'm embarrasingly in love with).
Traditional SLAP says that all the code in a function should be at the
same level of abstraction - code at a lower level of abstraction
should be split out into other functions or methods. A refinement
could could state that code at a lower level of abstraction should be
either moved to another function, or indented in a where block.
> Anyway, I don't feel strongly either way. I see the benefits of
> "where" clauses and I'd use them if they were available, but I can
> certainly do without.
I would use them too, and I can't do without. I'm going to hold my
breath until we get them. ;)
As for the precise specification, I see only a few issues.
As a first approximation, assume that this code:
[LHS OP] EXPR where:
BODY
is expanded into the equivalent of this:
def TEMP():
BODY
return EXPR
[LHS OP] TEMP()
That seems like the most obvious way to get a nested scope to run the body in.
Problem 1) return
Since there will be an implicit return injected into the body of TEMP,
what happens if the user puts an explicit return in the body of the
block:
x = foo(y) where:
return 23
expands to
def TEMP():
return 23
return foo(y)
x = TEMP()
So return should be disallowed.
Problem 2) break/continue
for a in range(b):
x = foo(y) where:
break #or continue
expands to:
for a in range(b):
def TEMP():
break # or continue
return foo(y)
x = TEMP()
Which is an error. However:
x = foo(y) where:
y = []
for i in range(n):
if cond(i):
break
y.append(i)
must be allowed (right?). So we can't make a blanket restriction on
break/continue within the body.
Problem 3) yield
x = foo(y) where:
yield 3
expands to:
def TEMP():
yield 3
return foo(y)
x = TEMP()
which is an error. So yield must be disallowed.
I can't think of any other constructs that would need special
handling, and I'm sure the issues I have listed are solvable.
Chris Perkins
Paul Moore
> As for the precise specification, I see only a few issues.
>
> As a first approximation, assume that this code:
>
> [LHS OP] EXPR where:
> BODY
>
> is expanded into the equivalent of this:
>
> def TEMP():
> BODY
> return EXPR
> [LHS OP] TEMP()
So where constructs are only allowed for code of the form [LHS OP] EXPR?
What does that correspond to in the grammar? (If it doesn't correspond
to something at the moment, you'll need to extend the grammar to
define a "thing that can have a where clause" production...)
You've also disallowed
x[i] = 12 where:
i = some_complicated_expression()
Was that deliberate? If so, could you explain why now, so that it's on
record for the legions of people who will ask after it's implemented?
:-)
Paul.
Chris Perkins
>
> So where constructs are only allowed for code of the form [LHS OP] EXPR?
>
> What does that correspond to in the grammar? (If it doesn't correspond
> to something at the moment, you'll need to extend the grammar to
> define a "thing that can have a where clause" production...)
Without having thought too hard about it, I would allow a where block
after a "simple_stmt" - same thing I did with my "do block" proposal.
That means you could have a where after any of these:
small_stmt: (expr_stmt | print_stmt | del_stmt | pass_stmt | flow_stmt |
import_stmt | global_stmt | exec_stmt | assert_stmt)
So by EXPR above, I guess I was thinking of any "small_stmt"
production. Of course, it makes no sense for some of them, like
pass_stmt, global_stmt, etc, but I don't see any harm in allowing the
block to be there - I don't think I could have any effect though.
> You've also disallowed
>
> x[i] = 12 where:
> i = some_complicated_expression()
>
> Was that deliberate? If so, could you explain why now, so that it's on
> record for the legions of people who will ask after it's implemented?
> :-)
Well, my original answer was "no, I just didnt' think of that'. But
now that you mention it, I think in the case of an "augassign"
production, it makes sense to have the left hand side evaluated in the
current scope (not in the 'where block" scope). So in your examle,
you would either get a NameError for i, the code would do somthing
that you didn't expect.
Chris Perkins
Daniel Stutzbach
x[i] = 12 where:
i = some_complicated_expression()
Was that deliberate? If so, could you explain why now, so that it's on
record for the legions of people who will ask after it's implemented?
:-)
Because only the right-hand side is in the where-block's scope. If the left-hand side were in the where-block's scope, then:
x = 0
x = i where:
i = 5
print x
Would print "0" not "5".
George Sakkis
> This idea is really growing on me.
>
> Here's another example - how do you calculate sample variance? You're
> probably thinking "Um, well, I think it's the square root of the sum
> of the squares minus the square of the sums". Great, that's a high-ish
> level, intuitive description of the algorithm. The precise details of
> how to go about calculating that are not important yet. So write it
> that way:
>
> def variance(data):
> return sqrt(sum_of_squares - square_of_sums) where:
> ... elided ...
>
> There - we've written the core of the algorithm at the same level of
> abstraction as we think about it. The code matches our mental model.
> Only then do we go about filling in the boring details.
>
> def variance(data):
> return sqrt(sum_of_squares - square_of_sums) where:
> def sqrt(v): return v ** 0.5
> sum_of_squares = sum(v**2 for v in data)
> square_of_sums = sum(data) ** 2
>
> The advantage is that now readers of the code, having developed some
> intuition about what "where blocks" mean, can see at a glance that the
> code at the first level of indentation is the important stuff, and the
> indented block is the implementation details; the stuff that's at a
> lower level of abstraction.
I realize the appeal of this but I see two issues with it:
- What you call an advantage can be a disadvantage for many
programmers used to the standard bottom-up way most procedural and
object-oriented languages support. For better or for worse, thinking
sequentially feels more natural than in terms of top-down levels of
abstractions. Unless one is (or gets) used to this style, "where"
blocks would be the code equivalent of top-posting; it doesn't make
sense unless you read it backwards.
- TIMTOWTDI. In any case, bottom-up won't go away so for any
non-trivial piece of code one has to choose between two equivalent
ways of expressing it. Think of Perl's "STATEMENT (if|unless) (EXPR)".
I'm not saying that these are necessarily show stoppers but they
should be addressed to see whether the benefits are worth the added
complexity.
George
Chris Perkins
> - What you call an advantage can be a disadvantage for many
> programmers used to the standard bottom-up way most procedural and
> object-oriented languages support. For better or for worse, thinking
> sequentially feels more natural than in terms of top-down levels of
> abstractions. Unless one is (or gets) used to this style, "where"
> blocks would be the code equivalent of top-posting; it doesn't make
> sense unless you read it backwards.
That's interesting - I see it as exactly the opposite. For me,
bottom-up approach appears backwards, and I get annoyed by having to
read backwards. It clearly depends on what you're used to, I guess.
> - TIMTOWTDI. In any case, bottom-up won't go away so for any
> non-trivial piece of code one has to choose between two equivalent
> ways of expressing it. Think of Perl's "STATEMENT (if|unless) (EXPR)".
>
> I'm not saying that these are necessarily show stoppers but they
> should be addressed to see whether the benefits are worth the added
> complexity.
Yes, I'm very interested to see what proportion of people find the
top-down style more readable.
Chris Perkins
Jan Kaliszewski
> On Thu, Jul 16, 2009 at 10:06 AM, Paul Moore <p.f....@gmail.com> wrote:
>
>> x[i] = 12 where:
>> i = some_complicated_expression()
>>
>> Was that deliberate? If so, could you explain why now, so that it's on
>> record for the legions of people who will ask after it's implemented?
>> :-)
>
> Because only the right-hand side is in the where-block's scope.
Yeah. But I believe it should be possible using global/nonlocal keywords,
e.g.:
x[i] = 12 where:
global i
i = some_complicated_expression()
I think that many rules that apply for functions should also apply for
where-blocks. (But obviously not all of them, e.g. return, yield and such
stuff don't make sense)
16-07-2009, 18:16 Chris Perkins <chrispe...@gmail.com> wrote:
> Yes, I'm very interested to see what proportion of people find the
> top-down style more readable.
I think even for the same person it can strongly depend on particular
application. It'll be good for some things, and useless for other.
But it isn't a strong argument against the feature. IMHO it'll be
useful in many situations.
15-07-2009, 14:49 Daniel Stutzbach <dan...@stutzbachenterprises.com> wrote:
> How about the following as additional syntactic sugar for the common
> one-function case?
> x = blah(f) where def f(item):
> body_of_f
+1 from me also.
Cheers,
--
Jan Kaliszewski <z...@chopin.edu.pl>
Chris Perkins
>> How about the following as additional syntactic sugar for the common
>> one-function case?
>> x = blah(f) where def f(item):
>> body_of_f
>
> +1 from me also.
>
I'm not sold on that one yet, only because someone is inevitably going
to ask why they can't create a singleton-ish sort of thing like this:
thing = Thing() where class Thing(object):
def __init__(...): etc
Or maybe that should be allowed too?
Chris Perkins
Steven D'Aprano
> Here's another example - how do you calculate sample variance? You're
> probably thinking "Um, well, I think it's the square root of the sum
> of the squares minus the square of the sums". Great, that's a
> high-ish level, intuitive description of the algorithm. The precise
> details of how to go about calculating that are not important yet. So
> write it that way:
>
> def variance(data):
> return sqrt(sum_of_squares - square_of_sums) where:
> ... elided ...
That would actually be the standard deviation, not the variance.
> There - we've written the core of the algorithm at the same level of
> abstraction as we think about it. The code matches our mental model.
> Only then do we go about filling in the boring details.
>
> def variance(data):
> return sqrt(sum_of_squares - square_of_sums) where:
> def sqrt(v): return v ** 0.5
> sum_of_squares = sum(v**2 for v in data)
> square_of_sums = sum(data) ** 2
I don't know what you've calculated, but it's not the variance, or even
the standard deviation. The variance is actually the mean of the
squares of the deviations from the mean of the data. Written another
way, that's the mean of the squares of the data minus the square of the
mean. But putting all that aside, let's pretend that the formula you
gave is correct. It's certainly calculating *something* -- let's
pretend it's the variance.
I'm now going to try to convince you that this entire approach is
actually harmful and should be avoided. You said:
"...sum of the squares minus the square of the sums".
That's all very well, but what squares? What sums? You have
under-specified what the function should do. Is it squares of the first
fifty prime numbers? Sums of odd-positioned data? You can't just
pretend that this is an implementation detail -- you have to face up to
what the function actually does at the design phase.
If you had, you would have noted that the so-called "variance" is the
sum of the squares *of the data*, minus the square of the sums *of the
data*. The sums-and-squares are functions of data, not expressions or
constants, which suggests writing them as functions.
This gives us:
def variance(data): # for some definition of "variance"
return sqrt(sum_of_squares(data) - square_of_sums(data)) where:
...
This naturally suggests that they are (or could be) reusable functions
that belong in the module namespace, not repeated inside each function
that uses them:
def sqrt(v):
return v**0.5
def sum_of_squares(data):
return sum(v**2 for v in data)
def square_of_sums(data):
return sum(data)**2
These can now easily be documented and tested, which is a major win.
Having done this, the "where" statement is redundant:
def variance(data):
return sqrt(sum_of_squares(data) - square_of_sums(data))
Not just redundant, but actively harmful: it encourages the coder to
duplicate common code inside functions instead of factoring it out into
a single external function. This not only violates Don't Repeat
Yourself, but it makes it harder to test and document the code.
Now of course it's possible to inappropriately inline common code inside
functions without "where". For instance, I might have written:
def variance(data):
def sqrt(v):
return v**0.5
def sum_of_squares(data):
return sum(v**2 for v in data)
def square_of_sums(data):
return sum(data)**2
return sqrt(sum_of_squares(data) - square_of_sums(data))
which is a micro-pessimation (not only do I lose the opportunity to
re-use and test the internal functions, but I also pay the micro-cost
of re-creating them every time I call the function).
Or:
def variance(data):
sum_of_squares = sum(v**2 for v in data)
square_of_sums = sum(data)**2
return (sum_of_squares - square_of_sums)**0.5
In either case, writing the sums-and-squares before the return suggests
that these are common code best written as re-usable functions.
But "where" changes the focus of the coder away from factoring code
*out* of functions into placing code *inside* internal blocks. It
encourages the coder to think of sum_of_squares as a private
implementation detail instead of a re-usable component. And that is, I
believe, harmful.
--
Steven D'Aprano
Jan Kaliszewski
(in thread 'Immemorial desire for "do-while"-likeconstruction'):
> Steven D'Aprano wrote:
>> On Thu, 16 Jul 2009 08:50:06 am Jan Kaliszewski wrote:
>>> Hello,
>>>
>>> The issue has been coming back repeatedly:
>>
>> And not very long ago at that.
>
> So often that there's even a PEP* that documents the pros and cons of
> different ideas and points out that a viable syntactic challenger to the
> "while True with inner break" approach has yet to be found.
Maybe the where-statement could be a candidate? Consider an example:
while foo(x) and bar() where:
foo = SOMETHING
def bar():
SOMETHING ELSE
do:
THE
LOOP
BODY
or maybe better:
while foo(x) and bar():
where:
foo = SOMETHING
def bar():
SOMETHING ELSE
THE
LOOP
BODY
and Daniel's idea of one-def-shortcut also could apply:
while setup(x < y + 9 or bzzz):
where def setup(cond):
SETUP
ACTIONS
return cond
THE
LOOP
BODY
And what about for-loop?
for i in smartgen():
where def smartgen():
SOME ACTIONS
yield SOMETHING
THE
LOOP
BODY
Cheers,
--
Jan Kaliszewski <z...@chopin.edu.pl>
Jan Kaliszewski
> Not just redundant, but actively harmful: it encourages the coder to
> duplicate common code inside functions instead of factoring it out into
> a single external function.
But note that, if it's necessary, refactoring such code moving
appropriate parts e.g. into global scope could be done very easily.
> Now of course it's possible to inappropriately inline common code
> inside functions without "where".
Every language feature can be misused. More powerful feature -- larger
possibilities of misusing. But language is not for substituting
programmer's sense.
Cheers,
--
Jan Kaliszewski <z...@chopin.edu.pl>
Jan Kaliszewski
>
> while foo(x) and bar():
> where:
> foo = SOMETHING
> def bar():
> SOMETHING ELSE
> THE
> LOOP
> BODY
Another possible variant:
while foo(x) and bar() # no colon
where:
foo = SOMETHING
def bar():
SOMETHING ELSE
do:
THE
LOOP
BODY
But the former has such an advantage that where-block content is
indented deeper than loop body -- what reflects that it (where-block)
creates nested scope (contrary to loop body).
Chris Perkins
> On Fri, 17 Jul 2009 12:51:54 am Chris Perkins wrote:
>
>> def variance(data):
>> return sqrt(sum_of_squares - square_of_sums) where:
>> ... elided ...
>
> That would actually be the standard deviation, not the variance.
Hence my use of "Um, I think it's..." :)
Yeah, I couldn't remember if this was right, but as you say below,
it's not the point of the example, so I didn't bother to look it up.
> I'm now going to try to convince you that this entire approach is
> actually harmful and should be avoided. You said:
[ .. snip me being almost - but not quite - convinced ...]
>
> def variance(data):
> def sqrt(v):
> return v**0.5
> def sum_of_squares(data):
> return sum(v**2 for v in data)
> def square_of_sums(data):
> return sum(data)**2
> return sqrt(sum_of_squares(data) - square_of_sums(data))
This is the case that I was trying to present an alternative to. The
fact is, people _do_ write code like this. Sometimes pulling the local
stuff out into reusable functions is the right thing to do, but I
don't think it always is.
> In either case, writing the sums-and-squares before the return suggests
> that these are common code best written as re-usable functions.
> But "where" changes the focus of the coder away from factoring code
> *out* of functions into placing code *inside* internal blocks. It
> encourages the coder to think of sum_of_squares as a private
> implementation detail instead of a re-usable component. And that is, I
> believe, harmful.
But surely you agree that _sometimes_ there is code that really is an
implementation detail - not everything is reusable, or at least not
usefully reusable. Taking your argument to the extreme, every function
would be only one line of code. I prefer to trust programmers to make
the best use of the tools available to create good code.
In other words:
Jan Kaliszewski:
>Every language feature can be misused. More powerful feature -- larger
>possibilities of misusing. But language is not for substituting
>programmer's sense.
Yeah, ditto to what he said.
Chris Perkins
MRAB
When defining a function, any name that's the target of an assignment
will default to being local; other names will be non-local.
In a 'where' construct the rule could be that any name that's the
target of an assignment before the 'where' will be non-local; other
names will default to local if they are the target of an assignment
within the 'where' construct, but non-local otherwise.
For example:
bar = "outside"
foo = bar where:
bar = "inside"
'foo' is non-local. There's an assignment to 'bar' with the 'where'
construct, so 'bar' is local in the statement and hides the non-local 'bar'.
Therefore 'foo' becomes bound to "inside".
More clearly:
bar = "outside"
local:
nonlocal foo
bar = "inside"
foo = bar
Another example:
x[i] = 12 where:
i = some_complicated_expression()
'x' is non-local because there's no assignment to 'x' within the 'where'
construct'. 'i' is local because there is an assignment to 'i' within
the 'where' construct'.
More clearly:
local:
i = some_complicated_expression()
x[i] = 12
Paul Moore
>> You've also disallowed
>>
>> x[i] = 12 where:
>> i = some_complicated_expression()
>>
>> Was that deliberate? If so, could you explain why now, so that it's on
>> record for the legions of people who will ask after it's implemented?
>> :-)
>
> Well, my original answer was "no, I just didnt' think of that'. But
> now that you mention it, I think in the case of an "augassign"
> production, it makes sense to have the left hand side evaluated in the
> current scope (not in the 'where block" scope). So in your examle,
> you would either get a NameError for i, the code would do somthing
> that you didn't expect.
And yet the user's intent is obviously to use the i defined inside the
where (that's how functional languages work, so it's hardly an
unrealistic intent).
I'd argue that having this example use a previously defined value of i, when
x = i where:
i = some_complicated_expression()
uses the i defined in the where clause, is a disaster waiting to happen.
On another point that came up -
i = 5
print i # 5
x[i] = 12 where:
global i
i = 3
print i # 3
seems to me to be so unexpected and non-obvious that it makes the
whole construct suspect. Yes, I know that the semantics, as given in
terms of a rewrite, define what happens, but it seems to me that it
goes counter to intuition.
Thinking about it, the issue is that you're making where the *only*
example in Python (that i can think of) of a non-defining construct
which creates a new scope. Currently, scopes are created by the class
and def statements. Both of these define objects (classes and
functions, respectively). The where clause as you've stated it defines
a scope, but no object. That doesn't make it ill-defined, but it does
make it (very, in my view) non-intuitive.
Paul.
Daniel Stutzbach
bar = "outside"
local:
nonlocal foo
bar = "inside"
foo = bar
Carl Johnson
> Which is an error. However:
> x = foo(y) where:
> y = []
> for i in range(n):
> if cond(i):
> break
> y.append(i)
> must be allowed (right?). So we can't make a blanket restriction on
> break/continue within the body.
Same with return and yield if there's a def inside the where. (Seems
obvious, but you never mentioned it.)
-- Carl Johnson
Jan Kaliszewski
>>> x[i] = 12 where:
>>> i = some_complicated_expression()
[snip]
> And yet the user's intent is obviously to use the i defined inside the
> where (that's how functional languages work, so it's hardly an
> unrealistic intent).
[snip]
> Yes, I know that the semantics, as given in
> terms of a rewrite, define what happens, but it seems to me that it
> goes counter to intuition.
>
> Thinking about it, the issue is that you're making where the *only*
> example in Python (that i can think of) of a non-defining construct
> which creates a new scope. Currently, scopes are created by the class
> and def statements. Both of these define objects (classes and
> functions, respectively). The where clause as you've stated it defines
> a scope, but no object. That doesn't make it ill-defined, but it does
> make it (very, in my view) non-intuitive.
Yeah, It is an issue :-/
I can imagine some solutions based on name overriding rules, e.g.:
* in the "target line" only those names are treated as part of nested
where-block scope, which are targets of assignment within the where-block
(as MRAB proposes, if I understood well), or
* in the "target line" only those names are treated as part of nested
where-block scope, which are not targets of assignment within the target
line.
But it seems to me obscure at first sight, and I don't know if
implementation wouldn't be problematic...
Maybe the solution is to make 'where' creating an object, somehow
similarly to the way the 'class' statement works? Some examples:
x[somename.i] = 12 where somename:
i = some_complicated_expression()
def variance(data): # for some definition of "variance"
return var.sqrt(var.sum_of_squares(data)
- var.square_of_sums(data)) where var:
def sqrt...
def sum_of_squares...
def square_of_sums...
foo(numbers.x, numbers.y) where numbers:
x = a * 42
y = b / 17
a = {}
a[fn.key] = bar(fn.f, fn.g) where fn:
key = ...
def f(x):
...
def g(y):
...
Another possibility is to allow using "anonymus name":
x[.index] = 12 where:
index = some_complicated_expression()
foo(.x, .y) where:
x = a * 42
y = b / 17
Cheers,
zuo
Daniel Stutzbach
Thinking about it, the issue is that you're making where the *only*
example in Python (that i can think of) of a non-defining construct
which creates a new scope.
Paul Moore
> On Thu, Jul 16, 2009 at 1:51 PM, Paul Moore <p.f....@gmail.com> wrote:
>>
>> Thinking about it, the issue is that you're making where the *only*
>> example in Python (that i can think of) of a non-defining construct
>> which creates a new scope.
>
> I think that's why the "where" clause is being proposed: because there's no
> construct to create a new scope without defining a new class or function.
That's a very interesting perspective. Maybe then it would make more
sense to simply bite the bullet and define a scope-creating statement
-
scope:
i = a_complicated_expression()
x[i] = 15
Without a way of having some values "escape" the scope, this may be a
bit limited, though. And it's almost certainly not going to appeal to
the people who want to see the expression first, and the subordinate
names defined afterwards. But that's a different underlying concept -
a new construct to execute code in something other than top-to-bottom
order (and one I have less interest in, personally).
MRAB
> 2009/7/16 Daniel Stutzbach <dan...@stutzbachenterprises.com>:
>> On Thu, Jul 16, 2009 at 1:51 PM, Paul Moore <p.f....@gmail.com> wrote:
>>> Thinking about it, the issue is that you're making where the *only*
>>> example in Python (that i can think of) of a non-defining construct
>>> which creates a new scope.
>> I think that's why the "where" clause is being proposed: because there's no
>> construct to create a new scope without defining a new class or function.
>
> That's a very interesting perspective. Maybe then it would make more
> sense to simply bite the bullet and define a scope-creating statement
> -
>
> scope:
> i = a_complicated_expression()
> x[i] = 15
>
> Without a way of having some values "escape" the scope, this may be a
> bit limited, though. And it's almost certainly not going to appeal to
> the people who want to see the expression first, and the subordinate
> names defined afterwards. But that's a different underlying concept -
> a new construct to execute code in something other than top-to-bottom
> order (and one I have less interest in, personally).
>
Perhaps:
scope except foo:
...
:-)
Mike Meyer
On Thu, 16 Jul 2009 10:51:54 -0400
Chris Perkins <chrispe...@gmail.com> wrote:
> As a first approximation, assume that this code:
This seems reasonable, but there are a couple of questions. Key words:
"first approximation".
> [LHS OP] EXPR where:
> BODY
>
> is expanded into the equivalent of this:
>
> def TEMP():
> BODY
> return EXPR
> [LHS OP] TEMP()
>
> Problem 1) return
> Since there will be an implicit return injected into the body of TEMP,
> what happens if the user puts an explicit return in the body of the
> block:
> x = foo(y) where:
> return 23
> expands to
> def TEMP():
> return 23
> return foo(y)
> x = TEMP()
> So return should be disallowed.
. [ ...]
> Problem 3) yield
> x = foo(y) where:
> yield 3
> expands to:
> def TEMP():
> yield 3
> return foo(y)
> x = TEMP()
> which is an error. So yield must be disallowed.
I disagree with both these conclusions. Clearly, they are paired, as
they serve a similar purpose, but I don't see that they need to be
disallowed at all. That they cause problems with your first attempt at
describing an implementation certainly isn't reason to disallow them
all by itself.
On the other hand, using a where block to create some temporary names
doesn't in any way prevent me from discovering in the middle of
calculating their values that I have a value to return (or yield) from
the enclosing function. This would pretty clearly be a wart should it
be so.
That said - if where can't be implemented in such a way as to allow a
return or yield from inside a where, I don't believe that's sufficient
reason to reject where out of hand. But we shouldn't start by assuming
we have to break it.
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Jan Kaliszewski
> That they cause problems with your first attempt at
> describing an implementation certainly isn't reason to disallow them
> all by itself.
But, it isn't a problem of implementation. I don't see any reasonable
*semantics* of return or yield within where-block in proposed form.
return/yield statements seem to be completely unnecessary in this context.
(Of course, I don't say about return/yield statements within functions
defined within where-block).
--
Jan Kaliszewski <z...@chopin.edu.pl>
Chris Perkins
> 2009/7/16 Chris Perkins <chrispe...@gmail.com>:
>>> You've also disallowed
>>>
>>> x[i] = 12 where:
>>> i = some_complicated_expression()
>
> where (that's how functional languages work, so it's hardly an
> unrealistic intent).
I don't follow this argument - in what way is that how functional
languages work?
I see that code as analogous to the following Clojure, for example:
(def x (into-array [1 2]))
(def i 0)
(aset x i (let [i 1] 99))
You wouldn't expect x to now be [1 99], rather than [99 2], would you?
> I'd argue that having this example use a previously defined value of i, when
>
> x = i where:
> i = some_complicated_expression()
>
> uses the i defined in the where clause, is a disaster waiting to happen.
I don't really see the problem with it. If you told people that the
where block only applies to the right-hand-side of an assignment, and
that the left-hand-side is always evaluated outside the scope of the
where block, I doubt there would be much confusion.
> On another point that came up -
>
> i = 5
> print i # 5
> x[i] = 12 where:
> global i
> i = 3
> print i # 3
>
> seems to me to be so unexpected and non-obvious that it makes the
> whole construct suspect. Yes, I know that the semantics, as given in
> terms of a rewrite, define what happens, but it seems to me that it
> goes counter to intuition.
Again, I don't think that's a big deal. One solution is to disallow
global statements (if we're already disallowing stuff like break and
return, why not). Another is just "don't do that".
> Thinking about it, the issue is that you're making where the *only*
> example in Python (that i can think of) of a non-defining construct
> which creates a new scope. Currently, scopes are created by the class
> and def statements. Both of these define objects (classes and
> functions, respectively). The where clause as you've stated it defines
> a scope, but no object. That doesn't make it ill-defined, but it does
> make it (very, in my view) non-intuitive.
"Namespaces are one honking great idea -- let's do more of those!"
And everyone knows, you can't argue with the Zen. ;)
Chris Perkins
MRAB
> 17-07-2009, 00:51 Mike Meyer <mwm-keyword-...@mired.org> wrote:
>
>> That they cause problems with your first attempt at
>> describing an implementation certainly isn't reason to disallow them
>> all by itself.
>
> But, it isn't a problem of implementation. I don't see any reasonable
> *semantics* of return or yield within where-block in proposed form.
> return/yield statements seem to be completely unnecessary in this context.
> (Of course, I don't say about return/yield statements within functions
> defined within where-block).
>
in a sense be no different to one in a normal suite, except, of course,
that there happens to be a local namespace.
foo = bar where:
if condition:
return
bar = "bar"
would be equivalent to:
local:
nonlocal foo
if condition:
return
bar = "bar"
foo = bar
(I accept that 'nonlocal' might not be the right keyword, but you get
the idea.)
Jim Jewett
> ... I don't see any reasonable *semantics* of
> return or yield within where-block in proposed form.
next_batch = myfilter(candidates, failed) where:
failed=[]
for x in this_batch:
if magical(x):
# We actually found one!!!
return x
# more failures, of course.
# bias the next batch to a different section
# of the search space.
failed.append(x)
I leave to others the judgment on whether or not any function
containing code is already too long...
alternatively,
provide(foo(item)) where:
start_to_generate_an_item()
if flag():
# Are there reasons to return instead of throwing?
raise HigherPriority
item=finish_generation()
Again, I'm not certain that the style should be encouraged, but
neither I am sure it shouldn't be.
-jJ
Greg Ewing
> If the
> left-hand side were in the where-block's scope, then:
>
> x = 0
> x = i where:
> i = 5
> print x
>
> Would print "0" not "5".
My intuitive expectations are different for assignment
to a bare name vs. assignment to an item or attribute.
In
x = i where:
i = 5
I would expect x to be bound in the current scope,
not the where-block scope. But I would expect
x[i] = 5 where:
x = y
i = 7
to be equivalent to
y[7] = 5
i.e. both x and i are *evaluated* in the where-block
scope.
I'm not sure where this leaves us with respect to
augmented assignment, though. If you follow it to
its logical conclusion, then
x += i where:
i = 5
should be equivalent to
x = x.__iadd__(i) where:
i = 5
and then the evaluation and rebinding of 'x' would
happen in different scopes!
--
Greg
Bruce Frederiksen
> I disagree with both these conclusions. Clearly, they are paired, as
> they serve a similar purpose, but I don't see that they need to be
> disallowed at all. That they cause problems with your first attempt at
> describing an implementation certainly isn't reason to disallow them
> all by itself.
>
statement where:
body
creates a new temporary local context and evaluates the body in it. Any
variables assigned to within the body are set in this temporary context
(as you'd expect). Variables in the outer context are still visible to
the body.
Now the difference:
Then the statement is run within this temporary local context, but with
a change in that now all variables assigned to bypass the temporary
context (perhaps unbinding the variable there, if it's set, so that it
no longer hides the outer value?) and are set in the outer context. But
variables within the temporary context are still visible to the
statement. This is a new use of contexts for Python.
Once the statement is finished executing, the temporary context is
discarded.
Thus:
i = 2
x[27] = 2
x[i] = 4 where:
i = 27
print(x[27]) # 4
print(i) # 2
has access to the temporary i (set to 27).
While:
x = 1
x = i where:
i = 27
print(x) # 27
assigns 27 to x in the outer context.
Thus, the where clause applies to the whole statement, without limiting
the statement's ability to bind variables in the outer context.
-Bruce
Mike Meyer
That evaluating an expression to the right of an augmented assignment
might return something different than what you would bind that
expression to has always been an issue with augmented assignments. I
believe this particular problem belongs has more to do with them than
with the proposed where statement.
I think the intuitive behavior doesn't involve position relative to
the assignment, but binding vs. non-bindings. What you *expect* is
that expressions left of the where will be evaluated in the inner
scope (created by the where), but that any bindings that happen there
will happen in the outer scope (before the where).
That makes what you want to happen happen - *most* of the time:
i = 3
x += i where:
i = 5
increments x by 5.
i = 'a'
x[i] = 'b' where:
i = 'c'
sets x['c'] to 'b'.
Things get really ugly when the bound variables start appearing in
both scopes combined with augmented assignments:
x = 3
x += 4 where:
x = 5
print x
What should this print?
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Greg Ewing
> while foo(x) and bar() where:
> foo = SOMETHING
> def bar():
> SOMETHING ELSE
> do:
> THE
> LOOP
> BODY
That's kind of clever, although it does require
modifying the while-loop syntax, and if you're
doing that, there's not a lot of gain over
introducing a dedicated loop-and-a-half syntax.
Also, once we're allowed 'where' clauses on
while statements, people are going to want them
on other kinds of statements as well:
if a == b where:
a = foo()
b = blarg()
then:
...
class Ham(Spam) where:
class Spam:
...
is:
...
def f(x = default) where:
default = whatever
as:
...
try:
...
except eels where:
eels = something_bad()
do:
...
Where do we stop?
> while foo(x) and bar():
> where:
> foo = SOMETHING
> def bar():
> SOMETHING ELSE
> THE
> LOOP
> BODY
I think I liked the first version better. There's
something jarring about the unindent half way
through without any keyword to mark the boundary.
Also the bare 'where' relating to the previous line
suggests some weird grammatical gymnastics going
on.
--
Greg
Greg Ewing
> Thinking about it, the issue is that you're making where the *only*
> example in Python (that i can think of) of a non-defining construct
> which creates a new scope.
Not quite true any more, since LCs and generator
expressions now use a local scope for their
iteration variables.
However in this case, it might just be less
confusing overall to not have a new scope for
the where-block. It's not necessary in order
to get the main intended benefit.
--
Greg
Greg Ewing
> I think that's why the "where" clause is being proposed: because there's no
> construct to create a new scope without defining a new class or function.
No, the reason is to provide a way to put
implementation details in a convenient place
after where they're used, but not too far
away.
--
Greg
Steven D'Aprano
> > In either case, writing the sums-and-squares before the return
> > suggests that these are common code best written as re-usable
> > functions. But "where" changes the focus of the coder away from
> > factoring code *out* of functions into placing code *inside*
> > internal blocks. It encourages the coder to think of sum_of_squares
> > as a private implementation detail instead of a re-usable
> > component. And that is, I believe, harmful.
>
> But surely you agree that _sometimes_ there is code that really is an
> implementation detail - not everything is reusable, or at least not
> usefully reusable.
Of course. What your suggestion gives us, though, is a second way of
including multi-line functions:
def parrot():
x = 1
y = 2
return x + y
versus
def parrot():
return x + y where:
x = 1
y = 2
I'm not convinced that allowing the second is either necessary or
beneficial.
> Taking your argument to the extreme, every
> function would be only one line of code. I prefer to trust
> programmers to make the best use of the tools available to create
> good code.
The Python community takes the attitude that we're all consenting adults
here, and that if you want to shoot yourself in the foot, you should be
allowed to. And that's fine. But it's only fine because Python, by and
large, is a safe language, one with very few jagged edges, which
encourages good practices rather than bad.
Which brings me to this:
> In other words:
>
> Jan Kaliszewski:
> >Every language feature can be misused. More powerful feature --
> > larger possibilities of misusing. But language is not for
> > substituting programmer's sense.
>
> Yeah, ditto to what he said.
Some language features are begging to be misused, and although I've
occasionally wanted a way to define code blocks on the fly outside of a
function, I'm leaning towards the conclusion that this suggestion is a
feature where the abuses outweigh the benefits.
--
Steven D'Aprano
Jan Kaliszewski
Mike Meyer <mwm-keyword-...@mired.org> wrote:
[snip]
> I think the intuitive behavior doesn't involve position relative to
> the assignment, but binding vs. non-bindings. What you *expect* is
> that expressions left of the where will be evaluated in the inner
> scope (created by the where), but that any bindings that happen there
> will happen in the outer scope (before the where).
>
> That makes what you want to happen happen - *most* of the time:
>
> i = 3
> x += i where:
> i = 5
>
> increments x by 5.
>
> i = 'a'
> x[i] = 'b' where:
> i = 'c'
>
> sets x['c'] to 'b'.
>
> Things get really ugly when the bound variables start appearing in
> both scopes combined with augmented assignments:
>
> x = 3
> x += 4 where:
> x = 5
> print x
>
> What should this print?
I think, a good rule could be that:
All *expressions* of the target line are evaluated within the
where-block scope; and -- moreover -- those names (variables) that
are *bound* in the target line, are "leaked" to the outer scope.
[target line == the line with where statement]
Then the former example code should print 9.
--
Jan Kaliszewski <z...@chopin.edu.pl>
Daniel Stutzbach
I would expect x to be bound in the current scope,
not the where-block scope. But I would expect
x[i] = 5 where:
x = y
i = 7
to be equivalent to
y[7] = 5
>>> x = 5
>>> def foo():
... blah = x
... x = 6
...
>>> foo()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in foo
UnboundLocalError: local variable 'x' referenced before assignment
(rebinding is allowed if "global" or "nonlocal" is explicitly used, of course)
David Stanek
This thread is a little confusing to me. I am failing to see how the where
block makes anything clearer. The fact that a new scope is created and names
are available during a singe line of execution scares me.
I also worry that this will encourage code duplication. Any substantial logic
should be rolling into a function not into a where block.
--
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
Ben Finney
> This thread is a little confusing to me. I am failing to see how the
> where block makes anything clearer.
+1. I haven't found that any of the examples improve the clarity of the
code over using a named function.
--
\ “If I melt dry ice, can I swim without getting wet?” —Steven |
`\ Wright |
_o__) |
Ben Finney
MRAB
x = 3
local:
nonlocal x
x = 5
x += 4 # The target line.
MRAB
while ... where:
...
do:
...
if a == b where:
...
do:
...
etc.
Paul Moore
> On Thu, Jul 16, 2009 at 2:51 PM, Paul Moore<p.f....@gmail.com> wrote:
>> 2009/7/16 Chris Perkins <chrispe...@gmail.com>:
>>>> You've also disallowed
>>>>
>>>> x[i] = 12 where:
>>>> i = some_complicated_expression()
>>
>> And yet the user's intent is obviously to use the i defined inside the
>> where (that's how functional languages work, so it's hardly an
>> unrealistic intent).
>
> I don't follow this argument - in what way is that how functional
> languages work?
Sorry, I should have said "Haskell, and some other functional
languages I vaguely recall working similarly" :-)
In Haskell, the expressions given names in the where clause are used
throughout the expression to the left of the where. Of course
immutability and referential transparency makes that a lot less
controversial...
> I don't really see the problem with it. If you told people that the
> where block only applies to the right-hand-side of an assignment, and
> that the left-hand-side is always evaluated outside the scope of the
> where block, I doubt there would be much confusion.
That somewhat assumes that a where block is only used for assignments
(or more accurately, that there's a special exception for the left
side of an assignment not being included). That's at best a wart.
> Again, I don't think that's a big deal. One solution is to disallow
> global statements (if we're already disallowing stuff like break and
> return, why not). Another is just "don't do that".
More warts. At some point, the weight of the various exceptions make
the basic idea not worth it.
> "Namespaces are one honking great idea -- let's do more of those!"
>
> And everyone knows, you can't argue with the Zen. ;)
"If the implementation is hard to explain, it's a bad idea."
"Simple is better than complex."
But the Zen can have arguments with itself :-)
Paul.
Gerald Britton
However, I don't like unnecessary function calls since they are
expensive in Python compared to other languages. Still I might do
this:
def foo(x):
def bar(y):
...
return ... #something depending on y
...
return bar(z)
If I had a "where" clause I could write:
def foo(x):
...
return y where:
#compute y
Using "where" would have the advantage of avoiding a function call,
though I suppose it wouldn't save (much) stack space since it needs a
new scope anyway. I see the "where" clause as a useful way to
abstract a section of code to keep the high level in view, then break
it down as you go.
Paul Moore
> Using "where" would have the advantage of avoiding a function call,
> though I suppose it wouldn't save (much) stack space since it needs a
> new scope anyway. I see the "where" clause as a useful way to
> abstract a section of code to keep the high level in view, then break
> it down as you go.
You're assuming that the implementation of where is faster than a
function call. Given that the defined semantics are in terms of
defining then calling a temporary function, this may well be an
invalid assumption. (And if it isn't, then can the tricks which get
used to speed up the where cause please be used to speed up function
calls in general, too - as that would be far more generally useful!
:-))
Paul.
Steven D'Aprano
> Sometimes I use helper functions to make things easier to read.
> However, I don't like unnecessary function calls since they are
> expensive in Python compared to other languages.
That's only a problem if:
(1) Your code actually is too slow; and
(2) You have profiled your code and discovered that the biggest
contributor to your code's slowness is the cost of calling functions,
not the execution time of the functions.
Unless both of these cases hold, then I suggest you are guilty of
premature optimization. Possibly even pessimation -- given the
complexities of CPU caches, memory, swap, etc, it's possible that your
code could be *slower*, not faster, due to your "optimization".
People's intuitions about what's fast and what's slow in Python code are
often completely at odds with reality. Unless you actually have a
problem with slow code that needs solving, my advice is to not worry
about it. Use good algorithms, use tried and tested fast data
structures, and worry about micro-optimizations when you *actually*
need them.
--
Steven D'Aprano
Stephen J. Turnbull
> But the Zen can have arguments with itself :-)
If it didn't, it wouldn't be the Zen.
Arnaud Delobelle
On 17 Jul 2009, at 03:02, Greg Ewing wrote:
>
> [...] I would expect
>
> x[i] = 5 where:
> x = y
> i = 7
>
> to be equivalent to
>
> y[7] = 5
>
> i.e. both x and i are *evaluated* in the where-block
> scope.
>
So would I, given that the statement
x[i] = 5
is the really the same as the expression
x.__setitem__(i, 5)
> I'm not sure where this leaves us with respect to
> augmented assignment, though. If you follow it to
> its logical conclusion, then
>
> x += i where:
> i = 5
>
> should be equivalent to
>
> x = x.__iadd__(i) where:
> i = 5
>
> and then the evaluation and rebinding of 'x' would
> happen in different scopes!
Indeed. But why not simply evaluate the whole statement in the where
scope, with all names not bound in the where scope declared as nonlocal?
i.e
x[i], y = foo(a), b where:
i = 7
def foo(x):
return bar(x, x)
would be equivalent to:
def _():
nonlocal x, y, b
i = 7
def foo(x):
return bar(x, x)
x[i], y = foo(a), b
_()
--
Arnaud
Terry Reedy
> David Stanek <dst...@dstanek.com> writes:
>
>> This thread is a little confusing to me. I am failing to see how the
>> where block makes anything clearer.
>
> +1. I haven't found that any of the examples improve the clarity of the
> code over using a named function.
I presume you mean -1 on the proposal, as am I (strongly).
Gerald Britton
"where" possible actually makes a measurable difference. This is
especially the case with little getters and setters in class defs.
Anyway, I like the "where" idea not because of real or imagined
performance gains, but because of its cleanness when expressing
problems. A big use for me would be in list comprehensions. In one
project I work on, I see things like:
for i in [item in self.someobject.get_generator() if self.important_test(item)]
and other really long object references.
which I would like to write:
mylist = [item in f if g(item)] where:
f = self.someobject.get_generator()
g = self.important_test
To my eyes, the first is harder to read than the second one. Of
course I can do this:
f = self.someobject.get_generator()
g = self.important_test
mylist = [item in f if g(item)]:
but then "f" and "g" pollute the calling context's namespace.
Anyway, I don't have a strong preference, just a "nice to have" feeling.
--
Gerald Britton
Chris Perkins
>
>> I don't really see the problem with it. If you told people that the
>> where block only applies to the right-hand-side of an assignment, and
>> that the left-hand-side is always evaluated outside the scope of the
>> where block, I doubt there would be much confusion.
>
> That somewhat assumes that a where block is only used for assignments
> (or more accurately, that there's a special exception for the left
> side of an assignment not being included). That's at best a wart.
The "special case for assignments" is what I was assuming. In fact, I
liked this proposal partly because the semantics seemed so obvious
that you could guess them and be right - but looking through this
thread, I see that the semantics are obvious to different people in
different ways.
For example, I see that many people seem to expect:
i = 1
x[i] = 3 where:
i = 2
to mean "x[2] = 3", which I would find utterly surprising.
>> Again, I don't think that's a big deal. One solution is to disallow
>> global statements (if we're already disallowing stuff like break and
>> return, why not). Another is just "don't do that".
>
> More warts. At some point, the weight of the various exceptions make
> the basic idea not worth it.
That does worry me, I'll admit - if the code that you can legally
write in a where block is only a subset of Python, then, well...
that's just ugly.
Nevertheless, I find several of the examples to have such a
readability advantage - similar to the way that list comprehensions
are immediately and wonderfully more readable than building a list in
a for loop - that I'm still in favor of the idea.
Chris Perkins
Terry Reedy
> To my eyes, the first is harder to read than the second one. Of
> course I can do this:
>
> f = self.someobject.get_generator()
> g = self.important_test
> mylist = [item in f if g(item)]:
>
> but then "f" and "g" pollute the calling context's namespace.
So what? Seriously, what harm does a temporary association do?
Especially when one can easily 'clean up' with 'del f,g' if it actually
make a functional difference or if one just wants to cater to an obsession.
In particular, why is this reason to *PERMANENTLYT pollute the language
with an otherwise useless new feature?
Unlike namespace associations, deleting language features is really
difficult.
Terry Jan Reedy
Jan Kaliszewski
> David Stanek <dst...@dstanek.com> writes:
>
>> This thread is a little confusing to me. I am failing to see how the
>> where block makes anything clearer.
>
> +1. I haven't found that any of the examples improve the clarity of the
> code over using a named function.
And what about that:
class SomeClass:
...
def summarize_proust(self, objects, args, method_name)
"Summarize Proust in 15 seconds"
return map(action, objects, args, alts) where:
def action(obj, arg, alt):
result = getattr(item, method_name)(arg) or alt
obj.set_last_result(result)
return result
alts = map(weirdtrans, objects) where def weirdtrans(obj):
# some transformations not very
# useful in more general context
...is more logical and readable for me, than:
class SomeClass:
...
def _weirdtrans(self, obj):
# some transformations useful
# only for somefilter method
def summarize_proust(self, objects, args, method_name)
"Summarize Proust in 15 seconds"
def action(obj, arg, alt):
result = getattr(item, method_name)(arg) or alt
obj.set_last_result(result)
return result
alts = map(self._weirdtrans, objects)
return map(action, objects, args, alts)
17-07-2009, 22:28 Gerald Britton <gerald....@gmail.com> wrote:
> which I would like to write:
>
> mylist = [item in f if g(item)] where:
Obviously you ment: "[item for item if g(item)] where:", didn't you?
In that particular case you could write: filter(g, f).
Anyway still I'd prefer:
mylist = filter(g, f) where:
f = self.someobject.get_generator()
g = self.important_test
Than:
f = self.someobject.get_generator()
g = self.important_test
mylist = filter(g, f) where:
> then "f" and "g" pollute the calling context's namespace.
+1
Generally one of main applications of 'where' would be using it
for functions what use function(s) as argument(s) (map, filter
et consortes...).
After all it's something like lambda, but:
* more beautiful (IMHO),
* not limited to expression.
Regards,
--
Jan Kaliszewski <z...@chopin.edu.pl>
Mike Meyer
"Jan Kaliszewski" <z...@chopin.edu.pl> wrote:
> 17-07-2009, 00:51 Mike Meyer <mwm-keyword-...@mired.org> wrote:
>
> > That they cause problems with your first attempt at
> > describing an implementation certainly isn't reason to disallow them
> > all by itself.
>
> But, it isn't a problem of implementation. I don't see any reasonable
> *semantics* of return or yield within where-block in proposed form.
> return/yield statements seem to be completely unnecessary in this context.
> (Of course, I don't say about return/yield statements within functions
> defined within where-block).
Seems obvious to me:
def foo(x):
bar = foundbar where:
if x is None:
return -1
foundbar = x * 2
return bar
The return will return -1 from foo if it's invoked; the assignment to
bar (and foundbar, for that matter) will never happen.
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Ben Finney
writes:
> Anyway, I like the "where" idea not because of real or imagined
> performance gains, but because of its cleanness when expressing
> problems.
(Speaking of cwclean expression, please don't top-post; instead, remove
irrelevant quoted material and respond beneath each remaining point
<URL:http://en.wikipedia.org/wiki/Posting_style#Inline_replying>.)
> A big use for me would be in list comprehensions. In one
> project I work on, I see things like:
>
> for i in [item in self.someobject.get_generator() if self.important_test(item)]
>
> and other really long object references.
>
> which I would like to write:
>
> mylist = [item in f if g(item)] where:
> f = self.someobject.get_generator()
> g = self.important_test
I presume these two are supposed to be equivalent (and syntactically
valid), so I'll take it the first one should be something like::
mylist = [item for item in self.someobject.get_generator() if self.important_test(item)]
> To my eyes, the first is harder to read than the second one.
That's largely attributable to the fact that you've got one alternative
all on a single line, and the other broken into more easily-readable
lines. I don't think the ‘where’ syntax is much help there.
I would write the single-statement version as::
mylist = [
item for item in self.someobject.get_generator()
if self.important_test(item)]
which makes it far more readable. I argue that this does away with
pretty much any justification for your use case above.
--
\ “… a Microsoft Certified System Engineer is to information |
`\ technology as a McDonalds Certified Food Specialist is to the |
_o__) culinary arts.” —Michael Bacarella |
Ben Finney