[Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

20924 views
Skip to first unread message

Nick Coghlan

unread,
Jun 22, 2018, 10:24:27 AM6/22/18
to Antoine Pitrou, python-dev
On 22 June 2018 at 02:26, Antoine Pitrou <soli...@pitrou.net> wrote:
> Indeed. But, for a syntax addition such as PEP 572, I think it would be
> a good idea to ask their opinion to teaching/education specialists.
>
> As far as I'm concerned, if teachers and/or education specialists were
> to say PEP 572 is not a problem, my position would shift from negative
> towards neutral.

I asked a handful of folks at the Education Summit the next day about it:

* for the basic notion of allowing expression level name binding using
the "NAME := EXPR" notation, the reactions ranged from mildly negative
(I read it as only a "-0" rather than a "-1") to outright positive.
* for the reactions to my description of the currently proposed parent
local scoping behaviour in comprehensions, I'd use the word
"horrified", and feel I wasn't overstating the response :)

While I try to account for the fact that I implemented the current
comprehension semantics for the 3.x series, and am hence biased
towards considering them the now obvious interpretation, it's also the
case that generator expressions have worked like nested functions
since they were introduced in Python 2.4 (more than 13 years ago now),
and comprehensions have worked the same way as generator expressions
since Python 3.0 (which has its 10th birthday coming up in December
this year).

This means that I take any claims that the legacy Python 2.x
interpretation of comprehension behaviour is intuitively obvious with
an enormous grain of salt - for the better part of a decade now, every
tool at a Python 3 user's disposal (the fact that the iteration
variable is hidden from the current scope, reading the language
reference [1], printing out locals(), using the dis module, stepping
through code in a debugger, writing their own tracing function, and
even observing the quirky interaction with class scopes) will have
nudged them towards the "it's a hidden nested function" interpretation
of expected comprehension behaviour.

Acquiring the old mental model for the way comprehensions work pretty
much requires a developer to have started with Python 2.x themselves
(perhaps even before comprehensions and lexical closures were part of
the language), or else have been taught the Python 2 comprehension
model by someone else - there's nothing in Python 3's behaviour to
encourage that point of view, and plenty of
functional-language-inspired documentation to instead encourage folks
to view comprehensions as tightly encapsulated declarative container
construction syntax.

I'm currently working on a concept proposal at
https://github.com/ncoghlan/peps/pull/2 that's much closer to PEP 572
than any of my previous `given` based suggestions: for already
declared locals, it devolves to being the same as PEP 572 (except that
expressions are allowed as top level statements), but for any names
that haven't been previously introduced, it prohibits assigning to a
name that doesn't already have a defined scope, and instead relies on
a new `given` clause on various constructs that allows new target
declarations to be introduced into the current scope (such that "if
x:= f():" implies "x" is already defined as a target somewhere else in
the current scope, while "if x := f() given x:" potentially introduces
"x" as a new local target the same way a regular assignment statement
does).

One of the nicer features of the draft proposal is that if all you
want to do is export the iteration variable from a comprehension, you
don't need to use an assignment expression at all: you can just append
"... given global x" or "... given nonlocal x" and export the
iteration variable directly to the desired outer scope, the same way
you can in the fully spelled out nested function equivalent.

Cheers,
Nick.

[1] From https://docs.python.org/3.0/reference/expressions.html#displays-for-lists-sets-and-dictionaries:
'Note that the comprehension is executed in a separate scope, so names
assigned to in the target list don’t “leak” in the enclosing scope.'
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

Antoine Pitrou

unread,
Jun 22, 2018, 11:08:44 AM6/22/18
to pytho...@python.org
On Sat, 23 Jun 2018 00:22:33 +1000
Nick Coghlan <ncog...@gmail.com> wrote:
> On 22 June 2018 at 02:26, Antoine Pitrou <soli...@pitrou.net> wrote:
> > Indeed. But, for a syntax addition such as PEP 572, I think it would be
> > a good idea to ask their opinion to teaching/education specialists.
> >
> > As far as I'm concerned, if teachers and/or education specialists were
> > to say PEP 572 is not a problem, my position would shift from negative
> > towards neutral.
>
> I asked a handful of folks at the Education Summit the next day about it:
>
> * for the basic notion of allowing expression level name binding using
> the "NAME := EXPR" notation, the reactions ranged from mildly negative
> (I read it as only a "-0" rather than a "-1") to outright positive.

Thank you. Personally, I'd like to see feedback from
educators/teachers after they take the time to read the PEP and take
some time to think about its consequences.

My main concern is we're introducing a second different way of doing
something which is really fundamental.

> * for the reactions to my description of the currently proposed parent
> local scoping behaviour in comprehensions, I'd use the word
> "horrified", and feel I wasn't overstating the response :) [...]

Hmm... I don't think conflating the assignment expression proposal
with comprehension semantics issues is helping the discussion.

Regards

Antoine.

Michael Selik

unread,
Jun 22, 2018, 1:04:48 PM6/22/18
to Antoine Pitrou, pytho...@python.org
On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <soli...@pitrou.net> wrote:
Thank you.  Personally, I'd like to see feedback from
educators/teachers after they take the time to read the PEP and take
some time to think about its consequences.

I've started testing the proposed syntax when I teach. I don't have a large sample yet, but most students either dislike it or don't appreciate the benefits. They state a clear preference for shorter, simpler lines at the consequence of more lines of code. This may partially be caused by the smaller screen real estate on a projector or large TV than a desktop monitor.

My intuition is that one strength of Python for beginners is the relative lack of punctuation and operators compared with most other languages. This proposal encourages denser lines with more punctuation. Because of the order of operations, many uses of ``:=`` will also require parentheses. Even relatively simple uses, like ``if (match := pattern.search(data)) is not None:`` require doubled parentheses on one side or the other. Beginners are especially prone to typographical errors with mismatched parentheses and missing colons and get easily frustrated by the associated syntax errors.


Given the following options:

A.

    if (row := cursor.fetchone()) is None:
        raise NotFound
    return row


B.

    row = cursor.fetchone()
    if row is None:
        raise NotFound
    return row


C.

    if (row := cursor.fetchone()) is not None:
        return row
    raise NotFound


D.

    row = cursor.fetchone()
    if row is not None:
        return row
    raise NotFound


The majority of students preferred option B. I also tested some regex match examples. Results were similar.


 
My main concern is we're introducing a second different way of doing
something which is really fundamental.

The few students who like the proposal ask why it requires creating a new operator instead of repurposing the ``=`` operator.

I'll reserve my personal opinions for a different thread.

Michael Selik

unread,
Jun 22, 2018, 1:12:00 PM6/22/18
to Antoine Pitrou, pytho...@python.org
On Fri, Jun 22, 2018 at 10:02 AM Michael Selik <mi...@selik.org> wrote:
On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <soli...@pitrou.net> wrote:
Thank you.  Personally, I'd like to see feedback from
educators/teachers after they take the time to read the PEP and take
some time to think about its consequences.

I forgot to add that I don't anticipate changing my lesson plans if this proposal is accepted. There's already not enough time to teach everything I'd like. Including a new assignment operator would distract from the learning objectives.
 

Chris Angelico

unread,
Jun 22, 2018, 1:18:35 PM6/22/18
to python-dev
On Sat, Jun 23, 2018 at 3:02 AM, Michael Selik <mi...@selik.org> wrote:
> On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <soli...@pitrou.net> wrote:
>>
>> Thank you. Personally, I'd like to see feedback from
>> educators/teachers after they take the time to read the PEP and take
>> some time to think about its consequences.
>
>
> I've started testing the proposed syntax when I teach. I don't have a large
> sample yet, but most students either dislike it or don't appreciate the
> benefits. They state a clear preference for shorter, simpler lines at the
> consequence of more lines of code.

This is partly because students, lacking the experience to instantly
recognize larger constructs, prefer a more concrete approach to
coding. "Good code" is code where the concrete behaviour is more
easily understood. As a programmer gains experience, s/he learns to
grok more complex expressions, and is then better able to make use of
the more expressive constructs such as list comprehensions.

ChrisA

Michael Selik

unread,
Jun 22, 2018, 2:01:49 PM6/22/18
to Chris Angelico, python-dev
On Fri, Jun 22, 2018 at 10:19 AM Chris Angelico <ros...@gmail.com> wrote:
On Sat, Jun 23, 2018 at 3:02 AM, Michael Selik <mi...@selik.org> wrote:
> On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <soli...@pitrou.net> wrote:
>>
>> Thank you.  Personally, I'd like to see feedback from
>> educators/teachers after they take the time to read the PEP and take
>> some time to think about its consequences.
>
>
> I've started testing the proposed syntax when I teach. I don't have a large
> sample yet, but most students either dislike it or don't appreciate the
> benefits. They state a clear preference for shorter, simpler lines at the
> consequence of more lines of code.

This is partly because students, lacking the experience to instantly
recognize larger constructs, prefer a more concrete approach to
coding. "Good code" is code where the concrete behaviour is more
easily understood. As a programmer gains experience, s/he learns to
grok more complex expressions, and is then better able to make use of
the more expressive constructs such as list comprehensions.

I don't think that's the only dynamic going on here. List comprehensions are more expressive, but also more declarative and in Python they have nice parallels with SQL and speech patterns in natural language. The concept of a comprehension is separate from its particular expression in Python. For example, Mozilla's array comprehensions in Javascript are/were ugly [0].

Students who are completely new to programming can see the similarity of list comprehensions to spoken language. They also appreciate the revision of certain 3-line and 4-line for-loops to comprehensions. I didn't get the same sense of "Oh! That looks better!" from my students when revising code with an assignment expression.

Despite my best efforts to cheerlead, some students initially dislike list comprehensions. However, they come around to the idea that there's a tradeoff between line density and code block density. Comprehensions have a 3-to-1 or 4-to-1 ratio of code line shrinkage. They're also often used in sequence, like piping data through a series of transforms. Even if students dislike a single comprehension, they agree that turning 15 lines into 5 lines improves the readability.

In contrast, an assignment expression only has a 2-to-1 code line compression ratio. It might save a level of indentation, but I think there are usually alternatives. Also, the assignment expression is less likely to be used several times in the same block.

A good pitch for an assignment expression is refactoring a cascade of regular expressions:


    for line in f:
        mo = foo_re.search(line)
        if mo is not None:
            foo(mo.groups())
            continue

        mo = bar_re.search(line)
        if mo is not None:
            bar(mo.groups())
            continue

        mo = baz_re.search(line)
        if mo is not None:
            baz(mo.groups())
            continue


Here the assignment operator makes a clear improvement:

    for line in f:
        if (mo := foo_re.search(line)) is not None:
            foo(mo.groups())
        elif (mo := bar_re.search(line)) is not None:
            bar(mo.groups())
        elif (mo := baz_re.search(line)) is not None:
            baz(mo.groups())


However, I think this example is cheating a bit. While I've written similar code many times, it's almost never just a function call in each if-block. It's nearly always a handful of lines of logic which I wouldn't want to cut out into a separate function. The refactor is misleading, because I'd nearly always make a visual separation with a newline and the code would still look similar to the initial example.


Chris Barker via Python-Dev

unread,
Jun 22, 2018, 2:31:22 PM6/22/18
to Michael Selik, Antoine Pitrou, Python Dev
On Fri, Jun 22, 2018 at 10:09 AM, Michael Selik <mi...@selik.org> wrote:
I forgot to add that I don't anticipate changing my lesson plans if this proposal is accepted. There's already not enough time to teach everything I'd like. Including a new assignment operator would distract from the learning objectives.

nor would I. For a while, anyway....

But once it becomes a more common idiom, students will see it in the wild pretty early in their path to learning python. So we'll need to start introducing it earlier than later.

I think this reflects that the "smaller" a language is, the easier it is to learn.

Python has already grown a fair bit since 1.5 (when I started using it :-) ). Some things, like generators, are special purpose enough that I can wait pretty far into the program before teaching them. But others, like comprehensions (and lambda) are common enough that I have to introduce them pretty early on.

Adding := is not a HUGE change, but it IS an expansion of the language, and one that we WILL have to introduce in an introductory course once it starts seeing common use.

I really have no idea how much harder thats going to make the langauge to teach, but it will make it a bit harder -- I see enough confusion with "is" vs == already...

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

Greg Ewing

unread,
Jun 22, 2018, 7:08:13 PM6/22/18
to python-dev
Nick Coghlan wrote:
> x:= f():" implies "x" is already defined as a target somewhere else in
> the current scope, while "if x := f() given x:" potentially introduces
> "x" as a new local target

Noooo..... this is just taking a bad idea and making it
worse, IMO.

I'm -1 on any contortions designed to allow comprehensions
to assign to things in outer scopes. All the proposed use
cases I've seen for this have not improved readability
over writing a function that does things the usual way.

Can we please leave comprehensions as declarative
constructs? The best tools do just one thing and do
it well. These proposals seem to be trying to turn
comprehensions into swiss army knives.

--
Greg

Steven D'Aprano

unread,
Jun 22, 2018, 10:25:14 PM6/22/18
to pytho...@python.org
On Fri, Jun 22, 2018 at 11:28:45AM -0700, Chris Barker via Python-Dev wrote:
> On Fri, Jun 22, 2018 at 10:09 AM, Michael Selik <mi...@selik.org> wrote:
>
> > I forgot to add that I don't anticipate changing my lesson plans if this
> > proposal is accepted. There's already not enough time to teach everything
> > I'd like. Including a new assignment operator would distract from the
> > learning objectives.
> >
>
> nor would I. For a while, anyway....
>
> But once it becomes a more common idiom, students will see it in the wild
> pretty early in their path to learning python. So we'll need to start
> introducing it earlier than later.

Students see many features early in their path. I've had people still
struggling with writing functions ask about metaclasses. People
will see async code everywhere. We don't have to teach *everything* at
once.

The *subtleties* of assignment expressions might have some funny corner
cases, but the high-level overview is simple. It is like ordinary
assignment, but it is an expression that returns the value being
assigned. So if you absolutely need to teach it to a beginner, it
shouldn't be difficult once they understand the difference between an
expression and a statement.


[...]
> I really have no idea how much harder thats going to make the langauge to
> teach, but it will make it a bit harder -- I see enough confusion with "is"
> vs == already...

I think that the biggest source of confusion with "is" is that it
*sometimes* seems to do what is wanted, i.e. test equality, but other
times doesn't. It is that inconsistency that bites.

Whereas with assignment expressions, there's no such inconsistency:

- regular assignment using = only works as a statement, always;
- assignment expression can go anywhere an expression can go, always;
- regular assignment never returns a value;
- assignment expression always returns a value;
- regular assignments have lots of complex forms, such as sequence
unpacking, and complex targets like spam[eggs](arg).attr;
- assignment expressions only takes a plain name, always.

Although there is some overlap in behaviour between the two, unlike
"is", there's no inconsist behaviour to lead people astray.

A better syntax error for things like this:

py> if mo = regex.match(string):
File "<stdin>", line 1
if mo = regex.match(string):
^
SyntaxError: invalid syntax

will also help, although of course some users won't read error messages
for love or money.



--
Steve

Steven D'Aprano

unread,
Jun 22, 2018, 10:48:18 PM6/22/18
to pytho...@python.org
On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:

> > > I've started testing the proposed syntax when I teach. I don't have a
> > > large
> > > sample yet, but most students either dislike it or don't appreciate the
> > > benefits. They state a clear preference for shorter, simpler lines at the
> > > consequence of more lines of code.

Of course they do -- they're less fluent at reading code. They don't
have the experience to judge good code from bad.

The question we should be asking is, do we only add features to Python
if they are easy for beginners? It's not that I especially want to add
features which *aren't* easy for beginners, but Python isn't Scratch and
"easy for beginners" should only be a peripheral concern.


> > This is partly because students, lacking the experience to instantly
> > recognize larger constructs, prefer a more concrete approach to
> > coding. "Good code" is code where the concrete behaviour is more
> > easily understood. As a programmer gains experience, s/he learns to
> > grok more complex expressions, and is then better able to make use of
> > the more expressive constructs such as list comprehensions.
> >
>
> I don't think that's the only dynamic going on here. List comprehensions
> are more expressive, but also more declarative and in Python they have nice
> parallels with SQL and speech patterns in natural language. The concept of
> a comprehension is separate from its particular expression in Python. For
> example, Mozilla's array comprehensions in Javascript are/were ugly [0].

Mozilla's array comprehensions are almost identical to Python's, aside
from a couple of trivial differences:

evens = [for (i of numbers) if (i % 2 === 0) i];

compared to:

evens = [i for i in numbers if (i % 2 == 0)]

- the inexplicable (to me) decision to say "for x of array" instead of
"for x in array";

- moving the expression to the end, instead of the beginning.

The second one is (arguably, though not by me) an improvement, since it
preserves a perfect left-to-right execution order within the
comprehension.


> Students who are completely new to programming can see the similarity of
> list comprehensions to spoken language.

o_O

I've been using comprehensions for something like a decade, and I can't
:-)

The closest analogy to comprehensions I know of is set builder notation
in mathematics, which is hardly a surprise. That's where Haskell got the
inspiration from, and their syntax is essentially an ASCIIfied version
of set builder notation:

Haskell: [(i,j) | i <- [1,2], j <- [1..4]]

Maths: {(i,j) : i ∈ {1, 2}, j ∈ {1...4}}

I teach secondary school children maths, and if there's a plain English
natural language equivalent to list builder notation, neither I nor any
of my students, nor any of the text books I've read, have noticed it.


--
Steve

Steven D'Aprano

unread,
Jun 22, 2018, 11:50:39 PM6/22/18
to pytho...@python.org
On Sat, Jun 23, 2018 at 12:22:33AM +1000, Nick Coghlan wrote:

[...]
> * for the reactions to my description of the currently proposed parent
> local scoping behaviour in comprehensions, I'd use the word
> "horrified", and feel I wasn't overstating the response :)

Without knowing how you worded the question, and the reasons for this
horrified reaction, I'm afraid that isn't really helpful. It is nothing
more than an appeal to emotion:

https://en.wikipedia.org/wiki/Wisdom_of_repugnance

Such strong emotions as "horrified" are typically a sign of an
immediate, emotional gut reaction, not careful thought. We often see
those sorts of reactions attached to the most objectively trivial
matters. Immediate gut reactions are rarely a good guide because they
tend to over-value the status quo, exaggerate the difficulty and costs
of change, and under-estimate the benefits.

Speaking personally, I've learned to question my immediately gut
reaction. (And I remember to do so at least half the time.) PEP 572 is
an example: when the issue was first raised back in February, my gut
reaction was "Not in MY Python!!!" but by taking it seriously and
running through some examples over the course of the discussion, I
realised that, actually, I cautiously favour the idea.

Of course, matters of *personal taste* cannot be anything but gut
reaction, but in those matters, what one person holds strongly another
can legitimately reject strongly. We ought to try to look beyond
personal taste, and try (even if only imperfectly) to consider rational
reasons for and against a proposal. If we do, reactions like "horrified"
are rarely justified. It's just a minor feature in a programming
language, the world will go on one way or the other, and Python already
has trickier gotchas.


> While I try to account for the fact that I implemented the current
> comprehension semantics for the 3.x series, and am hence biased
> towards considering them the now obvious interpretation,

While we certainly don't want to make "non-obvious" a virtue for its own
sake, obviousness (obvious to who?) ought to take a distant second place
to *useful*. Otherwise we'd have to give up an awful lot of existing
Python, starting with the fundamental execution model.

(Oh, the number and length of arguments about whether Python uses call
by value or call by reference, why mutable defaults and [[]]*3 are
"broken"... if you think Python's execution model is "obvious" you've
been using Python too long ;-)

But as Tim Peters has said on a number of occasions, nobody is
suggesting changing the interpretation of current comprehension
semantics. Comprehension loop variables will continue to remain
isolated to the comprehension.

(And for the record, that makes *comprehensions* a weird special case,
not assignment expressions. All other expressions run in the current
lexical scope. Comprehensions introduce an implicit, invisible,
sub-local scope that doesn't match up with a change in indentation as
class and def statements do.)

The behaviour in question is a matter of *assignment expression*
semantics, not comprehensions. And honestly, I don't see why the
proposed behaviour is "horrifying". Here's the high-level overview:

- at the top level of a module, assignment expressions assign in
the global scope;

- inside a class, assignment expressions assign in the class scope;

- inside a function, assignment expressions assign in the function
local scope (unless declared global or nonlocal);

- inside a comprehension, assignment expressions assign in the
surrounding lexical scope (the surrounding function, class or
module).


The first three are the same as ordinary statement assignment. The last
one is what you would expect if you treat comprehensions as any other
expression which run in the current lexical scope. (The current function
or class or module.) Even if we treat it as a "weird special case" (I
don't think it is, but for the sake of the argument let's say it is) its
not hard to explain.

As I discuss below, you can get a very long way indeed working with
comprehensions without once thinking about the scope they run in. By the
time you need to think about comprehension scope, it shouldn't be hard
to deal with the rule:

- loop variables are hidden in a comprehension private scope;
- explicit assignment expression variables are not.

This is not async, or metaclasses, or even Unicode.


[...]
> plenty of
> functional-language-inspired documentation to instead encourage folks
> to view comprehensions as tightly encapsulated declarative container
> construction syntax.

I can't say I've done a broad survey, but the third-party documentation
I've read on comprehensions typically glosses over the scoping issues
without mentioning them. To the extent that scoping is even hinted at,
comprehensions are treated as expressions which are exactly equivalent
to re-writing them as a for-loop in the current scope.

This is a typical example, found as the top result on googling for
"python comprehensions":

https://www.google.com/search?q=python+comprehensions

http://www.pythonforbeginners.com/basics/list-comprehensions-in-python

Nothing is mentioned about scope, and it repeats the inaccurate but
simple equivalency:

for item in list:
if conditional:
expression

But perhaps that tutorial is too old. Okay this recent one is only a
little more than a year old:

https://hackernoon.com/list-comprehension-in-python-8895a785550b

Again, no mention of scoping issues, comprehensions are simply
expressions which presumably run in the same scope as any other
expression.

I think you over-estimate how many newcomers to Python are even aware
that the scope of comprehensions is something to consider.


> I'm currently working on a concept proposal at
> https://github.com/ncoghlan/peps/pull/2 that's much closer to PEP 572
> than any of my previous `given` based suggestions:
[...]

I look forward to reading it, and I promise I won't go by my gut
reaction :-)



--
Steve

Chris Angelico

unread,
Jun 23, 2018, 12:08:12 AM6/23/18
to python-dev
On Sat, Jun 23, 2018 at 1:48 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> I can't say I've done a broad survey, but the third-party documentation
> I've read on comprehensions typically glosses over the scoping issues
> without mentioning them. To the extent that scoping is even hinted at,
> comprehensions are treated as expressions which are exactly equivalent
> to re-writing them as a for-loop in the current scope.

Even first-party documentation elides that distinction. The same
inaccurate-but-simple equivalency - even using the word "equivalent" -
comes up here:

https://docs.python.org/3/howto/functional.html?highlight=equivalent#generator-expressions-and-list-comprehensions

So I'm very sympathetic to the desire to have assignment expressions
inside comprehensions behave like assignment expressions outside
comprehensions. The trouble is that they are then _not_ the same as
other names inside comprehensions. One way or another, there's a
confusing distinction, especially at class scope. Unless this comes
with an actual semantic change that affects existing code, there is
going to be a bizarre disconnect *somewhere*, and it's just a matter
of where.

ChrisA

Chris Barker via Python-Dev

unread,
Jun 23, 2018, 12:13:42 AM6/23/18
to Steven D'Aprano, Python Dev
On Fri, Jun 22, 2018 at 7:23 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> But once it becomes a more common idiom, students will see it in the wild
> pretty early in their path to learning python. So we'll need to start
> introducing it earlier than later.

Students see many features early in their path. I've had people still
struggling with writing functions ask about metaclasses. People
will see async code everywhere. We don't have to teach *everything* at
once.

These are not similar at all -- if you want similar examples, I"d say comprehensions, and lambda, both of which I DO introduce fairly early

While newbies will *ask* about metaclasses, it's probably because they read about them somewhere, not because someone actually used a metaclass in a simple script or answer to a common question on SO.

As for async, you are either doing async or not -- you can't even run an async def function without an event loop -- so again, it won't show up in real code newbies need to understand (at least until async becomes common practice with python...)

-CHB

 So if you absolutely need to teach it to a beginner, it
shouldn't be difficult once they understand the difference between an
expression and a statement.

probably not, though that's a distinction that's mostly academic in the early stages of learning, it may become more critical now...

again, not a huge deal, just a little bit more complexity

-CHB

Steven D'Aprano

unread,
Jun 23, 2018, 1:53:56 AM6/23/18
to pytho...@python.org
On Fri, Jun 22, 2018 at 09:08:37PM -0700, Chris Barker wrote:

> > So if you absolutely need to teach it to a beginner, it
> > shouldn't be difficult once they understand the difference between an
> > expression and a statement.
> >
>
> probably not, though that's a distinction that's mostly academic in the
> early stages of learning,

I don't think so.

People do try to use assignment in expressions, even if only by mistake
writing = when they meant == and need to distinguish between them. In
Python 2, the most common clash between statements and expressions was
print, but at least that's gone.

https://www.quora.com/Whats-the-difference-between-a-statement-and-an-expression-in-Python-Why-is-print-%E2%80%98hi%E2%80%99-a-statement-while-other-functions-are-expressions

https://stackoverflow.com/questions/4728073/what-is-the-difference-between-an-expression-and-a-statement-in-python

https://stackoverflow.com/questions/43435850/what-is-the-difference-between-a-statement-and-a-function-in-python

Even without assignment expressions, people still need to know why they
can't write "if mo = re.match(pattern, text)".


> again, not a huge deal, just a little bit more complexity

Every new feature is added complexity.

Ivan Pozdeev via Python-Dev

unread,
Jun 23, 2018, 6:25:40 AM6/23/18
to pytho...@python.org
On 23.06.2018 5:46, Steven D'Aprano wrote:
> On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:
>
>>>> I've started testing the proposed syntax when I teach. I don't have a
>>>> large
>>>> sample yet, but most students either dislike it or don't appreciate the
>>>> benefits. They state a clear preference for shorter, simpler lines at the
>>>> consequence of more lines of code.
> Of course they do -- they're less fluent at reading code. They don't
> have the experience to judge good code from bad.
>
> The question we should be asking is, do we only add features to Python
> if they are easy for beginners? It's not that I especially want to add
> features which *aren't* easy for beginners, but Python isn't Scratch and
> "easy for beginners" should only be a peripheral concern.

Python's design principles are expressed in the Zen. They rather focus
on being no more complex than absolutely necessary, without prioritizing
either beginners or old-timers ("simple is better than complex",
"complex is better than complicated").

--
Regards,
Ivan

Nick Coghlan

unread,
Jun 24, 2018, 12:35:56 AM6/24/18
to Greg Ewing, python-dev
On 23 June 2018 at 09:06, Greg Ewing <greg....@canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>>
>> x:= f():" implies "x" is already defined as a target somewhere else in
>> the current scope, while "if x := f() given x:" potentially introduces
>> "x" as a new local target
>
>
> Noooo..... this is just taking a bad idea and making it
> worse, IMO.
>
> I'm -1 on any contortions designed to allow comprehensions
> to assign to things in outer scopes. All the proposed use
> cases I've seen for this have not improved readability
> over writing a function that does things the usual way.
>
> Can we please leave comprehensions as declarative
> constructs? The best tools do just one thing and do
> it well. These proposals seem to be trying to turn
> comprehensions into swiss army knives.

If PEP 572 was proposing the use of regular local scoping for
assignment expressions in comprehensions, such that they could still
be used to avoiding repeating subexpressions within an iteration, but
couldn't be used to export progress data, or to maintain a partial sum
without having to rely on `locals().get("total", 0)` to provide an
initial value, then I wouldn't be feeling obliged to present an
alternative that offers the same state export capabilities in a more
explicit form.

Given that PEP 572 *is* proposing implicit comprehension state export,
though, then I think it's important to make the case that seeing the
proposed semantics as intuitive is only going to be the case for folks
that have used Python 2 style comprehensions extensively - anyone
that's never encountered the old state-leaking behaviour for iteration
variables is going to be surprised when assignment expressions ignore
the existence of the comprehension scope (even though the iteration
variable pays attention to it).

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

Mike Miller

unread,
Jun 24, 2018, 12:59:12 AM6/24/18
to pytho...@python.org

On 2018-06-22 19:46, Steven D'Aprano wrote:
> - the inexplicable (to me) decision to say "for x of array" instead of
> "for x in array";

Believe JavaScript has for…in, but as usual in the language it is broken and
they needed a few more tries to get it right. for…of is the latest version and
works as expected.

-Mike

Nick Coghlan

unread,
Jun 24, 2018, 1:30:17 AM6/24/18
to Steven D'Aprano, python-dev
On 23 June 2018 at 13:48, Steven D'Aprano <st...@pearwood.info> wrote:
> On Sat, Jun 23, 2018 at 12:22:33AM +1000, Nick Coghlan wrote:
> [...]
I put quite a bit of work into making it possible for folks to gloss
over the distinction and still come to mostly-correct conclusions
about how particular code snippets would behave.

I was only able to achieve it because the folks that designed lexical
scoping before me had made *read* access to lexical scopes almost
entirely transparent, and because generator expressions were designed
to fail fast if there was a bug in the expression defining the
outermost iterable (which meant that even at class scope, the
outermost iterable expression still had access to class level
variables, because it was evaluated *outside* the nested scope).

*Write* access to lexically nested scopes, by contrast, was omitted
entirely from the original lexical scoping design, and when it was
later added by https://www.python.org/dev/peps/pep-3104/, it was done
using an explicit "nonlocal" declaration statement (expressly akin to
"global"), and PEP 3099 explicitly ruled out the use of ":=" to
implicitly declare the target name as being non-local.

PEP 572 is thus taking the position that:

- we now want to make write access to outer scopes implicit (despite
PEP 3099 explicitly ruling that out as desired design feature)
- but only in comprehensions and generator expressions (not lambda
expressions, and not full nested functions)
- and only for assignment expressions, not for loop iteration variables
- and we want it to implicitly choose between a "global NAME"
declaration and a "nonlocal NAME" declaration based on where the
comprehension is defined
- and this is OK because "nobody" actually understands how
comprehensions hide the iteration variable in practice, and
"everybody" thinks they're still a simple for loop like they were in
Python 2
- the fact that the language reference, the behaviour at class scopes,
the disassembly output, and the behaviour in a debugger all indicate
that comprehensions are full nested scopes isn't important

This level of additional complication and complexity in the scoping
semantics simply isn't warranted for such a minor readability
enhancement as assignment expressions.

Cheers,
Nick.

P.S. "You did such a good job of minimising the backwards
compatibility breakage when we changed the semantics of scoping in
comprehensions that we now consider your opinion on reasonable scoping
semantics for comprehensions to be irrelevant, because everybody else
still thinks they work the same way as they did in Python 2" is such a
surreal position for folks to be taking that I'm having trouble
working out how to effectively respond to it.

Guido has complained that "I keep changing my mind about what I want",
but that's not what's actually going on: what I want is to keep folks
from taking our already complicated scoping semantics and making it
close to impossible for anyone to ever infer how they work from
experimentation at the interactive prompt. That goal has pretty much
stayed consistent since the parent local scoping proposal was first
put forward.

What keeps changing is my tactics in pursuing that goal, based on my
current perception of what the folks pushing that proposal are
actually trying to achieve (which seems to be some combination of "We
want to pretend that the Python 3 scoping changes in comprehensions
never happened, but we still want to avoid leaking the iteration
variables somehow" and "We want to enable new clever tricks with state
export from comprehensions and generator expressions"), as well as
repeatedly asking myself what *actually* bothers me about the proposal
(which I've now distilled down to just the comprehension scoping
issue, and the reliance on an arbitrary syntactic restriction against
top level usage to avoid competing with traditional assignment
statements).

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

Steven D'Aprano

unread,
Jun 24, 2018, 1:58:42 AM6/24/18
to pytho...@python.org
On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:

> Given that PEP 572 *is* proposing implicit comprehension state export,

"Implicit" and "explicit" are two terms which often get misused to mean
"I don't like it" and "I do like it".

Making the intentional choice to use an assignment expression is not
really "implicit" in any meaningful sense. One might as well complain
that "import this" implicitly creates a local variable "this". Well,
yes, it does, in a very loose sense, but that's what imports are
defined as do and it is the whole purpose for making them.

If PEP 572's proposal goes ahead, the behaviour of assignment
expressions will be *defined* as creating assignments in the local scope
rather than the sublocal comprehension scope. To call that "implicit"
is rather like saying that regular assignment is implicit.


> though, then I think it's important to make the case that seeing the
> proposed semantics as intuitive is only going to be the case for folks
> that have used Python 2 style comprehensions extensively - anyone
> that's never encountered the old state-leaking behaviour for iteration
> variables is going to be surprised when assignment expressions ignore
> the existence of the comprehension scope (even though the iteration
> variable pays attention to it).

You are making the assumption that most people are even aware of
"comprehension scope". I don't think that is the case.

In my experience, scoping in Python is still typically seen as the LGB
rule (locals/globals/builtins). See for example this StackOverflow post
from 2016:

https://stackoverflow.com/questions/37211910/override-lgb-scope-rule

Sometimes people remember the E/N (enclosing function/nonlocal) part.
Hardly anyone remembers the C (class) part unless they are actively
thinking in terms of code running inside a class definition, and even if
they do, they typically aren't sure of exactly how it interacts with the
rest.

And I predict that even fewer think of comprehensions as a separate
scope, except by ommission: they *don't think about* the scope of the
loop variable until it bites them.

But as Tim Peters has previously discussed, the loop variable is
special, and is especially prone to accidental shadowing. That won't be
the case for assignment expressions. If there's shadowing going on, it
will be deliberate.


Aside: I've said before that I'm not a fan of sublocal comprehension
scoping, since I personally found it helpful on occasion for the loop
variable to be visible outside of the comprehension. But given that the
only experience most people apparently had with comprehension scoping
was to be bitten by it, I grudgingly accept that encapsulating the loop
variable was the right decision to make, even if it personally
inconvenienced me more than it saved me.

Nor was I the only one: others have been bitten by the change to
comprehension scope, see for example:

https://www.reddit.com/r/Python/comments/425qmb/strange_python_27_34_difference/

There is no consensus that the change to comprehensions was a good thing
or justified.

The bottom line is that I don't think people will be surprised by
assignment expression scope being local instead of sublocal. Rather I
expect that they won't even think about it, until they do, and then
*whatever* behaviour we pick, we'll annoy somebody.



--
Steve

Nick Coghlan

unread,
Jun 24, 2018, 2:35:37 AM6/24/18
to Steven D'Aprano, python-dev
On 24 June 2018 at 15:56, Steven D'Aprano <st...@pearwood.info> wrote:
> On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
>
>> Given that PEP 572 *is* proposing implicit comprehension state export,
>
> "Implicit" and "explicit" are two terms which often get misused to mean
> "I don't like it" and "I do like it".
>
> Making the intentional choice to use an assignment expression is not
> really "implicit" in any meaningful sense.

No, it's actually implicit: there's an extra "global NAME" or
"nonlocal NAME" in the equivalent code for a comprehension that isn't
there in the as-written source code, and doesn't get emitted for a
regular assignment expression or for the iteration variable in a
comprehension - it only shows up due to the defined interaction
between comprehensions and assignment expressions.

> One might as well complain
> that "import this" implicitly creates a local variable "this". Well,
> yes, it does, in a very loose sense, but that's what imports are
> defined as do and it is the whole purpose for making them.

And they behave the same way in every context where they're permitted to appear.

> If PEP 572's proposal goes ahead, the behaviour of assignment
> expressions will be *defined* as creating assignments in the local scope
> rather than the sublocal comprehension scope. To call that "implicit"
> is rather like saying that regular assignment is implicit.

I do say that regular assignments implicitly declare a name as local.
"Python has implicit local variable declarations" is also regularly
cited as one of the differences between it and languages that require
explicit declarations, like C. Even augmented assignments implicitly
declare a name as being a local (hence the infamous UnboundLocalError
that arises when you attempt to use an augmented assignment to rebind
a name from an outer scope).

The problem I have with PEP 572 is that it proposes *breaking that
otherwise universal pattern* - instead of having assignment
expressions in comprehensions implicitly declare the name as local in
the nested comprehension scope, it instead has them:

1. implicitly declare the name as global or as nonlocal in the
comprehension (or else raise an exception), depending on the nature of
the parent scope where the comprehension is used
2. in the nonlocal reference case, amend the symbol table analysis to
act like there was a preceding "if 0:\n for NAME in ():\n pass" in the
parent scope (so the compiler knows which outer function scope to
target)

The rationale being given for why that is OK is:

1. "Everyone" thinks comprehensions are just a for loop (even though
that hasn't been true for the better part of a decade, and was never
true for generator expressions)
2. If comprehensions are just a for loop, then assignment expressions
inside them should be local to the containing scope
3. Therefore the implicit declarations required to tie everything
together and allow folks to continue with an incorrect understanding
of how comprehensions work aren't really implicit - they're explicit
in the inaccurate expansion of the construct!

Can you imagine the reaction if anyone other than Guido or Tim was
attempting to argue for a change to the language that only makes sense
if we grant a completely inaccurate understanding of how a particular
language construct works as being a credible starting point?

Because that's how this comprehension scoping argument feels to me:

Proposal author: "If the language worked in a way other than it does,
then this proposal would make perfect sense."
Proposal reviewer: "Yes, but it doesn't work that way, it works this
way. We deliberately changed it because the old way caused problems."
Proposal author: "Ah, but it *used* to work that way, and a lot of
people still think it works that way, and we can get the compiler to
jump through hoops to pretend it still works that way, except for the
parts of the new way that we want to keep."
Proposal reviewer: "..."

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

Chris Angelico

unread,
Jun 24, 2018, 2:55:48 AM6/24/18
to python-dev
On Sun, Jun 24, 2018 at 4:33 PM, Nick Coghlan <ncog...@gmail.com> wrote:
> On 24 June 2018 at 15:56, Steven D'Aprano <st...@pearwood.info> wrote:
>> On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
>>
>>> Given that PEP 572 *is* proposing implicit comprehension state export,
>>
>> "Implicit" and "explicit" are two terms which often get misused to mean
>> "I don't like it" and "I do like it".
>>
>> Making the intentional choice to use an assignment expression is not
>> really "implicit" in any meaningful sense.
>
> No, it's actually implicit: there's an extra "global NAME" or
> "nonlocal NAME" in the equivalent code for a comprehension that isn't
> there in the as-written source code, and doesn't get emitted for a
> regular assignment expression or for the iteration variable in a
> comprehension - it only shows up due to the defined interaction
> between comprehensions and assignment expressions.

The implicit "nonlocal NAME" is only because there is an equally
implicit function boundary. Why is there a function boundary marked by
square brackets? It's not saying "def" or "lambda", which obviously
create functions. It's a 'for' loop wrapped inside a list display.
What part of that says "hey, I'm a nested function"?

So if there's an implicit function, with implicit declaration of a
magical parameter called ".0", why can't it have an equally implicit
declaration that "spam" is a nonlocal name?

ChrisA

Nick Coghlan

unread,
Jun 24, 2018, 3:12:24 AM6/24/18
to Chris Angelico, python-dev
On 24 June 2018 at 16:53, Chris Angelico <ros...@gmail.com> wrote:
> On Sun, Jun 24, 2018 at 4:33 PM, Nick Coghlan <ncog...@gmail.com> wrote:
>> On 24 June 2018 at 15:56, Steven D'Aprano <st...@pearwood.info> wrote:
>>> On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
>>>
>>>> Given that PEP 572 *is* proposing implicit comprehension state export,
>>>
>>> "Implicit" and "explicit" are two terms which often get misused to mean
>>> "I don't like it" and "I do like it".
>>>
>>> Making the intentional choice to use an assignment expression is not
>>> really "implicit" in any meaningful sense.
>>
>> No, it's actually implicit: there's an extra "global NAME" or
>> "nonlocal NAME" in the equivalent code for a comprehension that isn't
>> there in the as-written source code, and doesn't get emitted for a
>> regular assignment expression or for the iteration variable in a
>> comprehension - it only shows up due to the defined interaction
>> between comprehensions and assignment expressions.
>
> The implicit "nonlocal NAME" is only because there is an equally
> implicit function boundary. Why is there a function boundary marked by
> square brackets? It's not saying "def" or "lambda", which obviously
> create functions. It's a 'for' loop wrapped inside a list display.
> What part of that says "hey, I'm a nested function"?

Nothing - that's why I refer to them as implicitly nested scopes (vs
the explicitly nested scopes in functions and lambda expressions,
where the scope is introduced via keyword).

However, there's still a major behavioural tell at runtime that
they're running in a nested scope: the iteration variables don't leak.
(There are other tells as well, but not ones that most folks are
likely to encounter)

> So if there's an implicit function, with implicit declaration of a
> magical parameter called ".0", why can't it have an equally implicit
> declaration that "spam" is a nonlocal name?

Because comprehensions don't do that for their iteration variables,
because assignment expressions don't do that when used in explicitly
nested scopes, because the required implicit scope declarations are
context dependent, and because even such gyrations still can't hide
the existence of the comprehension's implicitly nested scope when
dealing with classes and the two-argument form of exec().

Since the implicitly nested scopes can't be hidden, it makes far more
sense to me to just admit that they're there, and provide explicit
syntax for cases where folks decide they really do want name bindings
to leak out of that scope (whether those name bindings are assignment
expression targets or the iteration variables themselves).

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

Steven D'Aprano

unread,
Jun 24, 2018, 7:49:38 AM6/24/18
to pytho...@python.org
On Sun, Jun 24, 2018 at 03:56:47PM +1000, Steven D'Aprano wrote:

> There is no consensus that the change to comprehensions was a good thing
> or justified.

On re-reading that, I think its wrong -- it wasn't really what I
intended to say. What I intended to say was, in hindsight, more like:

*Despite the consensus to change comprehension scope*, there's a
contingent of people who are not convinced that the change was a good
thing or justified.

Sorry for the inaccurate comment. Mea culpa.

Ivan Pozdeev via Python-Dev

unread,
Jun 24, 2018, 10:27:04 AM6/24/18
to pytho...@python.org
On 24.06.2018 9:53, Chris Angelico wrote:
> On Sun, Jun 24, 2018 at 4:33 PM, Nick Coghlan <ncog...@gmail.com> wrote:
>> On 24 June 2018 at 15:56, Steven D'Aprano <st...@pearwood.info> wrote:
>>> On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
>>>
>>>> Given that PEP 572 *is* proposing implicit comprehension state export,
>>> "Implicit" and "explicit" are two terms which often get misused to mean
>>> "I don't like it" and "I do like it".
>>>
>>> Making the intentional choice to use an assignment expression is not
>>> really "implicit" in any meaningful sense.

My 2c.
An expression is intuitively thought to be self-contained i.e. without
side effects.
if I write `a=b+1`, I'm not expecting it to do anything except assigning
`a'.

Expressions with side effects has long since proven to be problematic
because of the implicit (thus hard to see and track) links they create
(and because the result depends on the order of evaluation).
Moreover, Python's other design elements have been consistently
discouraging expressions with side effects, too (e.g. mutator methods
intentionally return None instead of the new value, making them useless
in expressions), so the proposition is in direct conflict with the
language's design.

Assignment expressions are a grey area: they carry the full implications
of expressions with side effects described above, but their side effect
is their only effect, i.e. they are explicit and prominent about the
"evil" they do.

>> No, it's actually implicit: there's an extra "global NAME" or
>> "nonlocal NAME" in the equivalent code for a comprehension that isn't
>> there in the as-written source code, and doesn't get emitted for a
>> regular assignment expression or for the iteration variable in a
>> comprehension - it only shows up due to the defined interaction
>> between comprehensions and assignment expressions.
> The implicit "nonlocal NAME" is only because there is an equally
> implicit function boundary. Why is there a function boundary marked by
> square brackets? It's not saying "def" or "lambda", which obviously
> create functions. It's a 'for' loop wrapped inside a list display.
> What part of that says "hey, I'm a nested function"?
>
> So if there's an implicit function, with implicit declaration of a
> magical parameter called ".0", why can't it have an equally implicit
> declaration that "spam" is a nonlocal name?
>
> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

--
Regards,
Ivan

Steven D'Aprano

unread,
Jun 24, 2018, 10:54:03 AM6/24/18
to pytho...@python.org
On Sun, Jun 24, 2018 at 05:24:12PM +0300, Ivan Pozdeev via Python-Dev wrote:

> An expression is intuitively thought to be self-contained i.e. without
> side effects.
> if I write `a=b+1`, I'm not expecting it to do anything except assigning
> `a'.

a = d.pop(1)
a = d.setdefault(key, 0)
chars_written = file.write(text)


> Expressions with side effects has long since proven to be problematic
> because of the implicit (thus hard to see and track) links they create
> (and because the result depends on the order of evaluation).

If you're going to take a hard-core functional approach to side-effects,
I think you are using the wrong language. Nearly everything in Python
*could* have side-effects (even if usually it won't).

Even your own example of "b+1" (depending on what b.__add__ does).


> Moreover, Python's other design elements have been consistently
> discouraging expressions with side effects, too (e.g. mutator methods
> intentionally return None instead of the new value, making them useless
> in expressions),

I don't think that's the reason why mutator methods return None. They
return None rather than self to avoid confusion over whether they return
a copy or not.

https://docs.python.org/3/faq/design.html#why-doesn-t-list-sort-return-the-sorted-list


> so the proposition is in direct conflict with the
> language's design.

Python is full of operations with side-effects.

Besides, they're not quite useless:

(alist.append() or alist)

is functionally equivalent to alist.append returning self. Just a bit
more verbose.

Methods (and functions) all return a value, even if that value is None,
so they can be used in expressions. If Guido wanted Pascal style
procedures, which cannot be used in expressions, we would have them by
now :-)


--
Steve

Guido van Rossum

unread,
Jun 24, 2018, 12:25:53 PM6/24/18
to Steven D'Aprano, Nick Coghlan, Tim Peters, Python-Dev
A quick follow-up: PEP 572 currently has two ideas: (a) introduce := for inline assignment, (b) when := is used in a comprehension, set the scope for the target as if the assignment occurred outside any comprehensions. It seems we have more support for (a) than for (b) -- at least Nick and Greg seem to be +0 or better for (a) but -1 for (b). IIRC (b) originated with Tim. But his essay on the topic, included as Appendix A (https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings) does not even mention comprehensions. However, he did post his motivation for (b) on python-ideas, IIRC a bit before PyCon; and the main text of the PEP gives a strong motivation (https://www.python.org/dev/peps/pep-0572/#scope-of-the-target). Nevertheless, maybe we should compromise and drop (b)?

--
--Guido van Rossum (python.org/~guido)

Steven D'Aprano

unread,
Jun 24, 2018, 2:08:33 PM6/24/18
to pytho...@python.org
On Sun, Jun 24, 2018 at 04:33:38PM +1000, Nick Coghlan wrote:
[...]
> > Making the intentional choice to use an assignment expression is not
> > really "implicit" in any meaningful sense.
>
> No, it's actually implicit: there's an extra "global NAME" or
> "nonlocal NAME" in the equivalent code for a comprehension that isn't
> there in the as-written source code, and doesn't get emitted for a
> regular assignment expression or for the iteration variable in a
> comprehension - it only shows up due to the defined interaction
> between comprehensions and assignment expressions.

You seem to be talking about an implementation which could change in the
future. I'm talking semantics of the proposed language feature. As a
programmer writing Python code, I have no visibility into the
implementation. The implementation could change ten times a day for all
I care, so long as the semantics remain the same.

I want the desired semantics to drive the implementation, not the
other way around.

You seem to want the implementation to drive the semantics, by
eliminating the proposed feature because it doesn't match your
deep understanding of the implementation as a nested function.

I want this feature because its useful, and without it the use-cases
for assignment expressions are significantly reduced.

As far as "implicit", for the sake of the discussion, I'll grant you
that one. Okay, the proposed behaviour will implicitly enable
comprehensions to export their state.

Now what? Is that a good thing or a bad thing?

If "implicit" (with or without the scare quotes) is such a bad thing to
be avoided, why are comprehensions implemented using an implicit
function?


> The problem I have with PEP 572 is that it proposes *breaking that
> otherwise universal pattern* - instead of having assignment
> expressions in comprehensions implicitly declare the name as local in
> the nested comprehension scope, it instead has them:

You talk about "nested comprehension scope", and that's a critical
point, but I'm going to skip answering that for now. I have a draft
email responding to another of your posts on that topic, which I hope to
polish in the next day.


> 1. implicitly declare the name as global or as nonlocal in the
> comprehension (or else raise an exception), depending on the nature of
> the parent scope where the comprehension is used
> 2. in the nonlocal reference case, amend the symbol table analysis to
> act like there was a preceding "if 0:\n for NAME in ():\n pass" in the
> parent scope (so the compiler knows which outer function scope to
> target)

If it is okay for you to amend the list comprehension to behave as if it
were wrapped in an implicit nested function, why shouldn't it be okay to
behave as if assignments inside the comprehension included an implicit
nonlocal declaration?


> The rationale being given for why that is OK is:
>
> 1. "Everyone" thinks comprehensions are just a for loop (even though
> that hasn't been true for the better part of a decade, and was never
> true for generator expressions)

Obviously "everyone" is an exaggeration, but, yes, I stand by that --
most people don't even give comprehension scope a thought until they
get bitten by it.

Either because (Python 2) they don't realise the loop variable is local
to their current scope:

http://www.librador.com/2014/07/10/Variable-scope-in-list-comprehension-vs-generator-expression/

or (Python 3) they get bitten by the change:

https://old.reddit.com/r/Python/comments/425qmb/strange_python_27_34_difference/

(As is so often the case, whatever behaviour we choose, we're going to
surprise somebody.)

It is hardly surprising that people don't think too hard about scoping
of comprehensions. Without a way to perform assignments inside
comprehensions, aside from the loop variables themselves, there's
nothing going on inside a comprehension where it makes a visible
difference whether it is a local scope or a sublocal scope.

*IF* assignment expressions are introduced, that is going to change. We
have some choices:

1. Keep assignment expressions encapsulated in their implicit function,
and be prepared for people to be annoyed because (with no way to declare
them global or non-local inside an expression), they can't use them to
get data in and out of the comprehension.


2. Allow assignment expressions to be exported out of the comprehension,
and be prepared for people to be annoyed because they clobbered a local.

(But for the reasons Tim Peters has already set out, I doubt this will
happen often.)


3. Allow some sort of extra comprehension syntax to allow
global/nonlocal declarations inside comprehensions.

x = 1
[nonlocal x := x+i for i in sequence]

(Hmmm... I thought I would hate that more than I actually do.)


4. Have some sort of cunning plan whereby if the variable in question
exists in the local scope, it is implicitly local inside the
comprehension:

x = 1
[x := i+1 for i in (1, 2)]
assert x == 3

but if it doesn't, then the variable is implicitly sublocal inside the
comprehension:

del x
[x := i+1 for i in (1, 2)]
x # raises NameError



Remember, the driving use-case which started this (ever-so-long)
discussion was the ability to push data into a comprehension and then
update it on each iteration, something like this:

x = initial_value()
results = [x := transform(x, i) for i in sequence]

Please, Nick, take your implementor's hat off, forget everything you
know about the implementation of comprehensions and their implicit
nested function, and tell me that doesn't look like it should work.




--
Steve

Steven D'Aprano

unread,
Jun 24, 2018, 2:50:13 PM6/24/18
to pytho...@python.org
On Sun, Jun 24, 2018 at 09:24:39AM -0700, Guido van Rossum wrote:
> A quick follow-up: PEP 572 currently has two ideas: (a) introduce := for
> inline assignment, (b) when := is used in a comprehension, set the scope
> for the target as if the assignment occurred outside any comprehensions. It
> seems we have more support for (a) than for (b) -- at least Nick and Greg
> seem to be +0 or better for (a) but -1 for (b). IIRC (b) originated with
> Tim.

I'm not sure who came up with the idea first, but as I remember it, the
first mention of this came in a separate thread on Python-Ideas:

https://mail.python.org/pipermail/python-ideas/2018-April/049631.html

so possibly I'm to blame :-)

That thread starts here:

https://mail.python.org/pipermail/python-ideas/2018-April/049622.html

If I did get the idea from Tim, I don't remember doing so.


> But his essay on the topic, included as Appendix A (
> https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings)
> does not even mention comprehensions. However, he did post his motivation
> for (b) on python-ideas, IIRC a bit before PyCon; and the main text of the
> PEP gives a strong motivation (
> https://www.python.org/dev/peps/pep-0572/#scope-of-the-target).
> Nevertheless, maybe we should compromise and drop (b)?

I will have more to say about the whole "comprehensions are their own
scope" issue later. But I'd like to see Nick's proposed PEP, or at least
a draft of it, before making any final decisions.

If it came down to it, I'd be happy with the ability to declare an
assignment target nonlocal in the comprehension if that's what it takes.
What do you think of this syntax?

[global|nonlocal] simple_target := expression


Inside a comprehension, without a declaration, the target would be
sublocal (comprehension scope); that should make Nick happier :-)

Tim Peters

unread,
Jun 24, 2018, 3:08:21 PM6/24/18
to Guido van Rossum, Nick Coghlan, Python Dev
[Guido]

> A quick follow-up: PEP 572 currently has two ideas: (a) introduce := for inline
> assignment, (b) when := is used in a comprehension, set the scope for the
> target as if the assignment occurred outside any comprehensions. It seems
> we have more support for (a) than for (b) -- at least Nick and Greg seem to
> be +0 or better for (a) but -1 for (b). IIRC (b) originated with Tim. But his
> essay on the topic, included as Appendix A
> (https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings)
> does not even mention comprehensions.

I was writing up my observations about simple changes to existing code.  Since there's nothing sanely akin to binding non-for-targets possible in comprehensions now, comprehensions were out of scope for that effort (which was limited to staring at existing code already doing bindings).


:> However, he did post his motivation for (b) on python-ideas, IIRC a bit

> before PyCon; and the main text of the PEP gives a strong motivation
> (https://www.python.org/dev/peps/pep-0572/#scope-of-the-target). Nevertheless,
> maybe we should compromise and drop (b)?

Two things to say about that.  First, the original example I gave would be approximately as well addressed by allowing to declare intended scopes in magically synthesized functions; like (say)

p = None # to establish the intended scope of `p`
while any(<nonlocal p>  # split across lines just for readability
                n % p == 0 for p in small_primes):
    n //= p

It didn't really require an inline assignment, just a way to override the unwanted (in this case) "all `for` targets are local to the invisible function" rigid consequence of the implementation du jour.

Second, if it's dropped, then the PEP needs more words to define what happens in cases like the following, because different textual parts of a comprehension execute in different scopes, and that can  become visible when bindings can be embedded:

def f():
    y = -1
    ys = [y for _ in range(y := 5)]
    print(y, ys)

Here `range(y := 5)` is executed in f's scope.  Presumably the `y` in `y for` also refers to f's scope, despite that `y` textually _appears_ to be assigned to in the body of the listcomp, and so would - for that reason - expected to be local to the synthesized function, and so raise `UnboundLocalError` when referenced.  It's incoherent without detailed knowledge of the implementation.

def g():
    y = -1
    ys = [y for y in range(y := 5)]
    print(y, ys)

And here the `y` in `y for y` is local to the synthesized function, and presumably has nothing to do with the `y` in the `range()` call.  That's incoherent in its own way.

Under the current PEP, all instances of `y` in `f` refer to the f-local `y`, and the listcomp in `g` is a compile-time error.

Chris Angelico

unread,
Jun 24, 2018, 5:09:42 PM6/24/18
to python-dev
On Mon, Jun 25, 2018 at 4:06 AM, Steven D'Aprano <st...@pearwood.info> wrote:
>
> Remember, the driving use-case which started this (ever-so-long)
> discussion was the ability to push data into a comprehension and then
> update it on each iteration, something like this:
>
> x = initial_value()
> results = [x := transform(x, i) for i in sequence]

Which means there is another option.

5. Have the assignment be local to the comprehension, but the initial
value of ANY variable is looked up from the surrounding scopes.

That is: you will NEVER get UnboundLocalError from a
comprehension/genexp; instead, you will look up the name as if it were
in the surrounding scope, either getting a value or bombing with
regular old NameError.

Or possibly variations on this such as "the immediately surrounding
scope only", rather than full name lookups. It'd have an awkward
boundary somewhere, whichever way you do it.

This isn't able to send information *out* of a comprehension, but it
is able to send information *in*.

ChrisA

Michael Selik

unread,
Jun 24, 2018, 5:41:30 PM6/24/18
to pytho...@python.org
This thread started with a request for educator feedback, which I took to mean observations of student reactions. I've only had the chance to test the proposal on ~20 students so far, but I'd like the chance to gather more data for your consideration before the PEP is accepted or rejected.



On Sun, Jun 24, 2018 at 11:09 AM Steven D'Aprano <st...@pearwood.info> wrote:
Remember, the driving use-case which started this (ever-so-long)
discussion was the ability to push data into a comprehension and then
update it on each iteration, something like this:

    x = initial_value()
    results = [x := transform(x, i) for i in sequence]

If that is the driving use-case, then the proposal should be rejected. The ``itertools.accumulate`` function has been available for a little while now and it handles this exact case. The accumulate function may even be more readable, as it explains the purpose explicitly, not merely the algorithm. And heck, it's a one-liner.

    results = accumulate(sequence, transform)


The benefits for ``any`` and ``all`` seem useful. Itertools has "first_seen" in the recipes section. While it feels intuitively useful, I can't recall ever writing something similar myself. For some reason, I (almost?) always want to find all (counter-)examples and aggregate them in some way -- min or max, perhaps -- rather than just get the first.

Even so, if it turns out those uses are quite prevalent, wouldn't a new itertool be better than a new operator? It's good to solve the general problem, but so far the in-comprehension usage seems to have only a handful of cases.



On Fri, Jun 22, 2018 at 9:14 PM Chris Barker via Python-Dev <pytho...@python.org> wrote:
again, not a huge deal, just a little bit more complexity

I worry that Python may experience something of a "death by a thousand cuts" along the lines of the "Remember the Vasa" warning. Python's greatest strength is its appeal to beginners. Little bits of added complexity have a non-linear effect. One day, we may wake up and Python won't be recommended as a beginner's language.




On Fri, Jun 22, 2018 at 7:48 PM Steven D'Aprano <st...@pearwood.info> wrote:
On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:

Of course they do -- they're less fluent at reading code. They don't 
have the experience to judge good code from bad.

On the other hand, an "expert" may be so steeped in a particular subculture that he no longer can distinguish esoteric from intuitive. Don't be so fast to reject the wisdom of the inexperienced.

 
The question we should be asking is, do we only add features to Python 
if they are easy for beginners? It's not that I especially want to add 
features which *aren't* easy for beginners, but Python isn't Scratch and 
"easy for beginners" should only be a peripheral concern.

On the contrary, I believe that "easy for beginners" should be a major concern.  Ease of use has been and is a, or even the main reason for Python's success. When some other language becomes a better teaching language, it will eventually take over in business and science as well. Right now, Python is Scratch for adults. That's a great thing. Given the growth of the field, there are far more beginner programmers working today than there ever have been experts.


Mozilla's array comprehensions are almost identical to Python's, aside 
from a couple of trivial differences:

I can't prove it, but I think the phrase ordering difference is not trivial.


> Students who are completely new to programming can see the similarity of
> [Python] list comprehensions to spoken language. 


I've been using comprehensions for something like a decade, and I can't 

Python: any(line.startswith('#') for line in file)
English: Any line starts with "#" in the file?

Guido van Rossum

unread,
Jun 24, 2018, 7:05:04 PM6/24/18
to Steven D'Aprano, Python-Dev
On Sun, Jun 24, 2018 at 11:50 AM Steven D'Aprano <st...@pearwood.info> wrote:
[Guido]
> [...] IIRC (b) originated with Tim.

I'm not sure who came up with the idea first, but as I remember it, the
first mention of this came in a separate thread on Python-Ideas:

https://mail.python.org/pipermail/python-ideas/2018-April/049631.html

so possibly I'm to blame :-)

Actually that post sounds like the OP of that thread (Peter O'Connor) is to blame --  he proposed a similar thing using `=` for the assignment and custom syntax (`from <value>`) to specify the initial value, and it looks like that inspired you.
 
> But his essay on the topic, included as Appendix A (
> https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings)
> does not even mention comprehensions. However, he did post his motivation
> for (b) on python-ideas, IIRC a bit before PyCon; and the main text of the
> PEP gives a strong motivation (
> https://www.python.org/dev/peps/pep-0572/#scope-of-the-target).
> Nevertheless, maybe we should compromise and drop (b)?

I will have more to say about the whole "comprehensions are their own
scope" issue later. But I'd like to see Nick's proposed PEP, or at least
a draft of it, before making any final decisions.

Agreed, though I assume it's just `given` again.
 
If it came down to it, I'd be happy with the ability to declare an
assignment target nonlocal in the comprehension if that's what it takes.
What do you think of this syntax?

    [global|nonlocal] simple_target := expression

Inside a comprehension, without a declaration, the target would be
sublocal (comprehension scope); that should make Nick happier :-)

It's more special syntax. Just taking part (a) of PEP 572 would make most people happy enough.
 

Guido van Rossum

unread,
Jun 24, 2018, 7:29:17 PM6/24/18
to Tim Peters, Nick Coghlan, Python-Dev
On Sun, Jun 24, 2018 at 12:03 PM Tim Peters <tim.p...@gmail.com> wrote:
[Guido]
:> However, [Tim] did post his motivation for (b) on python-ideas, IIRC a bit

> before PyCon; and the main text of the PEP gives a strong motivation
> (https://www.python.org/dev/peps/pep-0572/#scope-of-the-target). Nevertheless,
> maybe we should compromise and drop (b)?

Two things to say about that.  First, the original example I gave would be approximately as well addressed by allowing to declare intended scopes in magically synthesized functions; like (say)

p = None # to establish the intended scope of `p`
while any(<nonlocal p>  # split across lines just for readability
                n % p == 0 for p in small_primes):
    n //= p

It didn't really require an inline assignment, just a way to override the unwanted (in this case) "all `for` targets are local to the invisible function" rigid consequence of the implementation du jour.

Hm, that's more special syntax. The nice bit about (b) as currently specified is that it adds no syntax -- it adds a scope rule, but (as IIRC Steven has convincingly argued) few people care about those. Python's scope rules, when fully specified, are intricate to the point of being arcane (e.g. for class scopes) but all that has a purpose -- to make them so DWIM ("Do what I Mean") that in practice you almost never have to worry about them, *especially* when reading non-obfuscated code (and also when writing, except for a few well-known patterns).
 
Second, if it's dropped, then the PEP needs more words to define what happens in cases like the following, because different textual parts of a comprehension execute in different scopes, and that can  become visible when bindings can be embedded:

def f():
    y = -1
    ys = [y for _ in range(y := 5)]
    print(y, ys)

Here `range(y := 5)` is executed in f's scope.  Presumably the `y` in `y for` also refers to f's scope, despite that `y` textually _appears_ to be assigned to in the body of the listcomp, and so would - for that reason - expected to be local to the synthesized function, and so raise `UnboundLocalError` when referenced.  It's incoherent without detailed knowledge of the implementation.

That code should have the same meaning regardless of whether we accept (b) or not -- there is only one `y`, in f's scope. I don't mind if we have to add more words to the PEP's scope rules to make this explicit, though I doubt it -- the existing weirdness (in the comprehension spec) about the "outermost iterable" being evaluated in the surrounding scope specifies this. I wouldn't call it incoherent -- I think what I said about scope rules above applies here, it just does what you expect.
 
def g():
    y = -1
    ys = [y for y in range(y := 5)]
    print(y, ys)

And here the `y` in `y for y` is local to the synthesized function, and presumably has nothing to do with the `y` in the `range()` call.  That's incoherent in its own way.

Under the current PEP, all instances of `y` in `f` refer to the f-local `y`, and the listcomp in `g` is a compile-time error.

And under the (b)-less proposal, `g` would interpret `y for y` as both referring to a new variable created just for the comprehension, and `y := 5` as referring to g's scope. Again I don't think it needs extra words in the spec. And the end user docs might just say "don't do that" (with a link to the reference manual's rule about the "outermost iterable").

Even if in the end we did find a case where we'd have to write an explicit rule to make what happens here a consequence of the spec rather than the implementation, that doesn't count as an argument for keeping (b) to me.

In favor of (b) we have a few examples (see https://www.python.org/dev/peps/pep-0572/#scope-of-the-target) that require it, and more that you described on python-ideas (and also the motivating use case from the thread that Steven dug up, starting here: https://mail.python.org/pipermail/python-ideas/2018-April/049622.html).

A "neutral" argument about (b) is that despite the "horrified" reactions that Nick saw, in practice it's going to confuse very few people (again, due to my point about Python's scope rules). I'd wager that the people who might be most horrified about it would be people who feel strongly that the change to the comprehension scope rules in Python 3 is a big improvement, and who are familiar with the difference in implementation of comprehensions (though not generator expressions) in Python 2 vs. 3.

Greg Ewing

unread,
Jun 24, 2018, 7:32:19 PM6/24/18
to Python-Dev
Guido van Rossum wrote:
> Greg seem to be +0 or better for (a)

Actually, I'm closer to -1 on (a) as well. I don't like := as a
way of getting assignment in an expression. The only thing I would
give a non-negative rating is some form of "where" or "given".

Brief summary of reasons for disliking ":=":

* Cryptic use of punctuation

* Too much overlap in functionality with "="

* Asymmetry between first and subsequent uses of the bound value

* Makes expressions cluttered and hard to read to my eyes

--
Greg

Guido van Rossum

unread,
Jun 24, 2018, 7:39:31 PM6/24/18
to Chris Angelico, Python-Dev
On Sun, Jun 24, 2018 at 2:10 PM Chris Angelico <ros...@gmail.com> wrote:
On Mon, Jun 25, 2018 at 4:06 AM, Steven D'Aprano <st...@pearwood.info> wrote:
>
> Remember, the driving use-case which started this (ever-so-long)
> discussion was the ability to push data into a comprehension and then
> update it on each iteration, something like this:
>
>     x = initial_value()
>     results = [x := transform(x, i) for i in sequence]

Which means there is another option.

5. Have the assignment be local to the comprehension, but the initial
value of ANY variable is looked up from the surrounding scopes.

That is: you will NEVER get UnboundLocalError from a
comprehension/genexp; instead, you will look up the name as if it were
in the surrounding scope, either getting a value or bombing with
regular old NameError.

Or possibly variations on this such as "the immediately surrounding
scope only", rather than full name lookups. It'd have an awkward
boundary somewhere, whichever way you do it.

This isn't able to send information *out* of a comprehension, but it
is able to send information *in*.

But this "horrifies" me for a slightly different reason: it effectively introduces a new case of dynamic scoping, which Python used to do everywhere but has long switched away from, with the exception of class scopes (whose difference with function scopes sometimes confuses people -- usually people who put too much code in their class scope).

Greg Ewing

unread,
Jun 24, 2018, 7:42:52 PM6/24/18
to pytho...@python.org
Steven D'Aprano wrote:
> You seem to be talking about an implementation which could change in the
> future. I'm talking semantics of the proposed language feature.

The way I see it, it's not about implementation details,
it's about having a mental model that's easy to reason
about.

"Comprehensions run in their own scope, like a def or
lambda" is a clear and simple mental model. It's easy to
explain and keep in your head.

The proposed semantics are much more complicated, and as
far as I can see, are only motivated by use cases that
you shouldn't really be doing in the first place.

--
Greg

Ben Finney

unread,
Jun 24, 2018, 7:44:06 PM6/24/18
to pytho...@python.org
Greg Ewing <greg....@canterbury.ac.nz> writes:

> Actually, I'm closer to -1 on (a) as well. I don't like := as a
> way of getting assignment in an expression. The only thing I would
> give a non-negative rating is some form of "where" or "given".

+1 to this; ‘:=’ doesn't convey the meaning well. Python's syntax
typically eschews cryptic punctuation in faviour of existing words that
convey an appropriate meaning, and I agree with Greg that would be a
better way to get this effect.

--
\ “Self-respect: The secure feeling that no one, as yet, is |
`\ suspicious.” —Henry L. Mencken |
_o__) |
Ben Finney

Guido van Rossum

unread,
Jun 24, 2018, 8:07:33 PM6/24/18
to Michael Selik, Python-Dev
On Sun, Jun 24, 2018 at 2:41 PM Michael Selik <mi...@selik.org> wrote:
This thread started with a request for educator feedback, which I took to mean observations of student reactions. I've only had the chance to test the proposal on ~20 students so far, but I'd like the chance to gather more data for your consideration before the PEP is accepted or rejected.

Sure. Since the target for the PEP is Python 3.8 I am in no particular hurry. It would be important to know how you present it to your students.
 
On Sun, Jun 24, 2018 at 11:09 AM Steven D'Aprano <st...@pearwood.info> wrote:
Remember, the driving use-case which started this (ever-so-long)
discussion was the ability to push data into a comprehension and then
update it on each iteration, something like this:

    x = initial_value()
    results = [x := transform(x, i) for i in sequence]

If that is the driving use-case, then the proposal should be rejected. The ``itertools.accumulate`` function has been available for a little while now and it handles this exact case. The accumulate function may even be more readable, as it explains the purpose explicitly, not merely the algorithm. And heck, it's a one-liner.

    results = accumulate(sequence, transform)

I think that's a misunderstanding. At the very least the typical use case is *not* using an existing transform function which is readily passed to accumulate -- instead, it's typically written as a simple expression (e.g. `total := total + v` in the PEP) which would require a lambda.

Plus, I don't know what kind of students you are teaching, but for me, whenever the solution requires a higher-order function (like accumulate), this implies a significant speed bump -- both when writing and when reading code. (Honestly, whenever I read code that uses itertools, I end up making a trip to StackOverflow :-).
 
The benefits for ``any`` and ``all`` seem useful. Itertools has "first_seen" in the recipes section.

(I think you mean first_true().)
 
While it feels intuitively useful, I can't recall ever writing something similar myself. For some reason, I (almost?) always want to find all (counter-)examples and aggregate them in some way -- min or max, perhaps -- rather than just get the first.

I trust Tim's intuition here, he's written about this.

Also, Python's predecessor, ABC, had quantifiers (essentially any() and all()) built into the language, and the semantics included making the first (counter-)example available (https://homepages.cwi.nl/~steven/abc/qr.html#TESTS). Essentially

IF SOME x IN values HAS x < 0:
    WRITE "Found a negative x:", x

equivalently

IF EACH x IN values HAS x >= 0:
    # ...
ELSE:
    WRITE "Found a negative x:", x

and even

IF NO x IN values HAS x < 0:
    # ...
ELSE:
    WRITE "Found a negative x:", x
 
Even so, if it turns out those uses are quite prevalent, wouldn't a new itertool be better than a new operator? It's good to solve the general problem, but so far the in-comprehension usage seems to have only a handful of cases.

Perhaps, but IMO the new itertool would be much less useful than the new syntax.
 
On Fri, Jun 22, 2018 at 9:14 PM Chris Barker via Python-Dev <pytho...@python.org> wrote:
again, not a huge deal, just a little bit more complexity

I worry that Python may experience something of a "death by a thousand cuts" along the lines of the "Remember the Vasa" warning. Python's greatest strength is its appeal to beginners. Little bits of added complexity have a non-linear effect. One day, we may wake up and Python won't be recommended as a beginner's language.

I don't think that appeal to beginners is Python's greatest strength. I'd rather say that it is its appeal to both beginners and experts (and everyone in between). If true appeal to beginners is needed, Scratch or Processing would probably be better.
 
On Fri, Jun 22, 2018 at 7:48 PM Steven D'Aprano <st...@pearwood.info> wrote:
On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:

Of course they do -- they're less fluent at reading code. They don't 
have the experience to judge good code from bad.

On the other hand, an "expert" may be so steeped in a particular subculture that [they] no longer can distinguish esoteric from intuitive. Don't be so fast to reject the wisdom of the inexperienced.

Nor should we cater to them excessively though. While the user is indeed king, it's also well known that most users when they are asking for a feature don't know what they want (same for kings, actually, that's why they have advisors :-).
 
The question we should be asking is, do we only add features to Python 
if they are easy for beginners? It's not that I especially want to add 
features which *aren't* easy for beginners, but Python isn't Scratch and 
"easy for beginners" should only be a peripheral concern.

On the contrary, I believe that "easy for beginners" should be a major concern.  Ease of use has been and is a, or even the main reason for Python's success. When some other language becomes a better teaching language, it will eventually take over in business and science as well. Right now, Python is Scratch for adults. That's a great thing. Given the growth of the field, there are far more beginner programmers working today than there ever have been experts.

I'm sorry, but this offends me, and I don't believe it's true at all. Python is *not* a beginners language, and you are mixing ease of use and ease of learning. Python turns beginners into experts at an unprecedented rate, and that's the big difference with Scratch.

Michael Selik

unread,
Jun 24, 2018, 8:36:06 PM6/24/18
to gu...@python.org, Python-Dev
On Sun, Jun 24, 2018 at 4:57 PM Guido van Rossum <gu...@python.org> wrote:
On Sun, Jun 24, 2018 at 2:41 PM Michael Selik <mi...@selik.org> wrote:
This thread started with a request for educator feedback, which I took to mean observations of student reactions. I've only had the chance to test the proposal on ~20 students so far, but I'd like the chance to gather more data for your consideration before the PEP is accepted or rejected.

Sure. Since the target for the PEP is Python 3.8 I am in no particular hurry. It would be important to know how you present it to your students.

Absolutely. Since this has come up, I'll make an effort to be more systematic in data collection. 


 
On Sun, Jun 24, 2018 at 11:09 AM Steven D'Aprano <st...@pearwood.info> wrote:
Remember, the driving use-case which started this (ever-so-long)
discussion was the ability to push data into a comprehension and then
update it on each iteration, something like this:

    x = initial_value()
    results = [x := transform(x, i) for i in sequence]

If that is the driving use-case, then the proposal should be rejected. The ``itertools.accumulate`` function has been available for a little while now and it handles this exact case. The accumulate function may even be more readable, as it explains the purpose explicitly, not merely the algorithm. And heck, it's a one-liner.

    results = accumulate(sequence, transform)

I think that's a misunderstanding. At the very least the typical use case is *not* using an existing transform function which is readily passed to accumulate -- instead, it's typically written as a simple expression (e.g. `total := total + v` in the PEP) which would require a lambda.

Plus, I don't know what kind of students you are teaching, but for me, whenever the solution requires a higher-order function (like accumulate), this implies a significant speed bump -- both when writing and when reading code. (Honestly, whenever I read code that uses itertools, I end up making a trip to StackOverflow :-).

Mostly mid-career professionals, of highly varying backgrounds. The higher-order functions do require some cushioning getting into, but I have some tricks I've learned over the years to make it go over pretty well.


On Fri, Jun 22, 2018 at 7:48 PM Steven D'Aprano <st...@pearwood.info> wrote:
On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:

Of course they do -- they're less fluent at reading code. They don't 
have the experience to judge good code from bad.

On the other hand, an "expert" may be so steeped in a particular subculture that [they] no longer can distinguish esoteric from intuitive. Don't be so fast to reject the wisdom of the inexperienced.

Nor should we cater to them excessively though. While the user is indeed king, it's also well known that most users when they are asking for a feature don't know what they want (same for kings, actually, that's why they have advisors :-).
 
The question we should be asking is, do we only add features to Python 
if they are easy for beginners? It's not that I especially want to add 
features which *aren't* easy for beginners, but Python isn't Scratch and 
"easy for beginners" should only be a peripheral concern.

On the contrary, I believe that "easy for beginners" should be a major concern.  Ease of use has been and is a, or even the main reason for Python's success. When some other language becomes a better teaching language, it will eventually take over in business and science as well. Right now, Python is Scratch for adults. That's a great thing. Given the growth of the field, there are far more beginner programmers working today than there ever have been experts.

I'm sorry, but this offends me, and I don't believe it's true at all. Python is *not* a beginners language, and you are mixing ease of use and ease of learning. Python turns beginners into experts at an unprecedented rate, and that's the big difference with Scratch.

By saying "Scratch for adults" I meant that Python is a language that can be adopted by beginners and rapidly make them professionals, not that it's exclusively a beginner's language.

Also, Scratch and similar languages, like NetLogo, have some interesting features that allow beginners to write some sophisticated parallelism. I don't mean "beginner's language" in that it's overly simplistic, but that it enables what would be complex in other languages.

I realize that my phrasing was likely to be misunderstood without knowing the context that I teach working professionals who are asked to be immediately productive at high-value tasks.

Tim Peters

unread,
Jun 24, 2018, 10:48:31 PM6/24/18
to Guido van Rossum, Nick Coghlan, Python Dev
[Tim]
.  First, the original example I gave would be approximately as well addressed by allowing to declare intended scopes in magically synthesized functions; like (say)

p = None # to establish the intended scope of `p`
while any(<nonlocal p>  # split across lines just for readability
                n % p == 0 for p in small_primes):
    n //= p

It didn't really require an inline assignment, just a way to override the unwanted (in this case) "all `for` targets are local to the invisible function" rigid consequence of the implementation du jour.

[Guido]
Hm, that's more special syntax.

Of course - I'm anticipating that the PEP will be changed to throw out useful assignment expressions in comprehensions, but I still want a way to "export" comprehension for-targets at times ;-)
 
The nice bit about (b) as currently specified is that it adds no syntax -- it adds a scope rule, but (as IIRC Steven has convincingly argued) few people care about those. Python's scope rules, when fully specified, are intricate to the point of being arcane (e.g. for class scopes) but all that has a purpose -- to make them so DWIM ("Do what I Mean") that in practice you almost never have to worry about them, *especially* when reading non-obfuscated code (and also when writing, except for a few well-known patterns).

You and Steven and i appear to be on the same page here - but it's in a book nobody else seems to own :-(  To me it's just screamingly obvious that

    total = 0
    cumsums = [total := total + value for value in data]

"should do" what it obviously intends to do - and that the only thing stopping that is a bass-ackwards focus on what most trivially falls out of the current implementation.

...

def f():
    y = -1
    ys = [y for _ in range(y := 5)]
    print(y, ys)

Here `range(y := 5)` is executed in f's scope.  Presumably the `y` in `y for` also refers to f's scope, despite that `y` textually _appears_ to be assigned to in the body of the listcomp, and so would - for that reason - expected to be local to the synthesized function, and so raise `UnboundLocalError` when referenced.  It's incoherent without detailed knowledge of the implementation.

That code should have the same meaning regardless of whether we accept (b) or not -- there is only one `y`, in f's scope. I don't mind if we have to add more words to the PEP's scope rules to make this explicit, though I doubt it -- the existing weirdness (in the comprehension spec) about the "outermost iterable" being evaluated in the surrounding scope specifies this. I wouldn't call it incoherent -- I think what I said about scope rules above applies here, it just does what you expect.

Remove "y = -1" and - voila! - we have the dreaded "parent local scoping" Nick finds so baffling to explain (or so he claims).  That is, "has exactly the same scope in the comprehension as in the parent block, and will create a local in the latter if the name is otherwise unknown in the parent" comes with assignment expressions, regardless of whether _all_ such targets "leak" (the current PEP) or only targets in the expression defining the iterable of the outermost `for` (the PEP without leaking assignment expressions in comprehensions).

As to whether it "does what you expect", no, not really! In a world where _all_ binding targets in a comprehension are claimed to be local to the comprehension, I _expect_ that `y := 5` appearing inside the listcomp means `y` is local to the listcomp.  "Oh - unless the binding appears in the expression defining the iterable of the outermost `for`" comes from Mars.

Not that it really matters much, but (b) provides consistent semantics in these cases.  No need to search Mars for weird exceptions ;-)

...
A "neutral" argument about (b) is that despite the "horrified" reactions that Nick saw, in practice it's going to confuse very few people (again, due to my point about Python's scope rules). I'd wager that the people who might be most horrified about it would be people who feel strongly that the change to the comprehension scope rules in Python 3 is a big improvement, and who are familiar with the difference in implementation of comprehensions (though not generator expressions) in Python 2 vs. 3.

I also doubt it will generally confuse people in practice (to the contrary, I expect they _will_ be confused if things like the cumulative sums example blow up with UnboundLocalError).

But I still don't get the source of the "horror".  Assignment expression semantics are wholly consistent with ordinary nested lexical scoping, with or without (b).  The only difference is in the scopes picked for assignment expression target names (except for those appearing in the expression defining the iterable yadda yadda yadda).

Nick Coghlan

unread,
Jun 25, 2018, 7:46:40 AM6/25/18
to Guido van Rossum, Tim Peters, Python-Dev
On 25 June 2018 at 02:24, Guido van Rossum <gu...@python.org> wrote:
> A quick follow-up: PEP 572 currently has two ideas: (a) introduce := for
> inline assignment, (b) when := is used in a comprehension, set the scope for
> the target as if the assignment occurred outside any comprehensions. It
> seems we have more support for (a) than for (b) -- at least Nick and Greg
> seem to be +0 or better for (a)

Right, the proposed blunt solution to "Should I use 'NAME = EXPR' or
'NAME := EXPR'?" bothers me a bit, but it's the implementation
implications of parent local scoping that I fear will create a
semantic tar pit we can't get out of later.

> but -1 for (b). IIRC (b) originated with
> Tim. But his essay on the topic, included as Appendix A
> (https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings)
> does not even mention comprehensions. However, he did post his motivation
> for (b) on python-ideas, IIRC a bit before PyCon; and the main text of the
> PEP gives a strong motivation
> (https://www.python.org/dev/peps/pep-0572/#scope-of-the-target).
> Nevertheless, maybe we should compromise and drop (b)?

Unfortunately, I think the key rationale for (b) is that if you
*don't* do something along those lines, then there's a different
strange scoping discrepancy that arises between the non-comprehension
forms of container displays and the comprehension forms:

(NAME := EXPR,) # Binds a local
tuple(NAME := EXPR for __ in range(1)) # Doesn't bind a local

[NAME := EXPR] # Binds a local
[NAME := EXPR for __ in range(1)] # Doesn't bind a local
list(NAME := EXPR for __ in range(1)) # Doesn't bind a local

{NAME := EXPR} # Binds a local
{NAME := EXPR for __ in range(1)} # Doesn't bind a local
set(NAME := EXPR for __ in range(1)) # Doesn't bind a local

{NAME := EXPR : EXPR2} # Binds a local
{NAME := EXPR : EXPR2 for __ in range(1)} # Doesn't bind a local
set((NAME := EXPR, EXPR2) for __ in range(1)) # Doesn't bind a local

Those scoping inconsistencies aren't *new*, but provoking them
currently involves either class scopes, or messing about with
locals().

The one virtue that choosing this particular set of discrepancies has
is that the explanation for why they happen is the same as the
explanation for how the iteration variable gets hidden from the
containing scope: because "(EXPR for ....)" et al create an implicitly
nested scope, and that nested scope behaves the same way as an
explicitly nested scope as far as name binding and name resolution is
concerned.

Parent local scoping tries to mitigate the surface inconsistency by
changing how write semantics are defined for implicitly nested scopes,
but that comes at the cost of making those semantics inconsistent with
explicitly nested scopes and with the read semantics of implicitly
nested scopes.

The early iterations of PEP 572 tried to duck this whole realm of
potential semantic inconsistencies by introducing sublocal scoping
instead, such that the scoping for assignment expression targets would
be unusual, but they'd be consistently unusual regardless of where
they appeared, and their quirks would clearly be the result of how
assignment expressions were defined, rather than only showing up in
how they interacted with other scoping design decisions made years
ago.

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

Nick Coghlan

unread,
Jun 25, 2018, 8:19:12 AM6/25/18
to Guido van Rossum, Tim Peters, Python-Dev
On 25 June 2018 at 09:25, Guido van Rossum <gu...@python.org> wrote:
> A "neutral" argument about (b) is that despite the "horrified" reactions
> that Nick saw, in practice it's going to confuse very few people (again, due
> to my point about Python's scope rules). I'd wager that the people who might
> be most horrified about it would be people who feel strongly that the change
> to the comprehension scope rules in Python 3 is a big improvement, and who
> are familiar with the difference in implementation of comprehensions (though
> not generator expressions) in Python 2 vs. 3.

FWIW, the most cryptic parent local scoping related exception I've
been able to devise so far still exhibits PEP 572's desired "Omitting
the comprehension scope entirely would give you the same name lookup
behaviour" semantics:

>>> def outer(x=1):
... def middle():
... return [x := x + i for i in range(10)]
... return middle()
...
>>> outer()
Traceback (most recent call last):
...
NameError: free variable 'x' referenced before assignment in enclosing scope

It isn't the parent local scoping, or even the assignment expression,
that's at fault there, since you'd get exactly the same exception for:

def outer(x=1):
def middle():
x = x +1
return x
return middle()

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

Paul Moore

unread,
Jun 25, 2018, 8:27:15 AM6/25/18
to Nick Coghlan, Tim Peters, Python-Dev
On 25 June 2018 at 12:44, Nick Coghlan <ncog...@gmail.com> wrote:
> Unfortunately, I think the key rationale for (b) is that if you
> *don't* do something along those lines, then there's a different
> strange scoping discrepancy that arises between the non-comprehension
> forms of container displays and the comprehension forms:

I've been mostly ignoring this proposal for a while now, so I'm going
to respond here in the context of someone with a bit of an idea of the
underlying complexities, but otherwise coming at it as a new proposal.

>
> (NAME := EXPR,) # Binds a local
> tuple(NAME := EXPR for __ in range(1)) # Doesn't bind a local
>
> [NAME := EXPR] # Binds a local
> [NAME := EXPR for __ in range(1)] # Doesn't bind a local
> list(NAME := EXPR for __ in range(1)) # Doesn't bind a local
>
> {NAME := EXPR} # Binds a local
> {NAME := EXPR for __ in range(1)} # Doesn't bind a local
> set(NAME := EXPR for __ in range(1)) # Doesn't bind a local
>
> {NAME := EXPR : EXPR2} # Binds a local
> {NAME := EXPR : EXPR2 for __ in range(1)} # Doesn't bind a local
> set((NAME := EXPR, EXPR2) for __ in range(1)) # Doesn't bind a local

None of those "discrepancies" bother me in the slightest, when taken
in isolation as you present them here. I suspect you could lead me
through a chain of logic that left me understanding why you describe
them as discrepancies, but without that explanation, I'm fine with all
of them.

I'd also say that they seem contrived (not just in the use of
artificial names, but also in the sense that I'm not sure why I'd want
to use this *pattern*) so I'd happily say "well, don't do that then"
if things started behaving non-intuitively.

> Those scoping inconsistencies aren't *new*, but provoking them
> currently involves either class scopes, or messing about with
> locals().

And to reinforce my point above, I already consider putting
significant code in class scopes, or using locals() to be techniques
that should only be used sparingly and with a clear understanding of
the subtleties. I'm sure you could say "but the examples above would
be much more common" in response to which I'd like to see real use
cases that behave non-intuitively in the way you're concerned about.

> The one virtue that choosing this particular set of discrepancies has
> is that the explanation for why they happen is the same as the
> explanation for how the iteration variable gets hidden from the
> containing scope: because "(EXPR for ....)" et al create an implicitly
> nested scope, and that nested scope behaves the same way as an
> explicitly nested scope as far as name binding and name resolution is
> concerned.

But that's precisely why I find the behaviour intuitive - the nested
scope is the *reason* things behave this way, not some sort of
easily-overlooked way the "problem" can be explained away.

> Parent local scoping tries to mitigate the surface inconsistency by
> changing how write semantics are defined for implicitly nested scopes,
> but that comes at the cost of making those semantics inconsistent with
> explicitly nested scopes and with the read semantics of implicitly
> nested scopes.
>
> The early iterations of PEP 572 tried to duck this whole realm of
> potential semantic inconsistencies by introducing sublocal scoping
> instead, such that the scoping for assignment expression targets would
> be unusual, but they'd be consistently unusual regardless of where
> they appeared, and their quirks would clearly be the result of how
> assignment expressions were defined, rather than only showing up in
> how they interacted with other scoping design decisions made years
> ago.

Those last two paragraphs made my head explode, as far as I can see by
virtue of the fact that they try to over-analyze the fairly simple
intuition I have that "there's a nested scope involved".

Disclaimer: I may well have got a *lot* of subtleties wrong here, and
it's quite likely that my impressions don't stand up to the harsh
reality of how the implementation works. But my comments are on the
basis of my *intuition*, whether that's right or wrong. And if the
reality violates my intuition, it's *other* constructs that I find
non-intuitive, not this one. (I'm perfectly happy to concede that it's
not possible to avoid *any* non-intuitive behaviour - all I'm trying
to say is that my intuition doesn't balk at this one, unlike yours).

Paul

Paul Moore

unread,
Jun 25, 2018, 8:30:21 AM6/25/18
to Nick Coghlan, Tim Peters, Python-Dev
Once again offering an "intuition" based response:

1. That definition of outer() is very complicated, I don't *expect* to
understand it without checking the details. So the NameError is simply
"hmm, wonder what triggered that?" not "OMG that's not what I'd
expect!" :-)
2. Given that your version with no assignment expression or
comprehension exhibits the same behaviour, I'm not sure what your
argument is here anyway...

Paul

Nick Coghlan

unread,
Jun 25, 2018, 8:33:33 AM6/25/18
to Guido van Rossum, Tim Peters, Python-Dev
On 25 June 2018 at 22:17, Nick Coghlan <ncog...@gmail.com> wrote:
> FWIW, the most cryptic parent local scoping related exception I've
> been able to devise so far still exhibits PEP 572's desired "Omitting
> the comprehension scope entirely would give you the same name lookup
> behaviour" semantics:
>
> >>> def outer(x=1):
> ... def middle():
> ... return [x := x + i for i in range(10)]
> ... return middle()
> ...
> >>> outer()
> Traceback (most recent call last):
> ...
> NameError: free variable 'x' referenced before assignment in enclosing scope
>
> It isn't the parent local scoping, or even the assignment expression,
> that's at fault there, since you'd get exactly the same exception for:
>
> def outer(x=1):
> def middle():
> x = x +1
> return x
> return middle()

Oops, I didn't mean to say "exactly the same exception" here, as the
whole reason I'd settled on this example as the most cryptic one I'd
found so far was the fact that the doubly nested version *doesn't*
give you the same exception as the singly nested version: the version
without the comprehension throws UnboundLocalError instead.

However, the resolution is the same either way: either 'x' has to be
declared as 'nonlocal x' in 'middle', or else it has to be passed in
to 'middle' as a parameter.

Nick Coghlan

unread,
Jun 25, 2018, 8:45:23 AM6/25/18
to Guido van Rossum, Python-Dev
On 25 June 2018 at 09:02, Guido van Rossum <gu...@python.org> wrote:
> On Sun, Jun 24, 2018 at 11:50 AM Steven D'Aprano <st...@pearwood.info>
> wrote:
>> I will have more to say about the whole "comprehensions are their own
>> scope" issue later. But I'd like to see Nick's proposed PEP, or at least
>> a draft of it, before making any final decisions.
>
>
> Agreed, though I assume it's just `given` again.

While I still have some TODO notes of my own to resolve before posting
it to python-ideas, the examples section at
https://github.com/ncoghlan/peps/pull/2/files#diff-7a25ca1769914c1141cb5c63dc781f32R202
already gives a pretty good idea of the differences relative to PEP
572: rebinding existing names is unchanged from PEP 572, but
introducing new names requires a bit of "Yes, I really do want to
introduce this new name here" repetition.

The big difference from previous iterations of the "given" idea is
that it doesn't try to completely replace the proposed inline
assignments, it just supplements them by providing a way to do inline
name *declarations* (which may include declaring targets as global or
nonlocal, just as regular function level declarations can).

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

Paul Moore

unread,
Jun 25, 2018, 9:48:23 AM6/25/18
to Nick Coghlan, Tim Peters, Python-Dev
At the level of "what my intuition says" the result is the same in
both cases - "it throws an exception". I have no intuition on *which*
exception would be raised and would experiment (or look up the
details) if I cared.

> However, the resolution is the same either way: either 'x' has to be
> declared as 'nonlocal x' in 'middle', or else it has to be passed in
> to 'middle' as a parameter.

Once someone told me that's what I needed, it's sufficiently obvious
that I'm fine with that. If no-one was able to tell me what to do, I'd
simply rewrite the code to be less obfuscated :-)

I've probably explained my intuition enough here. If we debate any
further I'll just end up knowing what's going on, destroying my value
as an "uninformed user" :-)
Paul

Ivan Pozdeev via Python-Dev

unread,
Jun 25, 2018, 10:21:22 AM6/25/18
to pytho...@python.org
On 25.06.2018 2:30, Greg Ewing wrote:
> Guido van Rossum wrote:
>> Greg seem to be +0 or better for (a)
>
> Actually, I'm closer to -1 on (a) as well. I don't like := as a
> way of getting assignment in an expression. The only thing I would
> give a non-negative rating is some form of "where" or "given".
>
"as" was suggested even before is became a keyword in `with'. ( if
(re.match(regex,line) as m) is not None: <do smth> )

The only objective objection I've heard is it's already used in `import'
and `with' -- but that's perfectly refutable.


> Brief summary of reasons for disliking ":=":
>
> * Cryptic use of punctuation
>
> * Too much overlap in functionality with "="
>
> * Asymmetry between first and subsequent uses of the bound value
>
> * Makes expressions cluttered and hard to read to my eyes
>

--
Regards,
Ivan

Ivan Pozdeev via Python-Dev

unread,
Jun 25, 2018, 11:04:02 AM6/25/18