[Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Nick Coghlan]

On 22 June 2018 at 02:26, Antoine Pitrou <soli...@pitrou.net> wrote:
> Indeed. But, for a syntax addition such as PEP 572, I think it would be
> a good idea to ask their opinion to teaching/education specialists.
>
> As far as I'm concerned, if teachers and/or education specialists were
> to say PEP 572 is not a problem, my position would shift from negative
> towards neutral.

I asked a handful of folks at the Education Summit the next day about it:

* for the basic notion of allowing expression level name binding using
the "NAME := EXPR" notation, the reactions ranged from mildly negative
(I read it as only a "-0" rather than a "-1") to outright positive.
* for the reactions to my description of the currently proposed parent
local scoping behaviour in comprehensions, I'd use the word
"horrified", and feel I wasn't overstating the response :)

While I try to account for the fact that I implemented the current
comprehension semantics for the 3.x series, and am hence biased
towards considering them the now obvious interpretation, it's also the
case that generator expressions have worked like nested functions
since they were introduced in Python 2.4 (more than 13 years ago now),
and comprehensions have worked the same way as generator expressions
since Python 3.0 (which has its 10th birthday coming up in December
this year).

This means that I take any claims that the legacy Python 2.x
interpretation of comprehension behaviour is intuitively obvious with
an enormous grain of salt - for the better part of a decade now, every
tool at a Python 3 user's disposal (the fact that the iteration
variable is hidden from the current scope, reading the language
reference [1], printing out locals(), using the dis module, stepping
through code in a debugger, writing their own tracing function, and
even observing the quirky interaction with class scopes) will have
nudged them towards the "it's a hidden nested function" interpretation
of expected comprehension behaviour.

Acquiring the old mental model for the way comprehensions work pretty
much requires a developer to have started with Python 2.x themselves
(perhaps even before comprehensions and lexical closures were part of
the language), or else have been taught the Python 2 comprehension
model by someone else - there's nothing in Python 3's behaviour to
encourage that point of view, and plenty of
functional-language-inspired documentation to instead encourage folks
to view comprehensions as tightly encapsulated declarative container
construction syntax.

I'm currently working on a concept proposal at
https://github.com/ncoghlan/peps/pull/2 that's much closer to PEP 572
than any of my previous `given` based suggestions: for already
declared locals, it devolves to being the same as PEP 572 (except that
expressions are allowed as top level statements), but for any names
that haven't been previously introduced, it prohibits assigning to a
name that doesn't already have a defined scope, and instead relies on
a new `given` clause on various constructs that allows new target
declarations to be introduced into the current scope (such that "if
x:= f():" implies "x" is already defined as a target somewhere else in
the current scope, while "if x := f() given x:" potentially introduces
"x" as a new local target the same way a regular assignment statement
does).

One of the nicer features of the draft proposal is that if all you
want to do is export the iteration variable from a comprehension, you
don't need to use an assignment expression at all: you can just append
"... given global x" or "... given nonlocal x" and export the
iteration variable directly to the desired outer scope, the same way
you can in the fully spelled out nested function equivalent.

Cheers,
Nick.

[1] From https://docs.python.org/3.0/reference/expressions.html#displays-for-lists-sets-and-dictionaries:
'Note that the comprehension is executed in a separate scope, so names
assigned to in the target list don’t “leak” in the enclosing scope.'
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Antoine Pitrou]

On Sat, 23 Jun 2018 00:22:33 +1000
Nick Coghlan <ncog...@gmail.com> wrote:
> On 22 June 2018 at 02:26, Antoine Pitrou <soli...@pitrou.net> wrote:
> > Indeed. But, for a syntax addition such as PEP 572, I think it would be
> > a good idea to ask their opinion to teaching/education specialists.
> >
> > As far as I'm concerned, if teachers and/or education specialists were
> > to say PEP 572 is not a problem, my position would shift from negative
> > towards neutral.
>
> I asked a handful of folks at the Education Summit the next day about it:
>
> * for the basic notion of allowing expression level name binding using
> the "NAME := EXPR" notation, the reactions ranged from mildly negative
> (I read it as only a "-0" rather than a "-1") to outright positive.

Thank you. Personally, I'd like to see feedback from
educators/teachers after they take the time to read the PEP and take
some time to think about its consequences.

My main concern is we're introducing a second different way of doing
something which is really fundamental.

> * for the reactions to my description of the currently proposed parent
> local scoping behaviour in comprehensions, I'd use the word

> "horrified", and feel I wasn't overstating the response :) [...]

Hmm... I don't think conflating the assignment expression proposal
with comprehension semantics issues is helping the discussion.

Regards

Antoine.

[Michael Selik]

On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <soli...@pitrou.net> wrote:

Thank you.  Personally, I'd like to see feedback from
educators/teachers after they take the time to read the PEP and take
some time to think about its consequences.

I've started testing the proposed syntax when I teach. I don't have a large sample yet, but most students either dislike it or don't appreciate the benefits. They state a clear preference for shorter, simpler lines at the consequence of more lines of code. This may partially be caused by the smaller screen real estate on a projector or large TV than a desktop monitor.

My intuition is that one strength of Python for beginners is the relative lack of punctuation and operators compared with most other languages. This proposal encourages denser lines with more punctuation. Because of the order of operations, many uses of ``:=`` will also require parentheses. Even relatively simple uses, like ``if (match := pattern.search(data)) is not None:`` require doubled parentheses on one side or the other. Beginners are especially prone to typographical errors with mismatched parentheses and missing colons and get easily frustrated by the associated syntax errors.

Given the following options:

A.

    if (row := cursor.fetchone()) is None:

        raise NotFound

    return row

B.

    row = cursor.fetchone()

    if row is None:

        raise NotFound

    return row

C.

    if (row := cursor.fetchone()) is not None:

        return row

    raise NotFound

D.

    row = cursor.fetchone()

    if row is not None:

        return row

    raise NotFound

The majority of students preferred option B. I also tested some regex match examples. Results were similar.

 

My main concern is we're introducing a second different way of doing
something which is really fundamental.

The few students who like the proposal ask why it requires creating a new operator instead of repurposing the ``=`` operator.

I'll reserve my personal opinions for a different thread.

[Michael Selik]

On Fri, Jun 22, 2018 at 10:02 AM Michael Selik <mi...@selik.org> wrote:

On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <soli...@pitrou.net> wrote:

Thank you.  Personally, I'd like to see feedback from
educators/teachers after they take the time to read the PEP and take
some time to think about its consequences.

I forgot to add that I don't anticipate changing my lesson plans if this proposal is accepted. There's already not enough time to teach everything I'd like. Including a new assignment operator would distract from the learning objectives.

 

[Chris Angelico]

On Sat, Jun 23, 2018 at 3:02 AM, Michael Selik <mi...@selik.org> wrote:
> On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <soli...@pitrou.net> wrote:
>>
>> Thank you. Personally, I'd like to see feedback from
>> educators/teachers after they take the time to read the PEP and take
>> some time to think about its consequences.
>
>
> I've started testing the proposed syntax when I teach. I don't have a large
> sample yet, but most students either dislike it or don't appreciate the
> benefits. They state a clear preference for shorter, simpler lines at the
> consequence of more lines of code.

This is partly because students, lacking the experience to instantly
recognize larger constructs, prefer a more concrete approach to
coding. "Good code" is code where the concrete behaviour is more
easily understood. As a programmer gains experience, s/he learns to
grok more complex expressions, and is then better able to make use of
the more expressive constructs such as list comprehensions.

ChrisA

[Michael Selik]

On Fri, Jun 22, 2018 at 10:19 AM Chris Angelico <ros...@gmail.com> wrote:

On Sat, Jun 23, 2018 at 3:02 AM, Michael Selik <mi...@selik.org> wrote:
> On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <soli...@pitrou.net> wrote:
>>
>> Thank you.  Personally, I'd like to see feedback from
>> educators/teachers after they take the time to read the PEP and take
>> some time to think about its consequences.
>
>
> I've started testing the proposed syntax when I teach. I don't have a large
> sample yet, but most students either dislike it or don't appreciate the
> benefits. They state a clear preference for shorter, simpler lines at the
> consequence of more lines of code.

This is partly because students, lacking the experience to instantly
recognize larger constructs, prefer a more concrete approach to
coding. "Good code" is code where the concrete behaviour is more
easily understood. As a programmer gains experience, s/he learns to
grok more complex expressions, and is then better able to make use of
the more expressive constructs such as list comprehensions.

I don't think that's the only dynamic going on here. List comprehensions are more expressive, but also more declarative and in Python they have nice parallels with SQL and speech patterns in natural language. The concept of a comprehension is separate from its particular expression in Python. For example, Mozilla's array comprehensions in Javascript are/were ugly [0].

Students who are completely new to programming can see the similarity of list comprehensions to spoken language. They also appreciate the revision of certain 3-line and 4-line for-loops to comprehensions. I didn't get the same sense of "Oh! That looks better!" from my students when revising code with an assignment expression.

Despite my best efforts to cheerlead, some students initially dislike list comprehensions. However, they come around to the idea that there's a tradeoff between line density and code block density. Comprehensions have a 3-to-1 or 4-to-1 ratio of code line shrinkage. They're also often used in sequence, like piping data through a series of transforms. Even if students dislike a single comprehension, they agree that turning 15 lines into 5 lines improves the readability.

In contrast, an assignment expression only has a 2-to-1 code line compression ratio. It might save a level of indentation, but I think there are usually alternatives. Also, the assignment expression is less likely to be used several times in the same block.

A good pitch for an assignment expression is refactoring a cascade of regular expressions:

    for line in f:

        mo = foo_re.search(line)

        if mo is not None:

            foo(mo.groups())

            continue

        mo = bar_re.search(line)

        if mo is not None:

            bar(mo.groups())

            continue

        mo = baz_re.search(line)

        if mo is not None:

            baz(mo.groups())

            continue

Here the assignment operator makes a clear improvement:

    for line in f:

        if (mo := foo_re.search(line)) is not None:

            foo(mo.groups())

        elif (mo := bar_re.search(line)) is not None:

            bar(mo.groups())

        elif (mo := baz_re.search(line)) is not None:

            baz(mo.groups())

However, I think this example is cheating a bit. While I've written similar code many times, it's almost never just a function call in each if-block. It's nearly always a handful of lines of logic which I wouldn't want to cut out into a separate function. The refactor is misleading, because I'd nearly always make a visual separation with a newline and the code would still look similar to the initial example.

[0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Array_comprehensions

[Chris Barker via Python-Dev]

On Fri, Jun 22, 2018 at 10:09 AM, Michael Selik <mi...@selik.org> wrote:

I forgot to add that I don't anticipate changing my lesson plans if this proposal is accepted. There's already not enough time to teach everything I'd like. Including a new assignment operator would distract from the learning objectives.

nor would I. For a while, anyway....

But once it becomes a more common idiom, students will see it in the wild pretty early in their path to learning python. So we'll need to start introducing it earlier than later.

I think this reflects that the "smaller" a language is, the easier it is to learn.

Python has already grown a fair bit since 1.5 (when I started using it :-) ). Some things, like generators, are special purpose enough that I can wait pretty far into the program before teaching them. But others, like comprehensions (and lambda) are common enough that I have to introduce them pretty early on.

Adding := is not a HUGE change, but it IS an expansion of the language, and one that we WILL have to introduce in an introductory course once it starts seeing common use.

I really have no idea how much harder thats going to make the langauge to teach, but it will make it a bit harder -- I see enough confusion with "is" vs == already...

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Greg Ewing]

Nick Coghlan wrote:
> x:= f():" implies "x" is already defined as a target somewhere else in
> the current scope, while "if x := f() given x:" potentially introduces
> "x" as a new local target

Noooo..... this is just taking a bad idea and making it
worse, IMO.

I'm -1 on any contortions designed to allow comprehensions
to assign to things in outer scopes. All the proposed use
cases I've seen for this have not improved readability
over writing a function that does things the usual way.

Can we please leave comprehensions as declarative
constructs? The best tools do just one thing and do
it well. These proposals seem to be trying to turn
comprehensions into swiss army knives.

--
Greg

[Steven D'Aprano]

On Fri, Jun 22, 2018 at 11:28:45AM -0700, Chris Barker via Python-Dev wrote:
> On Fri, Jun 22, 2018 at 10:09 AM, Michael Selik <mi...@selik.org> wrote:
>
> > I forgot to add that I don't anticipate changing my lesson plans if this
> > proposal is accepted. There's already not enough time to teach everything
> > I'd like. Including a new assignment operator would distract from the
> > learning objectives.
> >
>
> nor would I. For a while, anyway....
>
> But once it becomes a more common idiom, students will see it in the wild
> pretty early in their path to learning python. So we'll need to start
> introducing it earlier than later.

Students see many features early in their path. I've had people still
struggling with writing functions ask about metaclasses. People
will see async code everywhere. We don't have to teach *everything* at
once.

The *subtleties* of assignment expressions might have some funny corner
cases, but the high-level overview is simple. It is like ordinary
assignment, but it is an expression that returns the value being
assigned. So if you absolutely need to teach it to a beginner, it
shouldn't be difficult once they understand the difference between an
expression and a statement.


[...]

> I really have no idea how much harder thats going to make the langauge to
> teach, but it will make it a bit harder -- I see enough confusion with "is"
> vs == already...

I think that the biggest source of confusion with "is" is that it
*sometimes* seems to do what is wanted, i.e. test equality, but other
times doesn't. It is that inconsistency that bites.

Whereas with assignment expressions, there's no such inconsistency:

- regular assignment using = only works as a statement, always;
- assignment expression can go anywhere an expression can go, always;
- regular assignment never returns a value;
- assignment expression always returns a value;
- regular assignments have lots of complex forms, such as sequence
unpacking, and complex targets like spam[eggs](arg).attr;
- assignment expressions only takes a plain name, always.

Although there is some overlap in behaviour between the two, unlike
"is", there's no inconsist behaviour to lead people astray.

A better syntax error for things like this:

py> if mo = regex.match(string):
File "<stdin>", line 1
if mo = regex.match(string):
^
SyntaxError: invalid syntax

will also help, although of course some users won't read error messages
for love or money.



--
Steve

[Steven D'Aprano]

On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:

> > > I've started testing the proposed syntax when I teach. I don't have a
> > > large
> > > sample yet, but most students either dislike it or don't appreciate the
> > > benefits. They state a clear preference for shorter, simpler lines at the
> > > consequence of more lines of code.

Of course they do -- they're less fluent at reading code. They don't
have the experience to judge good code from bad.

The question we should be asking is, do we only add features to Python
if they are easy for beginners? It's not that I especially want to add
features which *aren't* easy for beginners, but Python isn't Scratch and
"easy for beginners" should only be a peripheral concern.

> > This is partly because students, lacking the experience to instantly
> > recognize larger constructs, prefer a more concrete approach to
> > coding. "Good code" is code where the concrete behaviour is more
> > easily understood. As a programmer gains experience, s/he learns to
> > grok more complex expressions, and is then better able to make use of
> > the more expressive constructs such as list comprehensions.
> >
>
> I don't think that's the only dynamic going on here. List comprehensions
> are more expressive, but also more declarative and in Python they have nice
> parallels with SQL and speech patterns in natural language. The concept of
> a comprehension is separate from its particular expression in Python. For
> example, Mozilla's array comprehensions in Javascript are/were ugly [0].

Mozilla's array comprehensions are almost identical to Python's, aside
from a couple of trivial differences:

evens = [for (i of numbers) if (i % 2 === 0) i];

compared to:

evens = [i for i in numbers if (i % 2 == 0)]

- the inexplicable (to me) decision to say "for x of array" instead of
"for x in array";

- moving the expression to the end, instead of the beginning.

The second one is (arguably, though not by me) an improvement, since it
preserves a perfect left-to-right execution order within the
comprehension.

> Students who are completely new to programming can see the similarity of
> list comprehensions to spoken language.

o_O

I've been using comprehensions for something like a decade, and I can't
:-)

The closest analogy to comprehensions I know of is set builder notation
in mathematics, which is hardly a surprise. That's where Haskell got the
inspiration from, and their syntax is essentially an ASCIIfied version
of set builder notation:

Haskell: [(i,j) | i <- [1,2], j <- [1..4]]

Maths: {(i,j) : i ∈ {1, 2}, j ∈ {1...4}}

I teach secondary school children maths, and if there's a plain English
natural language equivalent to list builder notation, neither I nor any
of my students, nor any of the text books I've read, have noticed it.

--
Steve

[Steven D'Aprano]

On Sat, Jun 23, 2018 at 12:22:33AM +1000, Nick Coghlan wrote:

[...]

> * for the reactions to my description of the currently proposed parent
> local scoping behaviour in comprehensions, I'd use the word
> "horrified", and feel I wasn't overstating the response :)

Without knowing how you worded the question, and the reasons for this
horrified reaction, I'm afraid that isn't really helpful. It is nothing
more than an appeal to emotion:

https://en.wikipedia.org/wiki/Wisdom_of_repugnance

Such strong emotions as "horrified" are typically a sign of an
immediate, emotional gut reaction, not careful thought. We often see
those sorts of reactions attached to the most objectively trivial
matters. Immediate gut reactions are rarely a good guide because they
tend to over-value the status quo, exaggerate the difficulty and costs
of change, and under-estimate the benefits.

Speaking personally, I've learned to question my immediately gut
reaction. (And I remember to do so at least half the time.) PEP 572 is
an example: when the issue was first raised back in February, my gut
reaction was "Not in MY Python!!!" but by taking it seriously and
running through some examples over the course of the discussion, I
realised that, actually, I cautiously favour the idea.

Of course, matters of *personal taste* cannot be anything but gut
reaction, but in those matters, what one person holds strongly another
can legitimately reject strongly. We ought to try to look beyond
personal taste, and try (even if only imperfectly) to consider rational
reasons for and against a proposal. If we do, reactions like "horrified"
are rarely justified. It's just a minor feature in a programming
language, the world will go on one way or the other, and Python already
has trickier gotchas.

> While I try to account for the fact that I implemented the current
> comprehension semantics for the 3.x series, and am hence biased
> towards considering them the now obvious interpretation,

While we certainly don't want to make "non-obvious" a virtue for its own
sake, obviousness (obvious to who?) ought to take a distant second place
to *useful*. Otherwise we'd have to give up an awful lot of existing
Python, starting with the fundamental execution model.

(Oh, the number and length of arguments about whether Python uses call
by value or call by reference, why mutable defaults and [[]]*3 are
"broken"... if you think Python's execution model is "obvious" you've
been using Python too long ;-)

But as Tim Peters has said on a number of occasions, nobody is
suggesting changing the interpretation of current comprehension
semantics. Comprehension loop variables will continue to remain
isolated to the comprehension.

(And for the record, that makes *comprehensions* a weird special case,
not assignment expressions. All other expressions run in the current
lexical scope. Comprehensions introduce an implicit, invisible,
sub-local scope that doesn't match up with a change in indentation as
class and def statements do.)

The behaviour in question is a matter of *assignment expression*
semantics, not comprehensions. And honestly, I don't see why the
proposed behaviour is "horrifying". Here's the high-level overview:

- at the top level of a module, assignment expressions assign in
the global scope;

- inside a class, assignment expressions assign in the class scope;

- inside a function, assignment expressions assign in the function
local scope (unless declared global or nonlocal);

- inside a comprehension, assignment expressions assign in the
surrounding lexical scope (the surrounding function, class or
module).


The first three are the same as ordinary statement assignment. The last
one is what you would expect if you treat comprehensions as any other
expression which run in the current lexical scope. (The current function
or class or module.) Even if we treat it as a "weird special case" (I
don't think it is, but for the sake of the argument let's say it is) its
not hard to explain.

As I discuss below, you can get a very long way indeed working with
comprehensions without once thinking about the scope they run in. By the
time you need to think about comprehension scope, it shouldn't be hard
to deal with the rule:

- loop variables are hidden in a comprehension private scope;
- explicit assignment expression variables are not.

This is not async, or metaclasses, or even Unicode.


[...]

> plenty of
> functional-language-inspired documentation to instead encourage folks
> to view comprehensions as tightly encapsulated declarative container
> construction syntax.

I can't say I've done a broad survey, but the third-party documentation
I've read on comprehensions typically glosses over the scoping issues
without mentioning them. To the extent that scoping is even hinted at,
comprehensions are treated as expressions which are exactly equivalent
to re-writing them as a for-loop in the current scope.

This is a typical example, found as the top result on googling for
"python comprehensions":

https://www.google.com/search?q=python+comprehensions

http://www.pythonforbeginners.com/basics/list-comprehensions-in-python

Nothing is mentioned about scope, and it repeats the inaccurate but
simple equivalency:

for item in list:
if conditional:
expression

But perhaps that tutorial is too old. Okay this recent one is only a
little more than a year old:

https://hackernoon.com/list-comprehension-in-python-8895a785550b

Again, no mention of scoping issues, comprehensions are simply
expressions which presumably run in the same scope as any other
expression.

I think you over-estimate how many newcomers to Python are even aware
that the scope of comprehensions is something to consider.

> I'm currently working on a concept proposal at
> https://github.com/ncoghlan/peps/pull/2 that's much closer to PEP 572
> than any of my previous `given` based suggestions:

[...]

I look forward to reading it, and I promise I won't go by my gut
reaction :-)



--
Steve

[Chris Angelico]

On Sat, Jun 23, 2018 at 1:48 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> I can't say I've done a broad survey, but the third-party documentation
> I've read on comprehensions typically glosses over the scoping issues
> without mentioning them. To the extent that scoping is even hinted at,
> comprehensions are treated as expressions which are exactly equivalent
> to re-writing them as a for-loop in the current scope.

Even first-party documentation elides that distinction. The same
inaccurate-but-simple equivalency - even using the word "equivalent" -
comes up here:

https://docs.python.org/3/howto/functional.html?highlight=equivalent#generator-expressions-and-list-comprehensions

So I'm very sympathetic to the desire to have assignment expressions
inside comprehensions behave like assignment expressions outside
comprehensions. The trouble is that they are then _not_ the same as
other names inside comprehensions. One way or another, there's a
confusing distinction, especially at class scope. Unless this comes
with an actual semantic change that affects existing code, there is
going to be a bizarre disconnect *somewhere*, and it's just a matter
of where.

ChrisA

[Chris Barker via Python-Dev]

On Fri, Jun 22, 2018 at 7:23 PM, Steven D'Aprano <st...@pearwood.info> wrote:

> But once it becomes a more common idiom, students will see it in the wild
> pretty early in their path to learning python. So we'll need to start
> introducing it earlier than later.

Students see many features early in their path. I've had people still
struggling with writing functions ask about metaclasses. People
will see async code everywhere. We don't have to teach *everything* at
once.

These are not similar at all -- if you want similar examples, I"d say comprehensions, and lambda, both of which I DO introduce fairly early

While newbies will *ask* about metaclasses, it's probably because they read about them somewhere, not because someone actually used a metaclass in a simple script or answer to a common question on SO.

As for async, you are either doing async or not -- you can't even run an async def function without an event loop -- so again, it won't show up in real code newbies need to understand (at least until async becomes common practice with python...)

-CHB

 So if you absolutely need to teach it to a beginner, it
shouldn't be difficult once they understand the difference between an
expression and a statement.

probably not, though that's a distinction that's mostly academic in the early stages of learning, it may become more critical now...

again, not a huge deal, just a little bit more complexity

-CHB

[Steven D'Aprano]

On Fri, Jun 22, 2018 at 09:08:37PM -0700, Chris Barker wrote:

> > So if you absolutely need to teach it to a beginner, it
> > shouldn't be difficult once they understand the difference between an
> > expression and a statement.
> >
>
> probably not, though that's a distinction that's mostly academic in the
> early stages of learning,

I don't think so.

People do try to use assignment in expressions, even if only by mistake
writing = when they meant == and need to distinguish between them. In
Python 2, the most common clash between statements and expressions was
print, but at least that's gone.

https://www.quora.com/Whats-the-difference-between-a-statement-and-an-expression-in-Python-Why-is-print-%E2%80%98hi%E2%80%99-a-statement-while-other-functions-are-expressions

https://stackoverflow.com/questions/4728073/what-is-the-difference-between-an-expression-and-a-statement-in-python

https://stackoverflow.com/questions/43435850/what-is-the-difference-between-a-statement-and-a-function-in-python

Even without assignment expressions, people still need to know why they
can't write "if mo = re.match(pattern, text)".

> again, not a huge deal, just a little bit more complexity

Every new feature is added complexity.

[Ivan Pozdeev via Python-Dev]

On 23.06.2018 5:46, Steven D'Aprano wrote:
> On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:
>
>>>> I've started testing the proposed syntax when I teach. I don't have a
>>>> large
>>>> sample yet, but most students either dislike it or don't appreciate the
>>>> benefits. They state a clear preference for shorter, simpler lines at the
>>>> consequence of more lines of code.
> Of course they do -- they're less fluent at reading code. They don't
> have the experience to judge good code from bad.
>
> The question we should be asking is, do we only add features to Python
> if they are easy for beginners? It's not that I especially want to add
> features which *aren't* easy for beginners, but Python isn't Scratch and
> "easy for beginners" should only be a peripheral concern.

Python's design principles are expressed in the Zen. They rather focus
on being no more complex than absolutely necessary, without prioritizing
either beginners or old-timers ("simple is better than complex",
"complex is better than complicated").

--
Regards,
Ivan

[Nick Coghlan]

On 23 June 2018 at 09:06, Greg Ewing <greg....@canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>>
>> x:= f():" implies "x" is already defined as a target somewhere else in
>> the current scope, while "if x := f() given x:" potentially introduces
>> "x" as a new local target
>
>
> Noooo..... this is just taking a bad idea and making it
> worse, IMO.
>
> I'm -1 on any contortions designed to allow comprehensions
> to assign to things in outer scopes. All the proposed use
> cases I've seen for this have not improved readability
> over writing a function that does things the usual way.
>
> Can we please leave comprehensions as declarative
> constructs? The best tools do just one thing and do
> it well. These proposals seem to be trying to turn
> comprehensions into swiss army knives.

If PEP 572 was proposing the use of regular local scoping for
assignment expressions in comprehensions, such that they could still
be used to avoiding repeating subexpressions within an iteration, but
couldn't be used to export progress data, or to maintain a partial sum
without having to rely on `locals().get("total", 0)` to provide an
initial value, then I wouldn't be feeling obliged to present an
alternative that offers the same state export capabilities in a more
explicit form.

Given that PEP 572 *is* proposing implicit comprehension state export,
though, then I think it's important to make the case that seeing the
proposed semantics as intuitive is only going to be the case for folks
that have used Python 2 style comprehensions extensively - anyone
that's never encountered the old state-leaking behaviour for iteration
variables is going to be surprised when assignment expressions ignore
the existence of the comprehension scope (even though the iteration
variable pays attention to it).

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Mike Miller]

On 2018-06-22 19:46, Steven D'Aprano wrote:
> - the inexplicable (to me) decision to say "for x of array" instead of
> "for x in array";

Believe JavaScript has for…in, but as usual in the language it is broken and
they needed a few more tries to get it right. for…of is the latest version and
works as expected.

-Mike

[Nick Coghlan]

On 23 June 2018 at 13:48, Steven D'Aprano <st...@pearwood.info> wrote:
> On Sat, Jun 23, 2018 at 12:22:33AM +1000, Nick Coghlan wrote:
> [...]

I put quite a bit of work into making it possible for folks to gloss
over the distinction and still come to mostly-correct conclusions
about how particular code snippets would behave.

I was only able to achieve it because the folks that designed lexical
scoping before me had made *read* access to lexical scopes almost
entirely transparent, and because generator expressions were designed
to fail fast if there was a bug in the expression defining the
outermost iterable (which meant that even at class scope, the
outermost iterable expression still had access to class level
variables, because it was evaluated *outside* the nested scope).

*Write* access to lexically nested scopes, by contrast, was omitted
entirely from the original lexical scoping design, and when it was
later added by https://www.python.org/dev/peps/pep-3104/, it was done
using an explicit "nonlocal" declaration statement (expressly akin to
"global"), and PEP 3099 explicitly ruled out the use of ":=" to
implicitly declare the target name as being non-local.

PEP 572 is thus taking the position that:

- we now want to make write access to outer scopes implicit (despite
PEP 3099 explicitly ruling that out as desired design feature)
- but only in comprehensions and generator expressions (not lambda
expressions, and not full nested functions)
- and only for assignment expressions, not for loop iteration variables
- and we want it to implicitly choose between a "global NAME"
declaration and a "nonlocal NAME" declaration based on where the
comprehension is defined
- and this is OK because "nobody" actually understands how
comprehensions hide the iteration variable in practice, and
"everybody" thinks they're still a simple for loop like they were in
Python 2
- the fact that the language reference, the behaviour at class scopes,
the disassembly output, and the behaviour in a debugger all indicate
that comprehensions are full nested scopes isn't important

This level of additional complication and complexity in the scoping
semantics simply isn't warranted for such a minor readability
enhancement as assignment expressions.

Cheers,
Nick.

P.S. "You did such a good job of minimising the backwards
compatibility breakage when we changed the semantics of scoping in
comprehensions that we now consider your opinion on reasonable scoping
semantics for comprehensions to be irrelevant, because everybody else
still thinks they work the same way as they did in Python 2" is such a
surreal position for folks to be taking that I'm having trouble
working out how to effectively respond to it.

Guido has complained that "I keep changing my mind about what I want",
but that's not what's actually going on: what I want is to keep folks
from taking our already complicated scoping semantics and making it
close to impossible for anyone to ever infer how they work from
experimentation at the interactive prompt. That goal has pretty much
stayed consistent since the parent local scoping proposal was first
put forward.

What keeps changing is my tactics in pursuing that goal, based on my
current perception of what the folks pushing that proposal are
actually trying to achieve (which seems to be some combination of "We
want to pretend that the Python 3 scoping changes in comprehensions
never happened, but we still want to avoid leaking the iteration
variables somehow" and "We want to enable new clever tricks with state
export from comprehensions and generator expressions"), as well as
repeatedly asking myself what *actually* bothers me about the proposal
(which I've now distilled down to just the comprehension scoping
issue, and the reliance on an arbitrary syntactic restriction against
top level usage to avoid competing with traditional assignment
statements).

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Steven D'Aprano]

On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:

> Given that PEP 572 *is* proposing implicit comprehension state export,

"Implicit" and "explicit" are two terms which often get misused to mean
"I don't like it" and "I do like it".

Making the intentional choice to use an assignment expression is not
really "implicit" in any meaningful sense. One might as well complain
that "import this" implicitly creates a local variable "this". Well,
yes, it does, in a very loose sense, but that's what imports are
defined as do and it is the whole purpose for making them.

If PEP 572's proposal goes ahead, the behaviour of assignment
expressions will be *defined* as creating assignments in the local scope
rather than the sublocal comprehension scope. To call that "implicit"
is rather like saying that regular assignment is implicit.

> though, then I think it's important to make the case that seeing the
> proposed semantics as intuitive is only going to be the case for folks
> that have used Python 2 style comprehensions extensively - anyone
> that's never encountered the old state-leaking behaviour for iteration
> variables is going to be surprised when assignment expressions ignore
> the existence of the comprehension scope (even though the iteration
> variable pays attention to it).

You are making the assumption that most people are even aware of
"comprehension scope". I don't think that is the case.

In my experience, scoping in Python is still typically seen as the LGB
rule (locals/globals/builtins). See for example this StackOverflow post
from 2016:

https://stackoverflow.com/questions/37211910/override-lgb-scope-rule

Sometimes people remember the E/N (enclosing function/nonlocal) part.
Hardly anyone remembers the C (class) part unless they are actively
thinking in terms of code running inside a class definition, and even if
they do, they typically aren't sure of exactly how it interacts with the
rest.

And I predict that even fewer think of comprehensions as a separate
scope, except by ommission: they *don't think about* the scope of the
loop variable until it bites them.

But as Tim Peters has previously discussed, the loop variable is
special, and is especially prone to accidental shadowing. That won't be
the case for assignment expressions. If there's shadowing going on, it
will be deliberate.


Aside: I've said before that I'm not a fan of sublocal comprehension
scoping, since I personally found it helpful on occasion for the loop
variable to be visible outside of the comprehension. But given that the
only experience most people apparently had with comprehension scoping
was to be bitten by it, I grudgingly accept that encapsulating the loop
variable was the right decision to make, even if it personally
inconvenienced me more than it saved me.

Nor was I the only one: others have been bitten by the change to
comprehension scope, see for example:

https://www.reddit.com/r/Python/comments/425qmb/strange_python_27_34_difference/

There is no consensus that the change to comprehensions was a good thing
or justified.

The bottom line is that I don't think people will be surprised by
assignment expression scope being local instead of sublocal. Rather I
expect that they won't even think about it, until they do, and then
*whatever* behaviour we pick, we'll annoy somebody.



--
Steve

[Nick Coghlan]

On 24 June 2018 at 15:56, Steven D'Aprano <st...@pearwood.info> wrote:
> On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
>
>> Given that PEP 572 *is* proposing implicit comprehension state export,
>
> "Implicit" and "explicit" are two terms which often get misused to mean
> "I don't like it" and "I do like it".
>
> Making the intentional choice to use an assignment expression is not
> really "implicit" in any meaningful sense.

No, it's actually implicit: there's an extra "global NAME" or
"nonlocal NAME" in the equivalent code for a comprehension that isn't
there in the as-written source code, and doesn't get emitted for a
regular assignment expression or for the iteration variable in a
comprehension - it only shows up due to the defined interaction
between comprehensions and assignment expressions.

> One might as well complain
> that "import this" implicitly creates a local variable "this". Well,
> yes, it does, in a very loose sense, but that's what imports are
> defined as do and it is the whole purpose for making them.

And they behave the same way in every context where they're permitted to appear.

> If PEP 572's proposal goes ahead, the behaviour of assignment
> expressions will be *defined* as creating assignments in the local scope
> rather than the sublocal comprehension scope. To call that "implicit"
> is rather like saying that regular assignment is implicit.

I do say that regular assignments implicitly declare a name as local.
"Python has implicit local variable declarations" is also regularly
cited as one of the differences between it and languages that require
explicit declarations, like C. Even augmented assignments implicitly
declare a name as being a local (hence the infamous UnboundLocalError
that arises when you attempt to use an augmented assignment to rebind
a name from an outer scope).

The problem I have with PEP 572 is that it proposes *breaking that
otherwise universal pattern* - instead of having assignment
expressions in comprehensions implicitly declare the name as local in
the nested comprehension scope, it instead has them:

1. implicitly declare the name as global or as nonlocal in the
comprehension (or else raise an exception), depending on the nature of
the parent scope where the comprehension is used
2. in the nonlocal reference case, amend the symbol table analysis to
act like there was a preceding "if 0:\n for NAME in ():\n pass" in the
parent scope (so the compiler knows which outer function scope to
target)

The rationale being given for why that is OK is:

1. "Everyone" thinks comprehensions are just a for loop (even though
that hasn't been true for the better part of a decade, and was never
true for generator expressions)
2. If comprehensions are just a for loop, then assignment expressions
inside them should be local to the containing scope
3. Therefore the implicit declarations required to tie everything
together and allow folks to continue with an incorrect understanding
of how comprehensions work aren't really implicit - they're explicit
in the inaccurate expansion of the construct!

Can you imagine the reaction if anyone other than Guido or Tim was
attempting to argue for a change to the language that only makes sense
if we grant a completely inaccurate understanding of how a particular
language construct works as being a credible starting point?

Because that's how this comprehension scoping argument feels to me:

Proposal author: "If the language worked in a way other than it does,
then this proposal would make perfect sense."
Proposal reviewer: "Yes, but it doesn't work that way, it works this
way. We deliberately changed it because the old way caused problems."
Proposal author: "Ah, but it *used* to work that way, and a lot of
people still think it works that way, and we can get the compiler to
jump through hoops to pretend it still works that way, except for the
parts of the new way that we want to keep."
Proposal reviewer: "..."

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Chris Angelico]

On Sun, Jun 24, 2018 at 4:33 PM, Nick Coghlan <ncog...@gmail.com> wrote:
> On 24 June 2018 at 15:56, Steven D'Aprano <st...@pearwood.info> wrote:
>> On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
>>
>>> Given that PEP 572 *is* proposing implicit comprehension state export,
>>
>> "Implicit" and "explicit" are two terms which often get misused to mean
>> "I don't like it" and "I do like it".
>>
>> Making the intentional choice to use an assignment expression is not
>> really "implicit" in any meaningful sense.
>
> No, it's actually implicit: there's an extra "global NAME" or
> "nonlocal NAME" in the equivalent code for a comprehension that isn't
> there in the as-written source code, and doesn't get emitted for a
> regular assignment expression or for the iteration variable in a
> comprehension - it only shows up due to the defined interaction
> between comprehensions and assignment expressions.

The implicit "nonlocal NAME" is only because there is an equally
implicit function boundary. Why is there a function boundary marked by
square brackets? It's not saying "def" or "lambda", which obviously
create functions. It's a 'for' loop wrapped inside a list display.
What part of that says "hey, I'm a nested function"?

So if there's an implicit function, with implicit declaration of a
magical parameter called ".0", why can't it have an equally implicit
declaration that "spam" is a nonlocal name?

ChrisA

[Nick Coghlan]

On 24 June 2018 at 16:53, Chris Angelico <ros...@gmail.com> wrote:
> On Sun, Jun 24, 2018 at 4:33 PM, Nick Coghlan <ncog...@gmail.com> wrote:
>> On 24 June 2018 at 15:56, Steven D'Aprano <st...@pearwood.info> wrote:
>>> On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
>>>
>>>> Given that PEP 572 *is* proposing implicit comprehension state export,
>>>
>>> "Implicit" and "explicit" are two terms which often get misused to mean
>>> "I don't like it" and "I do like it".
>>>
>>> Making the intentional choice to use an assignment expression is not
>>> really "implicit" in any meaningful sense.
>>
>> No, it's actually implicit: there's an extra "global NAME" or
>> "nonlocal NAME" in the equivalent code for a comprehension that isn't
>> there in the as-written source code, and doesn't get emitted for a
>> regular assignment expression or for the iteration variable in a
>> comprehension - it only shows up due to the defined interaction
>> between comprehensions and assignment expressions.
>
> The implicit "nonlocal NAME" is only because there is an equally
> implicit function boundary. Why is there a function boundary marked by
> square brackets? It's not saying "def" or "lambda", which obviously
> create functions. It's a 'for' loop wrapped inside a list display.
> What part of that says "hey, I'm a nested function"?

Nothing - that's why I refer to them as implicitly nested scopes (vs
the explicitly nested scopes in functions and lambda expressions,
where the scope is introduced via keyword).

However, there's still a major behavioural tell at runtime that
they're running in a nested scope: the iteration variables don't leak.
(There are other tells as well, but not ones that most folks are
likely to encounter)

> So if there's an implicit function, with implicit declaration of a
> magical parameter called ".0", why can't it have an equally implicit
> declaration that "spam" is a nonlocal name?

Because comprehensions don't do that for their iteration variables,
because assignment expressions don't do that when used in explicitly
nested scopes, because the required implicit scope declarations are
context dependent, and because even such gyrations still can't hide
the existence of the comprehension's implicitly nested scope when
dealing with classes and the two-argument form of exec().

Since the implicitly nested scopes can't be hidden, it makes far more
sense to me to just admit that they're there, and provide explicit
syntax for cases where folks decide they really do want name bindings
to leak out of that scope (whether those name bindings are assignment
expression targets or the iteration variables themselves).

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Steven D'Aprano]

On Sun, Jun 24, 2018 at 03:56:47PM +1000, Steven D'Aprano wrote:

> There is no consensus that the change to comprehensions was a good thing
> or justified.

On re-reading that, I think its wrong -- it wasn't really what I
intended to say. What I intended to say was, in hindsight, more like:

*Despite the consensus to change comprehension scope*, there's a
contingent of people who are not convinced that the change was a good
thing or justified.

Sorry for the inaccurate comment. Mea culpa.

[Ivan Pozdeev via Python-Dev]

On 24.06.2018 9:53, Chris Angelico wrote:
> On Sun, Jun 24, 2018 at 4:33 PM, Nick Coghlan <ncog...@gmail.com> wrote:
>> On 24 June 2018 at 15:56, Steven D'Aprano <st...@pearwood.info> wrote:
>>> On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
>>>
>>>> Given that PEP 572 *is* proposing implicit comprehension state export,
>>> "Implicit" and "explicit" are two terms which often get misused to mean
>>> "I don't like it" and "I do like it".
>>>
>>> Making the intentional choice to use an assignment expression is not
>>> really "implicit" in any meaningful sense.

My 2c.
An expression is intuitively thought to be self-contained i.e. without
side effects.
if I write `a=b+1`, I'm not expecting it to do anything except assigning
`a'.

Expressions with side effects has long since proven to be problematic
because of the implicit (thus hard to see and track) links they create
(and because the result depends on the order of evaluation).
Moreover, Python's other design elements have been consistently
discouraging expressions with side effects, too (e.g. mutator methods
intentionally return None instead of the new value, making them useless
in expressions), so the proposition is in direct conflict with the
language's design.

Assignment expressions are a grey area: they carry the full implications
of expressions with side effects described above, but their side effect
is their only effect, i.e. they are explicit and prominent about the
"evil" they do.

>> No, it's actually implicit: there's an extra "global NAME" or
>> "nonlocal NAME" in the equivalent code for a comprehension that isn't
>> there in the as-written source code, and doesn't get emitted for a
>> regular assignment expression or for the iteration variable in a
>> comprehension - it only shows up due to the defined interaction
>> between comprehensions and assignment expressions.
> The implicit "nonlocal NAME" is only because there is an equally
> implicit function boundary. Why is there a function boundary marked by
> square brackets? It's not saying "def" or "lambda", which obviously
> create functions. It's a 'for' loop wrapped inside a list display.
> What part of that says "hey, I'm a nested function"?
>
> So if there's an implicit function, with implicit declaration of a
> magical parameter called ".0", why can't it have an equally implicit
> declaration that "spam" is a nonlocal name?
>
> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> https://mail.python.org/mailman/listinfo/python-dev

> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

--
Regards,
Ivan

[Steven D'Aprano]

On Sun, Jun 24, 2018 at 05:24:12PM +0300, Ivan Pozdeev via Python-Dev wrote:

> An expression is intuitively thought to be self-contained i.e. without
> side effects.
> if I write `a=b+1`, I'm not expecting it to do anything except assigning
> `a'.

a = d.pop(1)
a = d.setdefault(key, 0)
chars_written = file.write(text)

> Expressions with side effects has long since proven to be problematic
> because of the implicit (thus hard to see and track) links they create
> (and because the result depends on the order of evaluation).

If you're going to take a hard-core functional approach to side-effects,
I think you are using the wrong language. Nearly everything in Python
*could* have side-effects (even if usually it won't).

Even your own example of "b+1" (depending on what b.__add__ does).

> Moreover, Python's other design elements have been consistently
> discouraging expressions with side effects, too (e.g. mutator methods
> intentionally return None instead of the new value, making them useless
> in expressions),

I don't think that's the reason why mutator methods return None. They
return None rather than self to avoid confusion over whether they return
a copy or not.

https://docs.python.org/3/faq/design.html#why-doesn-t-list-sort-return-the-sorted-list

> so the proposition is in direct conflict with the
> language's design.

Python is full of operations with side-effects.

Besides, they're not quite useless:

(alist.append() or alist)

is functionally equivalent to alist.append returning self. Just a bit
more verbose.

Methods (and functions) all return a value, even if that value is None,
so they can be used in expressions. If Guido wanted Pascal style
procedures, which cannot be used in expressions, we would have them by
now :-)


--
Steve

[Guido van Rossum]

A quick follow-up: PEP 572 currently has two ideas: (a) introduce := for inline assignment, (b) when := is used in a comprehension, set the scope for the target as if the assignment occurred outside any comprehensions. It seems we have more support for (a) than for (b) -- at least Nick and Greg seem to be +0 or better for (a) but -1 for (b). IIRC (b) originated with Tim. But his essay on the topic, included as Appendix A (https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings) does not even mention comprehensions. However, he did post his motivation for (b) on python-ideas, IIRC a bit before PyCon; and the main text of the PEP gives a strong motivation (https://www.python.org/dev/peps/pep-0572/#scope-of-the-target). Nevertheless, maybe we should compromise and drop (b)?

--

--Guido van Rossum (python.org/~guido)

[Steven D'Aprano]

On Sun, Jun 24, 2018 at 04:33:38PM +1000, Nick Coghlan wrote:
[...]

> > Making the intentional choice to use an assignment expression is not
> > really "implicit" in any meaningful sense.
>
> No, it's actually implicit: there's an extra "global NAME" or
> "nonlocal NAME" in the equivalent code for a comprehension that isn't
> there in the as-written source code, and doesn't get emitted for a
> regular assignment expression or for the iteration variable in a
> comprehension - it only shows up due to the defined interaction
> between comprehensions and assignment expressions.

You seem to be talking about an implementation which could change in the
future. I'm talking semantics of the proposed language feature. As a
programmer writing Python code, I have no visibility into the
implementation. The implementation could change ten times a day for all
I care, so long as the semantics remain the same.

I want the desired semantics to drive the implementation, not the
other way around.

You seem to want the implementation to drive the semantics, by
eliminating the proposed feature because it doesn't match your
deep understanding of the implementation as a nested function.

I want this feature because its useful, and without it the use-cases
for assignment expressions are significantly reduced.

As far as "implicit", for the sake of the discussion, I'll grant you
that one. Okay, the proposed behaviour will implicitly enable
comprehensions to export their state.

Now what? Is that a good thing or a bad thing?

If "implicit" (with or without the scare quotes) is such a bad thing to
be avoided, why are comprehensions implemented using an implicit
function?

> The problem I have with PEP 572 is that it proposes *breaking that
> otherwise universal pattern* - instead of having assignment
> expressions in comprehensions implicitly declare the name as local in
> the nested comprehension scope, it instead has them:

You talk about "nested comprehension scope", and that's a critical
point, but I'm going to skip answering that for now. I have a draft
email responding to another of your posts on that topic, which I hope to
polish in the next day.

> 1. implicitly declare the name as global or as nonlocal in the
> comprehension (or else raise an exception), depending on the nature of
> the parent scope where the comprehension is used
> 2. in the nonlocal reference case, amend the symbol table analysis to
> act like there was a preceding "if 0:\n for NAME in ():\n pass" in the
> parent scope (so the compiler knows which outer function scope to
> target)

If it is okay for you to amend the list comprehension to behave as if it
were wrapped in an implicit nested function, why shouldn't it be okay to
behave as if assignments inside the comprehension included an implicit
nonlocal declaration?

> The rationale being given for why that is OK is:
>
> 1. "Everyone" thinks comprehensions are just a for loop (even though
> that hasn't been true for the better part of a decade, and was never
> true for generator expressions)

Obviously "everyone" is an exaggeration, but, yes, I stand by that --
most people don't even give comprehension scope a thought until they
get bitten by it.

Either because (Python 2) they don't realise the loop variable is local
to their current scope:

http://www.librador.com/2014/07/10/Variable-scope-in-list-comprehension-vs-generator-expression/

or (Python 3) they get bitten by the change:

https://old.reddit.com/r/Python/comments/425qmb/strange_python_27_34_difference/

(As is so often the case, whatever behaviour we choose, we're going to
surprise somebody.)

It is hardly surprising that people don't think too hard about scoping
of comprehensions. Without a way to perform assignments inside
comprehensions, aside from the loop variables themselves, there's
nothing going on inside a comprehension where it makes a visible
difference whether it is a local scope or a sublocal scope.

*IF* assignment expressions are introduced, that is going to change. We
have some choices:

1. Keep assignment expressions encapsulated in their implicit function,
and be prepared for people to be annoyed because (with no way to declare
them global or non-local inside an expression), they can't use them to
get data in and out of the comprehension.


2. Allow assignment expressions to be exported out of the comprehension,
and be prepared for people to be annoyed because they clobbered a local.

(But for the reasons Tim Peters has already set out, I doubt this will
happen often.)


3. Allow some sort of extra comprehension syntax to allow
global/nonlocal declarations inside comprehensions.

x = 1
[nonlocal x := x+i for i in sequence]

(Hmmm... I thought I would hate that more than I actually do.)


4. Have some sort of cunning plan whereby if the variable in question
exists in the local scope, it is implicitly local inside the
comprehension:

x = 1
[x := i+1 for i in (1, 2)]
assert x == 3

but if it doesn't, then the variable is implicitly sublocal inside the
comprehension:

del x
[x := i+1 for i in (1, 2)]
x # raises NameError



Remember, the driving use-case which started this (ever-so-long)
discussion was the ability to push data into a comprehension and then
update it on each iteration, something like this:

x = initial_value()
results = [x := transform(x, i) for i in sequence]

Please, Nick, take your implementor's hat off, forget everything you
know about the implementation of comprehensions and their implicit
nested function, and tell me that doesn't look like it should work.




--
Steve

[Steven D'Aprano]

On Sun, Jun 24, 2018 at 09:24:39AM -0700, Guido van Rossum wrote:
> A quick follow-up: PEP 572 currently has two ideas: (a) introduce := for
> inline assignment, (b) when := is used in a comprehension, set the scope
> for the target as if the assignment occurred outside any comprehensions. It
> seems we have more support for (a) than for (b) -- at least Nick and Greg
> seem to be +0 or better for (a) but -1 for (b). IIRC (b) originated with
> Tim.

I'm not sure who came up with the idea first, but as I remember it, the
first mention of this came in a separate thread on Python-Ideas:

https://mail.python.org/pipermail/python-ideas/2018-April/049631.html

so possibly I'm to blame :-)

That thread starts here:

https://mail.python.org/pipermail/python-ideas/2018-April/049622.html

If I did get the idea from Tim, I don't remember doing so.

> But his essay on the topic, included as Appendix A (
> https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings)
> does not even mention comprehensions. However, he did post his motivation
> for (b) on python-ideas, IIRC a bit before PyCon; and the main text of the
> PEP gives a strong motivation (
> https://www.python.org/dev/peps/pep-0572/#scope-of-the-target).
> Nevertheless, maybe we should compromise and drop (b)?

I will have more to say about the whole "comprehensions are their own
scope" issue later. But I'd like to see Nick's proposed PEP, or at least
a draft of it, before making any final decisions.

If it came down to it, I'd be happy with the ability to declare an
assignment target nonlocal in the comprehension if that's what it takes.
What do you think of this syntax?

[global|nonlocal] simple_target := expression


Inside a comprehension, without a declaration, the target would be
sublocal (comprehension scope); that should make Nick happier :-)

[Tim Peters]

[Guido]

> A quick follow-up: PEP 572 currently has two ideas: (a) introduce := for inline
> assignment, (b) when := is used in a comprehension, set the scope for the
> target as if the assignment occurred outside any comprehensions. It seems
> we have more support for (a) than for (b) -- at least Nick and Greg seem to
> be +0 or better for (a) but -1 for (b). IIRC (b) originated with Tim. But his
> essay on the topic, included as Appendix A
> (https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings)
> does not even mention comprehensions.

I was writing up my observations about simple changes to existing code.  Since there's nothing sanely akin to binding non-for-targets possible in comprehensions now, comprehensions were out of scope for that effort (which was limited to staring at existing code already doing bindings).

:> However, he did post his motivation for (b) on python-ideas, IIRC a bit

> before PyCon; and the main text of the PEP gives a strong motivation
> (https://www.python.org/dev/peps/pep-0572/#scope-of-the-target). Nevertheless,
> maybe we should compromise and drop (b)?

Two things to say about that.  First, the original example I gave would be approximately as well addressed by allowing to declare intended scopes in magically synthesized functions; like (say)

p = None # to establish the intended scope of `p`
while any(<nonlocal p>  # split across lines just for readability

                n % p == 0 for p in small_primes):
    n //= p

It didn't really require an inline assignment, just a way to override the unwanted (in this case) "all `for` targets are local to the invisible function" rigid consequence of the implementation du jour.

Second, if it's dropped, then the PEP needs more words to define what happens in cases like the following, because different textual parts of a comprehension execute in different scopes, and that can  become visible when bindings can be embedded:

def f():

    y = -1

    ys = [y for _ in range(y := 5)]

    print(y, ys)

Here `range(y := 5)` is executed in f's scope.  Presumably the `y` in `y for` also refers to f's scope, despite that `y` textually _appears_ to be assigned to in the body of the listcomp, and so would - for that reason - expected to be local to the synthesized function, and so raise `UnboundLocalError` when referenced.  It's incoherent without detailed knowledge of the implementation.

def g():

    y = -1

    ys = [y for y in range(y := 5)]

    print(y, ys)

And here the `y` in `y for y` is local to the synthesized function, and presumably has nothing to do with the `y` in the `range()` call.  That's incoherent in its own way.

Under the current PEP, all instances of `y` in `f` refer to the f-local `y`, and the listcomp in `g` is a compile-time error.

[Chris Angelico]

On Mon, Jun 25, 2018 at 4:06 AM, Steven D'Aprano <st...@pearwood.info> wrote:
>
> Remember, the driving use-case which started this (ever-so-long)
> discussion was the ability to push data into a comprehension and then
> update it on each iteration, something like this:
>
> x = initial_value()
> results = [x := transform(x, i) for i in sequence]

Which means there is another option.

5. Have the assignment be local to the comprehension, but the initial
value of ANY variable is looked up from the surrounding scopes.

That is: you will NEVER get UnboundLocalError from a
comprehension/genexp; instead, you will look up the name as if it were
in the surrounding scope, either getting a value or bombing with
regular old NameError.

Or possibly variations on this such as "the immediately surrounding
scope only", rather than full name lookups. It'd have an awkward
boundary somewhere, whichever way you do it.

This isn't able to send information *out* of a comprehension, but it
is able to send information *in*.

ChrisA

[Michael Selik]

This thread started with a request for educator feedback, which I took to mean observations of student reactions. I've only had the chance to test the proposal on ~20 students so far, but I'd like the chance to gather more data for your consideration before the PEP is accepted or rejected.

On Sun, Jun 24, 2018 at 11:09 AM Steven D'Aprano <st...@pearwood.info> wrote:

Remember, the driving use-case which started this (ever-so-long)
discussion was the ability to push data into a comprehension and then
update it on each iteration, something like this:

    x = initial_value()
    results = [x := transform(x, i) for i in sequence]

If that is the driving use-case, then the proposal should be rejected. The ``itertools.accumulate`` function has been available for a little while now and it handles this exact case. The accumulate function may even be more readable, as it explains the purpose explicitly, not merely the algorithm. And heck, it's a one-liner.

    results = accumulate(sequence, transform)

The benefits for ``any`` and ``all`` seem useful. Itertools has "first_seen" in the recipes section. While it feels intuitively useful, I can't recall ever writing something similar myself. For some reason, I (almost?) always want to find all (counter-)examples and aggregate them in some way -- min or max, perhaps -- rather than just get the first.

Even so, if it turns out those uses are quite prevalent, wouldn't a new itertool be better than a new operator? It's good to solve the general problem, but so far the in-comprehension usage seems to have only a handful of cases.

On Fri, Jun 22, 2018 at 9:14 PM Chris Barker via Python-Dev <pytho...@python.org> wrote:

again, not a huge deal, just a little bit more complexity

I worry that Python may experience something of a "death by a thousand cuts" along the lines of the "Remember the Vasa" warning. Python's greatest strength is its appeal to beginners. Little bits of added complexity have a non-linear effect. One day, we may wake up and Python won't be recommended as a beginner's language.

On Fri, Jun 22, 2018 at 7:48 PM Steven D'Aprano <st...@pearwood.info> wrote:

On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:

Of course they do -- they're less fluent at reading code. They don't 
have the experience to judge good code from bad.

On the other hand, an "expert" may be so steeped in a particular subculture that he no longer can distinguish esoteric from intuitive. Don't be so fast to reject the wisdom of the inexperienced.

 

The question we should be asking is, do we only add features to Python 
if they are easy for beginners? It's not that I especially want to add 
features which *aren't* easy for beginners, but Python isn't Scratch and 
"easy for beginners" should only be a peripheral concern.

On the contrary, I believe that "easy for beginners" should be a major concern.  Ease of use has been and is a, or even the main reason for Python's success. When some other language becomes a better teaching language, it will eventually take over in business and science as well. Right now, Python is Scratch for adults. That's a great thing. Given the growth of the field, there are far more beginner programmers working today than there ever have been experts.

Mozilla's array comprehensions are almost identical to Python's, aside 
from a couple of trivial differences:

I can't prove it, but I think the phrase ordering difference is not trivial.

> Students who are completely new to programming can see the similarity of

> [Python] list comprehensions to spoken language. 

I've been using comprehensions for something like a decade, and I can't 

Python: any(line.startswith('#') for line in file)

English: Any line starts with "#" in the file?