[Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Nick Coghlan]

On 22 June 2018 at 02:26, Antoine Pitrou <soli...@pitrou.net> wrote:
> Indeed. But, for a syntax addition such as PEP 572, I think it would be
> a good idea to ask their opinion to teaching/education specialists.
>
> As far as I'm concerned, if teachers and/or education specialists were
> to say PEP 572 is not a problem, my position would shift from negative
> towards neutral.

I asked a handful of folks at the Education Summit the next day about it:

* for the basic notion of allowing expression level name binding using
the "NAME := EXPR" notation, the reactions ranged from mildly negative
(I read it as only a "-0" rather than a "-1") to outright positive.
* for the reactions to my description of the currently proposed parent
local scoping behaviour in comprehensions, I'd use the word
"horrified", and feel I wasn't overstating the response :)

While I try to account for the fact that I implemented the current
comprehension semantics for the 3.x series, and am hence biased
towards considering them the now obvious interpretation, it's also the
case that generator expressions have worked like nested functions
since they were introduced in Python 2.4 (more than 13 years ago now),
and comprehensions have worked the same way as generator expressions
since Python 3.0 (which has its 10th birthday coming up in December
this year).

This means that I take any claims that the legacy Python 2.x
interpretation of comprehension behaviour is intuitively obvious with
an enormous grain of salt - for the better part of a decade now, every
tool at a Python 3 user's disposal (the fact that the iteration
variable is hidden from the current scope, reading the language
reference [1], printing out locals(), using the dis module, stepping
through code in a debugger, writing their own tracing function, and
even observing the quirky interaction with class scopes) will have
nudged them towards the "it's a hidden nested function" interpretation
of expected comprehension behaviour.

Acquiring the old mental model for the way comprehensions work pretty
much requires a developer to have started with Python 2.x themselves
(perhaps even before comprehensions and lexical closures were part of
the language), or else have been taught the Python 2 comprehension
model by someone else - there's nothing in Python 3's behaviour to
encourage that point of view, and plenty of
functional-language-inspired documentation to instead encourage folks
to view comprehensions as tightly encapsulated declarative container
construction syntax.

I'm currently working on a concept proposal at
https://github.com/ncoghlan/peps/pull/2 that's much closer to PEP 572
than any of my previous `given` based suggestions: for already
declared locals, it devolves to being the same as PEP 572 (except that
expressions are allowed as top level statements), but for any names
that haven't been previously introduced, it prohibits assigning to a
name that doesn't already have a defined scope, and instead relies on
a new `given` clause on various constructs that allows new target
declarations to be introduced into the current scope (such that "if
x:= f():" implies "x" is already defined as a target somewhere else in
the current scope, while "if x := f() given x:" potentially introduces
"x" as a new local target the same way a regular assignment statement
does).

One of the nicer features of the draft proposal is that if all you
want to do is export the iteration variable from a comprehension, you
don't need to use an assignment expression at all: you can just append
"... given global x" or "... given nonlocal x" and export the
iteration variable directly to the desired outer scope, the same way
you can in the fully spelled out nested function equivalent.

Cheers,
Nick.

[1] From https://docs.python.org/3.0/reference/expressions.html#displays-for-lists-sets-and-dictionaries:
'Note that the comprehension is executed in a separate scope, so names
assigned to in the target list don’t “leak” in the enclosing scope.'
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Antoine Pitrou]

On Sat, 23 Jun 2018 00:22:33 +1000
Nick Coghlan <ncog...@gmail.com> wrote:
> On 22 June 2018 at 02:26, Antoine Pitrou <soli...@pitrou.net> wrote:
> > Indeed. But, for a syntax addition such as PEP 572, I think it would be
> > a good idea to ask their opinion to teaching/education specialists.
> >
> > As far as I'm concerned, if teachers and/or education specialists were
> > to say PEP 572 is not a problem, my position would shift from negative
> > towards neutral.
>
> I asked a handful of folks at the Education Summit the next day about it:
>
> * for the basic notion of allowing expression level name binding using
> the "NAME := EXPR" notation, the reactions ranged from mildly negative
> (I read it as only a "-0" rather than a "-1") to outright positive.

Thank you. Personally, I'd like to see feedback from
educators/teachers after they take the time to read the PEP and take
some time to think about its consequences.

My main concern is we're introducing a second different way of doing
something which is really fundamental.

> * for the reactions to my description of the currently proposed parent
> local scoping behaviour in comprehensions, I'd use the word

> "horrified", and feel I wasn't overstating the response :) [...]

Hmm... I don't think conflating the assignment expression proposal
with comprehension semantics issues is helping the discussion.

Regards

Antoine.

[Michael Selik]

On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <soli...@pitrou.net> wrote:

Thank you.  Personally, I'd like to see feedback from
educators/teachers after they take the time to read the PEP and take
some time to think about its consequences.

I've started testing the proposed syntax when I teach. I don't have a large sample yet, but most students either dislike it or don't appreciate the benefits. They state a clear preference for shorter, simpler lines at the consequence of more lines of code. This may partially be caused by the smaller screen real estate on a projector or large TV than a desktop monitor.

My intuition is that one strength of Python for beginners is the relative lack of punctuation and operators compared with most other languages. This proposal encourages denser lines with more punctuation. Because of the order of operations, many uses of ``:=`` will also require parentheses. Even relatively simple uses, like ``if (match := pattern.search(data)) is not None:`` require doubled parentheses on one side or the other. Beginners are especially prone to typographical errors with mismatched parentheses and missing colons and get easily frustrated by the associated syntax errors.

Given the following options:

A.

    if (row := cursor.fetchone()) is None:

        raise NotFound

    return row

B.

    row = cursor.fetchone()

    if row is None:

        raise NotFound

    return row

C.

    if (row := cursor.fetchone()) is not None:

        return row

    raise NotFound

D.

    row = cursor.fetchone()

    if row is not None:

        return row

    raise NotFound

The majority of students preferred option B. I also tested some regex match examples. Results were similar.

 

My main concern is we're introducing a second different way of doing
something which is really fundamental.

The few students who like the proposal ask why it requires creating a new operator instead of repurposing the ``=`` operator.

I'll reserve my personal opinions for a different thread.

[Michael Selik]

On Fri, Jun 22, 2018 at 10:02 AM Michael Selik <mi...@selik.org> wrote:

On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <soli...@pitrou.net> wrote:

Thank you.  Personally, I'd like to see feedback from
educators/teachers after they take the time to read the PEP and take
some time to think about its consequences.

I forgot to add that I don't anticipate changing my lesson plans if this proposal is accepted. There's already not enough time to teach everything I'd like. Including a new assignment operator would distract from the learning objectives.

 

[Chris Angelico]

On Sat, Jun 23, 2018 at 3:02 AM, Michael Selik <mi...@selik.org> wrote:
> On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <soli...@pitrou.net> wrote:
>>
>> Thank you. Personally, I'd like to see feedback from
>> educators/teachers after they take the time to read the PEP and take
>> some time to think about its consequences.
>
>
> I've started testing the proposed syntax when I teach. I don't have a large
> sample yet, but most students either dislike it or don't appreciate the
> benefits. They state a clear preference for shorter, simpler lines at the
> consequence of more lines of code.

This is partly because students, lacking the experience to instantly
recognize larger constructs, prefer a more concrete approach to
coding. "Good code" is code where the concrete behaviour is more
easily understood. As a programmer gains experience, s/he learns to
grok more complex expressions, and is then better able to make use of
the more expressive constructs such as list comprehensions.

ChrisA

[Michael Selik]

On Fri, Jun 22, 2018 at 10:19 AM Chris Angelico <ros...@gmail.com> wrote:

On Sat, Jun 23, 2018 at 3:02 AM, Michael Selik <mi...@selik.org> wrote:
> On Fri, Jun 22, 2018 at 8:09 AM Antoine Pitrou <soli...@pitrou.net> wrote:
>>
>> Thank you.  Personally, I'd like to see feedback from
>> educators/teachers after they take the time to read the PEP and take
>> some time to think about its consequences.
>
>
> I've started testing the proposed syntax when I teach. I don't have a large
> sample yet, but most students either dislike it or don't appreciate the
> benefits. They state a clear preference for shorter, simpler lines at the
> consequence of more lines of code.

This is partly because students, lacking the experience to instantly
recognize larger constructs, prefer a more concrete approach to
coding. "Good code" is code where the concrete behaviour is more
easily understood. As a programmer gains experience, s/he learns to
grok more complex expressions, and is then better able to make use of
the more expressive constructs such as list comprehensions.

I don't think that's the only dynamic going on here. List comprehensions are more expressive, but also more declarative and in Python they have nice parallels with SQL and speech patterns in natural language. The concept of a comprehension is separate from its particular expression in Python. For example, Mozilla's array comprehensions in Javascript are/were ugly [0].

Students who are completely new to programming can see the similarity of list comprehensions to spoken language. They also appreciate the revision of certain 3-line and 4-line for-loops to comprehensions. I didn't get the same sense of "Oh! That looks better!" from my students when revising code with an assignment expression.

Despite my best efforts to cheerlead, some students initially dislike list comprehensions. However, they come around to the idea that there's a tradeoff between line density and code block density. Comprehensions have a 3-to-1 or 4-to-1 ratio of code line shrinkage. They're also often used in sequence, like piping data through a series of transforms. Even if students dislike a single comprehension, they agree that turning 15 lines into 5 lines improves the readability.

In contrast, an assignment expression only has a 2-to-1 code line compression ratio. It might save a level of indentation, but I think there are usually alternatives. Also, the assignment expression is less likely to be used several times in the same block.

A good pitch for an assignment expression is refactoring a cascade of regular expressions:

    for line in f:

        mo = foo_re.search(line)

        if mo is not None:

            foo(mo.groups())

            continue

        mo = bar_re.search(line)

        if mo is not None:

            bar(mo.groups())

            continue

        mo = baz_re.search(line)

        if mo is not None:

            baz(mo.groups())

            continue

Here the assignment operator makes a clear improvement:

    for line in f:

        if (mo := foo_re.search(line)) is not None:

            foo(mo.groups())

        elif (mo := bar_re.search(line)) is not None:

            bar(mo.groups())

        elif (mo := baz_re.search(line)) is not None:

            baz(mo.groups())

However, I think this example is cheating a bit. While I've written similar code many times, it's almost never just a function call in each if-block. It's nearly always a handful of lines of logic which I wouldn't want to cut out into a separate function. The refactor is misleading, because I'd nearly always make a visual separation with a newline and the code would still look similar to the initial example.

[0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Array_comprehensions

[Chris Barker via Python-Dev]

On Fri, Jun 22, 2018 at 10:09 AM, Michael Selik <mi...@selik.org> wrote:

I forgot to add that I don't anticipate changing my lesson plans if this proposal is accepted. There's already not enough time to teach everything I'd like. Including a new assignment operator would distract from the learning objectives.

nor would I. For a while, anyway....

But once it becomes a more common idiom, students will see it in the wild pretty early in their path to learning python. So we'll need to start introducing it earlier than later.

I think this reflects that the "smaller" a language is, the easier it is to learn.

Python has already grown a fair bit since 1.5 (when I started using it :-) ). Some things, like generators, are special purpose enough that I can wait pretty far into the program before teaching them. But others, like comprehensions (and lambda) are common enough that I have to introduce them pretty early on.

Adding := is not a HUGE change, but it IS an expansion of the language, and one that we WILL have to introduce in an introductory course once it starts seeing common use.

I really have no idea how much harder thats going to make the langauge to teach, but it will make it a bit harder -- I see enough confusion with "is" vs == already...

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Greg Ewing]

Nick Coghlan wrote:
> x:= f():" implies "x" is already defined as a target somewhere else in
> the current scope, while "if x := f() given x:" potentially introduces
> "x" as a new local target

Noooo..... this is just taking a bad idea and making it
worse, IMO.

I'm -1 on any contortions designed to allow comprehensions
to assign to things in outer scopes. All the proposed use
cases I've seen for this have not improved readability
over writing a function that does things the usual way.

Can we please leave comprehensions as declarative
constructs? The best tools do just one thing and do
it well. These proposals seem to be trying to turn
comprehensions into swiss army knives.

--
Greg

[Steven D'Aprano]

On Fri, Jun 22, 2018 at 11:28:45AM -0700, Chris Barker via Python-Dev wrote:
> On Fri, Jun 22, 2018 at 10:09 AM, Michael Selik <mi...@selik.org> wrote:
>
> > I forgot to add that I don't anticipate changing my lesson plans if this
> > proposal is accepted. There's already not enough time to teach everything
> > I'd like. Including a new assignment operator would distract from the
> > learning objectives.
> >
>
> nor would I. For a while, anyway....
>
> But once it becomes a more common idiom, students will see it in the wild
> pretty early in their path to learning python. So we'll need to start
> introducing it earlier than later.

Students see many features early in their path. I've had people still
struggling with writing functions ask about metaclasses. People
will see async code everywhere. We don't have to teach *everything* at
once.

The *subtleties* of assignment expressions might have some funny corner
cases, but the high-level overview is simple. It is like ordinary
assignment, but it is an expression that returns the value being
assigned. So if you absolutely need to teach it to a beginner, it
shouldn't be difficult once they understand the difference between an
expression and a statement.


[...]

> I really have no idea how much harder thats going to make the langauge to
> teach, but it will make it a bit harder -- I see enough confusion with "is"
> vs == already...

I think that the biggest source of confusion with "is" is that it
*sometimes* seems to do what is wanted, i.e. test equality, but other
times doesn't. It is that inconsistency that bites.

Whereas with assignment expressions, there's no such inconsistency:

- regular assignment using = only works as a statement, always;
- assignment expression can go anywhere an expression can go, always;
- regular assignment never returns a value;
- assignment expression always returns a value;
- regular assignments have lots of complex forms, such as sequence
unpacking, and complex targets like spam[eggs](arg).attr;
- assignment expressions only takes a plain name, always.

Although there is some overlap in behaviour between the two, unlike
"is", there's no inconsist behaviour to lead people astray.

A better syntax error for things like this:

py> if mo = regex.match(string):
File "<stdin>", line 1
if mo = regex.match(string):
^
SyntaxError: invalid syntax

will also help, although of course some users won't read error messages
for love or money.



--
Steve

[Steven D'Aprano]

On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:

> > > I've started testing the proposed syntax when I teach. I don't have a
> > > large
> > > sample yet, but most students either dislike it or don't appreciate the
> > > benefits. They state a clear preference for shorter, simpler lines at the
> > > consequence of more lines of code.

Of course they do -- they're less fluent at reading code. They don't
have the experience to judge good code from bad.

The question we should be asking is, do we only add features to Python
if they are easy for beginners? It's not that I especially want to add
features which *aren't* easy for beginners, but Python isn't Scratch and
"easy for beginners" should only be a peripheral concern.

> > This is partly because students, lacking the experience to instantly
> > recognize larger constructs, prefer a more concrete approach to
> > coding. "Good code" is code where the concrete behaviour is more
> > easily understood. As a programmer gains experience, s/he learns to
> > grok more complex expressions, and is then better able to make use of
> > the more expressive constructs such as list comprehensions.
> >
>
> I don't think that's the only dynamic going on here. List comprehensions
> are more expressive, but also more declarative and in Python they have nice
> parallels with SQL and speech patterns in natural language. The concept of
> a comprehension is separate from its particular expression in Python. For
> example, Mozilla's array comprehensions in Javascript are/were ugly [0].

Mozilla's array comprehensions are almost identical to Python's, aside
from a couple of trivial differences:

evens = [for (i of numbers) if (i % 2 === 0) i];

compared to:

evens = [i for i in numbers if (i % 2 == 0)]

- the inexplicable (to me) decision to say "for x of array" instead of
"for x in array";

- moving the expression to the end, instead of the beginning.

The second one is (arguably, though not by me) an improvement, since it
preserves a perfect left-to-right execution order within the
comprehension.

> Students who are completely new to programming can see the similarity of
> list comprehensions to spoken language.

o_O

I've been using comprehensions for something like a decade, and I can't
:-)

The closest analogy to comprehensions I know of is set builder notation
in mathematics, which is hardly a surprise. That's where Haskell got the
inspiration from, and their syntax is essentially an ASCIIfied version
of set builder notation:

Haskell: [(i,j) | i <- [1,2], j <- [1..4]]

Maths: {(i,j) : i ∈ {1, 2}, j ∈ {1...4}}

I teach secondary school children maths, and if there's a plain English
natural language equivalent to list builder notation, neither I nor any
of my students, nor any of the text books I've read, have noticed it.

--
Steve

[Steven D'Aprano]

On Sat, Jun 23, 2018 at 12:22:33AM +1000, Nick Coghlan wrote:

[...]

> * for the reactions to my description of the currently proposed parent
> local scoping behaviour in comprehensions, I'd use the word
> "horrified", and feel I wasn't overstating the response :)

Without knowing how you worded the question, and the reasons for this
horrified reaction, I'm afraid that isn't really helpful. It is nothing
more than an appeal to emotion:

https://en.wikipedia.org/wiki/Wisdom_of_repugnance

Such strong emotions as "horrified" are typically a sign of an
immediate, emotional gut reaction, not careful thought. We often see
those sorts of reactions attached to the most objectively trivial
matters. Immediate gut reactions are rarely a good guide because they
tend to over-value the status quo, exaggerate the difficulty and costs
of change, and under-estimate the benefits.

Speaking personally, I've learned to question my immediately gut
reaction. (And I remember to do so at least half the time.) PEP 572 is
an example: when the issue was first raised back in February, my gut
reaction was "Not in MY Python!!!" but by taking it seriously and
running through some examples over the course of the discussion, I
realised that, actually, I cautiously favour the idea.

Of course, matters of *personal taste* cannot be anything but gut
reaction, but in those matters, what one person holds strongly another
can legitimately reject strongly. We ought to try to look beyond
personal taste, and try (even if only imperfectly) to consider rational
reasons for and against a proposal. If we do, reactions like "horrified"
are rarely justified. It's just a minor feature in a programming
language, the world will go on one way or the other, and Python already
has trickier gotchas.

> While I try to account for the fact that I implemented the current
> comprehension semantics for the 3.x series, and am hence biased
> towards considering them the now obvious interpretation,

While we certainly don't want to make "non-obvious" a virtue for its own
sake, obviousness (obvious to who?) ought to take a distant second place
to *useful*. Otherwise we'd have to give up an awful lot of existing
Python, starting with the fundamental execution model.

(Oh, the number and length of arguments about whether Python uses call
by value or call by reference, why mutable defaults and [[]]*3 are
"broken"... if you think Python's execution model is "obvious" you've
been using Python too long ;-)

But as Tim Peters has said on a number of occasions, nobody is
suggesting changing the interpretation of current comprehension
semantics. Comprehension loop variables will continue to remain
isolated to the comprehension.

(And for the record, that makes *comprehensions* a weird special case,
not assignment expressions. All other expressions run in the current
lexical scope. Comprehensions introduce an implicit, invisible,
sub-local scope that doesn't match up with a change in indentation as
class and def statements do.)

The behaviour in question is a matter of *assignment expression*
semantics, not comprehensions. And honestly, I don't see why the
proposed behaviour is "horrifying". Here's the high-level overview:

- at the top level of a module, assignment expressions assign in
the global scope;

- inside a class, assignment expressions assign in the class scope;

- inside a function, assignment expressions assign in the function
local scope (unless declared global or nonlocal);

- inside a comprehension, assignment expressions assign in the
surrounding lexical scope (the surrounding function, class or
module).


The first three are the same as ordinary statement assignment. The last
one is what you would expect if you treat comprehensions as any other
expression which run in the current lexical scope. (The current function
or class or module.) Even if we treat it as a "weird special case" (I
don't think it is, but for the sake of the argument let's say it is) its
not hard to explain.

As I discuss below, you can get a very long way indeed working with
comprehensions without once thinking about the scope they run in. By the
time you need to think about comprehension scope, it shouldn't be hard
to deal with the rule:

- loop variables are hidden in a comprehension private scope;
- explicit assignment expression variables are not.

This is not async, or metaclasses, or even Unicode.


[...]

> plenty of
> functional-language-inspired documentation to instead encourage folks
> to view comprehensions as tightly encapsulated declarative container
> construction syntax.

I can't say I've done a broad survey, but the third-party documentation
I've read on comprehensions typically glosses over the scoping issues
without mentioning them. To the extent that scoping is even hinted at,
comprehensions are treated as expressions which are exactly equivalent
to re-writing them as a for-loop in the current scope.

This is a typical example, found as the top result on googling for
"python comprehensions":

https://www.google.com/search?q=python+comprehensions

http://www.pythonforbeginners.com/basics/list-comprehensions-in-python

Nothing is mentioned about scope, and it repeats the inaccurate but
simple equivalency:

for item in list:
if conditional:
expression

But perhaps that tutorial is too old. Okay this recent one is only a
little more than a year old:

https://hackernoon.com/list-comprehension-in-python-8895a785550b

Again, no mention of scoping issues, comprehensions are simply
expressions which presumably run in the same scope as any other
expression.

I think you over-estimate how many newcomers to Python are even aware
that the scope of comprehensions is something to consider.

> I'm currently working on a concept proposal at
> https://github.com/ncoghlan/peps/pull/2 that's much closer to PEP 572
> than any of my previous `given` based suggestions:

[...]

I look forward to reading it, and I promise I won't go by my gut
reaction :-)



--
Steve

[Chris Angelico]

On Sat, Jun 23, 2018 at 1:48 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> I can't say I've done a broad survey, but the third-party documentation
> I've read on comprehensions typically glosses over the scoping issues
> without mentioning them. To the extent that scoping is even hinted at,
> comprehensions are treated as expressions which are exactly equivalent
> to re-writing them as a for-loop in the current scope.

Even first-party documentation elides that distinction. The same
inaccurate-but-simple equivalency - even using the word "equivalent" -
comes up here:

https://docs.python.org/3/howto/functional.html?highlight=equivalent#generator-expressions-and-list-comprehensions

So I'm very sympathetic to the desire to have assignment expressions
inside comprehensions behave like assignment expressions outside
comprehensions. The trouble is that they are then _not_ the same as
other names inside comprehensions. One way or another, there's a
confusing distinction, especially at class scope. Unless this comes
with an actual semantic change that affects existing code, there is
going to be a bizarre disconnect *somewhere*, and it's just a matter
of where.

ChrisA

[Chris Barker via Python-Dev]

On Fri, Jun 22, 2018 at 7:23 PM, Steven D'Aprano <st...@pearwood.info> wrote:

> But once it becomes a more common idiom, students will see it in the wild
> pretty early in their path to learning python. So we'll need to start
> introducing it earlier than later.

Students see many features early in their path. I've had people still
struggling with writing functions ask about metaclasses. People
will see async code everywhere. We don't have to teach *everything* at
once.

These are not similar at all -- if you want similar examples, I"d say comprehensions, and lambda, both of which I DO introduce fairly early

While newbies will *ask* about metaclasses, it's probably because they read about them somewhere, not because someone actually used a metaclass in a simple script or answer to a common question on SO.

As for async, you are either doing async or not -- you can't even run an async def function without an event loop -- so again, it won't show up in real code newbies need to understand (at least until async becomes common practice with python...)

-CHB

 So if you absolutely need to teach it to a beginner, it
shouldn't be difficult once they understand the difference between an
expression and a statement.

probably not, though that's a distinction that's mostly academic in the early stages of learning, it may become more critical now...

again, not a huge deal, just a little bit more complexity

-CHB

[Steven D'Aprano]

On Fri, Jun 22, 2018 at 09:08:37PM -0700, Chris Barker wrote:

> > So if you absolutely need to teach it to a beginner, it
> > shouldn't be difficult once they understand the difference between an
> > expression and a statement.
> >
>
> probably not, though that's a distinction that's mostly academic in the
> early stages of learning,

I don't think so.

People do try to use assignment in expressions, even if only by mistake
writing = when they meant == and need to distinguish between them. In
Python 2, the most common clash between statements and expressions was
print, but at least that's gone.

https://www.quora.com/Whats-the-difference-between-a-statement-and-an-expression-in-Python-Why-is-print-%E2%80%98hi%E2%80%99-a-statement-while-other-functions-are-expressions

https://stackoverflow.com/questions/4728073/what-is-the-difference-between-an-expression-and-a-statement-in-python

https://stackoverflow.com/questions/43435850/what-is-the-difference-between-a-statement-and-a-function-in-python

Even without assignment expressions, people still need to know why they
can't write "if mo = re.match(pattern, text)".

> again, not a huge deal, just a little bit more complexity

Every new feature is added complexity.

[Ivan Pozdeev via Python-Dev]

On 23.06.2018 5:46, Steven D'Aprano wrote:
> On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:
>
>>>> I've started testing the proposed syntax when I teach. I don't have a
>>>> large
>>>> sample yet, but most students either dislike it or don't appreciate the
>>>> benefits. They state a clear preference for shorter, simpler lines at the
>>>> consequence of more lines of code.
> Of course they do -- they're less fluent at reading code. They don't
> have the experience to judge good code from bad.
>
> The question we should be asking is, do we only add features to Python
> if they are easy for beginners? It's not that I especially want to add
> features which *aren't* easy for beginners, but Python isn't Scratch and
> "easy for beginners" should only be a peripheral concern.

Python's design principles are expressed in the Zen. They rather focus
on being no more complex than absolutely necessary, without prioritizing
either beginners or old-timers ("simple is better than complex",
"complex is better than complicated").

--
Regards,
Ivan

[Nick Coghlan]

On 23 June 2018 at 09:06, Greg Ewing <greg....@canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>>
>> x:= f():" implies "x" is already defined as a target somewhere else in
>> the current scope, while "if x := f() given x:" potentially introduces
>> "x" as a new local target
>
>
> Noooo..... this is just taking a bad idea and making it
> worse, IMO.
>
> I'm -1 on any contortions designed to allow comprehensions
> to assign to things in outer scopes. All the proposed use
> cases I've seen for this have not improved readability
> over writing a function that does things the usual way.
>
> Can we please leave comprehensions as declarative
> constructs? The best tools do just one thing and do
> it well. These proposals seem to be trying to turn
> comprehensions into swiss army knives.

If PEP 572 was proposing the use of regular local scoping for
assignment expressions in comprehensions, such that they could still
be used to avoiding repeating subexpressions within an iteration, but
couldn't be used to export progress data, or to maintain a partial sum
without having to rely on `locals().get("total", 0)` to provide an
initial value, then I wouldn't be feeling obliged to present an
alternative that offers the same state export capabilities in a more
explicit form.

Given that PEP 572 *is* proposing implicit comprehension state export,
though, then I think it's important to make the case that seeing the
proposed semantics as intuitive is only going to be the case for folks
that have used Python 2 style comprehensions extensively - anyone
that's never encountered the old state-leaking behaviour for iteration
variables is going to be surprised when assignment expressions ignore
the existence of the comprehension scope (even though the iteration
variable pays attention to it).

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Mike Miller]

On 2018-06-22 19:46, Steven D'Aprano wrote:
> - the inexplicable (to me) decision to say "for x of array" instead of
> "for x in array";

Believe JavaScript has for…in, but as usual in the language it is broken and
they needed a few more tries to get it right. for…of is the latest version and
works as expected.

-Mike

[Nick Coghlan]

On 23 June 2018 at 13:48, Steven D'Aprano <st...@pearwood.info> wrote:
> On Sat, Jun 23, 2018 at 12:22:33AM +1000, Nick Coghlan wrote:
> [...]

I put quite a bit of work into making it possible for folks to gloss
over the distinction and still come to mostly-correct conclusions
about how particular code snippets would behave.

I was only able to achieve it because the folks that designed lexical
scoping before me had made *read* access to lexical scopes almost
entirely transparent, and because generator expressions were designed
to fail fast if there was a bug in the expression defining the
outermost iterable (which meant that even at class scope, the
outermost iterable expression still had access to class level
variables, because it was evaluated *outside* the nested scope).

*Write* access to lexically nested scopes, by contrast, was omitted
entirely from the original lexical scoping design, and when it was
later added by https://www.python.org/dev/peps/pep-3104/, it was done
using an explicit "nonlocal" declaration statement (expressly akin to
"global"), and PEP 3099 explicitly ruled out the use of ":=" to
implicitly declare the target name as being non-local.

PEP 572 is thus taking the position that:

- we now want to make write access to outer scopes implicit (despite
PEP 3099 explicitly ruling that out as desired design feature)
- but only in comprehensions and generator expressions (not lambda
expressions, and not full nested functions)
- and only for assignment expressions, not for loop iteration variables
- and we want it to implicitly choose between a "global NAME"
declaration and a "nonlocal NAME" declaration based on where the
comprehension is defined
- and this is OK because "nobody" actually understands how
comprehensions hide the iteration variable in practice, and
"everybody" thinks they're still a simple for loop like they were in
Python 2
- the fact that the language reference, the behaviour at class scopes,
the disassembly output, and the behaviour in a debugger all indicate
that comprehensions are full nested scopes isn't important

This level of additional complication and complexity in the scoping
semantics simply isn't warranted for such a minor readability
enhancement as assignment expressions.

Cheers,
Nick.

P.S. "You did such a good job of minimising the backwards
compatibility breakage when we changed the semantics of scoping in
comprehensions that we now consider your opinion on reasonable scoping
semantics for comprehensions to be irrelevant, because everybody else
still thinks they work the same way as they did in Python 2" is such a
surreal position for folks to be taking that I'm having trouble
working out how to effectively respond to it.

Guido has complained that "I keep changing my mind about what I want",
but that's not what's actually going on: what I want is to keep folks
from taking our already complicated scoping semantics and making it
close to impossible for anyone to ever infer how they work from
experimentation at the interactive prompt. That goal has pretty much
stayed consistent since the parent local scoping proposal was first
put forward.

What keeps changing is my tactics in pursuing that goal, based on my
current perception of what the folks pushing that proposal are
actually trying to achieve (which seems to be some combination of "We
want to pretend that the Python 3 scoping changes in comprehensions
never happened, but we still want to avoid leaking the iteration
variables somehow" and "We want to enable new clever tricks with state
export from comprehensions and generator expressions"), as well as
repeatedly asking myself what *actually* bothers me about the proposal
(which I've now distilled down to just the comprehension scoping
issue, and the reliance on an arbitrary syntactic restriction against
top level usage to avoid competing with traditional assignment
statements).

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Steven D'Aprano]

On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:

> Given that PEP 572 *is* proposing implicit comprehension state export,

"Implicit" and "explicit" are two terms which often get misused to mean
"I don't like it" and "I do like it".

Making the intentional choice to use an assignment expression is not
really "implicit" in any meaningful sense. One might as well complain
that "import this" implicitly creates a local variable "this". Well,
yes, it does, in a very loose sense, but that's what imports are
defined as do and it is the whole purpose for making them.

If PEP 572's proposal goes ahead, the behaviour of assignment
expressions will be *defined* as creating assignments in the local scope
rather than the sublocal comprehension scope. To call that "implicit"
is rather like saying that regular assignment is implicit.

> though, then I think it's important to make the case that seeing the
> proposed semantics as intuitive is only going to be the case for folks
> that have used Python 2 style comprehensions extensively - anyone
> that's never encountered the old state-leaking behaviour for iteration
> variables is going to be surprised when assignment expressions ignore
> the existence of the comprehension scope (even though the iteration
> variable pays attention to it).

You are making the assumption that most people are even aware of
"comprehension scope". I don't think that is the case.

In my experience, scoping in Python is still typically seen as the LGB
rule (locals/globals/builtins). See for example this StackOverflow post
from 2016:

https://stackoverflow.com/questions/37211910/override-lgb-scope-rule

Sometimes people remember the E/N (enclosing function/nonlocal) part.
Hardly anyone remembers the C (class) part unless they are actively
thinking in terms of code running inside a class definition, and even if
they do, they typically aren't sure of exactly how it interacts with the
rest.

And I predict that even fewer think of comprehensions as a separate
scope, except by ommission: they *don't think about* the scope of the
loop variable until it bites them.

But as Tim Peters has previously discussed, the loop variable is
special, and is especially prone to accidental shadowing. That won't be
the case for assignment expressions. If there's shadowing going on, it
will be deliberate.


Aside: I've said before that I'm not a fan of sublocal comprehension
scoping, since I personally found it helpful on occasion for the loop
variable to be visible outside of the comprehension. But given that the
only experience most people apparently had with comprehension scoping
was to be bitten by it, I grudgingly accept that encapsulating the loop
variable was the right decision to make, even if it personally
inconvenienced me more than it saved me.

Nor was I the only one: others have been bitten by the change to
comprehension scope, see for example:

https://www.reddit.com/r/Python/comments/425qmb/strange_python_27_34_difference/

There is no consensus that the change to comprehensions was a good thing
or justified.

The bottom line is that I don't think people will be surprised by
assignment expression scope being local instead of sublocal. Rather I
expect that they won't even think about it, until they do, and then
*whatever* behaviour we pick, we'll annoy somebody.



--
Steve

[Nick Coghlan]

On 24 June 2018 at 15:56, Steven D'Aprano <st...@pearwood.info> wrote:
> On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
>
>> Given that PEP 572 *is* proposing implicit comprehension state export,
>
> "Implicit" and "explicit" are two terms which often get misused to mean
> "I don't like it" and "I do like it".
>
> Making the intentional choice to use an assignment expression is not
> really "implicit" in any meaningful sense.

No, it's actually implicit: there's an extra "global NAME" or
"nonlocal NAME" in the equivalent code for a comprehension that isn't
there in the as-written source code, and doesn't get emitted for a
regular assignment expression or for the iteration variable in a
comprehension - it only shows up due to the defined interaction
between comprehensions and assignment expressions.

> One might as well complain
> that "import this" implicitly creates a local variable "this". Well,
> yes, it does, in a very loose sense, but that's what imports are
> defined as do and it is the whole purpose for making them.

And they behave the same way in every context where they're permitted to appear.

> If PEP 572's proposal goes ahead, the behaviour of assignment
> expressions will be *defined* as creating assignments in the local scope
> rather than the sublocal comprehension scope. To call that "implicit"
> is rather like saying that regular assignment is implicit.

I do say that regular assignments implicitly declare a name as local.
"Python has implicit local variable declarations" is also regularly
cited as one of the differences between it and languages that require
explicit declarations, like C. Even augmented assignments implicitly
declare a name as being a local (hence the infamous UnboundLocalError
that arises when you attempt to use an augmented assignment to rebind
a name from an outer scope).

The problem I have with PEP 572 is that it proposes *breaking that
otherwise universal pattern* - instead of having assignment
expressions in comprehensions implicitly declare the name as local in
the nested comprehension scope, it instead has them:

1. implicitly declare the name as global or as nonlocal in the
comprehension (or else raise an exception), depending on the nature of
the parent scope where the comprehension is used
2. in the nonlocal reference case, amend the symbol table analysis to
act like there was a preceding "if 0:\n for NAME in ():\n pass" in the
parent scope (so the compiler knows which outer function scope to
target)

The rationale being given for why that is OK is:

1. "Everyone" thinks comprehensions are just a for loop (even though
that hasn't been true for the better part of a decade, and was never
true for generator expressions)
2. If comprehensions are just a for loop, then assignment expressions
inside them should be local to the containing scope
3. Therefore the implicit declarations required to tie everything
together and allow folks to continue with an incorrect understanding
of how comprehensions work aren't really implicit - they're explicit
in the inaccurate expansion of the construct!

Can you imagine the reaction if anyone other than Guido or Tim was
attempting to argue for a change to the language that only makes sense
if we grant a completely inaccurate understanding of how a particular
language construct works as being a credible starting point?

Because that's how this comprehension scoping argument feels to me:

Proposal author: "If the language worked in a way other than it does,
then this proposal would make perfect sense."
Proposal reviewer: "Yes, but it doesn't work that way, it works this
way. We deliberately changed it because the old way caused problems."
Proposal author: "Ah, but it *used* to work that way, and a lot of
people still think it works that way, and we can get the compiler to
jump through hoops to pretend it still works that way, except for the
parts of the new way that we want to keep."
Proposal reviewer: "..."

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Chris Angelico]

On Sun, Jun 24, 2018 at 4:33 PM, Nick Coghlan <ncog...@gmail.com> wrote:
> On 24 June 2018 at 15:56, Steven D'Aprano <st...@pearwood.info> wrote:
>> On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
>>
>>> Given that PEP 572 *is* proposing implicit comprehension state export,
>>
>> "Implicit" and "explicit" are two terms which often get misused to mean
>> "I don't like it" and "I do like it".
>>
>> Making the intentional choice to use an assignment expression is not
>> really "implicit" in any meaningful sense.
>
> No, it's actually implicit: there's an extra "global NAME" or
> "nonlocal NAME" in the equivalent code for a comprehension that isn't
> there in the as-written source code, and doesn't get emitted for a
> regular assignment expression or for the iteration variable in a
> comprehension - it only shows up due to the defined interaction
> between comprehensions and assignment expressions.

The implicit "nonlocal NAME" is only because there is an equally
implicit function boundary. Why is there a function boundary marked by
square brackets? It's not saying "def" or "lambda", which obviously
create functions. It's a 'for' loop wrapped inside a list display.
What part of that says "hey, I'm a nested function"?

So if there's an implicit function, with implicit declaration of a
magical parameter called ".0", why can't it have an equally implicit
declaration that "spam" is a nonlocal name?

ChrisA

[Nick Coghlan]

On 24 June 2018 at 16:53, Chris Angelico <ros...@gmail.com> wrote:
> On Sun, Jun 24, 2018 at 4:33 PM, Nick Coghlan <ncog...@gmail.com> wrote:
>> On 24 June 2018 at 15:56, Steven D'Aprano <st...@pearwood.info> wrote:
>>> On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
>>>
>>>> Given that PEP 572 *is* proposing implicit comprehension state export,
>>>
>>> "Implicit" and "explicit" are two terms which often get misused to mean
>>> "I don't like it" and "I do like it".
>>>
>>> Making the intentional choice to use an assignment expression is not
>>> really "implicit" in any meaningful sense.
>>
>> No, it's actually implicit: there's an extra "global NAME" or
>> "nonlocal NAME" in the equivalent code for a comprehension that isn't
>> there in the as-written source code, and doesn't get emitted for a
>> regular assignment expression or for the iteration variable in a
>> comprehension - it only shows up due to the defined interaction
>> between comprehensions and assignment expressions.
>
> The implicit "nonlocal NAME" is only because there is an equally
> implicit function boundary. Why is there a function boundary marked by
> square brackets? It's not saying "def" or "lambda", which obviously
> create functions. It's a 'for' loop wrapped inside a list display.
> What part of that says "hey, I'm a nested function"?

Nothing - that's why I refer to them as implicitly nested scopes (vs
the explicitly nested scopes in functions and lambda expressions,
where the scope is introduced via keyword).

However, there's still a major behavioural tell at runtime that
they're running in a nested scope: the iteration variables don't leak.
(There are other tells as well, but not ones that most folks are
likely to encounter)

> So if there's an implicit function, with implicit declaration of a
> magical parameter called ".0", why can't it have an equally implicit
> declaration that "spam" is a nonlocal name?

Because comprehensions don't do that for their iteration variables,
because assignment expressions don't do that when used in explicitly
nested scopes, because the required implicit scope declarations are
context dependent, and because even such gyrations still can't hide
the existence of the comprehension's implicitly nested scope when
dealing with classes and the two-argument form of exec().

Since the implicitly nested scopes can't be hidden, it makes far more
sense to me to just admit that they're there, and provide explicit
syntax for cases where folks decide they really do want name bindings
to leak out of that scope (whether those name bindings are assignment
expression targets or the iteration variables themselves).

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Steven D'Aprano]

On Sun, Jun 24, 2018 at 03:56:47PM +1000, Steven D'Aprano wrote:

> There is no consensus that the change to comprehensions was a good thing
> or justified.

On re-reading that, I think its wrong -- it wasn't really what I
intended to say. What I intended to say was, in hindsight, more like:

*Despite the consensus to change comprehension scope*, there's a
contingent of people who are not convinced that the change was a good
thing or justified.

Sorry for the inaccurate comment. Mea culpa.

[Ivan Pozdeev via Python-Dev]

On 24.06.2018 9:53, Chris Angelico wrote:
> On Sun, Jun 24, 2018 at 4:33 PM, Nick Coghlan <ncog...@gmail.com> wrote:
>> On 24 June 2018 at 15:56, Steven D'Aprano <st...@pearwood.info> wrote:
>>> On Sun, Jun 24, 2018 at 02:33:59PM +1000, Nick Coghlan wrote:
>>>
>>>> Given that PEP 572 *is* proposing implicit comprehension state export,
>>> "Implicit" and "explicit" are two terms which often get misused to mean
>>> "I don't like it" and "I do like it".
>>>
>>> Making the intentional choice to use an assignment expression is not
>>> really "implicit" in any meaningful sense.

My 2c.
An expression is intuitively thought to be self-contained i.e. without
side effects.
if I write `a=b+1`, I'm not expecting it to do anything except assigning
`a'.

Expressions with side effects has long since proven to be problematic
because of the implicit (thus hard to see and track) links they create
(and because the result depends on the order of evaluation).
Moreover, Python's other design elements have been consistently
discouraging expressions with side effects, too (e.g. mutator methods
intentionally return None instead of the new value, making them useless
in expressions), so the proposition is in direct conflict with the
language's design.

Assignment expressions are a grey area: they carry the full implications
of expressions with side effects described above, but their side effect
is their only effect, i.e. they are explicit and prominent about the
"evil" they do.

>> No, it's actually implicit: there's an extra "global NAME" or
>> "nonlocal NAME" in the equivalent code for a comprehension that isn't
>> there in the as-written source code, and doesn't get emitted for a
>> regular assignment expression or for the iteration variable in a
>> comprehension - it only shows up due to the defined interaction
>> between comprehensions and assignment expressions.
> The implicit "nonlocal NAME" is only because there is an equally
> implicit function boundary. Why is there a function boundary marked by
> square brackets? It's not saying "def" or "lambda", which obviously
> create functions. It's a 'for' loop wrapped inside a list display.
> What part of that says "hey, I'm a nested function"?
>
> So if there's an implicit function, with implicit declaration of a
> magical parameter called ".0", why can't it have an equally implicit
> declaration that "spam" is a nonlocal name?
>
> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> https://mail.python.org/mailman/listinfo/python-dev

> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

--
Regards,
Ivan

[Steven D'Aprano]

On Sun, Jun 24, 2018 at 05:24:12PM +0300, Ivan Pozdeev via Python-Dev wrote:

> An expression is intuitively thought to be self-contained i.e. without
> side effects.
> if I write `a=b+1`, I'm not expecting it to do anything except assigning
> `a'.

a = d.pop(1)
a = d.setdefault(key, 0)
chars_written = file.write(text)

> Expressions with side effects has long since proven to be problematic
> because of the implicit (thus hard to see and track) links they create
> (and because the result depends on the order of evaluation).

If you're going to take a hard-core functional approach to side-effects,
I think you are using the wrong language. Nearly everything in Python
*could* have side-effects (even if usually it won't).

Even your own example of "b+1" (depending on what b.__add__ does).

> Moreover, Python's other design elements have been consistently
> discouraging expressions with side effects, too (e.g. mutator methods
> intentionally return None instead of the new value, making them useless
> in expressions),

I don't think that's the reason why mutator methods return None. They
return None rather than self to avoid confusion over whether they return
a copy or not.

https://docs.python.org/3/faq/design.html#why-doesn-t-list-sort-return-the-sorted-list

> so the proposition is in direct conflict with the
> language's design.

Python is full of operations with side-effects.

Besides, they're not quite useless:

(alist.append() or alist)

is functionally equivalent to alist.append returning self. Just a bit
more verbose.

Methods (and functions) all return a value, even if that value is None,
so they can be used in expressions. If Guido wanted Pascal style
procedures, which cannot be used in expressions, we would have them by
now :-)


--
Steve

[Guido van Rossum]

A quick follow-up: PEP 572 currently has two ideas: (a) introduce := for inline assignment, (b) when := is used in a comprehension, set the scope for the target as if the assignment occurred outside any comprehensions. It seems we have more support for (a) than for (b) -- at least Nick and Greg seem to be +0 or better for (a) but -1 for (b). IIRC (b) originated with Tim. But his essay on the topic, included as Appendix A (https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings) does not even mention comprehensions. However, he did post his motivation for (b) on python-ideas, IIRC a bit before PyCon; and the main text of the PEP gives a strong motivation (https://www.python.org/dev/peps/pep-0572/#scope-of-the-target). Nevertheless, maybe we should compromise and drop (b)?

--

--Guido van Rossum (python.org/~guido)

[Steven D'Aprano]

On Sun, Jun 24, 2018 at 04:33:38PM +1000, Nick Coghlan wrote:
[...]

> > Making the intentional choice to use an assignment expression is not
> > really "implicit" in any meaningful sense.
>
> No, it's actually implicit: there's an extra "global NAME" or
> "nonlocal NAME" in the equivalent code for a comprehension that isn't
> there in the as-written source code, and doesn't get emitted for a
> regular assignment expression or for the iteration variable in a
> comprehension - it only shows up due to the defined interaction
> between comprehensions and assignment expressions.

You seem to be talking about an implementation which could change in the
future. I'm talking semantics of the proposed language feature. As a
programmer writing Python code, I have no visibility into the
implementation. The implementation could change ten times a day for all
I care, so long as the semantics remain the same.

I want the desired semantics to drive the implementation, not the
other way around.

You seem to want the implementation to drive the semantics, by
eliminating the proposed feature because it doesn't match your
deep understanding of the implementation as a nested function.

I want this feature because its useful, and without it the use-cases
for assignment expressions are significantly reduced.

As far as "implicit", for the sake of the discussion, I'll grant you
that one. Okay, the proposed behaviour will implicitly enable
comprehensions to export their state.

Now what? Is that a good thing or a bad thing?

If "implicit" (with or without the scare quotes) is such a bad thing to
be avoided, why are comprehensions implemented using an implicit
function?

> The problem I have with PEP 572 is that it proposes *breaking that
> otherwise universal pattern* - instead of having assignment
> expressions in comprehensions implicitly declare the name as local in
> the nested comprehension scope, it instead has them:

You talk about "nested comprehension scope", and that's a critical
point, but I'm going to skip answering that for now. I have a draft
email responding to another of your posts on that topic, which I hope to
polish in the next day.

> 1. implicitly declare the name as global or as nonlocal in the
> comprehension (or else raise an exception), depending on the nature of
> the parent scope where the comprehension is used
> 2. in the nonlocal reference case, amend the symbol table analysis to
> act like there was a preceding "if 0:\n for NAME in ():\n pass" in the
> parent scope (so the compiler knows which outer function scope to
> target)

If it is okay for you to amend the list comprehension to behave as if it
were wrapped in an implicit nested function, why shouldn't it be okay to
behave as if assignments inside the comprehension included an implicit
nonlocal declaration?

> The rationale being given for why that is OK is:
>
> 1. "Everyone" thinks comprehensions are just a for loop (even though
> that hasn't been true for the better part of a decade, and was never
> true for generator expressions)

Obviously "everyone" is an exaggeration, but, yes, I stand by that --
most people don't even give comprehension scope a thought until they
get bitten by it.

Either because (Python 2) they don't realise the loop variable is local
to their current scope:

http://www.librador.com/2014/07/10/Variable-scope-in-list-comprehension-vs-generator-expression/

or (Python 3) they get bitten by the change:

https://old.reddit.com/r/Python/comments/425qmb/strange_python_27_34_difference/

(As is so often the case, whatever behaviour we choose, we're going to
surprise somebody.)

It is hardly surprising that people don't think too hard about scoping
of comprehensions. Without a way to perform assignments inside
comprehensions, aside from the loop variables themselves, there's
nothing going on inside a comprehension where it makes a visible
difference whether it is a local scope or a sublocal scope.

*IF* assignment expressions are introduced, that is going to change. We
have some choices:

1. Keep assignment expressions encapsulated in their implicit function,
and be prepared for people to be annoyed because (with no way to declare
them global or non-local inside an expression), they can't use them to
get data in and out of the comprehension.


2. Allow assignment expressions to be exported out of the comprehension,
and be prepared for people to be annoyed because they clobbered a local.

(But for the reasons Tim Peters has already set out, I doubt this will
happen often.)


3. Allow some sort of extra comprehension syntax to allow
global/nonlocal declarations inside comprehensions.

x = 1
[nonlocal x := x+i for i in sequence]

(Hmmm... I thought I would hate that more than I actually do.)


4. Have some sort of cunning plan whereby if the variable in question
exists in the local scope, it is implicitly local inside the
comprehension:

x = 1
[x := i+1 for i in (1, 2)]
assert x == 3

but if it doesn't, then the variable is implicitly sublocal inside the
comprehension:

del x
[x := i+1 for i in (1, 2)]
x # raises NameError



Remember, the driving use-case which started this (ever-so-long)
discussion was the ability to push data into a comprehension and then
update it on each iteration, something like this:

x = initial_value()
results = [x := transform(x, i) for i in sequence]

Please, Nick, take your implementor's hat off, forget everything you
know about the implementation of comprehensions and their implicit
nested function, and tell me that doesn't look like it should work.




--
Steve

[Steven D'Aprano]

On Sun, Jun 24, 2018 at 09:24:39AM -0700, Guido van Rossum wrote:
> A quick follow-up: PEP 572 currently has two ideas: (a) introduce := for
> inline assignment, (b) when := is used in a comprehension, set the scope
> for the target as if the assignment occurred outside any comprehensions. It
> seems we have more support for (a) than for (b) -- at least Nick and Greg
> seem to be +0 or better for (a) but -1 for (b). IIRC (b) originated with
> Tim.

I'm not sure who came up with the idea first, but as I remember it, the
first mention of this came in a separate thread on Python-Ideas:

https://mail.python.org/pipermail/python-ideas/2018-April/049631.html

so possibly I'm to blame :-)

That thread starts here:

https://mail.python.org/pipermail/python-ideas/2018-April/049622.html

If I did get the idea from Tim, I don't remember doing so.

> But his essay on the topic, included as Appendix A (
> https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings)
> does not even mention comprehensions. However, he did post his motivation
> for (b) on python-ideas, IIRC a bit before PyCon; and the main text of the
> PEP gives a strong motivation (
> https://www.python.org/dev/peps/pep-0572/#scope-of-the-target).
> Nevertheless, maybe we should compromise and drop (b)?

I will have more to say about the whole "comprehensions are their own
scope" issue later. But I'd like to see Nick's proposed PEP, or at least
a draft of it, before making any final decisions.

If it came down to it, I'd be happy with the ability to declare an
assignment target nonlocal in the comprehension if that's what it takes.
What do you think of this syntax?

[global|nonlocal] simple_target := expression


Inside a comprehension, without a declaration, the target would be
sublocal (comprehension scope); that should make Nick happier :-)

[Tim Peters]

[Guido]

> A quick follow-up: PEP 572 currently has two ideas: (a) introduce := for inline
> assignment, (b) when := is used in a comprehension, set the scope for the
> target as if the assignment occurred outside any comprehensions. It seems
> we have more support for (a) than for (b) -- at least Nick and Greg seem to
> be +0 or better for (a) but -1 for (b). IIRC (b) originated with Tim. But his
> essay on the topic, included as Appendix A
> (https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings)
> does not even mention comprehensions.

I was writing up my observations about simple changes to existing code.  Since there's nothing sanely akin to binding non-for-targets possible in comprehensions now, comprehensions were out of scope for that effort (which was limited to staring at existing code already doing bindings).

:> However, he did post his motivation for (b) on python-ideas, IIRC a bit

> before PyCon; and the main text of the PEP gives a strong motivation
> (https://www.python.org/dev/peps/pep-0572/#scope-of-the-target). Nevertheless,
> maybe we should compromise and drop (b)?

Two things to say about that.  First, the original example I gave would be approximately as well addressed by allowing to declare intended scopes in magically synthesized functions; like (say)

p = None # to establish the intended scope of `p`
while any(<nonlocal p>  # split across lines just for readability

                n % p == 0 for p in small_primes):
    n //= p

It didn't really require an inline assignment, just a way to override the unwanted (in this case) "all `for` targets are local to the invisible function" rigid consequence of the implementation du jour.

Second, if it's dropped, then the PEP needs more words to define what happens in cases like the following, because different textual parts of a comprehension execute in different scopes, and that can  become visible when bindings can be embedded:

def f():

    y = -1

    ys = [y for _ in range(y := 5)]

    print(y, ys)

Here `range(y := 5)` is executed in f's scope.  Presumably the `y` in `y for` also refers to f's scope, despite that `y` textually _appears_ to be assigned to in the body of the listcomp, and so would - for that reason - expected to be local to the synthesized function, and so raise `UnboundLocalError` when referenced.  It's incoherent without detailed knowledge of the implementation.

def g():

    y = -1

    ys = [y for y in range(y := 5)]

    print(y, ys)

And here the `y` in `y for y` is local to the synthesized function, and presumably has nothing to do with the `y` in the `range()` call.  That's incoherent in its own way.

Under the current PEP, all instances of `y` in `f` refer to the f-local `y`, and the listcomp in `g` is a compile-time error.

[Chris Angelico]

On Mon, Jun 25, 2018 at 4:06 AM, Steven D'Aprano <st...@pearwood.info> wrote:
>
> Remember, the driving use-case which started this (ever-so-long)
> discussion was the ability to push data into a comprehension and then
> update it on each iteration, something like this:
>
> x = initial_value()
> results = [x := transform(x, i) for i in sequence]

Which means there is another option.

5. Have the assignment be local to the comprehension, but the initial
value of ANY variable is looked up from the surrounding scopes.

That is: you will NEVER get UnboundLocalError from a
comprehension/genexp; instead, you will look up the name as if it were
in the surrounding scope, either getting a value or bombing with
regular old NameError.

Or possibly variations on this such as "the immediately surrounding
scope only", rather than full name lookups. It'd have an awkward
boundary somewhere, whichever way you do it.

This isn't able to send information *out* of a comprehension, but it
is able to send information *in*.

ChrisA

[Michael Selik]

This thread started with a request for educator feedback, which I took to mean observations of student reactions. I've only had the chance to test the proposal on ~20 students so far, but I'd like the chance to gather more data for your consideration before the PEP is accepted or rejected.

On Sun, Jun 24, 2018 at 11:09 AM Steven D'Aprano <st...@pearwood.info> wrote:

Remember, the driving use-case which started this (ever-so-long)
discussion was the ability to push data into a comprehension and then
update it on each iteration, something like this:

    x = initial_value()
    results = [x := transform(x, i) for i in sequence]

If that is the driving use-case, then the proposal should be rejected. The ``itertools.accumulate`` function has been available for a little while now and it handles this exact case. The accumulate function may even be more readable, as it explains the purpose explicitly, not merely the algorithm. And heck, it's a one-liner.

    results = accumulate(sequence, transform)

The benefits for ``any`` and ``all`` seem useful. Itertools has "first_seen" in the recipes section. While it feels intuitively useful, I can't recall ever writing something similar myself. For some reason, I (almost?) always want to find all (counter-)examples and aggregate them in some way -- min or max, perhaps -- rather than just get the first.

Even so, if it turns out those uses are quite prevalent, wouldn't a new itertool be better than a new operator? It's good to solve the general problem, but so far the in-comprehension usage seems to have only a handful of cases.

On Fri, Jun 22, 2018 at 9:14 PM Chris Barker via Python-Dev <pytho...@python.org> wrote:

again, not a huge deal, just a little bit more complexity

I worry that Python may experience something of a "death by a thousand cuts" along the lines of the "Remember the Vasa" warning. Python's greatest strength is its appeal to beginners. Little bits of added complexity have a non-linear effect. One day, we may wake up and Python won't be recommended as a beginner's language.

On Fri, Jun 22, 2018 at 7:48 PM Steven D'Aprano <st...@pearwood.info> wrote:

On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:

Of course they do -- they're less fluent at reading code. They don't 
have the experience to judge good code from bad.

On the other hand, an "expert" may be so steeped in a particular subculture that he no longer can distinguish esoteric from intuitive. Don't be so fast to reject the wisdom of the inexperienced.

 

The question we should be asking is, do we only add features to Python 
if they are easy for beginners? It's not that I especially want to add 
features which *aren't* easy for beginners, but Python isn't Scratch and 
"easy for beginners" should only be a peripheral concern.

On the contrary, I believe that "easy for beginners" should be a major concern.  Ease of use has been and is a, or even the main reason for Python's success. When some other language becomes a better teaching language, it will eventually take over in business and science as well. Right now, Python is Scratch for adults. That's a great thing. Given the growth of the field, there are far more beginner programmers working today than there ever have been experts.

Mozilla's array comprehensions are almost identical to Python's, aside 
from a couple of trivial differences:

I can't prove it, but I think the phrase ordering difference is not trivial.

> Students who are completely new to programming can see the similarity of

> [Python] list comprehensions to spoken language. 

I've been using comprehensions for something like a decade, and I can't 

Python: any(line.startswith('#') for line in file)

English: Any line starts with "#" in the file?

[Guido van Rossum]

On Sun, Jun 24, 2018 at 11:50 AM Steven D'Aprano <st...@pearwood.info> wrote:

[Guido]
> [...] IIRC (b) originated with Tim.

I'm not sure who came up with the idea first, but as I remember it, the
first mention of this came in a separate thread on Python-Ideas:

https://mail.python.org/pipermail/python-ideas/2018-April/049631.html

so possibly I'm to blame :-)

Actually that post sounds like the OP of that thread (Peter O'Connor) is to blame --  he proposed a similar thing using `=` for the assignment and custom syntax (`from <value>`) to specify the initial value, and it looks like that inspired you.

 

> But his essay on the topic, included as Appendix A (
> https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings)
> does not even mention comprehensions. However, he did post his motivation
> for (b) on python-ideas, IIRC a bit before PyCon; and the main text of the
> PEP gives a strong motivation (
> https://www.python.org/dev/peps/pep-0572/#scope-of-the-target).
> Nevertheless, maybe we should compromise and drop (b)?

I will have more to say about the whole "comprehensions are their own
scope" issue later. But I'd like to see Nick's proposed PEP, or at least
a draft of it, before making any final decisions.

Agreed, though I assume it's just `given` again.

 

If it came down to it, I'd be happy with the ability to declare an
assignment target nonlocal in the comprehension if that's what it takes.
What do you think of this syntax?

    [global|nonlocal] simple_target := expression

Inside a comprehension, without a declaration, the target would be
sublocal (comprehension scope); that should make Nick happier :-)

It's more special syntax. Just taking part (a) of PEP 572 would make most people happy enough.

 

[Guido van Rossum]

On Sun, Jun 24, 2018 at 12:03 PM Tim Peters <tim.p...@gmail.com> wrote:

[Guido]

:> However, [Tim] did post his motivation for (b) on python-ideas, IIRC a bit

> before PyCon; and the main text of the PEP gives a strong motivation
> (https://www.python.org/dev/peps/pep-0572/#scope-of-the-target). Nevertheless,
> maybe we should compromise and drop (b)?

Two things to say about that.  First, the original example I gave would be approximately as well addressed by allowing to declare intended scopes in magically synthesized functions; like (say)

p = None # to establish the intended scope of `p`
while any(<nonlocal p>  # split across lines just for readability

                n % p == 0 for p in small_primes):
    n //= p

It didn't really require an inline assignment, just a way to override the unwanted (in this case) "all `for` targets are local to the invisible function" rigid consequence of the implementation du jour.

Hm, that's more special syntax. The nice bit about (b) as currently specified is that it adds no syntax -- it adds a scope rule, but (as IIRC Steven has convincingly argued) few people care about those. Python's scope rules, when fully specified, are intricate to the point of being arcane (e.g. for class scopes) but all that has a purpose -- to make them so DWIM ("Do what I Mean") that in practice you almost never have to worry about them, *especially* when reading non-obfuscated code (and also when writing, except for a few well-known patterns).

 

Second, if it's dropped, then the PEP needs more words to define what happens in cases like the following, because different textual parts of a comprehension execute in different scopes, and that can  become visible when bindings can be embedded:

def f():

    y = -1

    ys = [y for _ in range(y := 5)]

    print(y, ys)

Here `range(y := 5)` is executed in f's scope.  Presumably the `y` in `y for` also refers to f's scope, despite that `y` textually _appears_ to be assigned to in the body of the listcomp, and so would - for that reason - expected to be local to the synthesized function, and so raise `UnboundLocalError` when referenced.  It's incoherent without detailed knowledge of the implementation.

That code should have the same meaning regardless of whether we accept (b) or not -- there is only one `y`, in f's scope. I don't mind if we have to add more words to the PEP's scope rules to make this explicit, though I doubt it -- the existing weirdness (in the comprehension spec) about the "outermost iterable" being evaluated in the surrounding scope specifies this. I wouldn't call it incoherent -- I think what I said about scope rules above applies here, it just does what you expect.

 

def g():

    y = -1

    ys = [y for y in range(y := 5)]

    print(y, ys)

And here the `y` in `y for y` is local to the synthesized function, and presumably has nothing to do with the `y` in the `range()` call.  That's incoherent in its own way.

Under the current PEP, all instances of `y` in `f` refer to the f-local `y`, and the listcomp in `g` is a compile-time error.

And under the (b)-less proposal, `g` would interpret `y for y` as both referring to a new variable created just for the comprehension, and `y := 5` as referring to g's scope. Again I don't think it needs extra words in the spec. And the end user docs might just say "don't do that" (with a link to the reference manual's rule about the "outermost iterable").

Even if in the end we did find a case where we'd have to write an explicit rule to make what happens here a consequence of the spec rather than the implementation, that doesn't count as an argument for keeping (b) to me.

In favor of (b) we have a few examples (see https://www.python.org/dev/peps/pep-0572/#scope-of-the-target) that require it, and more that you described on python-ideas (and also the motivating use case from the thread that Steven dug up, starting here: https://mail.python.org/pipermail/python-ideas/2018-April/049622.html).

A "neutral" argument about (b) is that despite the "horrified" reactions that Nick saw, in practice it's going to confuse very few people (again, due to my point about Python's scope rules). I'd wager that the people who might be most horrified about it would be people who feel strongly that the change to the comprehension scope rules in Python 3 is a big improvement, and who are familiar with the difference in implementation of comprehensions (though not generator expressions) in Python 2 vs. 3.

[Greg Ewing]

Guido van Rossum wrote:
> Greg seem to be +0 or better for (a)

Actually, I'm closer to -1 on (a) as well. I don't like := as a
way of getting assignment in an expression. The only thing I would
give a non-negative rating is some form of "where" or "given".

Brief summary of reasons for disliking ":=":

* Cryptic use of punctuation

* Too much overlap in functionality with "="

* Asymmetry between first and subsequent uses of the bound value

* Makes expressions cluttered and hard to read to my eyes

--
Greg

[Guido van Rossum]

On Sun, Jun 24, 2018 at 2:10 PM Chris Angelico <ros...@gmail.com> wrote:

On Mon, Jun 25, 2018 at 4:06 AM, Steven D'Aprano <st...@pearwood.info> wrote:
>
> Remember, the driving use-case which started this (ever-so-long)
> discussion was the ability to push data into a comprehension and then
> update it on each iteration, something like this:
>
>     x = initial_value()
>     results = [x := transform(x, i) for i in sequence]

Which means there is another option.

5. Have the assignment be local to the comprehension, but the initial
value of ANY variable is looked up from the surrounding scopes.

That is: you will NEVER get UnboundLocalError from a
comprehension/genexp; instead, you will look up the name as if it were
in the surrounding scope, either getting a value or bombing with
regular old NameError.

Or possibly variations on this such as "the immediately surrounding
scope only", rather than full name lookups. It'd have an awkward
boundary somewhere, whichever way you do it.

This isn't able to send information *out* of a comprehension, but it
is able to send information *in*.

But this "horrifies" me for a slightly different reason: it effectively introduces a new case of dynamic scoping, which Python used to do everywhere but has long switched away from, with the exception of class scopes (whose difference with function scopes sometimes confuses people -- usually people who put too much code in their class scope).

[Greg Ewing]

Steven D'Aprano wrote:
> You seem to be talking about an implementation which could change in the
> future. I'm talking semantics of the proposed language feature.

The way I see it, it's not about implementation details,
it's about having a mental model that's easy to reason
about.

"Comprehensions run in their own scope, like a def or
lambda" is a clear and simple mental model. It's easy to
explain and keep in your head.

The proposed semantics are much more complicated, and as
far as I can see, are only motivated by use cases that
you shouldn't really be doing in the first place.

--
Greg

[Ben Finney]

Greg Ewing <greg....@canterbury.ac.nz> writes:

> Actually, I'm closer to -1 on (a) as well. I don't like := as a
> way of getting assignment in an expression. The only thing I would
> give a non-negative rating is some form of "where" or "given".

+1 to this; ‘:=’ doesn't convey the meaning well. Python's syntax
typically eschews cryptic punctuation in faviour of existing words that
convey an appropriate meaning, and I agree with Greg that would be a
better way to get this effect.

--
\ “Self-respect: The secure feeling that no one, as yet, is |
`\ suspicious.” —Henry L. Mencken |
_o__) |
Ben Finney

[Guido van Rossum]

On Sun, Jun 24, 2018 at 2:41 PM Michael Selik <mi...@selik.org> wrote:

This thread started with a request for educator feedback, which I took to mean observations of student reactions. I've only had the chance to test the proposal on ~20 students so far, but I'd like the chance to gather more data for your consideration before the PEP is accepted or rejected.

Sure. Since the target for the PEP is Python 3.8 I am in no particular hurry. It would be important to know how you present it to your students.

 

On Sun, Jun 24, 2018 at 11:09 AM Steven D'Aprano <st...@pearwood.info> wrote:

Remember, the driving use-case which started this (ever-so-long)
discussion was the ability to push data into a comprehension and then
update it on each iteration, something like this:

    x = initial_value()
    results = [x := transform(x, i) for i in sequence]

If that is the driving use-case, then the proposal should be rejected. The ``itertools.accumulate`` function has been available for a little while now and it handles this exact case. The accumulate function may even be more readable, as it explains the purpose explicitly, not merely the algorithm. And heck, it's a one-liner.

    results = accumulate(sequence, transform)

I think that's a misunderstanding. At the very least the typical use case is *not* using an existing transform function which is readily passed to accumulate -- instead, it's typically written as a simple expression (e.g. `total := total + v` in the PEP) which would require a lambda.

Plus, I don't know what kind of students you are teaching, but for me, whenever the solution requires a higher-order function (like accumulate), this implies a significant speed bump -- both when writing and when reading code. (Honestly, whenever I read code that uses itertools, I end up making a trip to StackOverflow :-).

 

The benefits for ``any`` and ``all`` seem useful. Itertools has "first_seen" in the recipes section.

(I think you mean first_true().)

 

While it feels intuitively useful, I can't recall ever writing something similar myself. For some reason, I (almost?) always want to find all (counter-)examples and aggregate them in some way -- min or max, perhaps -- rather than just get the first.

I trust Tim's intuition here, he's written about this.

Also, Python's predecessor, ABC, had quantifiers (essentially any() and all()) built into the language, and the semantics included making the first (counter-)example available (https://homepages.cwi.nl/~steven/abc/qr.html#TESTS). Essentially

IF SOME x IN values HAS x < 0:

    WRITE "Found a negative x:", x

equivalently

IF EACH x IN values HAS x >= 0:

    # ...

ELSE:

    WRITE "Found a negative x:", x

and even

IF NO x IN values HAS x < 0:

    # ...

ELSE:

    WRITE "Found a negative x:", x

 

Even so, if it turns out those uses are quite prevalent, wouldn't a new itertool be better than a new operator? It's good to solve the general problem, but so far the in-comprehension usage seems to have only a handful of cases.

Perhaps, but IMO the new itertool would be much less useful than the new syntax.

 

On Fri, Jun 22, 2018 at 9:14 PM Chris Barker via Python-Dev <pytho...@python.org> wrote:

again, not a huge deal, just a little bit more complexity

I worry that Python may experience something of a "death by a thousand cuts" along the lines of the "Remember the Vasa" warning. Python's greatest strength is its appeal to beginners. Little bits of added complexity have a non-linear effect. One day, we may wake up and Python won't be recommended as a beginner's language.

I don't think that appeal to beginners is Python's greatest strength. I'd rather say that it is its appeal to both beginners and experts (and everyone in between). If true appeal to beginners is needed, Scratch or Processing would probably be better.

 

On Fri, Jun 22, 2018 at 7:48 PM Steven D'Aprano <st...@pearwood.info> wrote:

On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:

Of course they do -- they're less fluent at reading code. They don't 
have the experience to judge good code from bad.

On the other hand, an "expert" may be so steeped in a particular subculture that [they] no longer can distinguish esoteric from intuitive. Don't be so fast to reject the wisdom of the inexperienced.

Nor should we cater to them excessively though. While the user is indeed king, it's also well known that most users when they are asking for a feature don't know what they want (same for kings, actually, that's why they have advisors :-).

 

The question we should be asking is, do we only add features to Python 
if they are easy for beginners? It's not that I especially want to add 
features which *aren't* easy for beginners, but Python isn't Scratch and 
"easy for beginners" should only be a peripheral concern.

On the contrary, I believe that "easy for beginners" should be a major concern.  Ease of use has been and is a, or even the main reason for Python's success. When some other language becomes a better teaching language, it will eventually take over in business and science as well. Right now, Python is Scratch for adults. That's a great thing. Given the growth of the field, there are far more beginner programmers working today than there ever have been experts.

I'm sorry, but this offends me, and I don't believe it's true at all. Python is *not* a beginners language, and you are mixing ease of use and ease of learning. Python turns beginners into experts at an unprecedented rate, and that's the big difference with Scratch.

[Michael Selik]

On Sun, Jun 24, 2018 at 4:57 PM Guido van Rossum <gu...@python.org> wrote:

On Sun, Jun 24, 2018 at 2:41 PM Michael Selik <mi...@selik.org> wrote:

This thread started with a request for educator feedback, which I took to mean observations of student reactions. I've only had the chance to test the proposal on ~20 students so far, but I'd like the chance to gather more data for your consideration before the PEP is accepted or rejected.

Sure. Since the target for the PEP is Python 3.8 I am in no particular hurry. It would be important to know how you present it to your students.

Absolutely. Since this has come up, I'll make an effort to be more systematic in data collection. 

 

On Sun, Jun 24, 2018 at 11:09 AM Steven D'Aprano <st...@pearwood.info> wrote:

Remember, the driving use-case which started this (ever-so-long)
discussion was the ability to push data into a comprehension and then
update it on each iteration, something like this:

    x = initial_value()
    results = [x := transform(x, i) for i in sequence]

If that is the driving use-case, then the proposal should be rejected. The ``itertools.accumulate`` function has been available for a little while now and it handles this exact case. The accumulate function may even be more readable, as it explains the purpose explicitly, not merely the algorithm. And heck, it's a one-liner.

    results = accumulate(sequence, transform)

I think that's a misunderstanding. At the very least the typical use case is *not* using an existing transform function which is readily passed to accumulate -- instead, it's typically written as a simple expression (e.g. `total := total + v` in the PEP) which would require a lambda.

Plus, I don't know what kind of students you are teaching, but for me, whenever the solution requires a higher-order function (like accumulate), this implies a significant speed bump -- both when writing and when reading code. (Honestly, whenever I read code that uses itertools, I end up making a trip to StackOverflow :-).

Mostly mid-career professionals, of highly varying backgrounds. The higher-order functions do require some cushioning getting into, but I have some tricks I've learned over the years to make it go over pretty well.

On Fri, Jun 22, 2018 at 7:48 PM Steven D'Aprano <st...@pearwood.info> wrote:

On Fri, Jun 22, 2018 at 10:59:43AM -0700, Michael Selik wrote:

Of course they do -- they're less fluent at reading code. They don't 
have the experience to judge good code from bad.

On the other hand, an "expert" may be so steeped in a particular subculture that [they] no longer can distinguish esoteric from intuitive. Don't be so fast to reject the wisdom of the inexperienced.

Nor should we cater to them excessively though. While the user is indeed king, it's also well known that most users when they are asking for a feature don't know what they want (same for kings, actually, that's why they have advisors :-).

 

The question we should be asking is, do we only add features to Python 
if they are easy for beginners? It's not that I especially want to add 
features which *aren't* easy for beginners, but Python isn't Scratch and 
"easy for beginners" should only be a peripheral concern.

On the contrary, I believe that "easy for beginners" should be a major concern.  Ease of use has been and is a, or even the main reason for Python's success. When some other language becomes a better teaching language, it will eventually take over in business and science as well. Right now, Python is Scratch for adults. That's a great thing. Given the growth of the field, there are far more beginner programmers working today than there ever have been experts.

I'm sorry, but this offends me, and I don't believe it's true at all. Python is *not* a beginners language, and you are mixing ease of use and ease of learning. Python turns beginners into experts at an unprecedented rate, and that's the big difference with Scratch.

By saying "Scratch for adults" I meant that Python is a language that can be adopted by beginners and rapidly make them professionals, not that it's exclusively a beginner's language.

Also, Scratch and similar languages, like NetLogo, have some interesting features that allow beginners to write some sophisticated parallelism. I don't mean "beginner's language" in that it's overly simplistic, but that it enables what would be complex in other languages.

I realize that my phrasing was likely to be misunderstood without knowing the context that I teach working professionals who are asked to be immediately productive at high-value tasks.

[Tim Peters]

[Tim]

.  First, the original example I gave would be approximately as well addressed by allowing to declare intended scopes in magically synthesized functions; like (say)

p = None # to establish the intended scope of `p`
while any(<nonlocal p>  # split across lines just for readability

                n % p == 0 for p in small_primes):
    n //= p

It didn't really require an inline assignment, just a way to override the unwanted (in this case) "all `for` targets are local to the invisible function" rigid consequence of the implementation du jour.

[Guido]

Hm, that's more special syntax.

Of course - I'm anticipating that the PEP will be changed to throw out useful assignment expressions in comprehensions, but I still want a way to "export" comprehension for-targets at times ;-)
 

The nice bit about (b) as currently specified is that it adds no syntax -- it adds a scope rule, but (as IIRC Steven has convincingly argued) few people care about those. Python's scope rules, when fully specified, are intricate to the point of being arcane (e.g. for class scopes) but all that has a purpose -- to make them so DWIM ("Do what I Mean") that in practice you almost never have to worry about them, *especially* when reading non-obfuscated code (and also when writing, except for a few well-known patterns).

You and Steven and i appear to be on the same page here - but it's in a book nobody else seems to own :-(  To me it's just screamingly obvious that

    total = 0

    cumsums = [total := total + value for value in data]

"should do" what it obviously intends to do - and that the only thing stopping that is a bass-ackwards focus on what most trivially falls out of the current implementation.

...

def f():

    y = -1

    ys = [y for _ in range(y := 5)]

    print(y, ys)

Here `range(y := 5)` is executed in f's scope.  Presumably the `y` in `y for` also refers to f's scope, despite that `y` textually _appears_ to be assigned to in the body of the listcomp, and so would - for that reason - expected to be local to the synthesized function, and so raise `UnboundLocalError` when referenced.  It's incoherent without detailed knowledge of the implementation.

That code should have the same meaning regardless of whether we accept (b) or not -- there is only one `y`, in f's scope. I don't mind if we have to add more words to the PEP's scope rules to make this explicit, though I doubt it -- the existing weirdness (in the comprehension spec) about the "outermost iterable" being evaluated in the surrounding scope specifies this. I wouldn't call it incoherent -- I think what I said about scope rules above applies here, it just does what you expect.

Remove "y = -1" and - voila! - we have the dreaded "parent local scoping" Nick finds so baffling to explain (or so he claims).  That is, "has exactly the same scope in the comprehension as in the parent block, and will create a local in the latter if the name is otherwise unknown in the parent" comes with assignment expressions, regardless of whether _all_ such targets "leak" (the current PEP) or only targets in the expression defining the iterable of the outermost `for` (the PEP without leaking assignment expressions in comprehensions).

As to whether it "does what you expect", no, not really! In a world where _all_ binding targets in a comprehension are claimed to be local to the comprehension, I _expect_ that `y := 5` appearing inside the listcomp means `y` is local to the listcomp.  "Oh - unless the binding appears in the expression defining the iterable of the outermost `for`" comes from Mars.

Not that it really matters much, but (b) provides consistent semantics in these cases.  No need to search Mars for weird exceptions ;-)

...

A "neutral" argument about (b) is that despite the "horrified" reactions that Nick saw, in practice it's going to confuse very few people (again, due to my point about Python's scope rules). I'd wager that the people who might be most horrified about it would be people who feel strongly that the change to the comprehension scope rules in Python 3 is a big improvement, and who are familiar with the difference in implementation of comprehensions (though not generator expressions) in Python 2 vs. 3.

I also doubt it will generally confuse people in practice (to the contrary, I expect they _will_ be confused if things like the cumulative sums example blow up with UnboundLocalError).

But I still don't get the source of the "horror".  Assignment expression semantics are wholly consistent with ordinary nested lexical scoping, with or without (b).  The only difference is in the scopes picked for assignment expression target names (except for those appearing in the expression defining the iterable yadda yadda yadda).

[Nick Coghlan]

On 25 June 2018 at 02:24, Guido van Rossum <gu...@python.org> wrote:
> A quick follow-up: PEP 572 currently has two ideas: (a) introduce := for
> inline assignment, (b) when := is used in a comprehension, set the scope for
> the target as if the assignment occurred outside any comprehensions. It
> seems we have more support for (a) than for (b) -- at least Nick and Greg
> seem to be +0 or better for (a)

Right, the proposed blunt solution to "Should I use 'NAME = EXPR' or
'NAME := EXPR'?" bothers me a bit, but it's the implementation
implications of parent local scoping that I fear will create a
semantic tar pit we can't get out of later.

> but -1 for (b). IIRC (b) originated with
> Tim. But his essay on the topic, included as Appendix A
> (https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings)
> does not even mention comprehensions. However, he did post his motivation
> for (b) on python-ideas, IIRC a bit before PyCon; and the main text of the
> PEP gives a strong motivation
> (https://www.python.org/dev/peps/pep-0572/#scope-of-the-target).
> Nevertheless, maybe we should compromise and drop (b)?

Unfortunately, I think the key rationale for (b) is that if you
*don't* do something along those lines, then there's a different
strange scoping discrepancy that arises between the non-comprehension
forms of container displays and the comprehension forms:

(NAME := EXPR,) # Binds a local
tuple(NAME := EXPR for __ in range(1)) # Doesn't bind a local

[NAME := EXPR] # Binds a local
[NAME := EXPR for __ in range(1)] # Doesn't bind a local
list(NAME := EXPR for __ in range(1)) # Doesn't bind a local

{NAME := EXPR} # Binds a local
{NAME := EXPR for __ in range(1)} # Doesn't bind a local
set(NAME := EXPR for __ in range(1)) # Doesn't bind a local

{NAME := EXPR : EXPR2} # Binds a local
{NAME := EXPR : EXPR2 for __ in range(1)} # Doesn't bind a local
set((NAME := EXPR, EXPR2) for __ in range(1)) # Doesn't bind a local

Those scoping inconsistencies aren't *new*, but provoking them
currently involves either class scopes, or messing about with
locals().

The one virtue that choosing this particular set of discrepancies has
is that the explanation for why they happen is the same as the
explanation for how the iteration variable gets hidden from the
containing scope: because "(EXPR for ....)" et al create an implicitly
nested scope, and that nested scope behaves the same way as an
explicitly nested scope as far as name binding and name resolution is
concerned.

Parent local scoping tries to mitigate the surface inconsistency by
changing how write semantics are defined for implicitly nested scopes,
but that comes at the cost of making those semantics inconsistent with
explicitly nested scopes and with the read semantics of implicitly
nested scopes.

The early iterations of PEP 572 tried to duck this whole realm of
potential semantic inconsistencies by introducing sublocal scoping
instead, such that the scoping for assignment expression targets would
be unusual, but they'd be consistently unusual regardless of where
they appeared, and their quirks would clearly be the result of how
assignment expressions were defined, rather than only showing up in
how they interacted with other scoping design decisions made years
ago.

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Nick Coghlan]

On 25 June 2018 at 09:25, Guido van Rossum <gu...@python.org> wrote:
> A "neutral" argument about (b) is that despite the "horrified" reactions
> that Nick saw, in practice it's going to confuse very few people (again, due
> to my point about Python's scope rules). I'd wager that the people who might
> be most horrified about it would be people who feel strongly that the change
> to the comprehension scope rules in Python 3 is a big improvement, and who
> are familiar with the difference in implementation of comprehensions (though
> not generator expressions) in Python 2 vs. 3.

FWIW, the most cryptic parent local scoping related exception I've
been able to devise so far still exhibits PEP 572's desired "Omitting
the comprehension scope entirely would give you the same name lookup
behaviour" semantics:

>>> def outer(x=1):
... def middle():
... return [x := x + i for i in range(10)]
... return middle()
...
>>> outer()
Traceback (most recent call last):
...
NameError: free variable 'x' referenced before assignment in enclosing scope

It isn't the parent local scoping, or even the assignment expression,
that's at fault there, since you'd get exactly the same exception for:

def outer(x=1):
def middle():
x = x +1
return x
return middle()

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Paul Moore]

On 25 June 2018 at 12:44, Nick Coghlan <ncog...@gmail.com> wrote:
> Unfortunately, I think the key rationale for (b) is that if you
> *don't* do something along those lines, then there's a different
> strange scoping discrepancy that arises between the non-comprehension
> forms of container displays and the comprehension forms:

I've been mostly ignoring this proposal for a while now, so I'm going
to respond here in the context of someone with a bit of an idea of the
underlying complexities, but otherwise coming at it as a new proposal.

>
> (NAME := EXPR,) # Binds a local
> tuple(NAME := EXPR for __ in range(1)) # Doesn't bind a local
>
> [NAME := EXPR] # Binds a local
> [NAME := EXPR for __ in range(1)] # Doesn't bind a local
> list(NAME := EXPR for __ in range(1)) # Doesn't bind a local
>
> {NAME := EXPR} # Binds a local
> {NAME := EXPR for __ in range(1)} # Doesn't bind a local
> set(NAME := EXPR for __ in range(1)) # Doesn't bind a local
>
> {NAME := EXPR : EXPR2} # Binds a local
> {NAME := EXPR : EXPR2 for __ in range(1)} # Doesn't bind a local
> set((NAME := EXPR, EXPR2) for __ in range(1)) # Doesn't bind a local

None of those "discrepancies" bother me in the slightest, when taken
in isolation as you present them here. I suspect you could lead me
through a chain of logic that left me understanding why you describe
them as discrepancies, but without that explanation, I'm fine with all
of them.

I'd also say that they seem contrived (not just in the use of
artificial names, but also in the sense that I'm not sure why I'd want
to use this *pattern*) so I'd happily say "well, don't do that then"
if things started behaving non-intuitively.

> Those scoping inconsistencies aren't *new*, but provoking them
> currently involves either class scopes, or messing about with
> locals().

And to reinforce my point above, I already consider putting
significant code in class scopes, or using locals() to be techniques
that should only be used sparingly and with a clear understanding of
the subtleties. I'm sure you could say "but the examples above would
be much more common" in response to which I'd like to see real use
cases that behave non-intuitively in the way you're concerned about.

> The one virtue that choosing this particular set of discrepancies has
> is that the explanation for why they happen is the same as the
> explanation for how the iteration variable gets hidden from the
> containing scope: because "(EXPR for ....)" et al create an implicitly
> nested scope, and that nested scope behaves the same way as an
> explicitly nested scope as far as name binding and name resolution is
> concerned.

But that's precisely why I find the behaviour intuitive - the nested
scope is the *reason* things behave this way, not some sort of
easily-overlooked way the "problem" can be explained away.

> Parent local scoping tries to mitigate the surface inconsistency by
> changing how write semantics are defined for implicitly nested scopes,
> but that comes at the cost of making those semantics inconsistent with
> explicitly nested scopes and with the read semantics of implicitly
> nested scopes.
>
> The early iterations of PEP 572 tried to duck this whole realm of
> potential semantic inconsistencies by introducing sublocal scoping
> instead, such that the scoping for assignment expression targets would
> be unusual, but they'd be consistently unusual regardless of where
> they appeared, and their quirks would clearly be the result of how
> assignment expressions were defined, rather than only showing up in
> how they interacted with other scoping design decisions made years
> ago.

Those last two paragraphs made my head explode, as far as I can see by
virtue of the fact that they try to over-analyze the fairly simple
intuition I have that "there's a nested scope involved".

Disclaimer: I may well have got a *lot* of subtleties wrong here, and
it's quite likely that my impressions don't stand up to the harsh
reality of how the implementation works. But my comments are on the
basis of my *intuition*, whether that's right or wrong. And if the
reality violates my intuition, it's *other* constructs that I find
non-intuitive, not this one. (I'm perfectly happy to concede that it's
not possible to avoid *any* non-intuitive behaviour - all I'm trying
to say is that my intuition doesn't balk at this one, unlike yours).

Paul

[Paul Moore]

Once again offering an "intuition" based response:

1. That definition of outer() is very complicated, I don't *expect* to
understand it without checking the details. So the NameError is simply
"hmm, wonder what triggered that?" not "OMG that's not what I'd
expect!" :-)
2. Given that your version with no assignment expression or
comprehension exhibits the same behaviour, I'm not sure what your
argument is here anyway...

Paul

[Nick Coghlan]

On 25 June 2018 at 22:17, Nick Coghlan <ncog...@gmail.com> wrote:
> FWIW, the most cryptic parent local scoping related exception I've
> been able to devise so far still exhibits PEP 572's desired "Omitting
> the comprehension scope entirely would give you the same name lookup
> behaviour" semantics:
>
> >>> def outer(x=1):
> ... def middle():
> ... return [x := x + i for i in range(10)]
> ... return middle()
> ...
> >>> outer()
> Traceback (most recent call last):
> ...
> NameError: free variable 'x' referenced before assignment in enclosing scope
>
> It isn't the parent local scoping, or even the assignment expression,
> that's at fault there, since you'd get exactly the same exception for:
>
> def outer(x=1):
> def middle():
> x = x +1
> return x
> return middle()

Oops, I didn't mean to say "exactly the same exception" here, as the
whole reason I'd settled on this example as the most cryptic one I'd
found so far was the fact that the doubly nested version *doesn't*
give you the same exception as the singly nested version: the version
without the comprehension throws UnboundLocalError instead.

However, the resolution is the same either way: either 'x' has to be
declared as 'nonlocal x' in 'middle', or else it has to be passed in
to 'middle' as a parameter.

[Nick Coghlan]

On 25 June 2018 at 09:02, Guido van Rossum <gu...@python.org> wrote:
> On Sun, Jun 24, 2018 at 11:50 AM Steven D'Aprano <st...@pearwood.info>
> wrote:
>> I will have more to say about the whole "comprehensions are their own
>> scope" issue later. But I'd like to see Nick's proposed PEP, or at least
>> a draft of it, before making any final decisions.
>
>
> Agreed, though I assume it's just `given` again.

While I still have some TODO notes of my own to resolve before posting
it to python-ideas, the examples section at
https://github.com/ncoghlan/peps/pull/2/files#diff-7a25ca1769914c1141cb5c63dc781f32R202
already gives a pretty good idea of the differences relative to PEP
572: rebinding existing names is unchanged from PEP 572, but
introducing new names requires a bit of "Yes, I really do want to
introduce this new name here" repetition.

The big difference from previous iterations of the "given" idea is
that it doesn't try to completely replace the proposed inline
assignments, it just supplements them by providing a way to do inline
name *declarations* (which may include declaring targets as global or
nonlocal, just as regular function level declarations can).

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Paul Moore]

At the level of "what my intuition says" the result is the same in
both cases - "it throws an exception". I have no intuition on *which*
exception would be raised and would experiment (or look up the
details) if I cared.

> However, the resolution is the same either way: either 'x' has to be
> declared as 'nonlocal x' in 'middle', or else it has to be passed in
> to 'middle' as a parameter.

Once someone told me that's what I needed, it's sufficiently obvious
that I'm fine with that. If no-one was able to tell me what to do, I'd
simply rewrite the code to be less obfuscated :-)

I've probably explained my intuition enough here. If we debate any
further I'll just end up knowing what's going on, destroying my value
as an "uninformed user" :-)
Paul

[Ivan Pozdeev via Python-Dev]

On 25.06.2018 2:30, Greg Ewing wrote:
> Guido van Rossum wrote:
>> Greg seem to be +0 or better for (a)
>
> Actually, I'm closer to -1 on (a) as well. I don't like := as a
> way of getting assignment in an expression. The only thing I would
> give a non-negative rating is some form of "where" or "given".
>

"as" was suggested even before is became a keyword in `with'. ( if
(re.match(regex,line) as m) is not None: <do smth> )

The only objective objection I've heard is it's already used in `import'
and `with' -- but that's perfectly refutable.

> Brief summary of reasons for disliking ":=":
>
> * Cryptic use of punctuation
>
> * Too much overlap in functionality with "="
>
> * Asymmetry between first and subsequent uses of the bound value
>
> * Makes expressions cluttered and hard to read to my eyes
>

--

Regards,
Ivan

[Ivan Pozdeev via Python-Dev]

I've got an idea about this.

The fact is, assignments don't make much sense in an arbitrary part of a
comprehension:
`for' variables are assigned every iteration, so when the result is
returned, only the final value will be seen.
(And if you need a value every iteration, just go the explicit way and
add it to the returned tuple.)

Contrary to that, the "feeder" expression is only evaluated once at the
start -- there, assignments do make sense.
Effectively, it's equivalent to an additional line:

seq = range(calculate_b() as bottom, calculate_t() as top)
results = [calculate_r(bottom,r,top) for r in seq]

So, I suggest to evaluate the "feeder" expression in a local scope but
expressions that are evaluated every iteration in a private scope.

--
Regards,
Ivan

[Baptiste Carvello]

Not giving a vote, as I'm just a lurker, but:

Le 25/06/2018 à 01:30, Greg Ewing a écrit :
>
> Actually, I'm closer to -1 on (a) as well. I don't like := as a
> way of getting assignment in an expression. The only thing I would
> give a non-negative rating is some form of "where" or "given".

This resonates with me for a yet different reason: expressing the
feature with a new operator makes it feel very important and
fundamental, so that beginners would feel compelled to learn it early,
and old-timers tend to have a strong gut reaction to it. Using merely a
keyword makes it less prominent.

Baptiste

[Guido van Rossum]

[This is my one reply in this thread today. I am trying to limit the amount of time I spend to avoid another overheated escalation.]

On Mon, Jun 25, 2018 at 4:44 AM Nick Coghlan <ncog...@gmail.com> wrote:

Right, the proposed blunt solution to "Should I use 'NAME = EXPR' or
'NAME := EXPR'?" bothers me a bit, but it's the implementation
implications of parent local scoping that I fear will create a
semantic tar pit we can't get out of later.

Others have remarked this too, but it really bother me that you are focusing so much on the implementation of parent local scoping rather than on the "intuitive" behavior which is super easy to explain -- especially to someone who isn't all that familiar (or interested) with the implicit scope created for the loop control variable(s). According to Steven (who noticed that this is barely mentioned in most tutorials about comprehensions) that is most people, however very few of them read python-dev.

It's not that much work for the compiler, since it just needs to do a little bit of (new) static analysis and then it can generate the bytecode to manipulate closure(s). The runtime proper doesn't need any new implementation effort. The fact that sometimes a closure must be introduced where no explicit initialization exists is irrelevant to the runtime -- this only affects the static analysis, at runtime it's no different than if the explicit initialization was inside `if 0`.

Unfortunately, I think the key rationale for (b) is that if you
*don't* do something along those lines, then there's a different
strange scoping discrepancy that arises between the non-comprehension
forms of container displays and the comprehension forms:

    (NAME := EXPR,) # Binds a local
    tuple(NAME := EXPR for __ in range(1)) # Doesn't bind a local

[...]

Those scoping inconsistencies aren't *new*, but provoking them
currently involves either class scopes, or messing about with
locals().

In what sense are they not new? This syntax doesn't exist yet.

 

The one virtue that choosing this particular set of discrepancies has
is that the explanation for why they happen is the same as the
explanation for how the iteration variable gets hidden from the
containing scope: because "(EXPR for ....)" et al create an implicitly
nested scope, and that nested scope behaves the same way as an
explicitly nested scope as far as name binding and name resolution is
concerned.

Yeah, but most people don't think much about that explanation.

You left out another discrepancy, which is more likely to hit people in the face: according to your doctrine, := used in the "outermost iterable" would create a local in the containing scope, since that's where the outermost iterable is evaluated. So in this example

    a = [x := i+1 for i in range(y := 2)]

the scope of x would be the implicit function (i.e. it wouldn't leak) while the scope of y would be the same as that of a. (And there's an even more cryptic example, where the same name is assigned in both places.)

This is another detail of comprehensions that I assume tutorials (rightly, IMO) gloss over because it's so rarely relevant. But it would make the explanation of how := works in comprehensions more cumbersome: you'd have to draw attention to the outermost iterable, otherwise "inline assignment in comprehensions has the same scope as the comprehension's loop control variable(s)" would lead one to believe that y's scope above would also be that of the implicit function.

 

Parent local scoping tries to mitigate the surface inconsistency by
changing how write semantics are defined for implicitly nested scopes,
but that comes at the cost of making those semantics inconsistent with
explicitly nested scopes and with the read semantics of implicitly
nested scopes.

Nobody thinks about write semantics though -- it's simply not the right abstraction to use here, you've introduced it because that's how *you* think about this.

 

The early iterations of PEP 572 tried to duck this whole realm of
potential semantic inconsistencies by introducing sublocal scoping
instead, such that the scoping for assignment expression targets would
be unusual, but they'd be consistently unusual regardless of where
they appeared, and their quirks would clearly be the result of how
assignment expressions were defined, rather than only showing up in
how they interacted with other scoping design decisions made years
ago.

There was also another variant in some iteration or PEP 572, after sublocal scopes were already eliminated -- a change to comprehensions that would evaluate the innermost iterable in the implicit function. This would make the explanation of inline assignment in comprehensions consistent again (they were always local to the comprehension in that iteration of the PEP), at the cost of a backward incompatibility that was ultimately withdrawn.

[Terry Reedy]

On 6/24/2018 7:25 PM, Guido van Rossum wrote:
> I'd wager that the people who might be most horrified about it

the (b) scoping rule change

> would be people who feel strongly that the change to the
> comprehension scope rules in Python 3 is a big improvement,

I might not be one of those 'most horrified' by (b), but I increasingly
don't like it, and I was at best -0 on the comprehension scope change.
To me, iteration variable assignment in the current scope is a
non-problem. So to me the change was mostly useless churn. Little
benefit, little harm. And not worth fighting when others saw a benefit.

However, having made the change to nested scopes, I think we should
stick with them. Or repeal them. (I believe there is another way to
isolate iteration names -- see below). To me, (b) amounts to half
repealing the nested scope change, making comprehensions half-fowl,
half-fish chimeras.

> and who are familiar with the difference in implementation
> of comprehensions (though not generator expressions) in Python 2 vs. 3.

That I pretty much am, I think. In Python 2, comprehensions (the fish)
were, at least in effect, expanded in-line to a normal for loop.
Generator expressions (the fowls) were different. They were, and still
are, expanded into a temporary generator function whose return value is
dropped back into the original namespace. Python 3 turned
comprehensions (with 2 news varieties thereof) into fowls also,
temporary functions whose return value is dropped back in the original
namespace. The result is that a list comprehension is equivalent to
list(generator_ expression), even though, for efficiency, it is not
implemented that way. (To me, this unification is more a benefit than
name hiding.)

(b) proposes to add extra hidden code in and around the temporary
function to partly undo the isolation. list comprehensions would no
longer be equivalent to list(generator_expression), unless
generator_expressions got the same treatment, in which case they would
no longer be equivalent to calling the obvious generator function.
Breaking either equivalence might break someone's code.
---

How loop variables might be isolated without a nested scope: After a
comprehension is parsed, so that names become strings, rename the loop
variables to something otherwise illegal. For instance, i could become
'<i>', just as lambda becomes '<lambda>' as the name of the resulting
function. Expand the comprehension as in Python 2, except for deleting
the loop names along with the temporary result name.

Assignment expressions within a comprehension would become assignment
expressions within the for loop expansion and would automatically add or
replace values in the namespace containing the comprehension. In other
words, I am suggesting that if we want name expressions in
comprehensions to act as they would in Python 2, then we should consider
reverting to an altered version of the Python 2 expansion.
---

In any case, I think (b) should be a separate PEP linked to a PEP for
(a). The decision for (a) could be reject (making (b) moot), accept
with (b), or accept unconditionally (but still consider (b)).

--
Terry Jan Reedy

[Chris Angelico]

On Tue, Jun 26, 2018 at 5:37 AM, Terry Reedy <tjr...@udel.edu> wrote:
> How loop variables might be isolated without a nested scope: After a
> comprehension is parsed, so that names become strings, rename the loop
> variables to something otherwise illegal. For instance, i could become
> '<i>', just as lambda becomes '<lambda>' as the name of the resulting
> function. Expand the comprehension as in Python 2, except for deleting the
> loop names along with the temporary result name.
>
> Assignment expressions within a comprehension would become assignment
> expressions within the for loop expansion and would automatically add or
> replace values in the namespace containing the comprehension. In other
> words, I am suggesting that if we want name expressions in comprehensions to
> act as they would in Python 2, then we should consider reverting to an
> altered version of the Python 2 expansion.

So..... sublocal scopes, like in the earliest versions of PEP 572?

The wheel turns round and round, and the same spokes come up.

ChrisA

[Terry Reedy]

On 6/25/2018 8:25 AM, Paul Moore wrote:
> On 25 June 2018 at 12:44, Nick Coghlan <ncog...@gmail.com> wrote:
>> Unfortunately, I think the key rationale for (b) is that if you
>> *don't* do something along those lines, then there's a different
>> strange scoping discrepancy that arises between the non-comprehension
>> forms of container displays and the comprehension forms:
>
> I've been mostly ignoring this proposal for a while now, so I'm going
> to respond here in the context of someone with a bit of an idea of the
> underlying complexities, but otherwise coming at it as a new proposal.
>
>>
>> (NAME := EXPR,) # Binds a local
>> tuple(NAME := EXPR for __ in range(1)) # Doesn't bind a local

Of course not, in local scopes where is it not executed. But it would,
in the nested function where the assignment *is* executed. Ditto for
all of the following.

>> [NAME := EXPR] # Binds a local
>> [NAME := EXPR for __ in range(1)] # Doesn't bind a local
>> list(NAME := EXPR for __ in range(1)) # Doesn't bind a local
>>
>> {NAME := EXPR} # Binds a local
>> {NAME := EXPR for __ in range(1)} # Doesn't bind a local
>> set(NAME := EXPR for __ in range(1)) # Doesn't bind a local
>>
>> {NAME := EXPR : EXPR2} # Binds a local
>> {NAME := EXPR : EXPR2 for __ in range(1)} # Doesn't bind a local
>> set((NAME := EXPR, EXPR2) for __ in range(1)) # Doesn't bind a local
>
> None of those "discrepancies" bother me in the slightest,

Me neither. I pretty much agree with the rest of what Paul said. If we
don't want comprehensions to execute in a nested scope, then we should
not create one. See my response to Guido for a possible alternative.

--
Terry Jan Reedy

[Steve Holden]

On Mon, Jun 25, 2018 at 8:37 PM, Terry Reedy <tjr...@udel.edu> wrote:

On 6/24/2018 7:25 PM, Guido van Rossum wrote:

I'd wager that the people who might be most horrified about it

the (b) scoping rule change

would be people who feel strongly that the change to the
comprehension scope rules in Python 3 is a big improvement,

I might not be one of those 'most horrified' by (b), but I increasingly don't like it, and I was at best -0 on the comprehension scope change. To me, iteration variable assignment in the current scope is a non-problem.  So to me the change was mostly useless churn.  Little benefit, little harm.  And not worth fighting when others saw a benefit.

However, having made the change to nested scopes, I think we should stick with them.  Or repeal them.  (I believe there is another way to isolate iteration names -- see  below).  To me, (b) amounts to half repealing the nested scope change, making comprehensions half-fowl, half-fish chimeras.

​[...]​

 

--
Terry Jan Reedy

​I'd like to ask: how many readers of ​

 

​this email have ever deliberately taken advantage of the limited Python 3 scope in comprehensions and generator expressions to use what would otherwise be a conflicting local variable name?​

I appreciate that the scope limitation can sidestep accidental naming errors, which is a good thing.

Unfortunately, unless we anticipate Python 4 (or whatever) also making for loops have an implicit scope, I am left wondering whether it's not too large a price to pay. After all, special cases aren't special enough to break the rules, and unless the language is headed towards implicit scope for all uses of "for" one could argue that the scope limitation is a special case too far. It certainly threatens to be yet another confusion for learners, and while that isn't the only consideration, it should be given due weight.

[Eric Fahlgren]

On Mon, Jun 25, 2018 at 2:16 PM Steve Holden <st...@holdenweb.com> wrote:

​I'd like to ask: how many readers of ​

 

​this email have ever deliberately taken advantage of the limited Python 3 scope in comprehensions and generator expressions to use what would otherwise be a conflicting local variable name?​

​No, never, but the opposite has bitten me in production code (as I related several months back, a class-level variable was being used on the lhs of a comprehension and that failed when it was run in Py3).

The caveat is that our code base is Py2+Py3, so we have the mindset that comprehension variables always leak.​

[David Mertz]

On Mon, Jun 25, 2018 at 5:14 PM Steve Holden <st...@holdenweb.com> wrote:

I'd like to ask: how many readers of ​

 

​this email have ever deliberately taken advantage of the limited Python 3 scope in comprehensions and generator expressions to use what would otherwise be a conflicting local variable name?​

I have never once *deliberately* utilized the Python 3 local scoping in comprehensions.  There were a few times in Python 2 where I made an error of overwriting a surrounding name by using it in a comprehension, and probably Python 3 has saved me from that a handful of times.

Where I ever made such an error, it was with names like 'x' and 'i' and 'n'.  They are useful for quick use, but "more important" variables always get more distinctive names anyway.  Had the Python 2 behavior remained, I would have been very little inconvenienced; and I suppose comprehensions would have been slightly less "magic" (but less functional-programming).

 

I appreciate that the scope limitation can sidestep accidental naming errors, which is a good thing.

Unfortunately, unless we anticipate Python 4 (or whatever) also making for loops have an implicit scope, I am left wondering whether it's not too large a price to pay. After all, special cases aren't special enough to break the rules, and unless the language is headed towards implicit scope for all uses of "for" one could argue that the scope limitation is a special case too far. It certainly threatens to be yet another confusion for learners, and while that isn't the only consideration, it should be given due weight.

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx

--

Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.

[Greg Ewing]

Ivan Pozdeev via Python-Dev wrote:
> "as" was suggested even before is became a keyword in `with'. ( if
> (re.match(regex,line) as m) is not None: <do smth> )

That's not equivalent where/given, though, since it still
has the asymmetry problem.

--
Greg

[Greg Ewing]

Terry Reedy wrote:
> How loop variables might be isolated without a nested scope: After a
> comprehension is parsed, so that names become strings, rename the loop
> variables to something otherwise illegal.

This doesn't change the situation conceptually, though, since
the question arises of why not do the same mangling for names
assigned within the comprehension.

A decision still needs to be made about whether we *want*
semantics that leak some things but not others.

--
Greg

[Greg Ewing]

Chris Angelico wrote:

> The wheel turns round and round, and the same spokes come up.

A discussion long past, and a discussion yet to come.

There are no beginnings or endings in the Wheel of Python...

--
Greg

[Ivan Pozdeev via Python-Dev]

On 26.06.2018 0:13, Steve Holden wrote:

On Mon, Jun 25, 2018 at 8:37 PM, Terry Reedy <tjr...@udel.edu> wrote:

On 6/24/2018 7:25 PM, Guido van Rossum wrote:

I'd wager that the people who might be most horrified about it

the (b) scoping rule change

would be people who feel strongly that the change to the
comprehension scope rules in Python 3 is a big improvement,

I might not be one of those 'most horrified' by (b), but I increasingly don't like it, and I was at best -0 on the comprehension scope change. To me, iteration variable assignment in the current scope is a non-problem.  So to me the change was mostly useless churn.  Little benefit, little harm.  And not worth fighting when others saw a benefit.

However, having made the change to nested scopes, I think we should stick with them.  Or repeal them.  (I believe there is another way to isolate iteration names -- see  below).  To me, (b) amounts to half repealing the nested scope change, making comprehensions half-fowl, half-fish chimeras.

​[...]​

 

--
Terry Jan Reedy

​I'd like to ask: how many readers of ​

 

​this email have ever deliberately taken advantage of the limited Python 3 scope in comprehensions and generator expressions to use what would otherwise be a conflicting local variable name?​

I did:

for l in (l.rstrip() for l in f):

The provisional unstripped line variable is totally unneeded in the following code.

I appreciate that the scope limitation can sidestep accidental naming errors, which is a good thing.

Unfortunately, unless we anticipate Python 4 (or whatever) also making for loops have an implicit scope, I am left wondering whether it's not too large a price to pay. After all, special cases aren't special enough to break the rules, and unless the language is headed towards implicit scope for all uses of "for" one could argue that the scope limitation is a special case too far. It certainly threatens to be yet another confusion for learners, and while that isn't the only consideration, it should be given due weight.

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

-- 
Regards,
Ivan

[Ivan Pozdeev via Python-Dev]

On 26.06.2018 1:58, Greg Ewing wrote:
> Chris Angelico wrote:
>
>> The wheel turns round and round, and the same spokes come up.
>

Unless there's a repository of prior discussion no-one can be bothered
to gather scraps from around the Net.
Wikis solve this by all the discussion being in one place, and even they
struggle is there were multiple.

> A discussion long past, and a discussion yet to come.
>
> There are no beginnings or endings in the Wheel of Python...
>

--

Regards,
Ivan

[Steven D'Aprano]

On Mon, Jun 25, 2018 at 11:30:24AM +1200, Greg Ewing wrote:
> Guido van Rossum wrote:

> >Greg seem to be +0 or better for (a)
>

> Actually, I'm closer to -1 on (a) as well. I don't like := as a
> way of getting assignment in an expression. The only thing I would
> give a non-negative rating is some form of "where" or "given".
>

> Brief summary of reasons for disliking ":=":
>
> * Cryptic use of punctuation

":=" is the second most common syntax used for assignment in common
programming languages, not just Pascal. Even modern languages like Go
use it. If that's "cryptic", what word would you use to describe
@decorator syntax? *wink*

Honestly Greg, can you put your hand on your heart and swear that if you
came across "name := expression" in source code you wouldn't be able to
hazard a guess as the meaning of the := operator?

> * Too much overlap in functionality with "="

If you are willing to consider a non-negative rating under the
"given/willing"spelling, presumably the "overlap in functionality" isn't
that important. (Otherwise it would be an argument against the feature
*regardless of spelling*.)

So why should it be an argument against the := spelling?

> * Asymmetry between first and subsequent uses of the bound value

I don't know what this means.

> * Makes expressions cluttered and hard to read to my eyes

And Nick's more verbose "given" proposal makes expressions less
cluttered?

result = process(first=(spam := ham or eggs), second=spam*5)

result = process(first=(spam given spam = ham or eggs), second=spam*5)


The := spelling has three syntactic elements: the target name, the :=
operator itself, and the expression being assigned.

The syntax you are willing to consider has five elements: an arbitrarily
complex return expression, the keyword "given", the target name, the =
operator, and the expression being assigned.

It isn't rational to say that adding extra complexity and more syntactic
elements *reduces* clutter. At the minimum, Nick's syntax requires:

- an extra keyword ("given" or "where")

- a repetitive, redundant, repeated use of the target name

just to save one : character. That adds, not subtracts, clutter.

Aside from the asymmetry issue (which I don't understand) it seems that
most of your arguments against := apply equally, or even more strongly,
to the "expr given name = expr" version.

I know matters of taste are deeply subjective, but we ought to
distinguish between *subjective* and *irrational* reasons for disliking
proposed features, and try to resist the irrational ones:

"We should change the spelling of set.add to set.append, as
that will remove the troublesome double-letter, and reduce
typing."

*wink*



--
Steve

[Guido van Rossum]

[This is my one response today]

On Mon, Jun 25, 2018 at 12:40 PM Terry Reedy <tjr...@udel.edu> wrote:

On 6/24/2018 7:25 PM, Guido van Rossum wrote:
> I'd wager that the people who might be most horrified about it

the (b) scoping rule change

> would be people who feel strongly that the change to the
> comprehension scope rules in Python 3 is a big improvement,

I might not be one of those 'most horrified' by (b), but I increasingly
don't like it, and I was at best -0 on the comprehension scope change.
To me, iteration variable assignment in the current scope is a
non-problem.  So to me the change was mostly useless churn.  Little
benefit, little harm.  And not worth fighting when others saw a benefit.

Fair enough, and by itself this might not have been enough reason to make the change. But see below.

 

However, having made the change to nested scopes, I think we should
stick with them.  Or repeal them.  (I believe there is another way to
isolate iteration names -- see  below).  To me, (b) amounts to half
repealing the nested scope change, making comprehensions half-fowl,
half-fish chimeras.

That depends on how you see it -- to me (b) just means that there's an implicit nonlocal[1] to make the assignment have the (desirable) side-effect.

The key thing to consider here is whether that side-effect is in fact desirable. For me, the side-effect of the comprehension's loop control variable was never desirable -- it was just an implementation detail leaking out. (And that's different from leaking a regular for-loop's control variable -- since we have 'break' (and 'else') there are some legitimate use cases. But comprehensions try to be expressions, and here the side effect is at best useless and at worst a nasty surprise.)

 

> and who are familiar with the difference in implementation
> of comprehensions (though not generator expressions) in Python 2 vs. 3.

That I pretty much am, I think.  In Python 2, comprehensions (the fish)
were, at least in effect, expanded in-line to a normal for loop.
Generator expressions (the fowls) were different.  They were, and still
are, expanded into a temporary generator function whose return value is
dropped back into the original namespace.  Python 3 turned
comprehensions (with 2 news varieties thereof) into fowls also,
temporary functions whose return value is dropped back in the original
namespace.  The result is that a list comprehension is equivalent to
list(generator_ expression), even though, for efficiency, it is not
implemented that way.  (To me, this unification is more a benefit than
name hiding.)

Right, and this consistency convinced me that the change was worth it. I just really like to be able to say "[... for ...]" is equivalent to "list(... for ...)", and similar for set and dict.

 

(b) proposes to add extra hidden code in and around the temporary
function to partly undo the isolation.

But it just adds a nonlocal declaration. There's always some hidden code ('def' and 'return' at the very least).

 

list comprehensions would no
longer be equivalent to list(generator_expression), unless
generator_expressions got the same treatment, in which case they would
no longer be equivalent to calling the obvious generator function.
Breaking either equivalence might break someone's code.

Ah, there's the rub! I should probably apologize for not clarifying my terminology more. In the context of PEP 572, when I say "comprehensions" I include generators! PEP 572 states this explicitly (https://github.com/python/peps/blame/master/pep-0572.rst#L201-L202).

Certainly PEP 572 intends to add that implicit nonlocal to both comprehensions and generator expressions. (I just got really tired of writing that phrase over and over, and at some point I forgot that this is only a parenthetical remark added in the PEP's latest revision, and not conventional terminology -- alas. :-)

Part (b) of PEP 572 does several things of things to *retain* consistency:

- The target of := lives in the same scope regardless of whether it occurs in a comprehension, a generator expression, or just in some other expression.

- When it occurs in a comprehension or generator expression, the scope is the same regardless of whether it occurs in the "outermost iterable" or not.

If we didn't have (b) the target would live in the comprehension/genexpr scope if it occurred in a comprehension/genexp but outside its "outermost iterable", and in the surrounding scope otherwise.

 

---

How loop variables might be isolated without a nested scope: After a
comprehension is parsed, so that names become strings, rename the loop
variables to something otherwise illegal.  For instance, i could become
'<i>', just as lambda becomes '<lambda>' as the name of the resulting
function.  Expand the comprehension as in Python 2, except for deleting
the loop names along with the temporary result name.

Assignment expressions within a comprehension would become assignment
expressions within the for loop expansion and would automatically add or
replace values in the namespace containing the comprehension.  In other
words, I am suggesting that if we want name expressions in
comprehensions to act as they would in Python 2, then we should consider
reverting to an altered version of the Python 2 expansion.

Possibly this is based on a misunderstanding of my use of "comprehensions". Also, since your trick can only be used for list/set/dict comprehensions, but not for generator expressions (at least I assume you don't want it there) it would actually *reduce* consistency between list/set/dict comprehensions and generator expressions.

 

---

In any case, I think (b) should be a separate PEP linked to a PEP for
(a).  The decision for (a) could be reject (making (b) moot), accept
with (b), or accept unconditionally (but still consider (b)).

For me personally, (b) makes the PEP more consistent, so I'm not in favor of breaking up the PEP. But we can certainly break up the discussion -- that's why I started using the labels (a) and (b).

 

----------

[1] Sometimes it's an implicit global instead of an implicit nonlocal -- when there's already a global for the same variable in the target scope.

[Greg Ewing]

Steven D'Aprano wrote:
> ":=" is the second most common syntax used for assignment in common
> programming languages,

Yes, but it represents an *ordinary* assigment in those
languages. The special way that's being proposed to use
it in Python is not obvious.

--
Greg

[Terry Reedy]

On 6/26/2018 10:36 PM, Guido van Rossum wrote:
> [This is my one response today]

Thank you for clearly presenting how you see 'comprehension', 'generator
expression' and by implication 'equivalent code'. The latter can either
be a definition or an explanation. The difference is subtle but real,
and, I believe, the essence of the disagreement over iteration
variables. If the code equivalent to a comprehension is its definition,
like a macro expansion, then survival of the iteration variable is to be
expected. If the equivalent code is an explanation of the *result* of
evaluating a *self-contained expression*, then leakage is easily seen a
wart, just as leakage of temporaries from any other expression would be.

My interpretation of what you say below is that you always wanted, for
instance, [i*i for i in iterable] == [j*j for j in iterable] to be true,
and saw the leakage making this not quite true as a wart. In other
words, within comprehensions (including generator expressions)
iterations names should be regarded as dummy placeholders and not part
of the value.

If this is correct, the list comprehension syntax could have been
[\0 * \0 for \0 in iterable]
with \1, \2, ... used as needed. (I am using the regex back-reference
notation in a way similar to the use of str.format forward reference
notation.)

I will stop here for now, as it is 1:30 am for me.

Terry

> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)

[Steven D'Aprano]

On Tue, Jun 26, 2018 at 05:42:43AM +1000, Chris Angelico wrote:

> So..... sublocal scopes, like in the earliest versions of PEP 572?
>
> The wheel turns round and round, and the same spokes come up.

It isn't as if comprehensions (and generator expressions) run in a
proper separate scope. It is more half-and-half, sometimes it is
seperate, sometimes it isn't:


py> def show_subscope():
... a, b = 1, 2
... print("Comprehension scope, Part A")
... print(next(locals() for x in (1,)))
... print("Comprehension scope, Part B")
... print(next(obj for obj in (locals(),)))
...
py> show_subscope()
Comprehension scope, Part A
{'x': 1, '.0': <tuple_iterator object at 0xb799cf8c>}
Comprehension scope, Part B
{'b': 2, 'a': 1}


Comprehensions already run partly in the surrounding scope.

I tried to take a survey of people on the Python-List mailing list, so
see what their expectations of comprehension scope was. Disappointingly,
not many people responded, but those who did, invariably think in terms
of comprehensions running inside their enclosing scope, like any other
expression:

https://mail.python.org/pipermail/python-list/2018-June/734838.html

(Please excuse the doubled-up posts, some misconfigured news server is
periodically sending duplicate posts.)

(Oh and ignore my comment about Python 2 -- I was thinking of
something else.)

Given the code shown:

def test():
a = 1
b = 2
result = [value for key, value in locals().items()]
return result


nobody suggested that the result ought to be the empty list, which is
what you should get if the comprehension ran in its own scope. Instead,
they all expected some variation of [1, 2], which is what you would get
if the comprehension ran in the enclosing scope.

A decade or more since generator expressions started running in their
own half-local-half-sublocal scope, people still think of scoping in
terms of LEGB and don't think of comprehensions as running in their own
scope *except* to the very limited degree that sometimes they are either
surprised or pleased that "the loop variable doesn't leak".

For example:

http://nbviewer.jupyter.org/github/rasbt/python_reference/blob/master/tutorials/scope_resolution_legb_rule.ipynb

doesn't mention comprehensions until the very end, almost in passing,
and doesn't describe them as a separate scope at all. Rather, they are
described as using closures "to prevent the for-loop variable to cut
[sic] into the global namespace."

This doesn't mention comprehension subscope at all:

https://www.python-course.eu/python3_global_vs_local_variables.php

Even the official documentation doesn't explicitly state that
comprehensions are a separate scope:

https://docs.python.org/3/reference/executionmodel.html#resolution-of-names

rather leaving it to an after thought, to mention in passing almost as
if it were an implementation-dependent accident, that comprehensions
cannot see variables defined in any surrounding class scope.

Aside from the loop variable (which PEP 572 will not change!) I see no
evidence that the average non-core developer Python programmer considers
comprehensions as a separate scope, or wants them to be a separate
scope. Regardless of comprehensions being implicitly wrapped in a
function or not, the average developer doesn't want the loop variable to
"leak", and that's as far as their consideration has needed to go until
now. But when pressed to explicitly consider the scope inside a
comprehension, the evidence I have seen is that they consider it the
same as the local scope surrounding it.

Which is not wrong, as can be seen from the example above.

Unlike the loop variable, I don't believe that assignment-expression
bindings quote-unquote "leaking" from comprehensions will come as a
surprise. On the contrary -- given that Nick et al have gone to great
lengths to ensure that as a first approximation, comprehensions are
equivalent to a simple for-loop running in the current scope:

result = [expr for a in seq]

# is almost the same as
result = []
for a in seq:
result.append(expr)

I expect that people will be surprised if explicit, non-loop
variable assignments *don't* occur in the current scope.

If all that takes to implement is something like an implicit "nonlocal",
that's hardly worse than the implicit functions already used.


--
Steve

[Paul Moore]

But test() returns [1, 2]. So does that say (as you claim above) that
"the comprehension ran in the enclosing scope"? Doesn't it just say
that the outermost iterable runs in the enclosing scope?

So everybody expected the actual behaviour? (Disclaimer: in my
response, I said that I had no clear expectation, which I stand by -
locals() exposes implementation details that I don't normally feel
that I need to know - but certainly the majority of respondents
expected 1 and 2 to appear).

On the other hand,

>>> def test2():
... a = 1
... b = 2
... result = [locals().items() for v in 'a']
... return result
...
>>> test2()
[dict_items([('v', 'a'), ('.0', <str_iterator object at 0x0000015AA0BDE8D0>)])]

and I bet no-one would have expected that if you'd posed that question
(I certainly wouldn't). Although some might have said [('v', 'a')]. I
suspect some would have expected a and b to appear there too, but
that's just a guess...

So yes, it's likely that people would have found the current behaviour
unexpected in respect of locals(). But I imagine most people only care
about the effective results when referencing variables, and

>>> def test3():
... a = 1
... b = 2
... result = [a for v in (1,)]
... return result
...
>>> test3()
[1]

i.e., thanks to scope nesting, you can still reference locals from the
enclosing scope.

The problem is that := allows you to *change* values in a scope, and
at that point you need to know *which* scope. So to that extent, the
locals() question is important. However, I still suspect that most
people would answer that they would like := to assign values *as if*
they were in the enclosing scope, which is not really something that I
think people would express in answer to a question about locals().
This can be achieved with an implicit "nonlocal" (and some extra
shenanigans if the enclosing scope has a nonlocal or global
declaration itself). Which, AIUI, is what the current proposal tries
to do.

IMO, the big question over the current PEP 572 proposal is whether it
goes too far in the direction of "do what I mean". Superficially, the
semantics are pretty clearly "what people would expect", and indeed
that's been the whole focus recently to capture and satisfy *expected*
behaviour. But there are edge cases (there always are when you work
from messy real-world requirements rather than nice clean mathematical
definitions ;-)) and the question is essentially whether any of those
are bad enough to be an issue.

I'm starting to feel that they aren't, and I'm moving towards a
cautious +0 (or even +1) on the proposal.
Paul

[Steven D'Aprano]

On Tue, Jun 26, 2018 at 10:54:12AM +1200, Greg Ewing wrote:

> A decision still needs to be made about whether we *want*
> semantics that leak some things but not others.

My sense (or bias, if you prefer) is that the answer to that depends
on how you word the question. If you talk about "leaking", or
give examples with trivial 1-character names that look all too
easy to accidentally clobber, people will say "No":

# Given this:
x = 999
[(x := i)*x for i in (1, 2)]

# should print(x) afterwards result in 4?


but if you show a useful example that doesn't look like an accident
waiting to happen, but a deliberate feature:

# Given this:
previous = 0
[previous + (previous := i) for i in (1, 2, 3)]

# what value would you expect previous to have
# at the completion of the loop?

they'll be more receptive to the idea. (If they're not opposed to
assignment expressions at all.)

Avoiding leading questions is *hard*, and I believe that in general
people don't know what they want until they've got it. I say that from
considering all the times I've made a radical about face, features which
I was *sure* would be awful actually turned out to be not awful at all
-- augmented assignment, for instance.


--
Steve

[Chris Angelico]

On Wed, Jun 27, 2018 at 5:30 PM, Paul Moore <p.f....@gmail.com> wrote:
> But test() returns [1, 2]. So does that say (as you claim above) that
> "the comprehension ran in the enclosing scope"? Doesn't it just say
> that the outermost iterable runs in the enclosing scope?

Yes - because the *outermost iterable* runs in the enclosing scope.
But suppose you worded it like this:

def test():
a = 1
b = 2

vars = {key: locals()[key] for key in locals()}
return vars

What would your intuition say? Should this be equivalent to dict(locals()) ?

ChrisA

[Paul Moore]

On 27 June 2018 at 08:52, Chris Angelico <ros...@gmail.com> wrote:
> On Wed, Jun 27, 2018 at 5:30 PM, Paul Moore <p.f....@gmail.com> wrote:
>> But test() returns [1, 2]. So does that say (as you claim above) that
>> "the comprehension ran in the enclosing scope"? Doesn't it just say
>> that the outermost iterable runs in the enclosing scope?
>
> Yes - because the *outermost iterable* runs in the enclosing scope.
> But suppose you worded it like this:
>
> def test():
> a = 1
> b = 2
> vars = {key: locals()[key] for key in locals()}
> return vars
>
> What would your intuition say? Should this be equivalent to dict(locals()) ?

As I said on python-list, my intuition doesn't apply to locals() - I
simply have no idea what I'd "expect" from that code, other than a
request to go back and write it more clearly :-)

*After* staring at it for a while and trying to interpret it base on
the detailed knowledge I've gained from this thread, I'd say it does
nothing remotely useful, and if you want dict(locals()) you should
write it. (No, test() is not equivalent, because the two instances of
locals() refer to different scopes, but I can't imagine why I'd ever
need to know that outside of solving artificial puzzles like this).

Paul

[Steven D'Aprano]

On Wed, Jun 27, 2018 at 08:30:00AM +0100, Paul Moore wrote:
> On 27 June 2018 at 07:54, Steven D'Aprano <st...@pearwood.info> wrote:
> > Comprehensions already run partly in the surrounding scope.

[...]

> > Given the code shown:
> >
> > def test():
> > a = 1
> > b = 2
> > result = [value for key, value in locals().items()]
> > return result

[...]

> But test() returns [1, 2]. So does that say (as you claim above) that
> "the comprehension ran in the enclosing scope"? Doesn't it just say
> that the outermost iterable runs in the enclosing scope?

I think I was careful enough to only say that this was the same result
you would get *if* the comprehension ran in the outer scope. Not to
specifically say it *did* run in the outer scope. (If I slipped up
anywhere, sorry.)

I did say that the comprehension runs *partly* in the surrounding scope,
and the example shows that the local namespace in the "... in iterable"
part is not the same as the (sub)local namespace in the "expr for x in
..." part.

*Parts* of the comprehension run in the surrounding scope, and parts of
it run in an implicit sublocal scope inside a hidden function, giving us
a quite complicated semantics for "comprehension scope":

[expression for a in first_sequence for b in second ... ]
|------sublocal-----|----local-----|------sublocal------|

Try fitting *that* in the LEGB (+class) acronym :-)


This becomes quite relevant once we include assignment expressions. To
make the point that this is not specific to := but applies equally to
Nick's "given" syntax as well, I'm going to use his syntax:

result = [a for a in (x given x = expensive_function(), x+1, 2*x, x**3)]

Here, the assignment to x runs in the local part. I can simulate that
right now, using locals, but only outside of a function due to CPython's
namespace optimization inside functions. (For simplicity, I'm just going
to replace the call to "expensive_function" with just a constant.)


py> del x
py> [a for a in (locals().__setitem__('x', 2) or x, x+1, 2*x, x**3)]
[2, 3, 4, 8]
py> x
2


This confirms that the first sequence part of the comprehension runs in
the surrounding local scope.

So far so good. What if we move that assignment one level deep?
Unfortunately, I can no longer use locals for this simulation, due to a
peculiarity of the CPython function implementation. But replacing the
call to locals() with globals() does the trick:

del x
# simulate [b*a for b in (1,) for a in (x given x = 2, x+1, 2*x, x**3)]
[b*a for b in (1,) for a in (globals().__setitem__('x', 2) or x, x+1, 2*x, x**3)]


That also works. But the problem comes if the user tries to assign to x
in both the local and a sublocal section:

# no simulation here, sorry
[b*a for b in (x given x = 2, x**2) for a in (x given x = x + 1, x**3)]

That looks like it should work. You're assigning to the same x in two
parts of the same expression. Where's the problem?

But given the "implicit function" implementation of comprehensions, I
expect that this ought to raise an UnboundLocalError. The local scope
part is okay:

# needs a fixed-width font for best results
[b*a for b in (x given x = 2, x**2) for a in (x given x = x + 1, x**3)]
..............|-----local part----|.....|--------sublocal part--------|

but the sublocal part defines x as a sublocal variable, shadowing the
surrounding local x, then tries to get a value for that sublocal x
before it is defined.

If we had assignment expressions before generator expressions and
comprehensions, I don't think this would have been the behaviour we
desired.

(We might, I guess, accept it as an acceptable cost of the implicit
function implementation. But we surely wouldn't argue for this
complicated scoping behaviour as a good thing in and of itself.)

In any case, we can work around this (at some cost of clarity and
unobviousness) by changing the name of the variable. Not a big burden
when the variable is a single character x:

[b*a for b in (x given x = 2, x**2) for a in (y given y = x + 1, y**3)]

but if x is a more descriptive name, that becomes more annoying.
Nevermind, it is a way around this.

Or we could Just Make It Work by treating the entire comprehension as
the same scope for assignment expressions. (I stress, not for the loop
variable.) Instead of having to remember which bits of the comprehension
run in which scope, we have a conceptually much simpler rule:

- comprehensions are expressions, and assignments inside them
bind to the enclosing local scope, just like other expressions:

- except for the loop variables, which are intentionally
encapsulated inside the comprehension and don't "leak".

The *implementation details* of how that works are not conceptually
relevant. We may or may not want to advertise the fact that
comprehensions use an implicit hidden function to do the encapsulation,
and implicit hidden nonlocal to undo the effects of that hidden
function. Or whatever implementation we happen to use.

> So everybody expected the actual behaviour?

More or less, if we ignore a few misapprehensions about how locals
works.

> On the other hand,
>
> >>> def test2():
> ... a = 1
> ... b = 2
> ... result = [locals().items() for v in 'a']
> ... return result
> ...
> >>> test2()
> [dict_items([('v', 'a'), ('.0', <str_iterator object at 0x0000015AA0BDE8D0>)])]
>
> and I bet no-one would have expected that if you'd posed that question

I suspect not. To be honest, I didn't even think of asking that question
until after I had asked the first.

> The problem is that := allows you to *change* values in a scope, and
> at that point you need to know *which* scope. So to that extent, the
> locals() question is important. However, I still suspect that most
> people would answer that they would like := to assign values *as if*
> they were in the enclosing scope,

That is my belief as well. But that was intentionally not the question I
was asking. I was interested in seeing whether people thought of
comprehensions as a separate scope, or part of the enclosing scope.


--
Steve

[Steven D'Aprano]

On Wed, Jun 27, 2018 at 05:52:16PM +1000, Chris Angelico wrote:

> def test():
> a = 1
> b = 2
> vars = {key: locals()[key] for key in locals()}
> return vars
>
> What would your intuition say? Should this be equivalent to dict(locals()) ?

That example is so elegant it makes me want to cry.

And not just because you shadowed the vars() builtin *wink*



--
Steve

[Ivan Pozdeev via Python-Dev]

On 26.06.2018 1:34, Greg Ewing wrote:
> Ivan Pozdeev via Python-Dev wrote:
>> "as" was suggested even before is became a keyword in `with'. ( if
>> (re.match(regex,line) as m) is not None: <do smth> )
>
> That's not equivalent where/given, though, since it still
> has the asymmetry problem.
>

What do you mean by "asymmetry"? The fact that the first time around,
it's the expression and after that, the variable?

If that, it's not a "problem". The whole idea is to assign the result of
a subexpression to something.
If you force any assignments to be outside, it won't be a subexpression
anymore, but effectively a separate statement -- if not syntactically,
then visually at least -- both of which are the things the feature's
purpose is to avoid.

If you seek to force assignments outside, you should've rather suggested
inline code blocks e.g. like anonymous methods in C# ( { a=foo();
b=bar(); return a+b;} ).

Using this assigned result elsewhere in the same expression (akin to
regex backreferences) is not a part of the basic idea actually.
It depends on the evaluation order (and whether something is evaluated
at all), so I doubt it should even be allowed -- but even if it is, it's
a side benefit at best.

--
Regards,
Ivan

[Chris Angelico]

On Wed, Jun 27, 2018 at 7:19 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> On Wed, Jun 27, 2018 at 05:52:16PM +1000, Chris Angelico wrote:
>
>> def test():
>> a = 1
>> b = 2
>> vars = {key: locals()[key] for key in locals()}
>> return vars
>>
>> What would your intuition say? Should this be equivalent to dict(locals()) ?
>
> That example is so elegant it makes me want to cry.
>
> And not just because you shadowed the vars() builtin *wink*

It gets funnier with nested loops. Or scarier. I've lost the ability
to distinguish those two.

def test():
spam = 1
ham = 2
vars = [key1+key2 for key1 in locals() for key2 in locals()]
return vars

Wanna guess what that's gonna return?

ChrisA

[Eric V. Smith]

On 6/27/2018 7:08 AM, Chris Angelico wrote:
> It gets funnier with nested loops. Or scarier. I've lost the ability
> to distinguish those two.
>
> def test():
> spam = 1
> ham = 2
> vars = [key1+key2 for key1 in locals() for key2 in locals()]
> return vars
>
> Wanna guess what that's gonna return?

I'm not singling out Chris here, but these discussions would be easier
to follow and more illuminating if the answers to such puzzles were
presented when they're posed.

Eric

[Nick Coghlan]

On 26 June 2018 at 02:27, Guido van Rossum <gu...@python.org> wrote:
> [This is my one reply in this thread today. I am trying to limit the amount
> of time I spend to avoid another overheated escalation.]

Aye, I'm trying to do the same, and deliberately spending some
evenings entirely offline is helping with that :)

> On Mon, Jun 25, 2018 at 4:44 AM Nick Coghlan <ncog...@gmail.com> wrote:
>>
>> Right, the proposed blunt solution to "Should I use 'NAME = EXPR' or
>> 'NAME := EXPR'?" bothers me a bit, but it's the implementation
>> implications of parent local scoping that I fear will create a
>> semantic tar pit we can't get out of later.
>
> Others have remarked this too, but it really bother me that you are focusing
> so much on the implementation of parent local scoping rather than on the
> "intuitive" behavior which is super easy to explain -- especially to someone
> who isn't all that familiar (or interested) with the implicit scope created
> for the loop control variable(s). According to Steven (who noticed that this
> is barely mentioned in most tutorials about comprehensions) that is most
> people, however very few of them read python-dev.
>
> It's not that much work for the compiler, since it just needs to do a little
> bit of (new) static analysis and then it can generate the bytecode to
> manipulate closure(s). The runtime proper doesn't need any new
> implementation effort. The fact that sometimes a closure must be introduced
> where no explicit initialization exists is irrelevant to the runtime -- this
> only affects the static analysis, at runtime it's no different than if the
> explicit initialization was inside `if 0`.

One of the things I prize about Python's current code generator is how
many of the constructs can be formulated as simple content-and-context
independent boilerplate removal, which is why parent local scoping (as
currently defined in PEP 572) bothers me: rather than being a new
primitive in its own right, the PEP instead makes the notion of "an
assignment expression in a comprehension or generator expression" a
construct that can't readily decomposed into lower level building
blocks the way that both assignment expressions on their own and
comprehensions and generator expressions on their own can be. Instead,
completely new language semantics arise from the interaction between
two otherwise independent features.

Even changes as complicated as PEP 343's with statement, PEP 380's
yield from, and PEP 492's native coroutines all include examples of
how they could be written *without* the benefit of the new syntax.

By contrast, PEP 572's parent local scoping can't currently be defined
that way. Instead, to explain how the code generator is going to be
expected to handle comprehensions, you have to take the current
comprehension semantics and add two new loops to link up the bound
names correctly::

[item := x for x in items]

becomes:

# Each bound name gets declared as local in the parent scope
if 0:
for item in (): pass
def _list_comp(_outermost_iter):
# Each bound name gets declared as:
# - nonlocal if outer scope is a function scope
# - global item if outer scope is a module scope
# - an error, otherwise
_result = []
for x in _outermost_iter:
_result.append(x)
return _result

_expr_result = _list_comp(items)

This is why my objections would be reduced significantly if the PEP
explicitly admitted that it was defining a new kind of scoping
semantics, and actually made those semantics available as an explicit
"parentlocal NAME" declaration (behind a "from __future__ import
parent_locals" guard), such that the translation of the above example
to an explicitly nested scope could just be the visually
straightforward::

def _list_comp(_outermost_iter):
parentlocal item
_result = []
for x in _outermost_iter:
item = x
_result.append(x)
return _result

_expr_result = _list_comp(items)

That splits up the learning process for anyone trying to really
understand how this particular aspect of Python's code generation
works into two distinct pieces:

- "assignment expressions inside comprehensions and generator
expressions use parent local scoping"
- "parent local scoping works <the way that PEP 572 defines it>"

If the PEP did that, we could likely even make parent locals work
sensibly for classes by saying that "parent local" for a method
definition in a class body refers to the closure namespace where we
already stash __class__ references for the benefit of zero-arg super
(this would also be a far more robust way of defining private class
variables than name mangling is able to offer).

Having parent locals available as a language level concept (rather
than solely as an interaction between assignment expressions and
implicitly nested scopes) also gets us to a point where
context-independent code thunks that work both at module level and
inside another function can be built as nested functions which declare
all their working variables as parentlocal (you still need to define
the thunks inline in the scope you want them to affect, since this
isn't dynamic scoping, but when describing the code, you don't need to
say "as a module level function define it this way, as a nested
function define it that way").

An explicit "parentlocal NAME" concept at the PEP 572 layer would also
change the nature of the draft "given" proposal from competing with
PEP 572, to instead being a follow-up proposal that focused on
providing control of target name declarations in lambda expressions,
comprehensions, and generator expressions such that:

- (lambda arg: value := arg given parentlocal value) # Exports "value"
to parent scope
- any(x for x in items given parentlocal x) # Exports "x" to parent scope
- [y for x in data if (y := f(x)) given y] # *Avoids* exporting "y" to
parent scope

With parent local scoping in the mix the proposed "given" syntax could
also dispense with initialiser and type hinting support entirely and
instead only allow:

- "... given NAME" (always local, no matter the default scoping)
- "... given parentlocal NAME" (always parent local, declaring if necessary)
- "... given nonlocal NAME" (always nonlocal, error if not declared in
outer scope)
- "... given global NAME" (always global, no matter how nested the
current scope is)
- "... given (TARGET1, TARGET2, ...)" (declaring multiple assignment targets)

If you want an initialiser or a type hint, then you'd use parentlocal
semantics. If you want to keep names local (e.g. to avoid exporting
them as part of a module's public API) then you can do that, too.

>> Unfortunately, I think the key rationale for (b) is that if you
>> *don't* do something along those lines, then there's a different
>> strange scoping discrepancy that arises between the non-comprehension
>> forms of container displays and the comprehension forms:
>>

>> (NAME := EXPR,) # Binds a local
>> tuple(NAME := EXPR for __ in range(1)) # Doesn't bind a local

>> [...]
>> Those scoping inconsistencies aren't *new*, but provoking them
>> currently involves either class scopes, or messing about with
>> locals().
>
> In what sense are they not new? This syntax doesn't exist yet.

The simplest way to illustrate the scope distinction today is with
"len(locals())":

>>> [len(locals()) for i in range(1)]
[2]
>>> [len(locals())]
[7]

But essentially nobody ever does that, so the distinction doesn't
currently matter.

By contrast, where assignment expressions bind their targets matters a
*lot*, so PEP 572 makes the existing scoping oddities a lot more
significant.

> You left out another discrepancy, which is more likely to hit people in the
> face: according to your doctrine, := used in the "outermost iterable" would
> create a local in the containing scope, since that's where the outermost
> iterable is evaluated. So in this example
>
> a = [x := i+1 for i in range(y := 2)]
>
> the scope of x would be the implicit function (i.e. it wouldn't leak) while
> the scope of y would be the same as that of a. (And there's an even more
> cryptic example, where the same name is assigned in both places.)

Yeah, the fact it deals with this problem nicely is one aspect of the
parent local scoping that I find genuinely attractive.

>> Parent local scoping tries to mitigate the surface inconsistency by
>> changing how write semantics are defined for implicitly nested scopes,
>> but that comes at the cost of making those semantics inconsistent with
>> explicitly nested scopes and with the read semantics of implicitly
>> nested scopes.
>
>
> Nobody thinks about write semantics though -- it's simply not the right
> abstraction to use here, you've introduced it because that's how *you* think
> about this.

The truth of the last part of that paragraph means that the only way
for the first part of it to be true is to decide that my way of
thinking is *so* unusual that nobody else in the 10 years that Python
3 has worked the way it does now has used the language reference, the
source code, the disassembler, or the debugger to formulate a similar
mental model of how they expect comprehensions and generator
expressions to behave.

I'll grant that I may be unusual in thinking about comprehensions and
generator expressions the way I do, and I definitely accept that most
folks simply don't think about the subtleties of how they handle
scopes in the first place, but I *don't* accept the assertion that I'm
unique in thinking about them that way. There are simply too many edge
cases in their current runtime behaviour where the "Aha!" moment at
the end of a debugging effort is going to be the realisation that
they're implemented as an implicitly nested scope, and we've had a
decade of Python 3 use where folks prone towards writing overly clever
comprehensions have been in a position to independently make that
discovery.

>> The early iterations of PEP 572 tried to duck this whole realm of
>> potential semantic inconsistencies by introducing sublocal scoping

> There was also another variant in some iteration or PEP 572, after sublocal
> scopes were already eliminated -- a change to comprehensions that would
> evaluate the innermost iterable in the implicit function. This would make
> the explanation of inline assignment in comprehensions consistent again
> (they were always local to the comprehension in that iteration of the PEP),
> at the cost of a backward incompatibility that was ultimately withdrawn.

Yeah, the current "given" draft has an open question around the idea
of having the presence of a "given" clause pull the outermost iterable
evaluation inside the nested scope. It still doesn't really solve the
problem, though, so I think I'd actually consider
PEP-572-with-explicit-parent-local-scoping-support the version of
assignment expressions that most cleanly handles the interaction with
comprehension scopes without making that interaction rely on opaque
magic (instead, it would be relying on an implicit target scope
declaration, the same as any other name binding - the only unusual
aspect is that the implicit declaration would be "parentlocal NAME"
rather than the more typical local variable declaration).

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Ivan Pozdeev via Python-Dev]

"A shorthand to list()/dict()/set()" is actually how I thought of comprehensions when I studied them. And I was actually using list() in my code for some time before I learned of their existence.

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

-- 
Regards,
Ivan

[Greg Ewing]

Ivan Pozdeev via Python-Dev wrote:
> Using this assigned result elsewhere in the same expression (akin to
> regex backreferences) is not a part of the basic idea actually.

If that's true, then the proposal has mutated into something
that has *no* overlap whatsoever with the use case that started
this whole discussion, which was about binding a temporary
variable in a comprehension, for use *within* the comprehension.

> It depends on the evaluation order (and whether something is evaluated
> at all),

Which to my mind is yet another reason not to like ":=".

--
Greg

[Greg Ewing]

Nick Coghlan wrote:
> actually made those semantics available as an explicit

> "parentlocal NAME" declaration ...:

>
> def _list_comp(_outermost_iter):
> parentlocal item
> _result = []
> for x in _outermost_iter:
> item = x
> _result.append(x)
> return _result
>
> _expr_result = _list_comp(items)

I'm not sure that's possible. If I understand correctly,
part of the definition of "parent local" is that "parent"
refers to the nearest enclosing *non-comprehension* scope,
to give the expected result for nested comprehensions.
If that's so, then it's impossible to fully decouple its
definition from comprehensions.

--
Greg

[Antoine Pitrou]

Why is this discussion talking about comprehensions at all?
Is there a decent use case for using assignments in comprehensions (as
opposed to language lawyering or deliberate obfuscation)?

Regards

Antoine.

On Thu, 28 Jun 2018 01:25:14 +1200
Greg Ewing <greg....@canterbury.ac.nz> wrote:
> Ivan Pozdeev via Python-Dev wrote:
> > Using this assigned result elsewhere in the same expression (akin to
> > regex backreferences) is not a part of the basic idea actually.
>
> If that's true, then the proposal has mutated into something
> that has *no* overlap whatsoever with the use case that started
> this whole discussion, which was about binding a temporary
> variable in a comprehension, for use *within* the comprehension.
>
> > It depends on the evaluation order (and whether something is evaluated
> > at all),
>
> Which to my mind is yet another reason not to like ":=".
>

[Steven D'Aprano]

On Wed, Jun 27, 2018 at 08:00:20AM -0400, Eric V. Smith wrote:
> On 6/27/2018 7:08 AM, Chris Angelico wrote:
> >It gets funnier with nested loops. Or scarier. I've lost the ability
> >to distinguish those two.
> >
> >def test():
> > spam = 1
> > ham = 2
> > vars = [key1+key2 for key1 in locals() for key2 in locals()]
> > return vars
> >
> >Wanna guess what that's gonna return?
>
> I'm not singling out Chris here, but these discussions would be easier
> to follow and more illuminating if the answers to such puzzles were
> presented when they're posed.

You can just copy and paste the function into the interactive
interpreter and run it :-)

But where's the fun in that? The point of the exercise is to learn first
hand just how complicated it is to try to predict the *current* scope
behaviour of comprehensions. Without the ability to perform assignment
inside them, aside from the loop variable, we've managed to avoid
thinking too much about this until now.

It also demonstrates the unrealisticness of treating comprehensions as a
separate scope -- they're hybrid scope, with parts of the comprehension
running in the surrounding local scope, and parts running in an sublocal
scope.

Earlier in this thread, Nick tried to justify the idea that
comprehensions run in their own scope, no matter how people think of
them -- but that's an over-simplification, as Chris' example above
shows. Parts of the comprehension do in fact behave exactly as the naive
model would suggest (even if Nick is right that other parts don't).

As complicated and hairy as the above example is, (1) it is a pretty
weird thing to do, so most of us will almost never need to consider it;
and (2) backwards compatibility requires that we live with it now (at
least unless we introduce a __future__ import).

If we can't simplify the scope of comprehensions, we can at least
simplify the parts that actually matters. What matters are the loop
variables (already guaranteed to be sublocal and not "leak" out of the
comprehension) and the behaviour of assignment expressions (open to
discussion).

Broadly speaking, there are two positions we can take:

1. Let the current implementation of comprehensions as an implicit
hidden function drive the functionality; that means we duplicate the
hairiness of the locals() behaviour seen above, although it won't be
obvious at first glance.

What this means in practice is that assignments will go to different
scopes depending on *where* they are in the comprehension:

[ expr for x in iter1 for y in iter2 if cond ...]
[ BBBBBB for x in AAAAAA for y in BBBBBB if BBBBBB ...]

Assignments in the section marked "AAAAAA" will be in the local scope;
assignments in the BBBBBB sections will be in the sublocal scope. That's
not too bad, up to the point you try to assign to the same name in
AAAAAA and BBBBBB. And then you are likely to get confusing hard to
debug UnboundLocalErrors.


2. Or we can keep the current behaviour for locals and the loop
variables, but we can keep assignment expressions simple by ensuring
they always bind to the enclosing scope. Compared to the complexity of
the above, we have the relatively straight forward:

[ AAAAAA for x in AAAAAA for y in AAAAAA if AAAAAA ...]

The loop variables continue to be hidden away in the invisible, implicit
comprehension function, where they can't leak out, while explicit
assignments to variables (using := or given or however it is spelled)
will always go into the surrounding local scope, like they do in every
other expression.

Does it matter that the implementation of this requires an implicit
nonlocal declaration for each assignment? No more than it matters that
comprehensions themselves require an implicit function.

And what we get out of this is simpler semantics at the Python level:

- Unless previous declared global, assignment expressions always bind to
the current scope, even if they're inside a comprehension;

- and we don't have to deal with the oddity that different bits of a
comprehension run in different scopes (unless we go out of our way to
use locals()); merely using assignment expressions will just work
consistently and simply, and loop variables will still be confined to
the comprehension as they are now.


--
Steve

[Steven D'Aprano]

On Wed, Jun 27, 2018 at 03:41:23PM +0200, Antoine Pitrou wrote:
>
> Why is this discussion talking about comprehensions at all?
> Is there a decent use case for using assignments in comprehensions (as
> opposed to language lawyering or deliberate obfuscation)?

Yes. The *very first* motivating example for this proposal came from a
comprehension.

I think it is both unfortunate and inevitable that the discussion bogged
down in comprehension-hell. Unfortunate because I don't think that the
most compelling use-cases involve comprehensions at all. But inevitable
because *comprehensions are the hard case*, thanks to the (justifiable!)
decision to implement them as implicit hidden functions.

In my opinion, the really two BIG wins for assignment expressions are
while loops and cascades of if... blocks. Tim Peters has also given a
couple of good examples of mathematical code that would benefit strongly
from this feature.

Going back a few months now, they were the examples that tipped me over
from the opinion

"Oh, just re-write the comprehension as a loop"

to the opinion

"You know, I think this feature actually is useful... and
as a bonus, you can keep using the comprehension"

But that requires that we get the comprehension scoping right. Not just
leave it as an unspecified implementation detail.



--
Steve

[Ivan Pozdeev via Python-Dev]

On 27.06.2018 16:25, Greg Ewing wrote:
> Ivan Pozdeev via Python-Dev wrote:
>> Using this assigned result elsewhere in the same expression (akin to
>> regex backreferences) is not a part of the basic idea actually.
>
> If that's true, then the proposal has mutated into something
> that has *no* overlap whatsoever with the use case that started
> this whole discussion,

I don't know what and where "started" it (AFAIK the idea has been around
for years) but for me, the primary use case for an assignment expression
is to be able to "catch" a value into a variable in places where I can't
put an assignment statement in, like the infamous `if re.match() is not
None'.

> which was about binding a temporary
> variable in a comprehension, for use *within* the comprehension.

Then I can't understand all the current fuss about scoping.
AFAICS, it's already like I described in
https://mail.python.org/pipermail/python-dev/2018-June/154067.html :
the outermost iterable is evaluated in the local scope while others in
the internal one:

In [13]: [(l,i) for l in list(locals())[:5] for i in locals()]
Out[13]:
[('__name__', 'l'),
 ('__name__', '.0'),
 ('__builtin__', 'l'),
 ('__builtin__', '.0'),
 ('__builtin__', 'i'),
 ('__builtins__', 'l'),
 ('__builtins__', '.0'),
 ('__builtins__', 'i'),
 ('_ih', 'l'),
 ('_ih', '.0'),
 ('_ih', 'i'),
 ('_oh', 'l'),
 ('_oh', '.0'),
 ('_oh', 'i')]

(note that `i' is bound after the first evaluation of internal
`locals()' btw, as to be expected)

If the "temporary variables" are for use inside the comprehension only,
the assignment expression needs to bind in the current scope like the
regular assignment statement, no changes are needed!

>> It depends on the evaluation order (and whether something is
>> evaluated at all),
>
> Which to my mind is yet another reason not to like ":=".
>

--
Regards,
Ivan

[Nick Coghlan]

On 27 June 2018 at 23:38, Greg Ewing <greg....@canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>>
>> actually made those semantics available as an explicit
>> "parentlocal NAME" declaration ...:
>>
>> def _list_comp(_outermost_iter):
>> parentlocal item
>> _result = []
>> for x in _outermost_iter:
>> item = x
>> _result.append(x)
>> return _result
>>
>> _expr_result = _list_comp(items)
>
>
> I'm not sure that's possible. If I understand correctly,
> part of the definition of "parent local" is that "parent"
> refers to the nearest enclosing *non-comprehension* scope,
> to give the expected result for nested comprehensions.
> If that's so, then it's impossible to fully decouple its
> definition from comprehensions.

I'm OK with a target scope declaration construct having
lexical-scope-dependent behaviour - exactly what "nonlocal NAME" will
do depends on both the nature of the current scope, and on which names
are declared as local in which outer scopes, and that's also
implicitly the case for all name lookups.

However, PEP 572 in its current form takes the position "parent local
scoping is sufficiently useful to make it a required pre-requisite for
adding assignment expressions, but not useful enough to expose as a
new scope declaration primitive", and I've come to the view that it
really is the "A+B=MAGIC!" aspect of the current proposal that bothers
me, whereas "A+B implies C for <pragmatic reasons>" doesn't bother me
any more than the implicit non-local references introduced as part of
the original lexical scoping changes bother me.

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Ivan Pozdeev via Python-Dev]

This isn't as messy as you make it sound if you remember that the
outermost iterable is evaluated only once at the start and all the
others -- each iteration.
Anyone using comprehensions has to know this fact.
The very readable syntax also makes it rather straightforward (though
admittedly requiring some hand-tracing) to figure out what is evaluated
after what.

>
> 2. Or we can keep the current behaviour for locals and the loop
> variables, but we can keep assignment expressions simple by ensuring
> they always bind to the enclosing scope. Compared to the complexity of
> the above, we have the relatively straight forward:
>
> [ AAAAAA for x in AAAAAA for y in AAAAAA if AAAAAA ...]
>
> The loop variables continue to be hidden away in the invisible, implicit
> comprehension function, where they can't leak out, while explicit
> assignments to variables (using := or given or however it is spelled)
> will always go into the surrounding local scope, like they do in every
> other expression.
>
> Does it matter that the implementation of this requires an implicit
> nonlocal declaration for each assignment? No more than it matters that
> comprehensions themselves require an implicit function.
>
> And what we get out of this is simpler semantics at the Python level:
>
> - Unless previous declared global, assignment expressions always bind to
> the current scope, even if they're inside a comprehension;
>
> - and we don't have to deal with the oddity that different bits of a
> comprehension run in different scopes (unless we go out of our way to
> use locals()); merely using assignment expressions will just work
> consistently and simply, and loop variables will still be confined to
> the comprehension as they are now.
>
>

--

Regards,
Ivan

[Paul Moore]

On 27 June 2018 at 15:39, Nick Coghlan <ncog...@gmail.com> wrote:

> However, PEP 572 in its current form takes the position "parent local
> scoping is sufficiently useful to make it a required pre-requisite for
> adding assignment expressions, but not useful enough to expose as a
> new scope declaration primitive", and I've come to the view that it
> really is the "A+B=MAGIC!" aspect of the current proposal that bothers
> me, whereas "A+B implies C for <pragmatic reasons>" doesn't bother me
> any more than the implicit non-local references introduced as part of
> the original lexical scoping changes bother me.

From my reading, PEP 572 takes the position that "parent local
scoping" is what people expect from assignment expressions *in
comprehensions* and it's useful enough that there is no reason not to
make that the behaviour. The behaviour isn't generally useful enough
to be worth exposing as a primitive (it's not even useful enough for
the PEP to give it an explicit name!) so it's just a special case for
assignment expressions in comprehensions/generators.

That seems to me like a classic example of practicality beating purity.

Paul

[Tim Peters]

[Nick Coghlan]>

 actually made those semantics available as an explicit 

> "parentlocal NAME" declaration ...:
>
>     def _list_comp(_outermost_iter):
>         parentlocal item
>         _result = []
>         for x in _outermost_iter:
>             item = x
>             _result.append(x)
>         return _result
>
>     _expr_result = _list_comp(items)

[Greg Ewing]

I'm not sure that's possible. If I understand correctly,
part of the definition of "parent local" is that "parent"
refers to the nearest enclosing *non-comprehension* scope,
to give the expected result for nested comprehensions.
If that's so, then it's impossible to fully decouple its
definition from comprehensions.

 Nick's "parentlocal" does refer to the parent, but makes no distinction between synthesized and user-written functions.  If the parent has a matching parentlocal declaration for the same name then the original really refers to the grandparent - and so on.  Ultimately, it resolves to the closest enclosing scope in which the name is _not_ declared parentlocal.  In that scope, a "nonlocal" or "global" declaration settles it if one appears, else the name is local to that scope.

So a nested comprehension would declare its assignment expression targets as parentlocal in its synthesized function, and in all the containing synthesized functions generated for containing comprehensions.

This appears in some strained ;-) way "natural" only because there is no explicit way to declare something "local" in Python.  In just about any other language with closures and nested lexical scopes, comprehensions and generator expressions would have been implemented via nested functions that explicitly declared their "for" target names "local". and nothing else.  The only change needed then for PEP 572 (b) semantics would be to declare assignment expression target names local (if their scope wasn't already known) in the closest containing non-synthesized block.

None of which really matters.  The real question is which semantics are desired.

[Tim Peters]

[Nick Coghlan]

However, PEP 572 in its current form takes the position "parent local
scoping is sufficiently useful to make it a required pre-requisite for
adding assignment expressions, but not useful enough to expose as a
new scope declaration primitive",

Of course the PEP doesn't take that position at all:  it doesn't even contain the term "parent local scoping".  That's your term, which nobody else uses unless they're replying to you ;-)

What the PEP does say:

"""
an assignment expression occurring in a list, set or dict comprehension or in a generator expression (below collectively referred to as "comprehensions") binds the target in the containing scope, honoring a nonlocal or global declaration for the target in that scope, if one exists. For the purpose of this rule the containing scope of a nested comprehension is the scope that contains the outermost comprehension. A lambda counts as a containing scope.
"""

It's a small collection of plainly stated rules for specifying the intended semantics.  If you want to claim that this _is_ "useful enough to expose as a new scope declaration primitive", it's really on you to present use cases to justify that claim.  I'd present some for you, but I don't have any (I don't care that "by hand" conversion of nested comprehensions to workalike Python nested functions may require a bit of thought to establish the intended scope of assignment expression target names - all of which is easily doable without adding any new statements).

I don't _expect_ that other good use cases exist.  The gimmick's purpose is to make code that visually _appears_ to belong to a block act as if embedded assignments do occur in that block.  If there's an explicitly nested function, that fundamental motivation no longer applies.

[Eric V. Smith]

> On Jun 27, 2018, at 9:49 AM, Steven D'Aprano <st...@pearwood.info> wrote:
>
>> On Wed, Jun 27, 2018 at 08:00:20AM -0400, Eric V. Smith wrote:
>>> On 6/27/2018 7:08 AM, Chris Angelico wrote:
>>> It gets funnier with nested loops. Or scarier. I've lost the ability
>>> to distinguish those two.
>>>
>>> def test():
>>> spam = 1
>>> ham = 2
>>> vars = [key1+key2 for key1 in locals() for key2 in locals()]
>>> return vars
>>>
>>> Wanna guess what that's gonna return?
>>
>> I'm not singling out Chris here, but these discussions would be easier
>> to follow and more illuminating if the answers to such puzzles were
>> presented when they're posed.
>
> You can just copy and paste the function into the interactive
> interpreter and run it :-)

Not on my phone when I’m riding a bus, I can’t. I’m trying to more or less follow the discussion, but the “guess what this will do” aspect of the discussion makes it hard.

Eric

> Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com

[Guido van Rossum]

So IIUC you are okay with the behavior described by the PEP but you want an explicit language feature to specify it?

I don't particularly like adding a `parentlocal` statement to the language, because I don't think it'll be generally useful. (We don't have `goto` in the language even though it could be used in the formal specification of `if`, for example. :-)

But as a descriptive mechanism to make the PEP's spec clearer I'm fine with it. Let's call it `__parentlocal` for now. It would work a bit like `nonlocal` but also different, since in the normal case (when there's no matching `nonlocal` in the parent scope) it would make the target a local in that scope rather than trying to look for a definition of the target name in surrounding (non-class, non-global) scopes. Also if there's a matching `global` in the parent scope, `__parentlocal` itself changes its meaning to `global`. If you want to push a target through several level of target scopes you can do that by having a `__parentlocal` in each scope that it should push through (this is needed for nested comprehensions, see below).

Given that definition of `__parentlocal`, in first approximation the scoping rule proposed by PEP 572 would then be: In comprehensions (which in my use in the PEP 572 discussion includes generator expressions) the targets of inline assignments are automatically endowed with a `__parentlocal` declaration, except inside the "outermost iterable" (since that already runs in the parent scope).

There would have to be additional words when comprehensions themselves are nested (e.g. `[[a for a in range(i)] for i in range(10)]`) since the PEP's intention is that inline assignments anywhere there end up targeting the scope containing the outermost comprehension. But this can all be expressed by adding `__parentlocal` for various variables in various places (including in the "outermost iterable" of inner comprehensions).

I'd also like to keep the rule prohibiting use of the same name as a comprehension loop control variable and as an inline assignment target; this rule would also prohibit shenanigans with nested comprehensions (for any set of nested comprehensions, any name that's a loop control variable in any of them cannot be an inline assignment target in any of them). This would also apply to the "outermost iterable".

Does this help at all, or did I miss something?

--Guido

[Steven D'Aprano]

On Wed, Jun 27, 2018 at 05:52:16PM +0300, Ivan Pozdeev via Python-Dev wrote:

> >What this means in practice is that assignments will go to different
> >scopes depending on *where* they are in the comprehension:
> >
> > [ expr for x in iter1 for y in iter2 if cond ...]
> > [ BBBBBB for x in AAAAAA for y in BBBBBB if BBBBBB ...]
> >
> >Assignments in the section marked "AAAAAA" will be in the local scope;
> >assignments in the BBBBBB sections will be in the sublocal scope. That's
> >not too bad, up to the point you try to assign to the same name in
> >AAAAAA and BBBBBB. And then you are likely to get confusing hard to
> >debug UnboundLocalErrors.
>
> This isn't as messy as you make it sound if you remember that the
> outermost iterable is evaluated only once at the start and all the
> others -- each iteration.

The question isn't *how often* they are evaluated, or how many loops you
have, but *what scope* they are evaluated in. Even in a single loop
comprehension, parts of it are evaluated in the local scope and parts
are evaluated in an implicit sublocal scope.

The overlap between the two is the trap, if you try to assign to the
same variable in the loop header and then update it in the loop body.

Not to mention the inconsistency that some assignments are accessible
from the surrounding code:

[expr for a in (x := func(), ...) ]
print(x) # works

while the most useful ones, those in the body, will be locked up in an
implicit sublocal scope where they are unreachable from outside of the
comprehension:

[x := something ... for a in sequence ]
print(x) # fails


--
Steve

[Eric Fahlgren]

On Wed, Jun 27, 2018 at 9:27 AM Paul Moore <p.f....@gmail.com> wrote:

From my reading, PEP 572 takes the position that "parent local
scoping" is what people expect from assignment expressions *in
comprehensions* and it's useful enough that there is no reason not to
make that the behaviour. The behaviour isn't generally useful enough
to be worth exposing as a primitive (it's not even useful enough for
the PEP to give it an explicit name!) so it's just a special case for
assignment expressions in comprehensions/generators.

​So, my interpretation is that it will behave like this?

x = 2

y = [x := 3 for i in range(1)]

print(x)

3

def f():

    x = 4

    y = [x := 5 for i in range(1)]

    print(x)

f()

5

class C:

    x = 6

    y = [x := 7 for i in range(1)]

    print(x)

C()

6

print(x)

7​

[Greg Ewing]

Steven D'Aprano wrote:
> The *very first* motivating example for this proposal came from a
> comprehension.
>
> I think it is both unfortunate and inevitable that the discussion bogged
> down in comprehension-hell.

I think the unfortunateness started when we crossed over from
talking about binding a temporary name for use *within* a
comprehension or expression, to binding a name for use *outside*
the comprehension or expression where it's bound.

As long as it's for internal use, whether it's in a comprehension
or not isn't an issue.

> Tim Peters has also given a
> couple of good examples of mathematical code that would benefit strongly
> from this feature.
>
> Going back a few months now, they were the examples that tipped me over

Well, I remain profoundly unconvinced that writing comprehensions
with side effects is ever a good idea, and Tim's examples did
nothing to change that.

--
Greg

[Ivan Pozdeev via Python-Dev]

On 28.06.2018 1:42, Steven D'Aprano wrote:
> On Wed, Jun 27, 2018 at 05:52:16PM +0300, Ivan Pozdeev via Python-Dev wrote:
>
>>> What this means in practice is that assignments will go to different
>>> scopes depending on *where* they are in the comprehension:
>>>
>>> [ expr for x in iter1 for y in iter2 if cond ...]
>>> [ BBBBBB for x in AAAAAA for y in BBBBBB if BBBBBB ...]
>>>
>>> Assignments in the section marked "AAAAAA" will be in the local scope;
>>> assignments in the BBBBBB sections will be in the sublocal scope. That's
>>> not too bad, up to the point you try to assign to the same name in
>>> AAAAAA and BBBBBB. And then you are likely to get confusing hard to
>>> debug UnboundLocalErrors.
>> This isn't as messy as you make it sound if you remember that the
>> outermost iterable is evaluated only once at the start and all the
>> others -- each iteration.
> The question isn't *how often* they are evaluated, or how many loops you
> have, but *what scope* they are evaluated in. Even in a single loop
> comprehension, parts of it are evaluated in the local scope and parts
> are evaluated in an implicit sublocal scope.

All expressions inside the comprehension other than the initial iterable
have access to the loop variables generated by the previous parts. So
they are necessarily evaluated in the internal scope for that to be
possible.

Since this is too an essential semantics that one has to know to use the
construct sensibly, I kinda assumed you could make that connection...
E.g.:

[(x*y) for x in range(5) if x%2 for y in range(x,5) if not (x+y)%2]
   A              B          C              D               E

C and D have access to the current x; E and A to both x and y.

>
> The overlap between the two is the trap, if you try to assign to the
> same variable in the loop header and then update it in the loop body.
>
> Not to mention the inconsistency that some assignments are accessible
> from the surrounding code:
>
> [expr for a in (x := func(), ...) ]
> print(x) # works
>
> while the most useful ones, those in the body, will be locked up in an
> implicit sublocal scope where they are unreachable from outside of the
> comprehension:
>
> [x := something ... for a in sequence ]
> print(x) # fails
>
>

--
Regards,
Ivan

[Ivan Pozdeev via Python-Dev]

This means btw that users cannot rely on there being a single internal
scope, or a scope at all.
The public guarantee is only the access to the loop variables (and, with
the PEP, additional variables from assignments), of the current
iteration, generated by the previous parts.

[Greg Ewing]

Nick Coghlan wrote:
> I'm OK with a target scope declaration construct having
> lexical-scope-dependent behaviour - exactly what "nonlocal NAME" will
> do depends on both the nature of the current scope,

Yes, but my point is that having an explicit "parentlocal" scope
declaration doesn't help to make anything more orthogonal,
because there's no way it can have *exactly* the same effect
as a comprehension's implicit parent-local scoping.

In other words, taking a comprehension and manually expanding
it into a function with parentlocal declarations wouldn't
give you something exactly equivalent to the original.
If that's the purpose of having an explicit parentlocal,
then it fails at that purpose.

If that's *not* the purpose, then I'm not really sure what
the purpose is, because I can't think of a situation where
I'd choose to use parentlocal instead of nonlocal with an
explicit assignment in the outer scope. Except maybe for the
class-scope situation, which seems like an extremely obscure
reason to introduce a whole new scoping concept with its
own keyword.

--
Greg

[Ivan Pozdeev via Python-Dev]

The expressions in the comprehension just somehow automagically
determine which of the variables are internal and which are local. How
they do that is an implementation detail.
And the PEP doesn't need to (and probably shouldn't) make guarantees
here other than where the variables from expressions are promised to be
accessible.

[Greg Ewing]

Ivan Pozdeev via Python-Dev wrote:

> This isn't as messy as you make it sound if you remember that the
> outermost iterable is evaluated only once at the start and all the
> others -- each iteration.
> Anyone using comprehensions has to know this fact.

That fact alone doesn't imply anthing about the *scopes* in which
those iterators are evaluated, however.

Currently the only situation where the scoping makes a difference
is a generator expression that isn't immediately used, and you can
get a long way into your Python career without ever encountering
that case.

--
Greg

[Ivan Pozdeev via Python-Dev]

On 28.06.2018 2:31, Greg Ewing wrote:
> Steven D'Aprano wrote:
>> The *very first* motivating example for this proposal came from a
>> comprehension.
>>
>> I think it is both unfortunate and inevitable that the discussion bogged
>> down in comprehension-hell.
>
> I think the unfortunateness started when we crossed over from
> talking about binding a temporary name for use *within* a
> comprehension or expression, to binding a name for use *outside*
> the comprehension or expression where it's bound.
>

I've shown in <05f368c2-3cd2-d7e0...@mail.mipt.ru> (27
Jun 2018 17:07:24 +0300) that assignment expressions are fine in most
use cases without any changes to scoping whatsoever.

So, as Guido suggested in
<CAP7+vJ+xBAT4ZvAo4B7qSqxnnpj8jJ1VZ-Le7EwT8=n-UT...@mail.gmail.com> (26
Jun 2018 19:36:14 -0700), the scoping matter can be split into a
separate PEP and discussion.

> As long as it's for internal use, whether it's in a comprehension
> or not isn't an issue.
>
>> Tim Peters has also given a couple of good examples of mathematical
>> code that would benefit strongly from this feature.
>>
>> Going back a few months now, they were the examples that tipped me over
>
> Well, I remain profoundly unconvinced that writing comprehensions
> with side effects is ever a good idea, and Tim's examples did
> nothing to change that.
>

--

Regards,
Ivan