[Python-ideas] Fwd: unpacking generalisations for list comprehension

[Martti Kühne]

Hello list

I love the "new" unpacking generalisations as of pep448. And I found
myself using them rather regularly, both with lists and dict.
Today I somehow expected that [*foo for foo in bar] was equivalent to
itertools.chain(*[foo for foo in bar]), which it turned out to be a
SyntaxError.
The dict equivalent of the above might then be something along the
lines of {**v for v in dict_of_dicts.values()}. In case the values
(which be all dicts) are records with the same keys, one might go and
prepend the keys with their former keys using
{
**dict(
("{}_{}".format(k, k_sub), v_sub)
for k_sub, v_sub in v.items()
) for k, v in dict_of_dicts.items()
}
Was anyone able to follow me through this?

cheers!
mar77i
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[אלעזר‎]

I thought about it a lot recently. Specifically on your proposal, and in general. Unpacking expression can have a much more uniform treatment in the language, as an expression with special "bare tuple" type - like tuple, but "without the braces".

It also gives mental explanation for the conditional expression, where "a if cond" is an unpack expression whose value is "*[a]" if cond hold, and "*[]" otherwise. without context, this is an error. But with an else, "a if cond else b" becomes "*[] else b" which evaluates to b. The result is exactly like today, but gives the ability to build conditional elements in a list literal:

x = [foo(), bar() if cond, goo()]

y = [1, bar()?, 3] 

 
x is a list of 2 elements or three elements, depending of the truthness of cond.

y is a list of 2 elements or three elements, depending on whether bar() is None.

It also opens the gate for None-coercion operator (discussed recently), where "x?" is replaced with "*[x if x is None]". If operations on this expression are mapped into the elements, "x?.foo" becomes "*[x.foo if x is None]" which is "x.foo" if x is not None, and "*[]" otherwise.

It is similar to except-expression, but without actual explicit exception handling, and thus much more readable.

Elazar

[Nick Coghlan]

On 11 October 2016 at 23:50, אלעזר <ela...@gmail.com> wrote:
> I thought about it a lot recently. Specifically on your proposal, and in
> general. Unpacking expression can have a much more uniform treatment in the
> language, as an expression with special "bare tuple" type - like tuple, but
> "without the braces".

That's a recipe for much deeper confusion, as it would make "*expr"
and "*expr," semantically identical

>>> *range(3),
(0, 1, 2)

As things stand, the above makes tuple expansion the same as any other
expression: you need a comma to actually make it a tuple. If you allow
a bare "*" to imply the trailing comma, then it immediately becomes
confusing when you actually *do* have a comma present, as the "*" no
longer implies a new tuple, it gets absorbed into the surrounding one.
That's outright backwards incompatible with the status quo once you
take parentheses into account:

>>> (*range(3)),
(0, 1, 2)

Regards,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Sven R. Kunze]

Hi Martti,

On 11.10.2016 14:42, Martti Kühne wrote:
> Hello list
>
> I love the "new" unpacking generalisations as of pep448. And I found
> myself using them rather regularly, both with lists and dict.
> Today I somehow expected that [*foo for foo in bar] was equivalent to
> itertools.chain(*[foo for foo in bar]), which it turned out to be a
> SyntaxError.
> The dict equivalent of the above might then be something along the
> lines of {**v for v in dict_of_dicts.values()}. In case the values
> (which be all dicts) are records with the same keys, one might go and
> prepend the keys with their former keys using
> {
> **dict(
> ("{}_{}".format(k, k_sub), v_sub)
> for k_sub, v_sub in v.items()
> ) for k, v in dict_of_dicts.items()
> }
> Was anyone able to follow me through this?

Reading PEP448 it seems to me that it's already been considered:
https://www.python.org/dev/peps/pep-0448/#variations

The reason for not-inclusion were about concerns about acceptance
because of "strong concerns about readability" but also received "mild
support". I think your post strengthens the support given that you
"expected it to just work". This shows at least to me that the concerns
about readability/understandability are not justified much.

Personally, I find inclusion of */** expansion for comprehensions very
natural. It would again strengthen the meaning of */** for unpacking
which I am also in favor of.

Cheers,
Sven

[Nick Coghlan]

On 12 October 2016 at 23:58, Sven R. Kunze <srk...@mail.de> wrote:
> Reading PEP448 it seems to me that it's already been considered:
> https://www.python.org/dev/peps/pep-0448/#variations
>
> The reason for not-inclusion were about concerns about acceptance because of
> "strong concerns about readability" but also received "mild support". I
> think your post strengthens the support given that you "expected it to just
> work". This shows at least to me that the concerns about
> readability/understandability are not justified much.

Readability isn't about "Do some people guess the same semantics for
what it would mean?", as when there are only a few plausible
interpretations, all the possibilities are going to get a respectable
number of folks picking them as reasonable behaviour.

Instead, readability is about:

- Do people consistently guess the *same* interpretation?
- Is that interpretation consistent with other existing uses of the syntax?
- Is it more readily comprehensible than existing alternatives, or is
it brevity for brevity's sake?

This particular proposal fails on the first question (as too many
people would expect it to mean the same thing as either "[*expr, for
expr in iterable]" or "[*(expr for expr in iterable)]"), but it fails
on the other two grounds as well.

In most uses of *-unpacking it's adding entries to a comma-delimited
sequence, or consuming entries in a comma delimited sequence (the
commas are optional in some cases, but they're still part of the
relevant contexts). The expansions removed the special casing of
functions, and made these capabilities generally available to all
sequence definition operations.

Comprehensions and generator expressions, by contrast, dispense with
the comma delimited format entirely, and instead use a format inspired
by mathematical set builder notation (just modified to use keywords
and Python expressions rather than symbols and mathematical
expressions): https://en.wikipedia.org/wiki/Set-builder_notation#Sets_defined_by_a_predicate

However, set builder notation doesn't inherently include the notion of
flattening lists-of-lists. Instead, that's a *consumption* operation
that happens externally after the initial list-of-lists has been
built, and that's exactly how it's currently spelled in Python:
"itertools.chain.from_iterable(subiter for subiter in iterable)".

Regards,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Steven D'Aprano]

On Tue, Oct 11, 2016 at 02:42:54PM +0200, Martti Kühne wrote:
> Hello list
>
> I love the "new" unpacking generalisations as of pep448. And I found
> myself using them rather regularly, both with lists and dict.
> Today I somehow expected that [*foo for foo in bar] was equivalent to
> itertools.chain(*[foo for foo in bar]), which it turned out to be a
> SyntaxError.

To me, that's a very strange thing to expect. Why would you expect that
unpacking items in a list comprehension would magically lead to extra
items in the resulting list? I don't think that makes any sense.

Obviously we could program list comprehensions to act that way if we
wanted to, but that would not be consistent with the ordinary use of
list comprehensions. It would introduce a special case of magical
behaviour that people will have to memorise, because it doesn't follow
logically from the standard list comprehension design.

The fundamental design principle of list comps is that they are
equivalent to a for-loop with a single append per loop:

[expr for t in iterable]

is equivalent to:

result = []
for t in iterable:
result.append(expr)


If I had seen a list comprehension with an unpacked loop variable:

[*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]


I never in a million years would expect that running a list
comprehension over a three-item sequence would magically expand to six
items:

[1, 'a', 2, 'b', 3, 'c']


I would expect that using the unpacking operator would give some sort
of error, or *at best*, be a no-op and the result would be:

[(1, 'a'), (2, 'b'), (3, 'c')]


append() doesn't take multiple arguments, hence a error should be the
most obvious result. But if not an error, imagine the tuple unpacked to
two arguments 1 and 'a' (on the first iteration), then automatically
packed back into a tuple (1, 'a') just as you started with.

I think it is a clear, obvious and, most importantly, desirable property
of list comprehensions with a single loop that they cannot be longer
than the initial iterable that feeds them. They might be shorter, if you
use the form

[expr for t in iterable if condition]

but they cannot be longer.

So I'm afraid I cannot understand what reasoning lead you to
expect that unpacking would apply this way. Wishful thinking
perhaps?




--
Steve

[אלעזר‎]

Steve, you only need to allow multiple arguments to append(), then it makes perfect sense.

בתאריך יום ד׳, 12 באוק' 2016, 18:43, מאת Steven D'Aprano ‏<st...@pearwood.info>:

[Sven R. Kunze]

On 12.10.2016 17:41, Nick Coghlan wrote:
> This particular proposal fails on the first question (as too many
> people would expect it to mean the same thing as either "[*expr, for
> expr in iterable]" or "[*(expr for expr in iterable)]")

So, my reasoning would tell me: where have I seen * so far? *args and
**kwargs! [...] is just the list constructor. So, putting those two
pieces together is quite simple. I expect that Martti's reasoning was
similar.

Furthermore, your two "interpretations" would yield the very same result
as [expr for expr in iterable] which doesn't match with my experience
with Python so far; especially when it comes to special characters. They
must mean something. So, a simple "no-op" would not match my expectations.

> but it fails on the other two grounds as well.

Here I disagree with you. We use *args all the time, so we know what *
does. I don't understand why this should not work in between brackets [...].

Well, it works in between [...] sometimes but not always, to be precise.
And that's the problem, I guess.

> In most uses of *-unpacking it's adding entries to a comma-delimited
> sequence, or consuming entries in a comma delimited sequence (the
> commas are optional in some cases, but they're still part of the
> relevant contexts). The expansions removed the special casing of
> functions, and made these capabilities generally available to all
> sequence definition operations.

I don't know what you mean by comma-delimited sequence. There are no
commas. It's just a list of entries. * adds entries to this list. (At
least from my point of view.)

> Comprehensions ... [are] inspired by mathematical set builder notation.

Exactly. Inspired. I don't see any reason not to extend on this idea to
make it more useful.

> "itertools.chain.from_iterable(subiter for subiter in iterable)".

I have to admit that need to read that twice to get what it does. But
that might just be me.

Cheers,
Sven

[Stephen J. Turnbull]

אלעזר writes:

> Steve, you only need to allow multiple arguments to append(), then it makes
> perfect sense.

No, because that would be explicit. Here it's implicit and ambiguous.
Specifically, it requires guessing "operator associativity". That is
something people have different intuitions about.

> > On Tue, Oct 11, 2016 at 02:42:54PM +0200, Martti Kühne wrote:
> > > Hello list
> > >
> > > I love the "new" unpacking generalisations as of pep448. And I found
> > > myself using them rather regularly, both with lists and dict.
> > > Today I somehow expected that [*foo for foo in bar] was equivalent to
> > > itertools.chain(*[foo for foo in bar]), which it turned out to be a
> > > SyntaxError.

Which is what I myself would expect, same as *(1, 2) + 3 is a
SyntaxError. I could see Nick's interpretation that *foo in such a
context would actually mean (*foo,) (i.e., it casts iterables to
tuples). I would certainly want [i, j for i, j in [[1, 2], [3, 4]]]
to evaluate to [(1, 2), (3, 4)] (if it weren't a SyntaxError).

> > To me, that's a very strange thing to expect. Why would you expect that
> > unpacking items in a list comprehension would magically lead to extra
> > items in the resulting list? I don't think that makes any sense.

Well, that's what it does in display syntax for sequences. If you
think of a comprehension as a "macro" that expands to display syntax,
makes some sense. But as you and Nick point out, comprehensions are
real operations, not macros which implicitly construct displays, then
evaluate them to get the actual sequence.

> > Wishful thinking perhaps?

That was unnecessary. I know sometimes I fall into the trap of
thinking there really ought to be concise syntax for a "simple" idea,
and then making one up rather than looking it up.

[David Mertz]

I've followed this discussion some, and every example given so far completely mystifies me and I have no intuition about what they should mean.

[אלעזר‎]

What is the intuition behind [1, *x, 5]? The starred expression is replaced with a comma-separated sequence of its elements.

The trailing comma Nick referred to is there, with the rule that [1,, 5] is the same as [1, 5].

All the examples follow this intuition, IIUC.

Elazar

בתאריך יום ד׳, 12 באוק' 2016, 22:22, מאת David Mertz ‏<me...@gnosis.cx>:

[David Mertz]

On Wed, Oct 12, 2016 at 12:38 PM, אלעזר <ela...@gmail.com> wrote:

What is the intuition behind [1, *x, 5]? The starred expression is replaced with a comma-separated sequence of its elements.

I've never actually used the `[1, *x, 5]` form.  And therefore, of course, I've never taught it either (I teach Python for a living nowadays).  I think that syntax already perhaps goes too far, actually; but I can understand it relatively easily by analogy with:

     a, *b, c = range(10)

But the way I think about or explain either of those is "gather the extra items from the sequence." That works in both those contexts.  In contrast:

    >>> *b = range(10)

    SyntaxError: starred assignment target must be in a list or tuple

Since nothing was assigned to a non-unpacked variable, nothing is "extra items" in the same sense.  So failure feels right to me.  I understand that "convert an iterable to a list" is conceptually available for that line, but we already have `list(it)` around, so it would be redundant and slightly confusing.

What seems to be wanted with `[*foo for foo in bar]` is basically just `flatten(bar)`.  The latter feels like a better spelling, and the recipes in itertools docs give an implementation already (a one-liner).

We do have a possibility of writing this:

    >>>  [(*stuff,) for stuff in [range(-5,-1), range(5)]]

    [(-5, -4, -3, -2), (0, 1, 2, 3, 4)] 

That's not flattened, as it should not be.  But it is very confusing to have `[(*stuff) for stuff in ...]` behave differently than that.  It's much more natural—and much more explicit—to write:

    >>> [item for seq in [range(-5,-1), range(5)] for item in seq]

    [-5, -4, -3, -2, 0, 1, 2, 3, 4]

[Paul Moore]

On 12 October 2016 at 20:22, David Mertz <me...@gnosis.cx> wrote:
> I've followed this discussion some, and every example given so far
> completely mystifies me and I have no intuition about what they should mean.

Same here.

On 12 October 2016 at 20:38, אלעזר <ela...@gmail.com> wrote:
> What is the intuition behind [1, *x, 5]? The starred expression is replaced
> with a comma-separated sequence of its elements.
>
> The trailing comma Nick referred to is there, with the rule that [1,, 5] is
> the same as [1, 5].
>
> All the examples follow this intuition, IIUC.

But intuition is precisely that - it's not based on rules, but on
people's instinctive understanding. When evaluating whether something
is intuitive, the *only* thing that matters is what people tell you
they do or don't understand by a given construct. And in this case,
people have been expressing differing interpretations, and confusion.
That says "not intuitive" loud and clear to me.

And yes, I find [1, *x, 5] intuitive. And I can't tell you why I find
it OK, but I find {**x for x in d.items()} non-intuitive. But just
because I can't explain it doesn't mean it's not true, or you can
"change my mind" about how I feel.

Paul

[אלעזר‎]

On Wed, Oct 12, 2016 at 11:26 PM David Mertz <me...@gnosis.cx> wrote:

On Wed, Oct 12, 2016 at 12:38 PM, אלעזר <ela...@gmail.com> wrote:

What is the intuition behind [1, *x, 5]? The starred expression is replaced with a comma-separated sequence of its elements.

I've never actually used the `[1, *x, 5]` form.  And therefore, of course, I've never taught it either (I teach Python for a living nowadays).  I think that syntax already perhaps goes too far, actually; but I can understand it relatively easily by analogy with:

     a, *b, c = range(10)

It's not exactly "analogy" as such - it is the dual notion. Here you are using the "destructor" (functional terminology) but we are talking about "constructors". But nevermind.

 

But the way I think about or explain either of those is "gather the extra items from the sequence." That works in both those contexts.  In contrast:

    >>> *b = range(10)

    SyntaxError: starred assignment target must be in a list or tuple

Since nothing was assigned to a non-unpacked variable, nothing is "extra items" in the same sense.  So failure feels right to me.  I understand that "convert an iterable to a list" is conceptually available for that line, but we already have `list(it)` around, so it would be redundant and slightly confusing.

But that's not a uniform treatment. It might have good reasons from readability point of view, but it is an explicit exception for the rule. The desired behavior would be equivalent to

    b = tuple(range(10))

and yes, there are Two Ways To Do It. I would think it should have been prohibited by PEP-8 and not by the compiler. Oh well.

What seems to be wanted with `[*foo for foo in bar]` is basically just `flatten(bar)`.  The latter feels like a better spelling, and the recipes in itertools docs give an implementation already (a one-liner).

We do have a possibility of writing this:

    >>>  [(*stuff,) for stuff in [range(-5,-1), range(5)]]

    [(-5, -4, -3, -2), (0, 1, 2, 3, 4)] 

That's not flattened, as it should not be.  But it is very confusing to have `[(*stuff) for stuff in ...]` behave differently than that.  It's much more natural—and much more explicit—to write:

    >>> [item for seq in [range(-5,-1), range(5)] for item in seq]

    [-5, -4, -3, -2, 0, 1, 2, 3, 4]

The distinction between (x) and (x,) is already deep in the language. It has nothing to do with this thread

>>> [1, *([2],), 3]

[1, [2], 3]

>>> [1, *([2]), 3]

[1, 2, 3]

So there. Just like in this proposal.

Elazar.

[Sven R. Kunze]

On 12.10.2016 21:38, אלעזר wrote:
>
> What is the intuition behind [1, *x, 5]? The starred expression is
> replaced with a comma-separated sequence of its elements.
>
> The trailing comma Nick referred to is there, with the rule that [1,,
> 5] is the same as [1, 5].
>

I have to admit that I have my problems with this "comma-separated
sequence" idea. For me, lists are just collections of items. There are
no commas involved. I also think that thinking about commas here
complicates the matter.

What * does, it basically plugs in the items from the starred expression
into its surroundings:

[*[1,2,3]] = [1,2,3]

Let's plug in two lists into its surrounding list:

[*[1,2,3], *[1,2,3]] = [1,2,3,1,2,3]

So, as the thing goes, it looks like as if * could just work anywhere
inside those brackets:

[*[1,2,3] for _ in range(3)] = [*[1,2,3], *[1,2,3], *[1,2,3]] =
[1,2,3,1,2,3,1,2,3]

I have difficulties to understand the problem of understanding the
syntax. The * and ** variants just flow naturally whereas the "chain"
equivalent is bit "meh".

Cheers,
Sven

[אלעזר‎]

To be honest, I don't have a clear picture of what {**x for x in d.items()} should be. But I do have such picture for

    dict(**x for x in many_dictionaries)

Elazar

[Spencer Brown]

The semantics seem fairly obvious if you treat it as changing the method calls. For lists, * uses .extend() instead of .append(). Sets use * for .update() instead of .add(). Dicts use ** for .update() instead of __setitem__. In that case x should be a mapping (or iterable of pairs maybe), and all pairs in that should be added to the dict. In generator expressions * means yield from instead of just yield. The ** in dicts is needed to distinguish between set and dict comprehensions, since it doesn't use a colon.

Spencer

[Steven D'Aprano]

On Wed, Oct 12, 2016 at 06:32:12PM +0200, Sven R. Kunze wrote:
> On 12.10.2016 17:41, Nick Coghlan wrote:
> >This particular proposal fails on the first question (as too many
> >people would expect it to mean the same thing as either "[*expr, for
> >expr in iterable]" or "[*(expr for expr in iterable)]")
>
> So, my reasoning would tell me: where have I seen * so far? *args and
> **kwargs!

And multiplication. And sequence unpacking.

> [...] is just the list constructor.

Also indexing: dict[key] or sequence[item or slice].

The list constructor would be either list(...) or possibly
list.__new__.

[...] is either a list display:

[1, 2, 3, 4]

or a list comprehension. They are not the same thing, and they don't
work the same way. The only similarity is that they use [ ] as
delimiters, just like dict and sequence indexing. That doesn't mean that
you can write:

mydict[x for x in seq if condition]

Not everything with [ ] is the same.

> So, putting those two pieces together is quite simple.

I don't see that it is simple at all. I don't see any connection between
function *args and list comprehension loop variables.

> Furthermore, your two "interpretations" would yield the very same result
> as [expr for expr in iterable] which doesn't match with my experience
> with Python so far; especially when it comes to special characters. They
> must mean something. So, a simple "no-op" would not match my expectations.

Just because something would otherwise be a no-op doesn't mean that it
therefore has to have some magical meaning. Python has a few no-ops
which are allowed, or required, by syntax but don't do anything.

pass
(x) # same as just x
+1 # no difference between literals +1 and 1
-0
func((expr for x in iterable)) # redundant parens for generator expr

There may be more.

> >but it fails on the other two grounds as well.
>
> Here I disagree with you. We use *args all the time, so we know what *
> does. I don't understand why this should not work in between brackets [...].

By this logic, *t should work... everywhere?

while *args:
try:
raise *args
except *args:
del *args

That's not how Python works. Just because syntax is common, doesn't mean
it has to work everywhere. We cannot write:


for x in import math:
...

even though importing is common.

*t doesn't work as the expression inside a list comprehension because
that's not how list comps work. To make it work requires making this a
special case and mapping

[expr for t in iterable]

to a list append, while

[*expr for t in iterable]

gets mapped to a list extend.

Its okay to want that as a special feature, but understand what you are
asking for: you're not asking for some restriction to be lifted, which
will then automatically give you the functionality you expect. You're
asking for new functionality to be added.

Sequence unpacking inside list comprehensions as a way of flattening a
sequence is completely new functionality which does not logically follow
from the current semantics of comprehensions.

> >In most uses of *-unpacking it's adding entries to a comma-delimited
> >sequence, or consuming entries in a comma delimited sequence (the
> >commas are optional in some cases, but they're still part of the
> >relevant contexts). The expansions removed the special casing of
> >functions, and made these capabilities generally available to all
> >sequence definition operations.
>
> I don't know what you mean by comma-delimited sequence. There are no
> commas. It's just a list of entries. * adds entries to this list. (At
> least from my point of view.)

Not all points of view are equally valid.



--
Steve

[Steven D'Aprano]

On Wed, Oct 12, 2016 at 04:11:55PM +0000, אלעזר wrote:

> Steve, you only need to allow multiple arguments to append(), then it makes
> perfect sense.

I think you're missing a step. What will multiple arguments given to
append do? There are two obvious possibilities:

- collect all the arguments into a tuple, and append the tuple;

- duplicate the functionality of list.extend

neither of which appeals to me.

[אלעזר‎]

On Thu, Oct 13, 2016 at 2:35 AM Steven D'Aprano <st...@pearwood.info> wrote:

On Wed, Oct 12, 2016 at 04:11:55PM +0000, אלעזר wrote:

> Steve, you only need to allow multiple arguments to append(), then it makes
> perfect sense.

I think you're missing a step. What will multiple arguments given to
append do? There are two obvious possibilities:

- collect all the arguments into a tuple, and append the tuple;

- duplicate the functionality of list.extend


neither of which appeals to me. 

The latter, of course. Similar to max(). Not unheard of.

Elazar

[Sven R. Kunze]

On 13.10.2016 01:29, Steven D'Aprano wrote:
> On Wed, Oct 12, 2016 at 06:32:12PM +0200, Sven R. Kunze wrote:
>>
>> So, my reasoning would tell me: where have I seen * so far? *args and
>> **kwargs!
> And multiplication.

Multiplication with only a single argument? Come on.


> And sequence unpacking.

We are on the right side of the = if any and not no the left side.

>
>> [...] is just the list constructor.
> Also indexing: dict[key] or sequence[item or slice].

There's no name in front of [. So, I cannot be an index either.


Nothing else matches (in my head) and I also don't see any ambiguities.
YMMV.

I remember a new co-worker, I taught how to use *args and **kwargs. It
was unintuitive to him on the first time as well.

About the list constructor: we construct a list by writing [a,b,c] or by
writing [b for b in bs]. The end result is a list and that matters from
the end developer's point of view, no matter how fancy words you choose
for it.

Cheers,
Sven

[Steven D'Aprano]

On Thu, Oct 13, 2016 at 10:37:35AM +0200, Sven R. Kunze wrote:
> On 13.10.2016 01:29, Steven D'Aprano wrote:
> >On Wed, Oct 12, 2016 at 06:32:12PM +0200, Sven R. Kunze wrote:
> >>
> >>So, my reasoning would tell me: where have I seen * so far? *args and
> >>**kwargs!
> >And multiplication.
>
> Multiplication with only a single argument? Come on.

You didn't say anything about a single argument. Your exact words are
shown above: "where have I seen * so far?". I'm pretty sure you've seen
* used for multiplication. I also could have mentioned regexes, globs,
and exponentiation.

I cannot respond to your intended meaning, only to what you actually
write. Don't blame the reader if you failed to communicate clearly and
were misunderstood.


[...]

> About the list constructor: we construct a list by writing [a,b,c] or by
> writing [b for b in bs]. The end result is a list

I construct lists using all sorts of ways:

list(argument)
map(func, sequence)
zip(a, b)
file.readlines()
dict.items()
os.listdir('.')
sorted(values)

and so on. Should I call them all "list constructors" just because they
return a list? No, I don't think so. Constructor has a specific meaning,
and these are not all constructors -- and neither are list
comprehensions.

> and that matters from
> the end developer's point of view, no matter how fancy words you choose
> for it.

These "fancy words" that you dismiss are necessary technical terms.
Precision in language is something we should aim for, not dismiss as
unimportant.

List comprehensions and list displays have different names, not to show
off our knowledge of "fancy terms", but because they are different
things which just happen to both return lists. Neither of them are what
is commonly called a constructor. List displays are, in some senses,
like a literal; list comprehensions are not, and are better understood
as list builders, a process which builds a list. Emphasis should be on
the *process* part: a comprehension is syntactic sugar for building a
list using for-loop, not for a list display or list constructor.

The bottom line is that when you see a comprehension (list, set or dict)
or a generator expression, you shouldn't think of list displays, but of
a for-loop. That's one of the reasons why the analogy with argument
unpacking fails: it doesn't match what comprehensions *actually* are.



--
Steve

[אלעזר‎]

On Thu, Oct 13, 2016 at 5:10 PM Steven D'Aprano <st...@pearwood.info> wrote:

On Thu, Oct 13, 2016 at 10:37:35AM +0200, Sven R. Kunze wrote:
> About the list constructor: we construct a list by writing [a,b,c] or by
> writing [b for b in bs]. The end result is a list

I construct lists using all sorts of ways:

 

I think there is a terminology problem here (again). "Constructor" in OOP has a specific meaning, and "constructor" in functional terminology has a slightly different meaning. I guess Sven uses the latter terminology because pattern matching is the dual of the constructor - it is a "destructor" - and it feels appropriate, although admittedly confusing. In this terminology, map(), zip() etc. are definitely not constructors. there is only one "constructor" (list()), and there are functions that may use it as their implementation detail. In a way, [1, 2, 3] is just a syntactic shorthand for list construction, so it is reasonable to a call it a constructor.

This terminology is not a perfect fit into the object-oriented world of Python, but it is very helpful in discussion of patterns how to apply them uniformly, since they were pretty much invented in the functional world (ML, I think, and mathematics). One only needs to be aware of the two different meaning, and qualify if needed, so that we won't get lost in terminology arguments again.

Elazar

[Sven R. Kunze]

On 13.10.2016 16:10, Steven D'Aprano wrote:
> On Thu, Oct 13, 2016 at 10:37:35AM +0200, Sven R. Kunze wrote:
>> Multiplication with only a single argument? Come on.
> You didn't say anything about a single argument. Your exact words are
> shown above: "where have I seen * so far?". I'm pretty sure you've seen
> * used for multiplication. I also could have mentioned regexes, globs,
> and exponentiation.
>
> I cannot respond to your intended meaning, only to what you actually
> write. Don't blame the reader if you failed to communicate clearly and
> were misunderstood.

Steven, please. You seemed to struggle to understand the notion of the
[*....] construct and many people (not just me) here tried their best to
explain their intuition to you. But now it seems you don't even try to
come behind the idea and instead try hard not to understand the help
offered. If you don't want help or don't really want to understand the
proposal, that's fine but please, do us a favor and don't divert the
thread with nitpicking nonsensical details (like multiplication) and
waste everybody's time.


The context of the proposal is about lists/dicts and the */** unpacking
syntax. So, I actually expect you to put every post here into this very
context. Discussions without context don't make much sense. So, I won't
reply further.


Best,
Sven

[Martti Kühne]

On Wed, Oct 12, 2016 at 5:41 PM, Nick Coghlan <ncog...@gmail.com> wrote:
> However, set builder notation doesn't inherently include the notion of
> flattening lists-of-lists. Instead, that's a *consumption* operation
> that happens externally after the initial list-of-lists has been
> built, and that's exactly how it's currently spelled in Python:

> "itertools.chain.from_iterable(subiter for subiter in iterable)".

On Wed, Oct 12, 2016 at 5:42 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> The fundamental design principle of list comps is that they are
> equivalent to a for-loop with a single append per loop:
>
> [expr for t in iterable]
>
> is equivalent to:
>
> result = []
> for t in iterable:
> result.append(expr)
>
>
> If I had seen a list comprehension with an unpacked loop variable:
>

> [t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]
>
>


As it happens, python does have an external consumption operation that
happens externally with an iteration implied:

for t in iterable:
yield t

For your example [t for t in [(1, 'a'), (2, 'b'), (3, 'c')]] that would mean:

for t in [(1, 'a'), (2, 'b'), (3, 'c')]:
yield t

And accordingly, for the latter case [*t for t in [(1, 'a'), (2, 'b'),
(3, 'c')]] it would be:

for item in [(1, 'a'), (2, 'b'), (3, 'c')]:
for t in item:
yield t

cheers!
mar77i

[Paul Moore]

On 13 October 2016 at 15:32, Sven R. Kunze <srk...@mail.de> wrote:
> Steven, please. You seemed to struggle to understand the notion of the
> [*....] construct and many people (not just me) here tried their best to
> explain their intuition to you.

And yet, the fact that it's hard to explain your intuition to others
(Steven is not the only one who's finding this hard to follow) surely
implies that it's merely that - personal intuition - and not universal
understanding.

The *whole point* here is that not everyone understands the proposed
notation the way the proposers do, and it's *hard to explain* to those
people. Blaming the people who don't understand does not support the
position that this notation should be added to the language. Rather,
it reinforces the idea that the new proposal is hard to teach (and
consequently, it may be a bad idea for Python).

Paul

[אלעזר‎]

On Thu, Oct 13, 2016 at 6:19 PM Paul Moore <p.f....@gmail.com> wrote:

On 13 October 2016 at 15:32, Sven R. Kunze <srk...@mail.de> wrote:
> Steven, please. You seemed to struggle to understand the notion of the
> [*....] construct and many people (not just me) here tried their best to
> explain their intuition to you.

And yet, the fact that it's hard to explain your intuition to others
(Steven is not the only one who's finding this hard to follow) surely
implies that it's merely that - personal intuition - and not universal
understanding.

I fail to see this implication. Perhaps you mean that the universal understanding is hard to get, intuitively. And trying to explain them is the way to figure out howw hard can this difficulty be overcome.

 

The *whole point* here is that not everyone understands the proposed
notation the way the proposers do, and it's *hard to explain* to those
people. Blaming the people who don't understand does not support the
position that this notation should be added to the language. Rather,
it reinforces the idea that the new proposal is hard to teach (and
consequently, it may be a bad idea for Python).

It may also suggest that there are currently two ways to understand the *[...] construct, and only one of them can be generalized to lead the new proposal. So people that are *used* to the other way may have harder time than people coming with a clean slate. So it might or might not be hard to teach.

(I'm not saying that's necessarily the case)

I will be happy to understand that "other way" that is harder to generalize; I think this discussion may be fruitful in making these different understandings explicit.

Elazar

[Steven D'Aprano]

On Thu, Oct 13, 2016 at 03:28:27PM +0000, אלעזר wrote:

> It may also suggest that there are currently two ways to understand the
> *[...] construct,

This thread is about allowing sequence unpacking as the internal
expression of list comprehensions:

[ *(expr) for x in iterable ]

It isn't about unpacking lists:

*[...]

so I don't see what relevance your comment has.

There may be two or three or ten or 100 ways to (mis)understand list
comprehensions in Python, but only one of them is the correct way. List
comprehensions are (roughly) syntactic sugar for:

result = []
for x in iterable:
result.append(expression)

Any other understanding of them is incorrect.

Now if people wish to make an argument for changing the meaning of
comprehensions so that the suggested internal unpacking makes sense,
then by all means try making that argument! That's absolutely fine.

In the past, I've tried a similar thing: I argued for a variant list
comprehension that halts early:

[expr for x in iterable while condition]

(as found in at least one other language), but had that knocked back
because it doesn't fit the existing list comprehension semantics. I
wasn't able to convince people that the value of this new comprehension
was worth breaking the existing semantics of comprehensions.

Maybe you will be able to do better than me.

But understand that:

[*(expr) for x in iterable]

also fails to fit the existing list comprehension semantics. To make it
work requires changing the meaning of Python list comps. It isn't enough
to just deny the existing meaning and insist that your own personal
meaning is correct.

--
Steve

[Steven D'Aprano]

On Thu, Oct 13, 2016 at 04:34:49PM +0200, Martti Kühne wrote:

> > If I had seen a list comprehension with an unpacked loop variable:
> >
> > [t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]

Marttii, somehow you have lost the leading * when quoting me. What I
actually wrote was:

[*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]

> As it happens, python does have an external consumption operation that
> happens externally with an iteration implied:
>
> for t in iterable:
> yield t

If you replace the t with *t, you get a syntax error:


py> def gen():
... for t in [(1, 'a'), (2, 'b'), (3, 'c')]:
... yield *t
File "<stdin>", line 3
yield *t
^
SyntaxError: invalid syntax

Even if it was allowed, what would it mean? It could only mean "unpack
the sequence t, and collect the values into a tuple; then yield the
tuple".

> For your example [t for t in [(1, 'a'), (2, 'b'), (3, 'c')]] that would mean:
>
> for t in [(1, 'a'), (2, 'b'), (3, 'c')]:
> yield t
>
> And accordingly, for the latter case [*t for t in [(1, 'a'), (2, 'b'),
> (3, 'c')]] it would be:
>
> for item in [(1, 'a'), (2, 'b'), (3, 'c')]:
> for t in item:
> yield t

No it wouldn't. Where does the second for loop come from? The list
comprehension shown only has one loop, not nested loops.



--
Steve

[Martti Kühne]

On Thu, Oct 13, 2016 at 6:55 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> On Thu, Oct 13, 2016 at 04:34:49PM +0200, Martti Kühne wrote:
>
>> > If I had seen a list comprehension with an unpacked loop variable:
>> >
>> > [t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]
>

> Martti, somehow you have lost the leading * when quoting me. What I

> actually wrote was:
>
> [*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]
>

Sorry for misquoting you. Can I fix my name, though?
Also, this mail was too long in my outbox so the context was lost on it.
I reiterate it, risking that I would annoy some, but to be absolutely clear.

>
>> As it happens, python does have an external consumption operation that
>> happens externally with an iteration implied:
>>
>

> If you replace the t with *t, you get a syntax error:
>

I meant that statement in context of the examples which were brought up:
the occurrence of a list comprehension inside an array have the
following effect:

1) [ ..., [expr for t in iterable] ]

is equivalent to:

def expr_long(iterable, result):
result.append(iterable)
return result

expr_long(iterable, [ ..., ])

so, if you make the case for pep448, you might arrive at the following:

2) [ ..., *[expr for expr in iterable] ]

which would be, if I'm typing it correctly, equivalent to, what
resembles an external collection:

def expr_star(list_comp, result):
result.extend(list(list_comp))
return result

expr_star(iterable, [ ..., ])

Having this in mind, the step to making:

[ ..., [*expr for expr in iterable], ]

from:

def expr_insidestar(iterable, result):
for expr in iterable:
result.extend(expr)
return result

does not appear particularly far-fetched, at least not to me and a few
people on this list.

cheers!
mar77i

[David Mertz]

Exactly with Paul!

As I mentioned, I teach software developers and scientists Python for a living.  I get paid a lot of money to do that, and have a good sense of what learners can easily understand and not (I've also written hundred of articles and a few books about Python).  The people I write for and teach are educated, smart, and generally have familiarity with multiple programming languages.

In my opinion, this new construct—if added to the language—would be difficult to teach, and most of my students would get it wrong most of the time.

Yes, I understand the proposed semantics.  It is not *intuitive* to me, but I could file the rule about the behavior if I had to.  But if I were forced to teach it, it would always be "Here's a Python wart to look out for if you see it in other code... you should not ever use it yourself."

--

Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.

[David Mertz]

    [*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]

Another problem with this is that it is very hard to generalize to the case where the item included in a comprehension is a transformation on iterated values.  E.g. what does this do?

    [math.exp(*t) for t in [(1,2),(3,4)]]

Maybe that somehow magically gets us:

    [2.7182, 7.38905, 20.0855, 54.5981] 

 

Or maybe the syntax would be:

    [*math.exp(t) for t in [(1,2),(3,4)]]

Neither of those follows conventional Python semantics for function calling or sequence unpacking.  So maybe that remains a type error or syntax error.  But then we exclude a very common pattern of using comprehensions to create collections of *transformed* data, not simply of filtered data.

In contrast, either of these are unambiguous and obvious:

    [math.exp(t) for t in flatten([(1,2),(3,4)])]

Or:

    [math.exp(n) for t in [(1,2),(3,4)] for n in t]

Obviously, picking math.exp() is arbitrary and any unary function would be the same issue.

[Random832]

On Thu, Oct 13, 2016, at 14:50, David Mertz wrote:
> Neither of those follows conventional Python semantics for function
> calling
> or sequence unpacking. So maybe that remains a type error or syntax
> error. But then we exclude a very common pattern of using comprehensions
> to create collections of *transformed* data, not simply of filtered data.

[*map(math.exp, t) for t in [(1, 2), (3, 4)]]

[*(math.exp(x) for x in t) for t in [(1, 2), (3, 4)]]

I think "excluding" is a bit of a strong word - just because something
doesn't address a mostly unrelated need doesn't mean it doesn't have any
merit in its own right. Not every proposal is going to do everything. I
think the key is that the person originally asking this thought of *x as
a generalized "yield from x"-ish thing, for example:

"a, *b, c" becomes "def f(): yield a; yield from b; yield c;"

[a, *b, c] == list(f())
(a, *b, c) == tuple(f())

so, under a similar 'transformation', "*foo for foo in bar" likewise
becomes "def f(): for foo in bar: yield from foo"

bar = [(1, 2), (3, 4)]
(*(1, 2), *(3, 4)) == == tuple(f())
[*(1, 2), *(3, 4)] == == list(f())

> In contrast, either of these are unambiguous and obvious:
>
> [math.exp(t) for t in flatten([(1,2),(3,4)])]
>
> Or:
>
> [math.exp(n) for t in [(1,2),(3,4)] for n in t]
>
> Obviously, picking math.exp() is arbitrary and any unary function would
> be
> the same issue.
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons. Intellectual property is
> to the 21st century what the slave trade was to the 16th.

[Random832]

On Thu, Oct 13, 2016, at 15:46, Random832 wrote:
> so, under a similar 'transformation', "*foo for foo in bar" likewise
> becomes "def f(): for foo in bar: yield from foo"
>
> bar = [(1, 2), (3, 4)]
> (*(1, 2), *(3, 4)) == == tuple(f())
> [*(1, 2), *(3, 4)] == == list(f())

I accidentally hit ctrl-enter while copying and pasting, causing my
message to go out while my example was less thorough than intended and
containing syntax errors. It was intended to read as follows:

..."*foo for foo in bar" likewise becomes

def f():
for foo in bar:
yield from foo

a, b = (1, 2), (3, 4)
bar = [a, b]
(*a, *b) == (1, 2, 3, 4) == tuple(f()) # tuple(*foo for foo in bar)
[*a, *b] == [1, 2, 3, 4] == list(f()) # [*foo for foo in bar]

[Neil Girdhar]

First of all:

+1 to Sven's very well-expressed support of the proposal, and

+1 to Nick's very well-explained reasons for rejecting it.

As one of the main implementers of PEP 448, I have always liked this, but I suggested that we leave this out when there was opposition since there's no rush for it.

Regarding Steven's example, like Sven, I also see it this way:

    [*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]] 

should mean:

   [*(1, 'a'), *(2, 'b'), *(3, 'c')]] 

Which coincides with what the OP is asking for.

At the end of this discussion it might be good to get a tally of how many people think the proposal is reasonable and logical.  I imagine people will be asking this same question next year and the year after, and so it will be good to see if as familiarity with PEP 448 expands, more people will find this intuitive and logical.

From a CPython implementation standpoint, we specifically blocked this code path, and it is only a matter of unblocking it if we want to support this.

Best,

Neil

[Sjoerd Job Postmus]

After having followed this thread for a while, it occured to me that the
reason that the idea is confusing, is because the spelling is confusing.

I think the suggested spelling (`*`) is the confusing part. If it were
to be spelled `from ` instead, it would be less confusing.

Consider this:

g = (f(t) for t in iterable)

is "merely" sugar for

def gen():
for t in iterable:
yield f(t)
g = gen()

Likewise,

l = [f(t) for t in iterable]

can be seen as sugar for

def gen():
for t in iterable:
yield f(t)
l = list(gen())

Now the suggested spelling

l = [*f(t) for t in iterable]

is very confusing, from what I understand: what does the `*` even mean
here.

However, consider the following spelling:

l = [from f(t) for t in iterable]

To me, it does not seem far-fetched that this would mean:

def gen():
for t in iterable:
yield from f(t)
l = list(gen())

It follows the "rule" quite well: given a generator display, everything
before the first "for" gets placed after "yield ", and all the
`for`/`if`s are expanded to suites.

Now I'm not sure if I'm a fan of the idea, but I think that at least the
`from `-spelling is less confusing than the `*`-spelling.

(Unless I totally misunderstood what the `*`-spelling was about, given
how confusing it supposedly is. Maybe it confused me.)

[Paul Moore]

On 13 October 2016 at 20:51, Random832 <rand...@fastmail.com> wrote:
> On Thu, Oct 13, 2016, at 15:46, Random832 wrote:
>> so, under a similar 'transformation', "*foo for foo in bar" likewise
>> becomes "def f(): for foo in bar: yield from foo"
>>
>> bar = [(1, 2), (3, 4)]
>> (*(1, 2), *(3, 4)) == == tuple(f())
>> [*(1, 2), *(3, 4)] == == list(f())
>
>
> I accidentally hit ctrl-enter while copying and pasting, causing my
> message to go out while my example was less thorough than intended and
> containing syntax errors. It was intended to read as follows:
>
> ..."*foo for foo in bar" likewise becomes
>
> def f():
> for foo in bar:
> yield from foo
>
> a, b = (1, 2), (3, 4)
> bar = [a, b]
> (*a, *b) == (1, 2, 3, 4) == tuple(f()) # tuple(*foo for foo in bar)
> [*a, *b] == [1, 2, 3, 4] == list(f()) # [*foo for foo in bar]

I remain puzzled.

Given the well-documented and understood transformation:

[fn(x) for x in lst if cond]

translates to

result = []
for x in lst:
if cond:
result.append(fn(x))

please can you explain how to modify that translation rule to
incorporate the suggested syntax?

Personally, I'm not even sure any more that I can *describe* the
suggested syntax. Where in [fn(x) for x in lst if cond] is the *
allowed? fn(*x)? *fn(x)? Only as *x with a bare variable, but no
expression? Only in certain restricted types of construct which aren't
expressions but are some variation on an unpacking construct?

We've had a lot of examples. I think it's probably time for someone to
describe the precise syntax (as BNF, like the syntax in the Python
docs at https://docs.python.org/3.6/reference/expressions.html#displays-for-lists-sets-and-dictionaries
and following sections) and semantics (as an explanation of how to
rewrite any syntactically valid display as a loop). It'll have to be
done in the end, as part of any implementation, so why not now?

Paul

[אלעזר‎]

On Thu, Oct 13, 2016 at 11:42 PM Paul Moore <p.f....@gmail.com> wrote:

I remain puzzled.

Given the well-documented and understood transformation:

[fn(x) for x in lst if cond]

translates to

result = []
for x in lst:
   if cond:
      result.append(fn(x))

please can you explain how to modify that translation rule to
incorporate the suggested syntax?

if you allow result.append(1, 2, 3) to mean result.extend([1,2,3])  # which was discussed before

result = []
for x in lst:
   if cond:

      result.append(*fn(x))  

Or simply use result.extend([*fn(x)])

Personally, I'm not even sure any more that I can *describe* the
suggested syntax. Where in [fn(x) for x in lst if cond] is the *
allowed? fn(*x)? *fn(x)? Only as *x with a bare variable, but no
expression? Only in certain restricted types of construct which aren't
expressions but are some variation on an unpacking construct?

The star is always exactly at the place that should "handle" it. which means [*(fn(x)) for x in lst if cond]. fn(x) must be iterable as always.

 

We've had a lot of examples. I think it's probably time for someone to
describe the precise syntax (as BNF, like the syntax in the Python
docs at https://docs.python.org/3.6/reference/expressions.html#displays-for-lists-sets-and-dictionaries
and following sections) and semantics (as an explanation of how to
rewrite any syntactically valid display as a loop). It'll have to be
done in the end, as part of any implementation, so why not now?

I will be happy to do so, and will be happy to work with anyone else interested.

Elazar

[Paul Moore]

On 13 October 2016 at 21:40, Sjoerd Job Postmus <sjoe...@sjoerdjob.com> wrote:
> However, consider the following spelling:
>
> l = [from f(t) for t in iterable]
>
> To me, it does not seem far-fetched that this would mean:
>
> def gen():
> for t in iterable:
> yield from f(t)
> l = list(gen())

Thank you. This is the type of precise definition I was asking for in
my previous post (your timing was superb!)

I'm not sure I *like* the proposal, but I need to come up with some
reasonable justification for my feeling, whereas for previous
proposals the "I don't understand what you're suggesting" was the
overwhelming feeling, and stifled any genuine discussion of merits or
downsides.

Paul

PS I can counter a suggestion of using *f(t) rather than from f(t) in
the above, by saying that it adds yet another meaning to the already
heavily overloaded * symbol. The suggestion of "from" avoids this as
"from" only has a few meanings already. (You can agree or disagree
with my view, but at least we're debating the point objectively at
that point!)

[Paul Moore]

On 13 October 2016 at 21:47, אלעזר <ela...@gmail.com> wrote:
> if you allow result.append(1, 2, 3) to mean result.extend([1,2,3]) # which
> was discussed before

I don't (for the reasons raised before). But thank you for your
explanation, it clarifies what you were proposing. And it does so
within the *current* uses of the * symbol, which is good. But:

1. I'm not keen on extending append's meaning to overlap with extend's
like this.
2. Your proposal does not generalise to generator expressions, set
displays (without similarly modifying the set.add() method) or
dictionary displays.
3. *fn(x) isn't an expression, and yet it *looks* like it should be,
and in the current syntax, an expression is required in that position.
To me, that suggests it would be hard to teach. [1]

You can of course generalise Sjoerd's "from" proposal and then just
replace "from" with "*" throughout. That avoids your requirement to
change append, but at the cost of the translation no longer being a
parallel to an existing use of "*".

Paul

[1] On a purely personal note, I'd say it's confusing, but I don't
want to go back to subjective arguments, so I only note that here as
an opinion, not an argument.

[Random832]

On Thu, Oct 13, 2016, at 16:42, Paul Moore wrote:
> I remain puzzled.
>
> Given the well-documented and understood transformation:
>
> [fn(x) for x in lst if cond]
>
> translates to
>
> result = []
> for x in lst:
> if cond:
> result.append(fn(x))
>
> please can you explain how to modify that translation rule to
> incorporate the suggested syntax?

In this case * would change this to result.extend (or +=)

just as result = [a, *b, c] is equivalent to:
result = []
result.append(a)
result.extend(b)
result.append(c)

result = [*x for x in lst if cond] would become:

result = []
for x in lst:
if cond:

result.extend(x)

I used yield from as my original example to include generator
expressions, which should also support this.

> Personally, I'm not even sure any more that I can *describe* the
> suggested syntax. Where in [fn(x) for x in lst if cond] is the *
> allowed? fn(*x)?

This already has a meaning, so it's obviously "allowed", but not in a
way relevant to this proposal. The elements of x are passed to fn as
arguments rather than being inserted into the list. Ultimately the
meaning is the same.

> *fn(x)? Only as *x with a bare variable, but no expression?

Both of these would be allowed. Any expression would be allowed, but at
runtime its value must be iterable, the same as other places that you
can use *x.

[אלעזר‎]

On Thu, Oct 13, 2016 at 11:59 PM Paul Moore <p.f....@gmail.com> wrote:

On 13 October 2016 at 21:47, אלעזר <ela...@gmail.com> wrote:
> if you allow result.append(1, 2, 3) to mean result.extend([1,2,3])  # which
> was discussed before

I don't (for the reasons raised before). But thank you for your
explanation, it clarifies what you were proposing. And it does so
within the *current* uses of the * symbol, which is good. But:

1. I'm not keen on extending append's meaning to overlap with extend's
like this.
2. Your proposal does not generalise to generator expressions, set
displays (without similarly modifying the set.add() method) or
dictionary displays.
3. *fn(x) isn't an expression, and yet it *looks* like it should be,
and in the current syntax, an expression is required in that position.
To me, that suggests it would be hard to teach. [1]

You can of course generalise Sjoerd's "from" proposal and then just
replace "from" with "*" throughout. That avoids your requirement to
change append, but at the cost of the translation no longer being a
parallel to an existing use of "*".

I think it is an unfortunate accident of syntax, the use of "yield from foo()" instead of "yield *foo()". These "mean" the same: a syntactic context that directly handles iterable as repetition, (with some guarantees regarding exceptions etc.). Alternatively, we could be writing [1, 2, from [3, 4], 5, 6]. Whether it is "from x" or "*x" is just an accident. In my mind.

As you said, the proposal should be written in a much more formal way, so that it could be evaluated without confusion. I completely agree.

Elazar

[אלעזר‎]

On Fri, Oct 14, 2016 at 12:06 AM Neil Girdhar <miste...@gmail.com> wrote:

<snip>

Regarding Steven's example, like Sven, I also see it this way:

    [*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]] 

should mean:

   [*(1, 'a'), *(2, 'b'), *(3, 'c')]] 

Which coincides with what the OP is asking for.

<snip>

From a CPython implementation standpoint, we specifically blocked this code path, and it is only a matter of unblocking it if we want to support this.

This is *very, very* not surprising. And should be stressed.

Elazar

[Random832]

On Thu, Oct 13, 2016, at 16:59, Paul Moore wrote:
> I don't (for the reasons raised before). But thank you for your
> explanation, it clarifies what you were proposing. And it does so
> within the *current* uses of the * symbol, which is good. But:
>
> 1. I'm not keen on extending append's meaning to overlap with extend's
> like this.

I think the "append(*x)" bit was just a flourish to try to explain it in
terms of the current use of * since you don't seem to understand it any
other way, rather than an actual proposal to actually change the append
method.

> 2. Your proposal does not generalise to generator expressions, set
> displays (without similarly modifying the set.add() method) or
> dictionary displays.

Basically it would make the following substitutions in the conventional
"equivalent loops"
generator yield => yield from
list append => extend
set add => update
dict __setitem__ => update

dict comprehensions would need to use **x - {*x for x in y} would be a
set comprehension.

> 3. *fn(x) isn't an expression, and yet it *looks* like it should be,
> and in the current syntax, an expression is required in that position.
> To me, that suggests it would be hard to teach. [1]

I can think of another position an expression used to be required in:

Python 3.5.2
>>> [1, *(2, 3), 4]
[1, 2, 3, 4]

Python 2.7.11
>>> [1, *(2, 3), 4]
File "<stdin>", line 1
[1, *(2, 3), 4]
^
SyntaxError: invalid syntax

Was that hard to teach? Maybe. But it's a bit late to object now, and
every single expression on the right hand side in my examples below
already has a meaning.


Frankly, I don't see why the pattern isn't obvious [and why people keep
assuming there will be a new meaning of f(*x) as if it doesn't already
have a meaning]

Lists, present:
[x for x in [a, b, c]] == [a, b, c]
[f(x) for x in [a, b, c]] == [f(a), f(b), f(c)]
[f(*x) for x in [a, b, c]] == [f(*a), f(*b), f(*c)]
[f(**x) for x in [a, b, c]] == [f(**a), f(**b), f(**c)]
Lists, future:
[*x for x in [a, b, c]] == [*a, *b, *c]
[*f(x) for x in [a, b, c]] == [*f(a), *f(b), *f(c)]
[*f(*x) for x in [a, b, c]] == [*f(*a), *f(*b), *f(*c)]
[*f(**x) for x in [a, b, c]] == [*f(**a), *f(**b), *f(**c)]

Sets, present:
{x for x in [a, b, c]} == {a, b, c}
{f(x) for x in [a, b, c]} == {f(a), f(b), f(c)}
{f(*x) for x in [a, b, c]} == {f(*a), f(*b), f(*c)}
{f(**x) for x in [a, b, c]} == {f(**a), f(**b), f(**c)}
Sets, future:
{*x for x in [a, b, c]} == {*a, *b, *c}
{*f(x) for x in [a, b, c]} == {*f(a), *f(b), *f(c)}
{*f(*x) for x in [a, b, c]} == {*f(*a), *f(*b), *f(*c)}
{*f(**x) for x in [a, b, c]} == {*f(**a), *f(**b), *f(**c)}

Dicts, future:
{**x for x in [a, b, c]} == {**a, **b, **c}
{**f(x) for x in [a, b, c]} == {**f(a), **f(b), **f(c)}
{**f(*x) for x in [a, b, c]} == {**f(*a), **f(*b), **f(*c)}
{**f(**x) for x in [a, b, c]} == {**f(**a), **f(**b), **f(**c)}

[אלעזר‎]

Trying to restate the proposal, somewhat more formal following Random832 and Paul's suggestion.

I only speak about the single star.

---

The suggested change of syntax:

    comprehension ::=  starred_expression comp_for

Semantics:

(In the following, f(x) must always evaluate to an iterable)

1. List comprehension:

    result = [*f(x) for x in iterable if cond]

Translates to

    result = []

    for x in iterable:

        if cond:

            result.extend(f(x))

2. Set comprehension:

    result = {*f(x) for x in iterable if cond}

Translates to

    result = set()

    for x in iterable:

        if cond:

            result.update(f(x))

3. Generator expression:

    (*f(x) for x in iterable if cond)

Translates to

    for x in iterable:

        if cond:

            yield from f(x)

Elazar

[Steven D'Aprano]

On Thu, Oct 13, 2016 at 10:40:19PM +0200, Sjoerd Job Postmus wrote:

> Likewise,
>
> l = [f(t) for t in iterable]
>
> can be seen as sugar for
>
> def gen():
> for t in iterable:
> yield f(t)
> l = list(gen())

But that is *not* how list comprehensions are treated today. Perhaps
they should be?

https://docs.python.org/3.6/reference/expressions.html#displays-for-lists-sets-and-dictionaries


(Aside: earlier I contrasted "list display" from "list comprehension".
In fact according to the docs, a comprehension is a kind of display, a
special case of display. Nevertheless, my major point still holds: a
list display like [1, 2, 3] is not the same as a list comprehension like
[a+1 for a in (0, 1, 2)].)

There may be some conceptual benefits to switching to a model where
list/set/dict displays are treated as list(gen_expr) etc. But that still
leaves the question of what "yield *t" is supposed to mean?

Consider the analogy with f(*t), where t = (a, b, c). We *don't* have:

f(*t) is equivalent to f(a); f(b); f(c)

So why would yield *t give us this?

yield a; yield b; yield c

By analogy with the function call syntax, it should mean:

yield (a, b, c)

That is, t is unpacked, then repacked to a tuple, then yielded.

> Now the suggested spelling
>
> l = [*f(t) for t in iterable]
>
> is very confusing, from what I understand: what does the `*` even mean
> here.

Indeed. The reader may be forgiven for thinking that this is yet another
unrelated and arbitrary use of * to join the many other uses:

- mathematical operator;
- glob and regex wild-card;
- unpacking;
- import all
- and now yield from

> However, consider the following spelling:
>
> l = [from f(t) for t in iterable]
>
> To me, it does not seem far-fetched that this would mean:
>
> def gen():
> for t in iterable:
> yield from f(t)
> l = list(gen())

Now we're starting to move towards a reasonable proposal. It still
requires a conceptual shift in how list comprehensions are documented,
but at least now the syntax is no longer so arbitrary.

[Steven D'Aprano]

On Thu, Oct 13, 2016 at 08:15:36PM +0200, Martti Kühne wrote:

> Can I fix my name, though?

I don't understand what you mean. Your email address says your name is
Martti Kühne. Is that incorrect?



[...]

> I meant that statement in context of the examples which were brought up:
> the occurrence of a list comprehension inside an array have the
> following effect:
>
> 1) [ ..., [expr for t in iterable] ]
>
> is equivalent to:
>
> def expr_long(iterable, result):
> result.append(iterable)
> return result
>
> expr_long(iterable, [ ..., ])

The good thing about this example is that it is actual runnable code
that we can run to see if they are equivalent. They are not equivalent.

py> def expr_long(iterable, result):
... result.append(iterable)
... return result
...
py> iterable = (100, 200, 300)
py> a = [..., [2*x for x in iterable]]
py> b = expr_long(iterable, [...])
py> a == b
False
py> print(a, b)
[Ellipsis, [200, 400, 600]] [Ellipsis, (100, 200, 300)]

For this to work, you have to evaluate the list comprehension first,
then pass the resulting list to be appended to the result.

I don't think this is very insightful. All you have demonstrated is that
a list display [a, b, c, ...] is equivalent to:

result = []
for x in [a, b, c, ...]:
result.append(x)


except that you have written it in a slightly functional form.

> so, if you make the case for pep448, you might arrive at the following:
>
> 2) [ ..., *[expr for expr in iterable] ]

That syntax already works (in Python 3.5):

py> [1, 2, 3, *[x+1 for x in (100, 200, 300)], 4, 5]
[1, 2, 3, 101, 201, 301, 4, 5]

> which would be, if I'm typing it correctly, equivalent to, what
> resembles an external collection:
>
> def expr_star(list_comp, result):
> result.extend(list(list_comp))
> return result
>
> expr_star(iterable, [ ..., ])
>
> Having this in mind, the step to making:
>
> [ ..., [*expr for expr in iterable], ]
>
> from:
>
> def expr_insidestar(iterable, result):
> for expr in iterable:
> result.extend(expr)
> return result
>
> does not appear particularly far-fetched, at least not to me and a few
> people on this list.

But you don't have [..., list_comp, ] you just have the list comp.

You are saying:

(1) List displays [a, b, c, d, ...] are like this;
(2) we can sensibly extend that to the case [a, b, *c, d, ...]

I agree with (1) and (2). But then you have a leap:

(3) therefore [*t for t in iterable] should mean this.

There's a huge leap between the two.

To even begin to make sense of this, you have to unroll the list
comprehension into a list display. But that's not very helpful:

[expr for t in iterable]

Would you rather see that explained as:

[expr, expr, expr, expr, ...]

or as this?

result = []
for t in iterable:
result.append(expr)

The second form, the standard, documented explanation for
comprehensions, also applies easily to more complex examples:

[expr for t in iter1 for u in iter2 for v in iter3 if condition]

result = []
for t in iter1:
for u in iter2:
for v in iter3:
if condition:
result.append(expr)



--
Steve

[MRAB]

On 2016-10-14 02:04, Steven D'Aprano wrote:
> On Thu, Oct 13, 2016 at 08:15:36PM +0200, Martti Kühne wrote:
>
>> Can I fix my name, though?
>
> I don't understand what you mean. Your email address says your name is
> Martti Kühne. Is that incorrect?
>

[snip]

You wrote "Marttii" and he corrected it when he quoted you in his reply.

[Random832]

On Thu, Oct 13, 2016, at 18:15, Steven D'Aprano wrote:
> Consider the analogy with f(*t), where t = (a, b, c). We *don't* have:
>
> f(*t) is equivalent to f(a); f(b); f(c)

I don't know where this "analogy" is coming from.

f(*t) == f(a, b, c)
[*t] == [a, b, c]
{*t} == {a, b, c}

All of this is true *today*.

t, u, v = (a, b, c), (d, e, f), (g, h, i)
f(*t, *u, *v) == f(a, b, c, d, e, f, g, h, i)
[*t, *u, *v] == [a, b, c, d, e, f, g, h, i]

> > is very confusing, from what I understand: what does the `*` even mean
> > here.
>
> Indeed. The reader may be forgiven for thinking that this is yet another
> unrelated and arbitrary use of * to join the many other uses:

How is it arbitrary?

> - mathematical operator;
> - glob and regex wild-card;
> - unpacking;

This is unpacking. It unpacks the results into the destination.

There's a straight line from [*t, *u, *v] to [*x for x in (t, u, v)].
What's surprising is that it doesn't work now.

I think last month we even had someone who didn't know about 'yield
from' propose 'yield *x' for exactly this feature. It is intuitive - it
is a straight-line extension of the unpacking syntax.

> - import all
> - and now yield from

[Greg Ewing]

Steven D'Aprano wrote:

> py> def gen():
> ... for t in [(1, 'a'), (2, 'b'), (3, 'c')]:
> ... yield *t
> File "<stdin>", line 3
> yield *t
> ^
> SyntaxError: invalid syntax
>
> Even if it was allowed, what would it mean? It could only mean "unpack
> the sequence t, and collect the values into a tuple; then yield the
> tuple".

To maintain the identity

list(*x for x in y) == [*x for x in y]

it would be necessary for the *x in (*x for x in y) to expand
to "yield from x".

--
Greg

[Greg Ewing]

David Mertz wrote:
> it would always be "Here's a Python wart to look out
> for if you see it in other code... you should not ever use it yourself."

Do you currently tell them the same thing about the use
of * in a list display?

--
Greg

[David Mertz]

I've never used nor taught a * in a list display. I don't think they seem so bad, but it's a step down a slippery slope towards forms that might as well be Perl.

[Greg Ewing]

Random832 wrote:

> [*map(math.exp, t) for t in [(1, 2), (3, 4)]]
>
> [*(math.exp(x) for x in t) for t in [(1, 2), (3, 4)]]

Or more simply,

[math.exp(x) for t in [(1, 2), (3, 4)] for x in t]

I think this brings out an important point. While it
would be nice to allow * unpacking in comprehensions
for consistency with displays, it's not strictly
necessary, since you can always get the same effect
with another level of looping.

So it comes down to whether you think added conistency,
plus maybe some efficiency gains in some cases, are worth
making the change.

--
Greg

[Greg Ewing]

Neil Girdhar wrote:
> At the end of this discussion it might be good to get a tally of how
> many people think the proposal is reasonable and logical.

I think it's reasonable and logical.

[Greg Ewing]

Sjoerd Job Postmus wrote:
> I think the suggested spelling (`*`) is the confusing part. If it were
> to be spelled `from ` instead, it would be less confusing.

Are you suggesting this spelling just for generator
comprehensions, or for list comprehensions as well?
What about dict comprehensions?

--
Greg

[Greg Ewing]

Paul Moore wrote:
> please can you explain how to modify that translation rule to
> incorporate the suggested syntax?

It's quite simple: when there's a '*', replace 'append'
with 'extend':

[*fn(x) for x in lst if cond]

expands to

result = []
for x in lst:
if cond:

result.extend(fn(x))

The people thinking that you should just stick the '*x'
in as an argument to append() are misunderstanding the
nature of the expansion. You can't do that, because
the current expansion is based on the assumption that
the thing being substituted is an expression, and
'*x' is not a valid expression on its own. A new rule
is needed to handle that case.

And I'm the one who *invented* that expansion, so I
get to say what it means. :-)

--
Greg

[Greg Ewing]

Paul Moore wrote:
> Where in [fn(x) for x in lst if cond] is the *
> allowed? fn(*x)? *fn(x)?

Obviously you're *allowed* to put fn(*x), because that's
already a valid function call, but the only *new* place
we're talking about, and proposing new semantics for, is
in front of the expression representing items to be added
to the list, i.e. [*fn(x) for ...]

> I think it's probably time for someone to
> describe the precise syntax (as BNF, like the syntax in the Python
> docs at https://docs.python.org/3.6/reference/expressions.html#displays-for-lists-sets-and-dictionaries

Replace

comprehension ::= expression comp_for

with

comprehension ::= (expression | "*" expression) comp_for

> and semantics (as an explanation of how to
> rewrite any syntactically valid display as a loop).

The expansion of the "*" case is the same as currently except
that 'append' is replaced by 'extend' in a list comprehension,
'yield' is replaced by 'yield from' in a generator
comprehension.

If we decided to also allow ** in dict comprehensions, then
the expansion would use 'update'.

--
Greg

[Greg Ewing]

Paul Moore wrote:
> PS I can counter a suggestion of using *f(t) rather than from f(t) in
> the above, by saying that it adds yet another meaning to the already
> heavily overloaded * symbol.

We've *already* given it that meaning in non-comprehension
list displays, though, so we're not really adding any new
meanings for it -- just allowing it to have that meaning in
a place where it's currently disallowed.

Something I've just noticed -- the Language Reference actually
defines both ordinary list displays and list comprehensions
as "displays", and says that a display can contain either a
comprehension or an explicit list of values. It has to go
out of its way a bit to restrict the * form to
non-comprehensions.

--
Greg

[Greg Ewing]

Paul Moore wrote:
> 3. *fn(x) isn't an expression, and yet it *looks* like it should be ...

> To me, that suggests it would be hard to teach.

It's not an expression in any of the other places it's
used, either. Is it hard to to teach in those cases as
well?

--
Greg

[Greg Ewing]

אלעזר wrote:
> I think it is an unfortunate accident of syntax, the use of "yield from
> foo()" instead of "yield *foo()".

I think that was actually discussed back when yield-from
was being thrashed out, but as far as I remember we didn't
have * in list displays then, so the argument for it was
weaker. If we had, it might have been given more serious
consideration.

--
Greg

[Greg Ewing]

Steven D'Aprano wrote:

So why would yield *t give us this?
>
> yield a; yield b; yield c
>
> By analogy with the function call syntax, it should mean:
>
> yield (a, b, c)

This is a false analogy, because yield is not a function.

>>However, consider the following spelling:
>>
>> l = [from f(t) for t in iterable]

That sentence no verb!

In English, 'from' is a preposition, so one expects there
to be a verb associated with it somewhere. We currently
have 'from ... import' and 'yield from'.

But 'from f(t) for t in iterable' ... do what?

--
Greg

[Sjoerd Job Postmus]

On Fri, Oct 14, 2016 at 07:06:12PM +1300, Greg Ewing wrote:
> Sjoerd Job Postmus wrote:
> >I think the suggested spelling (`*`) is the confusing part. If it were
> >to be spelled `from ` instead, it would be less confusing.
>
> Are you suggesting this spelling just for generator
> comprehensions, or for list comprehensions as well?
> What about dict comprehensions?

For both generator, list and set comprehensions it makes sense, I think.
For dict comprehensions: not so much. That in itself is already sign
enough that probably the */** spelling would make more sense, while also
allowing the `yield *foo` alternative to `yield from foo`. But what
would be the meaning of `yield **foo`? Would that be `yield
*foo.items()`? I have no idea.

[Neil Girdhar]

Here's an interesting idea regarding yield **x:

Right now a function containing any yield returns a generator.  Therefore, it works like a generator expression, which is the lazy version of a list display.  lists can only contain elements x and unpackings *x.  Therefore, it would make sense to only have "yield x" and "yield *xs" (currently spelled "yield from xs")

If one day, there was a motivation to provide a lazy version of a dict display, then such a function would probably have "yield key: value" or "yield **d".   Such a lazy dictionary is the map stage of the famous mapreduce algorithm.  It might not make sense in single processor python, but it might in distributed Python.

Best,

Neil

--

---
You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/ROYNN7a5VAc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-ideas...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michel Desmoulin]

Regarding all those examples:

Le 14/10/2016 à 00:08, אלעזר a écrit :
> Trying to restate the proposal, somewhat more formal following Random832
> and Paul's suggestion.
>
> I only speak about the single star.
> ---
>

> *The suggested change of syntax:*
>
> comprehension ::= starred_expression comp_for
>
> *Semantics:*

>
> (In the following, f(x) must always evaluate to an iterable)
>
> 1. List comprehension:
>
> result = [*f(x) for x in iterable if cond]
>
> Translates to
>
> result = []
> for x in iterable:
> if cond:
> result.extend(f(x))
>
> 2. Set comprehension:
>
> result = {*f(x) for x in iterable if cond}
>
> Translates to
>
> result = set()
> for x in iterable:
> if cond:
> result.update(f(x))

Please note that we already have a way to do those. E.G:

result = [*f(x) for x in iterable if cond]

can currently been expressed as:

>>> iterable = range(10)
>>> f = lambda x: [x] * x
>>> [y for x in iterable if x % 2 == 0 for y in f(x)]
[2, 2, 4, 4, 4, 4, 6, 6, 6, 6, 6, 6, 8, 8, 8, 8, 8, 8, 8, 8]

Now I do like the new extension syntax. I find it more natural, and more
readable:

>>> [*f(x) for x in iterable if x % 2 == 0]

But it's not a missing feature, it's really just a (rather nice)
syntaxic improvement.

[אלעזר‎]

בתאריך יום ו׳, 14 באוק' 2016, 12:19, מאת Michel Desmoulin ‏<desmoul...@gmail.com>:

It is about lifting restrictions from an existing syntax. That this behavior is being *explicitly disabled* in the implementation is a strong evidence, in my mind.

(There are more restrictions I was asked not to divert this thread, which makes sense) 

Elazar

[Paul Moore]

On 14 October 2016 at 07:54, Greg Ewing <greg....@canterbury.ac.nz> wrote:
>> I think it's probably time for someone to
>> describe the precise syntax (as BNF, like the syntax in the Python
>> docs at
>> https://docs.python.org/3.6/reference/expressions.html#displays-for-lists-sets-and-dictionaries
>
>
> Replace
>
> comprehension ::= expression comp_for
>
> with
>
> comprehension ::= (expression | "*" expression) comp_for
>
>> and semantics (as an explanation of how to
>> rewrite any syntactically valid display as a loop).
>
>
> The expansion of the "*" case is the same as currently except
> that 'append' is replaced by 'extend' in a list comprehension,
> 'yield' is replaced by 'yield from' in a generator
> comprehension.

Thanks. That does indeed clarify. Part of my confusion was that I'm
sure I'd seen someone give an example along the lines of

[(x, *y, z) for ...]

which *doesn't* conform to the above syntax. OTOH, it is currently
valid syntax, just not an example of *this* proposal (that's part of
why all this got very confusing).

So now I understand what's being proposed, which is good. I don't
(personally) find it very intuitive, although I'm completely capable
of using the rules given to establish what it means. In practical
terms, I'd be unlikely to use or recommend it - not because of
anything specific about the proposal, just because it's "confusing". I
would say the same about [(x, *y, z) for ...].

IMO, combining unpacking and list (or other types of) comprehensions
leads to obfuscated code. Each feature is fine in isolation, but
over-enthusiastic use of the ability to combine them harms
readability. So I'm now -0 on this proposal. There's nothing *wrong*
with it, and I now see how it can be justified as a generalisation of
current rules. But I can't think of any real-world case where using
the proposed syntax would measurably improve code maintainability or
comprehensibility.

Paul

[Nick Coghlan]

On 13 October 2016 at 02:32, Sven R. Kunze <srk...@mail.de> wrote:
> Here I disagree with you. We use *args all the time, so we know what * does.
> I don't understand why this should not work in between brackets [...].

It does work between brackets:

>>> [*range(3)]
[0, 1, 2]

It doesn't work as part of the comprehension syntax, and that's the
case for function calls as well:

>>> f(*range(i) for i in range(3))

File "<stdin>", line 1

f(*range(i) for i in range(3))
^
SyntaxError: invalid syntax
>>> [*range(i) for i in range(3)]

File "<stdin>", line 1

SyntaxError: iterable unpacking cannot be used in comprehension

(With the less helpful error message in the function call case just
being due to the vagaries of CPython's parser and compiler
implementation, where things that don't even parse are just reported
as "invalid syntax", while problems detected later don't have the
helpful pointer to where in the line parsing failed, but do get a
better explanation of what actually went wrong)

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Steven D'Aprano]

On Fri, Oct 14, 2016 at 04:18:40AM +0100, MRAB wrote:
> On 2016-10-14 02:04, Steven D'Aprano wrote:
> >On Thu, Oct 13, 2016 at 08:15:36PM +0200, Martti Kühne wrote:
> >
> >>Can I fix my name, though?
> >
> >I don't understand what you mean. Your email address says your name is
> >Martti Kühne. Is that incorrect?
> >
> [snip]
>
> You wrote "Marttii" and he corrected it when he quoted you in his reply.

Ah, so I did! I'm sorry Martti, I read over my comment half a dozen
times and couldn't see the doubled "i". My apologies.


--
Steven

[Steven D'Aprano]

On Thu, Oct 13, 2016 at 05:30:49PM -0400, Random832 wrote:

> Frankly, I don't see why the pattern isn't obvious

*shrug*

Maybe your inability to look past your assumptions and see things from
other people's perspective is just as much a blind spot as our inability
to see why you think the pattern is obvious. We're *all* having
difficulty in seeing things from the other side's perspective here.

Let me put it this way: as far as I am concerned, sequence unpacking is
equivalent to manually replacing the sequence with its items:

t = (1, 2, 3)
[100, 200, *t, 300]

is equivalent to replacing "*t" with "1, 2, 3", which gives us:

[100, 200, 1, 2, 3, 300]

That's nice, simple, it makes sense, and it works in sufficiently recent
Python versions. It applies to function calls and assignments:

func(100, 200, *t) # like func(100, 200, 1, 2, 3)

a, b, c, d, e = 100, 200, *t # like a, b, c, d, e = 100, 200, 1, 2, 3

although it doesn't apply when the star is on the left hand side:

a, b, *x, e = 1, 2, 3, 4, 5, 6, 7

That requires a different model for starred names, but *that* model is
similar to its role in function parameters: def f(*args). But I digress.

Now let's apply that same model of "starred expression == expand the
sequence in place" to a list comp:

iterable = [t]

[*t for t in iterable]

If you do the same manual replacement, you get:

[1, 2, 3 for t in iterable]

which isn't legal since it looks like a list display [1, 2, ...]
containing invalid syntax. The only way to have this make sense is to
use parentheses:

[(1, 2, 3) for t in iterable]

which turns [*t for t in iterable] into a no-op.

Why should the OP's complicated, hard to understand (to many of us)
interpretation take precedence over the simple, obvious, easy to
understand model of sequence unpacking that I describe here?

That's not a rhetorical question. If you have a good answer, please
share it. But I strongly believe that on the evidence of this thread,

[a, b, *t, d]

is easy to explain, teach and understand, while:

[*t for t in iterable]

will be confusing, hard to teach and understand except as "magic syntax"
-- it works because the interpreter says it works, not because it
follows from the rules of sequence unpacking or comprehensions. It might
as well be spelled:

[ MAGIC!!!! HAPPENS!!!! HERE!!!! t for t in iterable]

except it is shorter.

Of course, ultimately all syntax is "magic", it all needs to be learned.
There's nothing about + that inherently means plus. But we should
strongly prefer to avoid overloading the same symbol with distinct
meanings, and * is one of the most heavily overloaded symbols in Python:

- multiplication and exponentiation
- wildcard imports
- globs, regexes
- collect arguments and kwargs
- sequence unpacking
- collect unused elements from a sequence

and maybe more. This will add yet another special meaning:

- expand the comprehension ("extend instead of append").

If we're going to get this (possibly useful?) functionality, I'd rather
see an explicit flatten() builtin, or see it spelled:

[from t for t in sequence]

which at least is *obviously* something magical, than yet another magic
meaning to the star operator. Its easy to look it up in the docs or
google for it, and doesn't look like Perlish line noise.



--
Steve

[Random832]

On Fri, Oct 14, 2016, at 22:38, Steven D'Aprano wrote:
> On Thu, Oct 13, 2016 at 05:30:49PM -0400, Random832 wrote:
>
> > Frankly, I don't see why the pattern isn't obvious
>
> *shrug*
>
> Maybe your inability to look past your assumptions and see things from
> other people's perspective is just as much a blind spot as our inability
> to see why you think the pattern is obvious. We're *all* having
> difficulty in seeing things from the other side's perspective here.
>
> Let me put it this way: as far as I am concerned, sequence unpacking is
> equivalent to manually replacing the sequence with its items:

And as far as I am concerned, comprehensions are equivalent to manually
creating a sequence/dict/set consisting of repeating the body of the
comprehension to the left of "for" with the iteration variable[s]
replaced in turn with each actual value.

> t = (1, 2, 3)
> [100, 200, *t, 300]
>
> is equivalent to replacing "*t" with "1, 2, 3", which gives us:
>
> [100, 200, 1, 2, 3, 300]

I don't understand why it's not _just as simple_ to say:

t = ('abc', 'def', 'ghi')
[*x for x in t]

is equivalent to replacing "x" in "*x" with, each in turn, 'abc', 'def',
and 'ghi', which gives us:

[*'abc', *'def', *'ghi']

just like [f(x) for x in t] would give you [f('abc'), f('def'),
f('ghi')]

> That's nice, simple, it makes sense, and it works in sufficiently recent
> Python versions.

That last bit is not an argument - every new feature works in
sufficiently recent python versions. The only difference for this
proposal (provided it is approved) is that the sufficiently recent
python versions simply don't exist yet.

[Steven D'Aprano]

On Thu, Oct 13, 2016 at 11:32:49PM -0400, Random832 wrote:
> On Thu, Oct 13, 2016, at 18:15, Steven D'Aprano wrote:
> > Consider the analogy with f(*t), where t = (a, b, c). We *don't* have:
> >
> > f(*t) is equivalent to f(a); f(b); f(c)
>
> I don't know where this "analogy" is coming from.

I'm explicitly saying that we DON'T have that behaviour with function
calls. f(*t) is NOT expanded to f(a), f(b), f(c). I even emphasised the
"don't" part of my sentence above.

And yet, this proposal wants to expand

[*t for t in iterable]

into the equivalent of:

result = []
for t in iterable:

a, b, c = *t
result.append(a)
result.append(b)
result.append(c)

Three separate calls to append, analogous to three separate calls to
f(). The point I am making is that this proposed change is *not*
analogous to the way sequence unpacking works in other contexts. I'm
sorry if I wasn't clear enough.


[...]

> > Indeed. The reader may be forgiven for thinking that this is yet another
> > unrelated and arbitrary use of * to join the many other uses:
>
> How is it arbitrary?

It is arbitrary because the suggested use of *t in list comprehensions
has no analogy to the use of *t in other contexts.

As far as I can see, this is not equivalent to the way sequence
(un)packing works on *either* side of assignment. It's not equivalent to
the way sequence unpacking works in function calls, or in list displays.
It's this magical syntax which turns a virtual append() into extend():

# [t for t in iterable]

result = []
for t in iterable:

result.append(t)

# but [*t for t in iterable]

result = []
for t in iterable:

result.extend(t)


or, if you prefer, keep the append but magical add an extra for-loop:

# [*t for t in iterable]

result = []
for t in iterable:

for x in t:
result.append(x)

> > - mathematical operator;
> > - glob and regex wild-card;
> > - unpacking;
>
> This is unpacking. It unpacks the results into the destination.

If it were unpacking as it is understood today, with no other changes,
it would be a no-op. (To be technical, it would convert whatever
iterable t is into a tuple.) I've covered that in an earlier post: if
you replace *t with the actual items of t, you DON'T get:

result = []
for t in iterable:

a, b, c = *t # assuming t has three items, as per above
result.append(a)
result.append(b)
result.append(c)

as desired, but:

result = []
for t in iterable:

a, b, c = *t
result.append((a, b, c))


which might as well be a no-op.

To make this work, the "unpacking operator" needs to do more than just
unpack. It has to either change append into extend, or equivalently, add
an extra for loop into the list comprehension.

> There's a straight line from [*t, *u, *v] to [*x for x in (t, u, v)].
> What's surprising is that it doesn't work now.

I'm not surprised that it doesn't work. I expected that it wouldn't
work. When I first saw the suggestion, I thought "That can't possibly be
meaningful, it should be an error."

Honestly Random832, I cannot comprehend how you see this as a
straightforward obvious extension from existing behaviour. To me, this
is nothing like the existing behaviour, and it contradicts the way
sequence unpacking works everywhere else.

I do not understand the reasoning you use to conclude that this is a
straight-line extension to the current behaviour. Nothing I have seen in
any of this discussion justifies that claim to me. I don't know what you
are seeing that I cannot see. My opinion is that you're seeing things
that aren't there -- I expect that your opinion is that I'm blind.

> I think last month we even had someone who didn't know about 'yield
> from' propose 'yield *x' for exactly this feature. It is intuitive - it
> is a straight-line extension of the unpacking syntax.

Except for all the folks who have repeatedly said that it is
counter-intuitive, that it is a twisty, unexpected, confusing path from
the existing behaviour to this proposal.



--
Steve

[Steven D'Aprano]

On Fri, Oct 14, 2016 at 08:29:28PM +1300, Greg Ewing wrote:
> Steven D'Aprano wrote:
>
> So why would yield *t give us this?
> >
> > yield a; yield b; yield c
> >
> >By analogy with the function call syntax, it should mean:
> >
> > yield (a, b, c)
>
> This is a false analogy, because yield is not a function.

Neither are list comprehensions or sequence unpacking in the context of
assignment:

a, b, c = *t

Not everything is a function. What's your point?

As far as I can see, in *every* other use of sequence unpacking, *t is
conceptually replaced by a comma-separated sequence of items from t. If
the starred item is on the left-hand side of the = sign, we might call
it "sequence packing" rather than unpacking, and it operates to collect
unused items, just like *args does in function parameter lists.

Neither of these are even close to what the proposed [*t for t in
iterable] will do.

> >>However, consider the following spelling:
> >>
> >> l = [from f(t) for t in iterable]
>
> That sentence no verb!
>
> In English, 'from' is a preposition, so one expects there
> to be a verb associated with it somewhere. We currently
> have 'from ... import' and 'yield from'.
>
> But 'from f(t) for t in iterable' ... do what?

*shrug*

I'm not married to this suggestion. It could be written
[MAGIC!!! HAPPENS!!! HERE!!! t for t in iterable] if you prefer.
The suggestion to use "from" came from Sjoerd Job Postmus, not me.


--
Steve

[Steven D'Aprano]

On Fri, Oct 14, 2016 at 07:51:18AM +0000, Neil Girdhar wrote:
> Here's an interesting idea regarding yield **x:
>
> Right now a function containing any yield returns a generator. Therefore,
> it works like a generator expression, which is the lazy version of a list
> display. lists can only contain elements x and unpackings *x. Therefore,
> it would make sense to only have "yield x" and "yield *xs" (currently
> spelled "yield from xs")

No, there's no "therefore" about it. "yield from x" is not the same as
"yield *x".


*x is conceptually equivalent to replacing "*x" with a
comma-separated sequence of individual items from x.

Given x = (1, 2, 3):


f(*x) is like f(1, 2, 3)

[100, 200, *x, 300] is like [100, 200, 1, 2, 3, 300]

a, b, c, d = 100, *x is like a, b, c, d = 100, 1, 2, 3

Now replace "yield *x" with "yield 1, 2, 3". Conveniently, that syntax
already works:

py> def gen():
... yield 1, 2, 3
...
py> it = gen()
py> next(it)
(1, 2, 3)


"yield *x" should not be the same as "yield from x". Yielding a starred
expression currently isn't allowed, but if it were allowed, it would be
pointless: it would be the same as unpacking x, then repacking it into a
tuple.

Either that, or we would have yet another special meaning for *
unrelated to the existing meanings.




--
Steve

[Random832]

On Sat, Oct 15, 2016, at 04:00, Steven D'Aprano wrote:
> > This is unpacking. It unpacks the results into the destination.
>
> If it were unpacking as it is understood today, with no other changes,
> it would be a no-op. (To be technical, it would convert whatever
> iterable t is into a tuple.)

If that were true, it would be a no-op everywhere.

> I've covered that in an earlier post: if
> you replace *t with the actual items of t, you DON'T get:

Replacing it _with the items_ is not the same thing as replacing it
_with a sequence containing the items_, and you're trying to pull a fast
one by claiming it is by using the fact that the "equivalent loop"
(which is and has always been a mere fiction, not a real transformation
actually performed by the interpreter) happens to use a sequence of
tokens that would cause a tuple to be created if a comma appears in the
relevant position.

> To make this work, the "unpacking operator" needs to do more than just
> unpack. It has to either change append into extend,

Yes, that's what unpacking does. In every context where unpacking means
anything at all, it does something to arrange for the sequence's
elements to be included "unbracketed" in the context it's being
ultimately used in. It's no different from changing BUILD_LIST
(equivalent to creating an empty list and appending each item) to
BUILD_LIST_UNPACK (equivalent to creating an empty list and extending
with each item).

Imagine that we were talking about ordinary list displays, and for some
reason had developed a tradition of explaining them in terms of
"equivalent" code the way we do for comprehensions.

x = [a, b, c] is equivalent to:
x = list()
x.append(a)
x.append(b)
x.append(c)

So now if we replace c with *c [where c == [d, e]], must we now say
this?
x = list()
x.append(a)
x.append(b)
x.append(d, e)

Well, that's just not valid at all. Clearly we must reject this
ridiculous notion of allowing starred expressions within list displays,
because we _can't possibly_ change the transformation to accommodate the
new feature.

[Steven D'Aprano]

On Fri, Oct 14, 2016 at 06:23:32PM +1300, Greg Ewing wrote:

> To maintain the identity
>
> list(*x for x in y) == [*x for x in y]
>
> it would be necessary for the *x in (*x for x in y) to expand
> to "yield from x".

Oh man, you're not even trying to be persuasive any more. You're just
assuming the result that you want, then declaring that it is
"necessary". :-(


I have a counter proposal: suppose *x is expanded to the string literal
"Nope!". Then, given y = (1, 2, 3) (say):

list(*x for x in y)

gives ["Nope!", "Nope!", "Nope!"], and

[*x for x in y]

also gives ["Nope!", "Nope!", "Nope!"]. Thus the identity is kept, and
your claim of "necessity" is disproven.

We already know what *x should expand to: nearly everywhere else, *x is
conceptually replaced by a comma-separated sequence of the items of x.
That applies to function calls, sequence unpacking and list displays.

The only exceptions I can think of are *args parameters in function
parameter lists, and sequence packing on the left side of an assignment,
both of which work in similar fashions.

But not this proposal: it wouldn't work like either of the above, hence
it would be yet another unrelated use of the * operator for some
special meaning.



--
Steve

[Steven D'Aprano]

On Thu, Oct 13, 2016 at 01:30:45PM -0700, Neil Girdhar wrote:

> From a CPython implementation standpoint, we specifically blocked this code
> path, and it is only a matter of unblocking it if we want to support this.

I find that difficult to believe. The suggested change seems like it
should be much bigger than just removing a block. Can you point us to
the relevant code?

In any case, it isn't really the difficulty of implementation that is
being questioned. Many things are easy to implement, but we still
don't do them. The real questions here are:

(1) Should we overload list comprehensions as sugar for a flatten()
function?

(2) If so, should we spell that [*t for t in iterable]?


Actually the answer to (1) should be "we already do". We just spell it:

[x for t in iterable for x in t]

[Martti Kühne]

On Sat, Oct 15, 2016 at 10:09 AM, Steven D'Aprano <st...@pearwood.info> wrote:
> Not everything is a function. What's your point?
>
> As far as I can see, in *every* other use of sequence unpacking, *t is
> conceptually replaced by a comma-separated sequence of items from t. If
> the starred item is on the left-hand side of the = sign, we might call
> it "sequence packing" rather than unpacking, and it operates to collect
> unused items, just like *args does in function parameter lists.
>

You brush over the fact that *t is not limited to a replacement by a
comma-separated sequence of items from t, but *t is actually a
replacement by that comma-separated sequence of items from t INTO an
external context. For func(*t) to work, all the elements of t are kind
of "leaked externally" into the function argument list's context, and
for {**{'a': 1, 'b': 2, ...}} the inner dictionary's items are kind of
"leaked externally" into the outer's context.

You can think of the */** operators as a promotion from append to
extend, but another way to see this is as a promotion from yield to
yield from. So if you want to instead of append items to a
comprehension, as is done with [yield_me for yield_me in iterator],
you can see this new piece as a means to [*yield_from_me for
yield_from_me in iterator]. FWIW, I think it's a bit confusing
that yield needs a different keyword if these asterisk operators
already have this outspoken promotion effect.

Besides, [*thing for thing in iterable_of_iters if cond] has this cool
potential for the existing any() and all() builtins for cond, where a
decision can be made based on the composition of the in itself
iterable thing.

cheers!
mar77i

[Martti Kühne]

On Sat, Oct 15, 2016 at 10:09 AM, Steven D'Aprano <st...@pearwood.info> wrote:

> Not everything is a function. What's your point?
>
> As far as I can see, in *every* other use of sequence unpacking, *t is
> conceptually replaced by a comma-separated sequence of items from t. If
> the starred item is on the left-hand side of the = sign, we might call
> it "sequence packing" rather than unpacking, and it operates to collect
> unused items, just like *args does in function parameter lists.
>

You brush over the fact that *t is not limited to a replacement by a
comma-separated sequence of items from t, but *t is actually a
replacement by that comma-separated sequence of items from t INTO an
external context. For func(*t) to work, all the elements of t are kind
of "leaked externally" into the function argument list's context, and
for {**{'a': 1, 'b': 2, ...}} the inner dictionary's items are kind of
"leaked externally" into the outer's context.

You can think of the */** operators as a promotion from append to
extend, but another way to see this is as a promotion from yield to
yield from. So if you want to instead of append items to a
comprehension, as is done with [yield_me for yield_me in iterator],
you can see this new piece as a means to [*yield_from_me for

yield_from_me in iterator]. Therefore I think it's a bit confusing

that yield needs a different keyword if these asterisk operators

already have this intuitive promotion effect.

Besides, [*thing for thing in iterable_of_iters if cond] has this cool
potential for the existing any() and all() builtins for cond, where a
decision can be made based on the composition of the in itself
iterable thing.

cheers!
mar77i

[Greg Ewing]

Steven D'Aprano wrote:

> t = (1, 2, 3)

> iterable = [t]
> [*t for t in iterable]
>
> If you do the same manual replacement, you get:
>
> [1, 2, 3 for t in iterable]

Um, no, you need to also *remove the for loop*, otherwise
you get complete nonsense, whether * is used or not.

Let's try a less degenerate example, both ways.

iterable = [1, 2, 3]
[t for t in iterable]

To expand that, we replace t with each of the values
generated by the loop and put commas between them:

[1, 2, 3]

Now with the star:

iterable = [(1, 2, 3), (4, 5, 6), (7, 8, 9)]

[*t for t in iterable]

Replace *t with each of the sequence generated by the
loop, with commas between:

[1,2,3 , 4,5,6 , 7,8,9]

> Maybe your inability to look past your assumptions and see things from
> other people's perspective is just as much a blind spot as our inability
> to see why you think the pattern is obvious.

It's obvious that you're having difficulty seeing what
we're seeing, but I don't know how to explain it any
more clearly, I'm sorry.

--
Greg

[Greg Ewing]

Martti Kühne wrote:
> You brush over the fact that *t is not limited to a replacement by a
> comma-separated sequence of items from t, but *t is actually a
> replacement by that comma-separated sequence of items from t INTO an
> external context.

Indeed. In situations where there isn't any context for
the interpretation of *, it's not allowed. For example:

>>> x = *(1, 2, 3)

File "<stdin>", line 1

SyntaxError: can't use starred expression here

But

>>> x = 1, *(2, 3)
>>> x
(1, 2, 3)

The * is allowed there because it's already in a context
where a comma-separated list has meaning.

--
Greg

[Steven D'Aprano]

On Sat, Oct 15, 2016 at 04:42:13AM -0400, Random832 wrote:
> On Sat, Oct 15, 2016, at 04:00, Steven D'Aprano wrote:
> > > This is unpacking. It unpacks the results into the destination.
> >
> > If it were unpacking as it is understood today, with no other changes,
> > it would be a no-op. (To be technical, it would convert whatever
> > iterable t is into a tuple.)
>
> If that were true, it would be a no-op everywhere.

That's clearly not the case.

x = (1, 2, 3)

[100, 200, *x, 300]

If you do it the way I say, and replace *x with the individual items of
x, you get this:

[100, 200, 1, 2, 3, 300]

which conveniently happens to be what you already get in Python. You
claim that if I were write it should be a no-op -- that doesn't follow.
Why would it be a no-op? I've repeatedly shown the transformation to
use, and it clearly does what I say it should. How could it not?

> > I've covered that in an earlier post: if
> > you replace *t with the actual items of t, you DON'T get:
>
> Replacing it _with the items_ is not the same thing as replacing it
> _with a sequence containing the items_,

I don't think I ever used the phrasing "a sequence containing the
items". I think that's *your* phrase, not mine.

I may have said "with the sequence of items" or words to that effect.
These two phrases do have different meanings:

x = (1, 2, 3)

[100, 200, *x, 300]

# Replace *x with "a sequence containing items of x"
[100, 200, [1, 2, 3], 300]

# Replace *x with "the sequence of items of x"

[100, 200, 1, 2, 3, 300]

Clearly they have different meanings. I'm confident that I've always
made it clear that I'm referring to the second, not the first, but I'm
only human and if I've been unclear or used the wrong phrasing, my
apologies.

But nit-picking about the exact phrasing used aside, it is clear that
expanding the *t in a list comprehension:

[*t for t in iterable]

to flatten the iterable cannot be analogous to this. Whatever
explanation you give for why *t expands the list comprehension, it
cannot be given in terms of replacing *t with the items of t. There has
to be some magic to give it the desired special behaviour.

> and you're trying to pull a fast
> one by claiming it is by using the fact that the "equivalent loop"
> (which is and has always been a mere fiction, not a real transformation
> actually performed by the interpreter) happens to use a sequence of
> tokens that would cause a tuple to be created if a comma appears in the
> relevant position.

I don't know what "equivalent loop" you are accusing me of misusing.

The core developers have made it absolutely clear that changing the
fundamental equivalence of list comps as syntactic sugar for:

result = []
for t in iterable:
result.append(t)

is NOT NEGOTIABLE. (That is much to my disappointment -- I would love to
introduce a "while" version of list comps to match the "if" version, but
that's not an option.)

So regardless of whether it is a fiction or an absolute loop, Python's
list comprehensions are categorically limited to behave equivalently to
the loop above (modulo scope, temporary variables, etc). If you want to
change that -- change the append to an extend, for example -- you need
to make a case for that change which is strong enough to overcome
Guido's ruling. (Maybe Guido will be willing to bend his ruling to allow
extend as well.)


There are three ways to get the desired flatten() behaviour from
a list comp. One way is to explicitly add a second loop, which has the
benefit of already working:

[x for t in iterable for x in t]


Another is to swap out the append for an extend:

[*t for t in iterable]

# real or virtual transformation, it doesn't matter

result = []
for t in iterable:
result.extend(t)

And the third is to keep the append but insert an extra virtual loop:

# real or virtual transformation, it still doesn't matter

result = []
for t in iterable:
for x in t:
result.append(x)

Neither of the second or third suggestions match the equivalent loop
form given above. Neither the second nor third is an obvious extension
of the way sequence unpacking works in other contexts.



[...]

> Imagine that we were talking about ordinary list displays, and for some
> reason had developed a tradition of explaining them in terms of
> "equivalent" code the way we do for comprehensions.
>
> x = [a, b, c] is equivalent to:
> x = list()
> x.append(a)
> x.append(b)
> x.append(c)
>
> So now if we replace c with *c [where c == [d, e]], must we now say
> this?
> x = list()
> x.append(a)
> x.append(b)
> x.append(d, e)
>
> Well, that's just not valid at all.

Right. And if we had a tradition of saying that list displays MUST be
equivalent to the unpacked sequence of appends, then sequence unpacking
inside a list display would be prohibited. But we have no such
tradition, and sequence unpacking inside the list really is an obvious
straight line extrapolation from (say) sequence unpacking inside a
function call.

Fortunately, we have a *different* tradition when it comes to list
displays, and no ruling that *c must turn into append with multiple
arguments. Our explanation of [a, b, *c] occurs at an earlier level:
replace the *c with the items of c:

c = [d, e]
[a, b, *c] ==> [a, b, d, e]

And there is no problem.

Unfortuantely for you, none of this is the case for list comps. We DO
have a tradition and a BDFL ruling that list comps are strictly
equivalent to a loop with append.

And the transformation of *t for the items of t (I don't care if it is a
real transformation in the implementation, or only a fictional
transformation) cannot work in a list comp. Let's make the number of
items of t explicit so we don't have to worry about variable item
counts:

[*t for t in iterable] # t has three items
[a, b, c for (a, b, c) in iterable]


That's a syntax error. To avoid the syntax error, we need parentheses:

[(a, b, c) for (a, b, c) in iterable]

and that's a no-op. So we're back to my first response to this thread:
why on earth would you expect *t in a list comprehension to flatten the
iterable? It should be either an error, or a no-op.

> Clearly we must reject this
> ridiculous notion of allowing starred expressions within list displays,
> because we _can't possibly_ change the transformation to accommodate the
> new feature.

Of course we can. I've repeatedly said we can do anything we want. If we
want, we can have *t in a list comprehension be sugar for importing the
sys module, or erasing your home directory. What we can't say is that
"erasing your home directory" is an obvious straight-line extrapolation
from existing uses of the star operator. There's nothing obvious here:
this thread is proof that whatever connection (if any) between the two
is non-obvious, twisted, even strange and bizarre.

I have never argued against this suggested functionality: flattening
iterables is obviously a useful thing to do. But:

- we can already use a list comp to flatten:

[x for t in iterable for x in t]


- there's no obvious or clear connection between the *t in the suggested
syntax and existing uses of the star operator; it might as well be
spelled [magic!!!! t for t in iterable] for all the relevance sequence
unpacking has;

- if anyone can explain the connection they see, I'm listening;

(believe me, I am *trying to understand* -- but none of the given
explanations for a connection hold up as far as I am concerned)

- even if we agree that there is a connection, this thread is
categorical proof that it is not obvious: it has taken DOZENS of
emails to (allegedly) get the message across;

- if we do get syntactic sugar for flatten(), why does it have to
overload the star operator for yet another meaning?

Hence my earlier questions: do we really need this, and if so, does it
have to be spelled *t? Neither of those questions are obviously answered
with a "Yes".



--
Steve

[Steven D'Aprano]

On Sat, Oct 15, 2016 at 11:36:28PM +1300, Greg Ewing wrote:

> Indeed. In situations where there isn't any context for
> the interpretation of *, it's not allowed.

You mean like in list comprehensions?

Are you now supporting my argument that starring the list comprehension
expression isn't meaningful? Not if star is defined as sequence
unpacking in the usual way. If you want to invent a new meaning for * to
make this work, to join all the other special case magic meanings for
the * symbol, that's another story.

> For example:
>
> >>> x = *(1, 2, 3)
> File "<stdin>", line 1
> SyntaxError: can't use starred expression here
>
> But
>
> >>> x = 1, *(2, 3)
> >>> x
> (1, 2, 3)
>
> The * is allowed there because it's already in a context
> where a comma-separated list has meaning.

Oh look, just like now:

py> iterable = [(1, 'a'), (2, 'b')]
py> [(100, *t) for t in iterable]
[(100, 1, 'a'), (100, 2, 'b')]

Hands up anyone who expected to flatten the iterable and get

[100, 1, 'a', 100, 2, 'b']

instead? Anyone? No?

Take out the (100, ...) and just leave the *t, and why should it be
different? It is my position that:

(quote)

there isn't any context for the interpretation of *

for [*t for t in iterable]. Writing that is the list comp equivalent of
writing x = *t.


--
Steve

[Martti Kühne]

On Sat, Oct 15, 2016 at 12:48 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> Oh look, just like now:
>
> py> iterable = [(1, 'a'), (2, 'b')]
> py> [(100, *t) for t in iterable]
> [(100, 1, 'a'), (100, 2, 'b')]
>
> Hands up anyone who expected to flatten the iterable and get
>
> [100, 1, 'a', 100, 2, 'b']
>
> instead? Anyone? No?
>

I don't know whether that should be provocating or beside the poinnt.
It's probably both. You're putting two expectations on the same
example: first, you make the reasonable expectation that results in
[(100, 1, 'a'), (100, 2, 'b')], and then you ask whether anyone
expected [100, 1, 'a', 100, 2, 'b'], but don't add or remove anything
from the same example. Did you forget to put a second example using
the new notation in there?
Then you'd have to spell it out and start out with [*(100, *t) for t
in iterable]. And then you can ask who expected [100, 1, 'a', 100, 2,
'b']. Which is what this thread is all about.

cheers!
mar77i

[Paul Moore]

On 14 October 2016 at 10:48, Paul Moore <p.f....@gmail.com> wrote:
> On 14 October 2016 at 07:54, Greg Ewing <greg....@canterbury.ac.nz> wrote:
>>> I think it's probably time for someone to
>>> describe the precise syntax (as BNF, like the syntax in the Python
>>> docs at
>>> https://docs.python.org/3.6/reference/expressions.html#displays-for-lists-sets-and-dictionaries
>>
>>
>> Replace
>>
>> comprehension ::= expression comp_for
>>
>> with
>>
>> comprehension ::= (expression | "*" expression) comp_for
>>
>>> and semantics (as an explanation of how to
>>> rewrite any syntactically valid display as a loop).
>>
>>
>> The expansion of the "*" case is the same as currently except
>> that 'append' is replaced by 'extend' in a list comprehension,
>> 'yield' is replaced by 'yield from' in a generator
>> comprehension.

[...]

> So now I understand what's being proposed, which is good. I don't
> (personally) find it very intuitive, although I'm completely capable
> of using the rules given to establish what it means. In practical
> terms, I'd be unlikely to use or recommend it - not because of
> anything specific about the proposal, just because it's "confusing". I
> would say the same about [(x, *y, z) for ...].

Thinking some more about this, is it not true that

[ *expression for var in iterable ]

is the same as

[ x for var in iterable for x in expression ]

?

If so, then this proposal adds no new expressiveness, merely a certain
amount of "compactness". Which isn't necessarily a bad thing, but it's
clearly controversial whether the compact version is more readable /
"intuitive" in this case. Given the lack of any clear improvement, I'd
be inclined to think that "explicit is better than implicit" applies
here, and reject the new proposal.

Paul.

[Martti Kühne]

On Sat, Oct 15, 2016 at 3:06 PM, Paul Moore <p.f....@gmail.com> wrote:
> is the same as
>
> [ x for var in iterable for x in expression ]
>

correction, that would be:

[var for expression in iterable for var in expression]

you are right, though. List comprehensions are already stackable.
TIL.

cheers!
mar77i

[Random832]

On Sat, Oct 15, 2016, at 06:38, Steven D'Aprano wrote:
> > Replacing it _with the items_ is not the same thing as replacing it
> > _with a sequence containing the items_,
>
> I don't think I ever used the phrasing "a sequence containing the
> items". I think that's *your* phrase, not mine.

It's not your phrasing, it's the actual semantic content of your claim
that it would have to wrap them in a tuple.

> The core developers have made it absolutely clear that changing the
> fundamental equivalence of list comps as syntactic sugar for:
>
> result = []
> for t in iterable:
> result.append(t)
>
>
> is NOT NEGOTIABLE.

I've never heard of this. It certainly never came up in this discussion.
And it was negotiable just fine when they got rid of the leaked loop
variable.

> (That is much to my disappointment -- I would love to
> introduce a "while" version of list comps to match the "if" version, but
> that's not an option.)
>
> So regardless of whether it is a fiction or an absolute loop, Python's
> list comprehensions are categorically limited to behave equivalently to
> the loop above (modulo scope, temporary variables, etc).

See, there it is. Why are *those* things that are allowed to be
differences, but this (which could be imagined as "result += [t]" if you
_really_ need a single statement where the left-hand clause is
substituted in, or otherwise) is not?

[אלעזר‎]

On Sat, Oct 15, 2016 at 1:49 PM Steven D'Aprano <st...@pearwood.info> wrote:

...

And the transformation of *t for the items of t (I don't care if it is a
real transformation in the implementation, or only a fictional
transformation) cannot work in a list comp. Let's make the number of
items of t explicit so we don't have to worry about variable item
counts:

    [*t for t in iterable]  # t has three items
    [a, b, c for (a, b, c) in iterable]


That's a syntax error. To avoid the syntax error, we need parentheses:

    [(a, b, c) for (a, b, c) in iterable]

and that's a no-op.

You are confusing here two distinct roles of the parenthesis: disambiguation as in "(1 + 2) * 2", and tuple construction as in (1, 2, 3). This overload is the reason that (1) is not a 1-tuple and we must write (1,).

You may argue that this overloading causes confusion and make this construct hard to understand, but please be explicit about that; even if <1, 2,3 > was the syntax for tuples, the expansion was still

 [(a, b, c) for (a, b, c) in iterable]

Since no tuple is constructed here.

 

Elazar

[Chris Angelico]

On Sun, Oct 16, 2016 at 4:33 AM, אלעזר <ela...@gmail.com> wrote:
> You are confusing here two distinct roles of the parenthesis: disambiguation
> as in "(1 + 2) * 2", and tuple construction as in (1, 2, 3). This overload
> is the reason that (1) is not a 1-tuple and we must write (1,).

Parentheses do not a tuple make. Commas do.

1, 2, 3, # three-element tuple
1, 2, # two-element tuple
1, # one-element tuple

The only time that a tuple requires parens is when it's the empty tuple, ().

ChrisA

[Neil Girdhar]

On Sat, Oct 15, 2016 at 5:01 AM Steven D'Aprano <st...@pearwood.info> wrote:

On Thu, Oct 13, 2016 at 01:30:45PM -0700, Neil Girdhar wrote:

> From a CPython implementation standpoint, we specifically blocked this code
> path, and it is only a matter of unblocking it if we want to support this.

I find that difficult to believe. The suggested change seems like it
should be much bigger than just removing a block. Can you point us to
the relevant code?

The Grammar specifies:

dictorsetmaker: ( ((test ':' test | '**' expr)

                   (comp_for | (',' (test ':' test | '**' expr))* [','])) |

                  ((test | star_expr)

                   (comp_for | (',' (test | star_expr))* [','])) )

In ast.c, you can find:

                if (is_dict) {

                    ast_error(c, n, "dict unpacking cannot be used in "

                            "dict comprehension");

                    return NULL;

                }

                res = ast_for_dictcomp(c, ch);

and ast_for_dictcomp supports dict unpacking.

Similarly:

    if (elt->kind == Starred_kind) {

        ast_error(c, ch, "iterable unpacking cannot be used in comprehension");

        return NULL;

    }

    comps = ast_for_comprehension(c, CHILD(n, 1));

and ast_for_comprehensions supports iterable unpacking.

In any case, it isn't really the difficulty of implementation that is
being questioned. Many things are easy to implement, but we still
don't do them.

If it doesn't matter, why bring it up?

 

The real questions here are:

(1) Should we overload list comprehensions as sugar for a flatten()
function?

(2) If so, should we spell that [*t for t in iterable]?


Actually the answer to (1) should be "we already do". We just spell it:

    [x for t in iterable for x in t]



--
Steve
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

--

---
You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/ROYNN7a5VAc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-ideas...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[אלעזר‎]

On Sat, Oct 15, 2016 at 8:36 PM Chris Angelico <ros...@gmail.com> wrote:

On Sun, Oct 16, 2016 at 4:33 AM, אלעזר <ela...@gmail.com> wrote:
> You are confusing here two distinct roles of the parenthesis: disambiguation
> as in "(1 + 2) * 2", and tuple construction as in (1, 2, 3). This overload
> is the reason that (1) is not a 1-tuple and we must write (1,).

Parentheses do not a tuple make. Commas do.

1, 2, 3, # three-element tuple
1, 2, # two-element tuple
1, # one-element tuple

And what [1, 2, 3] means? It's very different from [(1,2,3)].

Python explicitly allow 1, 2, 3 to mean tuple in certain contexts, I agree.

 

Elazar

[Chris Angelico]

Square brackets create a list. I'm not sure what you're not understanding, here.

The comma does have other meanings in other contexts (list/dict/set
display, function parameters), but outside of those, it means "create
tuple".

[אלעזר‎]

On Sat, Oct 15, 2016 at 8:45 PM Chris Angelico <ros...@gmail.com> wrote:

On Sun, Oct 16, 2016 at 4:38 AM, אלעזר <ela...@gmail.com> wrote:
> On Sat, Oct 15, 2016 at 8:36 PM Chris Angelico <ros...@gmail.com> wrote:
>>
>> On Sun, Oct 16, 2016 at 4:33 AM, אלעזר <ela...@gmail.com> wrote:
>> > You are confusing here two distinct roles of the parenthesis:
>> > disambiguation
>> > as in "(1 + 2) * 2", and tuple construction as in (1, 2, 3). This
>> > overload
>> > is the reason that (1) is not a 1-tuple and we must write (1,).
>>
>> Parentheses do not a tuple make. Commas do.
>>
>> 1, 2, 3, # three-element tuple
>> 1, 2, # two-element tuple
>> 1, # one-element tuple
>>
> And what [1, 2, 3] means? It's very different from [(1,2,3)].
>
> Python explicitly allow 1, 2, 3 to mean tuple in certain contexts, I agree.
>

Square brackets create a list. I'm not sure what you're not understanding, here.

The comma does have other meanings in other contexts (list/dict/set
display, function parameters), but outside of those, it means "create
tuple".

On a second thought you may decide whether the rule is tuple and there are exceptions, or the other way around.

The point was, conceptual expansion does not "fail" just because there is an overloading in the meaning of the tokens ( and ). It might make it harder to understand, though.

 

Elazar

[Sven R. Kunze]

On 15.10.2016 16:47, Martti Kühne wrote:
> [var for expression in iterable for var in expression]
> you are right, though. List comprehensions are already stackable.
> TIL.

Good catch, Paul. Comprehensions appear to be a special case when it
comes to unpacking as they provide an alternative path. So, nested
comprehensions seem to unintuitive to those who actually favor the
*-variant. ;) Anyway, I don't think that it's a strong argument against
the proposal. ~10 other ways are available to do what * does and this
kind of argument did not prevent PEP448.


What's more (and which I think is a more important response to the
nested comprehension alternative) is that nested comprehensions are
rarely used, and usually get long quite easily. To be practical here,
let's look at an example I remembered this morning (taken from
real-world code I needed to work with lately):

return [(language, text) for language, text in fulltext_tuples]

That's the minimum comprehension. So, you need to make it longer already
to do **actual** work like filtering or mapping (otherwise, just return
fulltext_tuples). So, we go even longer (and/or less readable):

return [t for t in tuple for tuple in fulltext_tuples if tuple[0] ==
'english']
return chain.from_iterable((language, text) for language, text in
fulltext_tuples if language == 'english'])

I still think the * variant would have its benefits here:

return [*(language, text) for language, text in fulltext_tuples if
language == 'english']

(Why it should be unpacked, you wonder? It's because of executemany of
psycopg2.]

Cheers,
Sven

[Greg Ewing]

Steven D'Aprano wrote:
> Are you now supporting my argument that starring the list comprehension
> expression isn't meaningful?

The context it's in (a form of list display) has a clear
meaning for a comma-separated list of values, so there
is a reasonable interpretation that it *could* be given.

> py> iterable = [(1, 'a'), (2, 'b')]
> py> [(100, *t) for t in iterable]
> [(100, 1, 'a'), (100, 2, 'b')]

The * there is in the context of constructing a tuple,
not the list into which the tuple is placed.

The difference is the same as the difference between
these:

>>> t = (10, 20)
>>> [1, (2, *t), 3]
[1, (2, 10, 20), 3]
>>> [1, 2, *t, 3]
[1, 2, 10, 20, 3]

--
Greg

[Steven D'Aprano]

On Sun, Oct 16, 2016 at 12:48:36PM +1300, Greg Ewing wrote:
> Steven D'Aprano wrote:
> >Are you now supporting my argument that starring the list comprehension
> >expression isn't meaningful?
>
> The context it's in (a form of list display) has a clear
> meaning for a comma-separated list of values, so there
> is a reasonable interpretation that it *could* be given.

This thread is a huge, multi-day proof that people do not agree that
this is a "reasonable" interpretation.

> >py> iterable = [(1, 'a'), (2, 'b')]
> >py> [(100, *t) for t in iterable]
> >[(100, 1, 'a'), (100, 2, 'b')]
>
> The * there is in the context of constructing a tuple,
> not the list into which the tuple is placed.

Right: the context of the star is meaningful. We all agree that *t in a
list display [a, b, c, ...] is meaningful; same for tuples; same for
function calls; same for sequence unpacking for assignment.

What is not meaningful (except as a Perlish line-noise special case to
be memorised) is *t as the list comprehension expression.

I've never disputed that we could *assert* that *t in a list comp means
"flatten". We could assert that it means anything we like. But it
doesn't follow from the usual meaning of sequence unpacking anywhere
else -- that's why it is currently a SyntaxError, and that's why people
reacted with surprise at the OP who assumed that *t would magically
flatten his iterable. Why would you assume that? It makes no sense to me
-- that's not how sequence unpacking works in any other context, it
isn't how list comprehensions work.

Right from the beginning I called this "wishful thinking", and *nothing*
since then has changed my mind. This proposal only makes even a little
bit of sense if you imagine list comprehensions

[*t for a in it1 for b in it2 for c in it3 ... for t in itN]

completely unrolled into a list display:

[*t, *t, *t, *t, ... ]

but who does that? Why would you reason about your list comps like that?
If you think about list comps as we're expected to think of them -- as
list builders equivalent to a for-loop -- the use of *t there is
invalid. Hence it is a SyntaxError.

You want a second way to flatten your iterables? A cryptic, mysterious,
Perlish line-noise way? Okay, fine, but don't pretend it is sequence
unpacking -- in the context of a list comprehension, sequence unpacking
doesn't make sense, it is invalid. Call it something else: the new
"flatten" operator:

[^t for t in iterable]

for example, which magically adds an second invisible for-loop to your
list comps:

# expands to
for t in iterable:
for x in t:
result.append(x)

Because as your own email inadvertently reinforces, if sequence
unpacking made sense in the context of a list comprehension, it would
already be allowed rather than a SyntaxError: it is intentionally
prohibited because it doesn't make sense in the context of list comps.


--
Steve

[Steven D'Aprano]

On Sun, Oct 16, 2016 at 04:36:05AM +1100, Chris Angelico wrote:
> On Sun, Oct 16, 2016 at 4:33 AM, אלעזר <ela...@gmail.com> wrote:
> > You are confusing here two distinct roles of the parenthesis: disambiguation
> > as in "(1 + 2) * 2", and tuple construction as in (1, 2, 3). This overload
> > is the reason that (1) is not a 1-tuple and we must write (1,).
>
> Parentheses do not a tuple make. Commas do.
>
> 1, 2, 3, # three-element tuple
> 1, 2, # two-element tuple
> 1, # one-element tuple
>
> The only time that a tuple requires parens is when it's the empty tuple, ().

Or to disambiguate a tuple from some other comma-separated syntax. Hence
why you need the parens here:

[(b, a) for a,b in sequence]

--
Steve

[David Mertz]

On Oct 15, 2016 6:42 PM, "Steven D'Aprano" <st...@pearwood.info> wrote:
> doesn't make sense, it is invalid. Call it something else: the new
> "flatten" operator:
>
>     [^t for t in iterable]
>
> for example, which magically adds an second invisible for-loop to your list comps:

This thread is a lot of work to try to save 8 characters in the spelling of `flatten(it)`. Let's just use the obvious and intuitive spelling.

We really don't need to be Perl. Folks who want to write Perl have a perfectly good interpreter available already.

The recipes in itertools give a nice implementation:

def flatten(listOfLists):
    "Flatten one level of nesting"
    return chain.from_iterable(listOfLists)

[Chris Angelico]

On Sun, Oct 16, 2016 at 12:10 PM, Steven D'Aprano <st...@pearwood.info> wrote:
> On Sun, Oct 16, 2016 at 04:36:05AM +1100, Chris Angelico wrote:
>> On Sun, Oct 16, 2016 at 4:33 AM, אלעזר <ela...@gmail.com> wrote:
>> > You are confusing here two distinct roles of the parenthesis: disambiguation
>> > as in "(1 + 2) * 2", and tuple construction as in (1, 2, 3). This overload
>> > is the reason that (1) is not a 1-tuple and we must write (1,).
>>
>> Parentheses do not a tuple make. Commas do.
>>
>> 1, 2, 3, # three-element tuple
>> 1, 2, # two-element tuple
>> 1, # one-element tuple
>>
>> The only time that a tuple requires parens is when it's the empty tuple, ().
>
> Or to disambiguate a tuple from some other comma-separated syntax. Hence
> why you need the parens here:
>
> [(b, a) for a,b in sequence]

Yes, in the same way that other operators can need to be
disambiguated. You need to say (1).bit_length() because otherwise "1."
will be misparsed. You need parens to say x = (yield 5) + 2, else it'd
yield 7. But that's not because a tuple fundamentally needs
parentheses.

ChrisA

[Greg Ewing]

Steven D'Aprano wrote:

> This thread is a huge, multi-day proof that people do not agree that
> this is a "reasonable" interpretation.

So far I've seen one very vocal person who disgrees, and
maybe one other who isn't sure.

> This proposal only makes even a little
> bit of sense if you imagine list comprehensions
>
> [*t for a in it1 for b in it2 for c in it3 ... for t in itN]
>
> completely unrolled into a list display:
>
> [*t, *t, *t, *t, ... ]
>
> but who does that? Why would you reason about your list comps like that?

Many people do, and it's a perfectly valid way to think
about them. They're meant to admit a declarative reading;
that's the reason they exist in the first place.

The expansion in terms of for-loops and appends is just
*one* way to describe the current semantics. It's not
written on stone tablets brought down from a mountain.
Any other way of thinking about it that gives the same
result is equally valid.

> magically adds an second invisible for-loop to your
> list comps:

You might as well say that the existing * in a list
display magically inserts a for-loop into it. You can
think of it that way if you want, but you don't have
to.

> it is intentionally
> prohibited because it doesn't make sense in the context of list comps.

I don't know why it's currently prohibited. You would
have to ask whoever put that code in, otherwise you're
just guessing about the motivation.

--
Greg

[Chris Angelico]

On Sun, Oct 16, 2016 at 3:44 PM, Greg Ewing <greg....@canterbury.ac.nz> wrote:
> Steven D'Aprano wrote:
>
>> This thread is a huge, multi-day proof that people do not agree that this
>> is a "reasonable" interpretation.
>
>
> So far I've seen one very vocal person who disgrees, and
> maybe one other who isn't sure.
>

And what you're NOT seeing is a whole lot of people (myself included)
who have mostly glazed over, unsure what is and isn't reasonable, and
not clear enough on either side of the debate to weigh in. (Or not
even clear what the two sides are.)

ChrisA