[Python-ideas] Temporary variables in comprehensions

[fhsxfhsx]

As far as I can see, a comprehension like
alist = [f(x) for x in range(10)]
is better than a for-loop
for x in range(10):
  alist.append(f(x))
because the previous one shows every element of the list explicitly so that we don't need to handle `append` mentally.

But when it comes to something like
[f(x) + g(f(x)) for x in range(10)]
you find you have to sacrifice some readableness if you don't want two f(x) which might slow down your code.

Someone may argue that one can write
[y + g(y) for y in [f(x) for x in range(10)]]
but it's not as clear as to show what `y` is in a subsequent clause, not to say there'll be another temporary list built in the process.
We can even replace every comprehension with map and filter, but that would face the same problems.

In a word, what I'm arguing is that we need a way to assign temporary variables in a comprehension.
In my opinion, code like
[y + g(y) for x in range(10) **some syntax for `y=f(x)` here**]
is more natural than any solution we now have.
And that's why I pro the new syntax, it's clear, explicit and readable, and is nothing beyond the functionality of the present comprehensions so it's not complicated.

And I hope the discussion could focus more on whether we should allow assigning temporary variables in comprehensions rather than how to solve the specific example I mentioned above.

 

[Jamie Willis]

I +1 this at surface level; Both Haskell list comprehensions and Scala for comprehensions have variable assignment in them, even between iterating and this is often very useful. Perhaps syntax can be generalised as:

[expr_using_x_and_y

 for i in is

  x = expr_using_i

 for j in is

  y = expr_using_j_and_x]

This demonstrates the scope of each assignment; available in main result and then every clause that follows it. 

Sorry to op who will receive twice, forgot reply to all

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Evpok Padding]

For simple cases such as `[y + g(y) for y in [f(x) for x in range(10)]]`,
I don't really see what the issue is, if you really want to make it shorter,

you can ``[y + g(y) for y in map(f,range(10))]` which is one of the rare

case where I like `map` more than comprehensions.

For more complex case, just define a intermediate generator along the lines

```

f_samples = (f(x) for x in range(10))

[y+g(y) for y in f_samples]

```

Which does exactly the same thing but

  - Is more readable and explicit

  - Has no memory overhead thanks to lazy evaluation

    (btw, you should consider generators for your nested comprenshions)

While I am sometimes in the same state of mind, wishing for variables in

comprehensions seems to me like a good indicator that your code needs

refactoring.

Best,

E

[Nils Bruin]

On Thursday, February 15, 2018 at 7:03:31 AM UTC, fhsxfhsx wrote:

In a word, what I'm arguing is that we need a way to assign temporary variables in a comprehension.
In my opinion, code like
[y + g(y) for x in range(10) **some syntax for `y=f(x)` here**]
is more natural than any solution we now have.

There is already a syntax that has that effect:

[y+g(x) for x in range(10) for y in (f(x),) ]

[Paul Moore]

On 15 February 2018 at 05:56, fhsxfhsx <fhsx...@126.com> wrote:
> As far as I can see, a comprehension like
> alist = [f(x) for x in range(10)]
> is better than a for-loop
> for x in range(10):
> alist.append(f(x))
> because the previous one shows every element of the list explicitly so that
> we don't need to handle `append` mentally.

... as long as the code inside the comprehension remains relatively
simple. It's easy to abuse comprehensions to the point where they are
less readable than a for loop, but that's true of a lot of things, so
isn't a specific problem with comprehensions.

> But when it comes to something like
> [f(x) + g(f(x)) for x in range(10)]
> you find you have to sacrifice some readableness if you don't want two f(x)
> which might slow down your code.

Agreed. I hit that quite often.

> Someone may argue that one can write
> [y + g(y) for y in [f(x) for x in range(10)]]
> but it's not as clear as to show what `y` is in a subsequent clause, not to
> say there'll be another temporary list built in the process.
> We can even replace every comprehension with map and filter, but that would
> face the same problems.

That is a workaround (and one I'd not thought of before) but I agree
it's ugly, and reduces readability. Actually, factoring out the inner
comprehension like Evpok Padding suggests:

f_samples = (f(x) for x in range(10))
[y+g(y) for y in f_samples]

is very readable and effective, IMO, so it's not *that* obvious that
local names are beneficial.

> In a word, what I'm arguing is that we need a way to assign temporary
> variables in a comprehension.

"We need" is a bit strong here. "It would be useful to have" is
probably true for some situations.

> In my opinion, code like
> [y + g(y) for x in range(10) **some syntax for `y=f(x)` here**]
> is more natural than any solution we now have.
> And that's why I pro the new syntax, it's clear, explicit and readable, and
> is nothing beyond the functionality of the present comprehensions so it's
> not complicated.

The problem is that you haven't proposed an actual syntax here, just
that one should be invented. There have been discussions on this in
the past (a quick search found
https://mail.python.org/pipermail/python-ideas/2011-April/009863.html
and https://mail.python.org/pipermail/python-ideas/2012-January/013468.html,
for example).

> And I hope the discussion could focus more on whether we should allow
> assigning temporary variables in comprehensions rather than how to solve the
> specific example I mentioned above.

The problem isn't so much "whether we should allow it" as "can we find
a syntax that is acceptable", and only then "does the new syntax give
sufficient benefit to be worth adding". New syntax has a pretty high
cost, and proposals that don't suggest explicit syntax will get stuck
because you can't judge whether adding the capability is "worth it"
without being clear on what the cost is - particularly when the
benefit is relatively small (which this is).

Agreed that it's important to focus on the general problem, but part
of the discussion *will* include arguing as to why the existing
workarounds and alternatives are less acceptable than new syntax. And
that will need to include discussion of specific cases. Generally, in
that sort of discussion, artificial examples like "y=f(x)" don't fare
well because it's too easy to end up just debating subjective views on
"readability". If you can provide examples from real-world code that
clearly demonstrate the cost in terms of maintainability of the
existing workarounds, that will help your argument a lot. Although
you'll need to be prepared for questions like "would you be willing to
drop support for versions of Python older than 3.8 in order to get
this improvement?" - it's surprisingly hard to justify language (as
opposed to library) changes when you really stop and think about it.
Which is not to say that it can't be done, just that it's easy to
underestimate the effort needed.

Paul

[Stephan Houben]

Note that you can already do:

 [y + g(y) for x in range(10) for y in [f(x)]]

i.e. for y  in [expr]

does exactly what the OP wants.

No new syntax needed.

If you hang out on python-list , you'll soon notice

that many newbies struggle already with the list comprehension

syntax.

It's a mini-language which is almost, but not entirely,

exactly unlike normal Python code.

Let's not complicate it further.

Stephan

 

[Jamie Willis]

I'm not sure it does indicate a need for refactoring, I'd argue it's quite a common pattern, at least in functional languages from which this construct arises.

In fact, in those languages, there are laws that govern interactions with the comprehensions (though this comes from monads and monads perhaps don't quite apply to pythons model). These laws define behaviour that is expected equivalent by users; 

[x for x in xs] = xs

[f(x) for x in [x]] = f(x)

[g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in f(x)]

Even though that last law isn't completely analogous to the given example from the OP, the transformation he wants to be able to do does arise from the laws. So it could be argued that not being able to flatten the comprehension down via law 3 is unexpected behaviour and in order to achieve this you'd need a form of assignment in the comprehension or suffer inefficiencies.

But that's probably just my functional brain talking...

[Chris Barker]

On Thu, Feb 15, 2018 at 2:13 AM, Jamie Willis <jw1489...@my.bristol.ac.uk> wrote:

 These laws define behaviour that is expected equivalent by users; 

[x for x in xs] = xs

OK -- that's the definition...

 

[f(x) for x in [x]] = f(x)

well, not quite:

[f(x) for x in [x]] = [f(x)]

Using x in two places where they mean different things makes this odd, but yes, again the definition (of a list comp, and a length-1 sequence)

[g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in f(x)]

well, no. using two for expressions yields the outer product -- all combinations:

In [14]: xs = range(3)

In [15]: [(x,y) for x in xs for y in xs]

Out[15]: [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2)]

so the result is a length len(seq1)*len(seq(2)) list. Or in this case, a len(xs)**2.

But nesting the comps applies one expression, and then the other, yielding  a length len(xs) list.

but you wrote:

[g(y) for x in xs for y in f(x)]

which I'm not sure what you were expecting, as f(x) is not a sequence (probably)...

To play with your examples:

Define some functions that make it clear what's been applied:

In [16]: def f(x):

    ...:     return "f({})".format(x)

    ...: 

In [17]: def g(x):

    ...:     return "g({})".format(x)

and a simple sequence to use:

In [18]: xs = range(3)

Now your examples:

In [19]: [x for x in xs]

Out[19]: [0, 1, 2]

In [20]: [f(x) for x in [x]]

Out[20]: ['f(5)']

In [21]: [g(y) for y in [f(x) for x in xs]]

Out[21]: ['g(f(0))', 'g(f(1))', 'g(f(2))']

OK -- all good f applied, then g, but then the last one:

In [27]: [g(y) for x in xs for y in f(x)]

Out[27]: 

['g(f)',

 'g(()',

 'g(0)',

 'g())',

 'g(f)',

 'g(()',

 'g(1)',

 'g())',

 'g(f)',

 'g(()',

 'g(2)',

 'g())']

in this case, f(x) is returning a string, which is a sequence, so you get that kind odd result. But what if f(x) was a simple scalr function:

In [29]: def f(x):

    ...:     return 2*x

Then you just get an error:

In [30]: [g(y) for x in xs for y in f(x)]

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-30-82ba3864654b> in <module>()

----> 1 [g(y) for x in xs for y in f(x)]

<ipython-input-30-82ba3864654b> in <listcomp>(.0)

----> 1 [g(y) for x in xs for y in f(x)]

TypeError: 'int' object is not iterable

The the nested comp is what is desired here:

In [31]: [g(y) for y in [f(x) for x in xs]]

Out[31]: ['g(0)', 'g(2)', 'g(4)']

Except you probably want a generator expression in the inner loop to avoid bulding an extra list:

In [33]:  [g(y) for y in (f(x) for x in xs)]

Out[33]: ['g(f(0))', 'g(f(1))', 'g(f(2))']

So back to the OP's example:

In [34]: [f(x) + g(f(x)) for x in range(10)]

Out[34]: 

['f(0)g(f(0))',

 'f(1)g(f(1))',

 'f(2)g(f(2))',

 'f(3)g(f(3))',

 'f(4)g(f(4))',

 'f(5)g(f(5))',

 'f(6)g(f(6))',

 'f(7)g(f(7))',

 'f(8)g(f(8))',

 'f(9)g(f(9))']

that is best done with comps as:

In [36]: [fx + g(fx) for fx in (f(x) for x in range(10))]

Out[36]: 

['f(0)g(f(0))',

 'f(1)g(f(1))',

 'f(2)g(f(2))',

 'f(3)g(f(3))',

 'f(4)g(f(4))',

 'f(5)g(f(5))',

 'f(6)g(f(6))',

 'f(7)g(f(7))',

 'f(8)g(f(8))',

 'f(9)g(f(9))']

which really doesn't seem bad to me. And if the function names are longer -- which they should be, you might want to use a temp as suggested earlier:

In [41]: fx = (f(x) for x in range(10))

In [42]: [x + g(x) for x in fx]

Out[42]: 

['f(0)g(f(0))',

 'f(1)g(f(1))',

 'f(2)g(f(2))',

 'f(3)g(f(3))',

 'f(4)g(f(4))',

 'f(5)g(f(5))',

 'f(6)g(f(6))',

 'f(7)g(f(7))',

 'f(8)g(f(8))',

 'f(9)g(f(9))']

The truth is, comprehensions really are a bit wordy, if you are doing a lot of this kind of thing (at least with numbers), you might be happier with an array-oriented language or library, such as numpy:

In [46]: import numpy as np

In [47]: def f(x):

    ...:     return x * 2

    ...: 

In [48]: def g(x):

    ...:     return x * 3

    ...: 

    ...: 

In [49]: xs = np.arange(3)

In [50]: f(xs) + g(f(xs))

Out[50]: array([ 0,  8, 16])

is pretty compact, and can be "optimized with a temp:

In [51]: fx = f(xs)

    ...: fx + g(fx)

    ...: 

Out[51]: array([ 0,  8, 16])

pretty simple  isn't it?

So this gets back to -- does anyone have a suggestion for a syntax for comprehensions that would make this substantially clearer, more readable, or more compact?

I'm guessing not :-)

(the compact bit comes from having to type the "for x in" part twice -- it does *feel* a bit unnecessary. which is why I like numpy -- no "for" at all :-)

(I'm still trying to figure out why folks think map() or filter() help here...)

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[jw1...@my.bristol.ac.uk]

I’d like to clarify that f(x) was indeed meant to be a sequence. As per monad law:

 

do { y <- do { x <- m;

               f x

             }

     g y

   }

===

 

do { x <- m;

     y <- f x;

     g y

   }

I think you might have misunderstood the types of things; f: Function[a, List[b]] and g: Function[b, List[c]]. m: List[a]

 

But I DO think my old the fly write up went wrong, converting do-notation to list comprehensions isn’t completely straightforward

 

The above is equivalent to

 

[g y | x <- m, y <- f x] in Haskell and the top is [g y | y <- [z |x <- m, z <- f x]]

 

These have analogous structures in python;

 

[g(y) for x in m for y in f(x)] and [g(y) for y in [z for x in m for z in f(x)]] (I think?)

 

And yes the left identity law I posted was missing the [f(x)] brackets.

 

If I’ve not made another mistake, that *should* now work?

[Steven D'Aprano]

Hi fhsxfhsx, and welcome.

My comments below, interleaved with yours.


On Thu, Feb 15, 2018 at 01:56:44PM +0800, fhsxfhsx wrote:

[quoted out of order]

> And I hope the discussion could focus more on whether we should allow
> assigning temporary variables in comprehensions rather than how to
> solve the specific example I mentioned above.

Whether or not to allow this proposal will depend on what alternate
solutions to the problem already exist, so your specific example is very
relevant. Any proposed change has to compete with existing solutions.

> As far as I can see, a comprehension like
> alist = [f(x) for x in range(10)]
> is better than a for-loop
> for x in range(10):
> alist.append(f(x))
> because the previous one shows every element of the list explicitly so
> that we don't need to handle `append` mentally.

While I personally agree with you, many others disagree. I know quite a
few experienced, competent Python programmers who avoid list
comprehensions because they consider them harder to read and reason
about. They consider a regular for-loop better precisely because you do
see the explicit call to append.

(In my experience, those of us who get functional-programming idioms
often forget that others find them tricky.)

The point is that list comprehensions are already complex enough that
they are difficult for many people to learn, and some people never come
to grips with them. Adding even more features comes with a cost.

The bottom line is that it isn't clear to me that allowing local
variables inside comprehensions will make them more readable.

> But when it comes to something like
> [f(x) + g(f(x)) for x in range(10)]
> you find you have to sacrifice some readableness if you don't want two
> f(x) which might slow down your code.

The usual comments about premature optimisation apply here.

Setting a new comprehension variable is not likely to be free, and may even be
more costly than calling f(x) twice if f() is a cheap expression:

[x+1 + some_func(x+1) for x in range(10)]

could be faster than

[y + some_func(y) for x in range(10) let y = x + 1]

or whatever syntax we come up with.

> Someone may argue that one can write
> [y + g(y) for y in [f(x) for x in range(10)]]

Indeed. This would be the functional-programming solution, and I
personally think it is an excellent one. The only changes are that I'd
use a generator expression for the intermediate value, avoiding the need
to make a full list, and I would lay it out more nicely, using
whitespace to make the structure more clear:

result = [y + g(y) for y in

(f(x) for x in range(10))
]

> but it's not as clear as to show what `y` is in a subsequent clause,
> not to say there'll be another temporary list built in the process.

There's no need to build the temporary list. Use a generator
comprehension. And I disagree that the value of y isn't as clear.

An alternative is simply to refactor your list comprehension. Move the
calls to f() and g() into a helper function:

def func(x):
y = f(x)
return y + g(y)

and now you can write the extremely clear comprehension

[func(x) for x in range(10)]

that needs no extra variable.



[...]

> In a word, what I'm arguing is that we need a way to assign temporary
> variables in a comprehension.

"Need" is very strong. I think that the two alternatives I mention above
cover 95% of the cases where might use a local variable in a
comprehension. And of the remaining cases, many of them will be so
complex that they should be re-written as an explicit for-loop. So in my
opinion, we're only talking about a "need" to solve the problem for a
small proportion of cases:

- most comprehensions don't need a local variable (apart from
the loop variable) at all;

- of those which do need a local variable, most can be easily
solved using a nested comprehension or a helper function;

- of those which cannot be solved that way, most are complicated
enough that they should use a regular for-loop;

- leaving only a small number of cases which are complicated enough
to genuinely benefit from local variables but not too complicated.

So this is very much a borderline feature. Occasionally it would be
"nice to have", but on the negative side:

- it adds complexity to the language;

- makes comprehensions harder to read;

- and people will use it unnecessarily where there is no readability
or speed benefit (premature optimization again).

It is not clear to me that we should burden *all* Python programmers
with additional syntax and complexity of an *already complex* feature
for such a marginal improvement.

> In my opinion, code like
> [y + g(y) for x in range(10) **some syntax for `y=f(x)` here**]
> is more natural than any solution we now have.
> And that's why I pro the new syntax, it's clear, explicit and readable

How can you say that the new syntax is "clear, explicit and readable"
when you haven't proposed any new syntax yet?

For lack of anything better, I'm going to suggest "let y = f(x)" as the
syntax, although personally I don't like it even a bit.

Where should the assignment go?

[(y, y**2) let y = x+1 for x in (1, 2, 3, 4)]

[(y, y**2) for x in (1, 2, 3, 4) let y = x+1]

I think they're both pretty ugly, but I can't think of anything else.

Can we rename the loop variable, or is that an error?

[(x, x**2) let x = x+1 for x in (1, 2, 3, 4)]

How do they interact when you have multiple loops and if-clauses?

[(w, w**2) for x in (1, 2, 3, 4) let y = x+1
for a in range(y) let z = a+1 if z > 2
for b in range(z) let w = z+1]


For simplicity, perhaps we should limit any such local assignments to
the very end of the comprehension:

[expression for name in sequence
<zero or more for-loops and if-clauses>
<zero or more let-clauses>
]

but that means we can't optimise this sort of comprehension:

[expression for x in sequence
for y in (something_expensive(x) + function(something_expensive(x))
]

Or these:

[expression for x in sequence
if something_expensive(x) or condition(something_expensive(x))
]


I think these are very hard questions to answer.


--
Steve

[Chris Barker - NOAA Federal]

I’d like to clarify that f(x) was indeed meant to be a sequence. As per monad law:

 

do { y <- do { x <- m;

               f x

             }

     g y

   }

===

 

do { x <- m;

     y <- f x;

     g y

   }

I think you might have misunderstood the types of things; f: Function[a, List[b]] and g: Function[b, List[c]]. m: List[a]

I’m still confused — f and g take a scale and return a list? Or take two parameters, a scalar and a list???

Either way, doesn’t really fit with python comprehensions, which are more or less expecting functions that except and return a single element (which could be a triple, but I digress)

And the key point is that in python:

for x in seq1 for y in seq2

Does all combinations, like a nested for loop would.

But I DO think my old the fly write up went wrong, converting do-notation to list comprehensions isn’t completely straightforward

Well, I don’t think I get the do notation at all....

The above is equivalent to

 

[g y | x <- m, y <- f x] in Haskell and the top is [g y | y <- [z |x <- m, z <- f x]]

 

These have analogous structures in python;

 

[g(y) for x in m for y in f(x)] and [g(y) for y in [z for x in m for z in f(x)]] (I think?)

Is this current Python or a new proposed syntax — ‘cause I can’t make any sense of it in python.

Maybe define some f,g,m that behave as you expect, and see what python does....

 

If I’ve not made another mistake, that *should* now work?

Now on a phone, so can’t test, but I also can’t tell what you are expecting the result to be.

Back to one of your examples:

[f(x) for x in [x]]

What does that mean???

for x in seq

Means iterate through seq, and assign each item to the name x.

If that seq has x in it — I’m not sure that is even legal python — the scope in a comprehension confuses me.

But that is the equivalent is something like:

it= iter(seq)

while True:

    Try:

        x = next(it)

    Except StopIteration:

        Break

(Excuse the caps — hard to write code on a phone)

So not sure how x gets into that sequence before the loop starts.

-CHB

[Chris Barker - NOAA Federal]

> Setting a new comprehension variable is not likely to be free, and may even be
> more costly than calling f(x) twice if f() is a cheap expression:
>
> [x+1 + some_func(x+1) for x in range(10)]
>
> could be faster than
>
> [y + some_func(y) for x in range(10) let y = x + 1]

A bit of a nit — function call overhead is substantial in python, so
if that is an actual function, rather than a simple expression, it’ll
likely be slower to call it twice for any but trivially small
iterables.

> [(y, y**2) let y = x+1 for x in (1, 2, 3, 4)]

Do we need the let?

[ g(y) for y = f(x) for c in seq]

Or, with expressions:

[y + y**2 for y = x+1 for x in (1,2,3)]

Maybe that would be ambiguous— I haven’t thought carefully about it.

-CHB

[Terry Reedy]

On 2/15/2018 8:37 PM, Chris Barker - NOAA Federal wrote:

> Back to one of your examples:
>
> [f(x) for x in [x]]
>
> What does that mean???
>
> for x in seq
>
> Means iterate through seq, and assign each item to the name x.
>
> If that seq has x in it — I’m not sure that is even legal python — the
> scope in a comprehension confuses me.
>
> But that is the equivalent is something like:
>
> it= iter(seq)
> while True:
>     Try:
>         x = next(it)
>     Except StopIteration:
>         Break
>
> (Excuse the caps — hard to write code on a phone)
>
> So not sure how x gets into that sequence before the loop starts.

Reusing a previously bound name as an iteration variable is a bad idea.
It works in 3.x because the outermost iterable, but only the outermost
iterable, is pre-calculated before executing the comprehension. Hence
'x in [x]' sometimes works, and sometimes not. ('Outermost' is topmost
in nested loops, left most in comprehension.)

>>> x = 2
>>> [x*x for x in [x]]
[4]
>>> [x*y for x in [3] for y in [x]] # here, local x is 3, not 2
[9]
>>> [x*y for y in [x] for x in [3]]
[6]
>>> [x*y for y in [3] for x in [x]]
Traceback (most recent call last):
File "<pyshell#4>", line 1, in <module>
[x*y for y in [3] for x in [x]]
File "<pyshell#4>", line 1, in <listcomp>
[x*y for y in [3] for x in [x]]
UnboundLocalError: local variable 'x' referenced before assignment
>>> [z*y for y in [3] for z in [x]] # no confusion here
[6]

To put it another way, l = [x for x in [x]] is actually calculated as
_temp = [x]; l = [x for x in _temp]. In general, other iterables cannot
be precalculated since they might depend on prior iteration variables.

--
Terry Jan Reedy

[Robert Vanden Eynde]

Hello, talking about this syntax :

[y+2 for x in range(5) let y = x+1]

*Previous talks*

I've had almost exactly the same idea in June 2017, see subject "variable assignment in functional context here: https://mail.python.org/pipermail/python-ideas/2017-June/subject.html.

Currently this can easily be done by iterating over a list of size 1:

[y+2 for x in range(5) for y in [x+1]]

(Other ways exist, see section "current possibilities" below).

This comparison would answer a lot of questions like "can we write [x+2 for x in range(5) for x in [x+1]], so [x+2 for x in range(5) let x = x+1]" the new variable would indeed shadow the old one.

In June 2017 I introduced the "let" syntax using the existing "for" keyword, using the "=" instead of a "in" like this:

[y+2 for x in range(5) for y = x+1]

The only difference I introduced was that it would be logical to accept the syntax in any expression:

x = 5

print(y+2 for y = x + 1)

or with the "let" keyword:

x = 5

print(y+2 let y = x + 1)

*Previous talk: Pep*

In the June conversation one conclusion was that someone wrote a pep (https://www.python.org/dev/peps/pep-3150/) in 2010 that's still pending (not exactly the same syntax but the same idea, he used a "given:" syntax) that looked like that :

print(y+2 given:

            y=x+1)

*Previous talk: GitHub implementation on Cython*

Another conclusion was that someone on GitHub implemented a "where" statement in Cython (url: https://github.com/thektulu/cpython/commit/9e669d63d292a639eb6ba2ecea3ed2c0c23f2636) where one could write :

print(y+2 where y = x+1)

So in a list comprehension:

[y+2 where y = x + 1 for x in range(5)]

As the author thektulu said "just compile and have fun".

*New syntax in list comprehension*

However, we probably would like to be able to use this "where" after the for:

[y+2 for x in range(5) where y = x+1]

This would allow the new variable to be used in further "for" and "if" statement :

[y+z for x in range(5) where y = x+1 for z in range(y+1)]

*Choice of syntax*

Initially I thought re-using the "for" keyword would be a good idea for backward comptability (a variable named "where" in old code wouldn't be a problem), however some people pointed out that the python Grammar wouldn't be LL1, when reading the "for" token it wouldn't be able to choose directly if the rest would be a "for in" or "for =" so actually introducing a dedicated keyword is probably better, like this :

[y+2 for x in range(5) where y = x+1]

print(y+2 where y = x+1)

The "where" keyword is very readable in my opinion, very close to English sentences we use. Sometimes we introduce a new variable before using it, generally using "let" or after using it using "where". For example "let y = x+1; print(y+2)". Or "print(y+2 where y = x+1)".

The first syntax is chosen in the "let in" syntax in haskell :

print(let y = x+2 in y+2)

Or chained:

print(let x = 2 in let y = x+1 in y+2)

But Haskell user would probably line break for clarity :

print(let x = 2 in

         let y = x+1 in

         y+2)

A postfix notation using "where" would probably be less verbose in my opinion. Another example of "postfix notation" python has is with the "a if condition else b" so adding a new one wouldn't be surprising. Furthermore, the postfix notation is preferred in the context of "presenting the result first, then the implementation" (context discussed already in the 2010 pep), the "presenting the result first" is also a goal of the list comprehension, indeed one does write [x+3 for x in range(5)] and not [for x in range(5): x+3], the latter would be more "imperative programming" style, and would be translated to a normal loop.

The problem of chaining without parenthesis, how to remove the parenthesis in the following statement ?

print((y+2 where y = x+1) where x = 2)

We have two options :

print(y+2 where x = 2 where y = x+1)

print(y+2 where y = x+1 where x = 2)

The first option would be probably closer to the way multiple "for" are linked in a list comprehension:

[y+2 for x in range(5) for y in [x+1]]

But the second option would be more "present result first" and more close to the parenthesized version, the user would create new variables as they go, "I want to compute y+2 but hey, what is y ? It's x+1 ! But what is x ? It's 5 !)). However, keeping the same order as in "multiple for in list comprehension" is better, so I'd choose first option.

In the implementation on GitHub of thektulu the parenthesis are mandatory.

Another syntax issue would probably surprise some users, the following statement, parenthesized :

[(y+2 where y = x+1) for x in range(5)]

Would have two totally legal ways to be done:

[y+2 where y = x+1 for x in range(5)]

[y+2 for x in range(5) where y = x+1]

The first one is a consequence of the "where" keyword usable in an expression and the second one is a consequence of using it in a list comprehension.

However I think it doesn't break the "there is only one obvious way to do it", because depending on the case, the "y" variable would be a consequence of the iteration or a consequence of the computation.

*Goals of the new syntax*

Personally, I find it very useful to do an assignment in such context, I use list comprehension for example to generate a big json with multiple for and if.

I don't have here a list of big real world example thar would be simplified using this syntax but people interested could search arguments here.

People could argue only the "list comprehension" case would be useful and not the "any expression" case:

[y+2 for x in range(5) where y = x+1]

would be accepted but not :

print(y+2 where y=x+1)

Because the latter could be written:

y = x + 1

print(y+2)

However, the idea of having an isolated scope can be a good idea.

*Current possibilities*

Currently we have multiple options :

[y+2 for x in range(5) for y in [x+1]]

[y+2 for (x+1 for x in range(5))]

[(lambda y:y+2)(y=x+1) for x in range(5)]

The first one works well but it's not obvious we just want to assign a new variable, especially when the expression is long, or multiple, or both:

[y+z for x in range(5) for y,z in [((x + 1) * 2, x ** 2 - 5)]]

The second one makes it impossible to reuse the "x" variable and the y = x+1 relation is not obvious.

The third example is what a functional programmer would think but is really too much complex for a beginner and very verbose.

*Proposed syntax : Conclusion*

As I said, I like the "where" syntax with the "where" keyword.

[y+2 for x in range(5) where y = x+1]

Also usable in any expression :

print(y+2 where y = x+1)

*Conclusion*

Here is all the talk/work/argument I've already found about this syntax. Apparently it's been a while (2010) since such an idea was thought but I think having a new pep listing all the pros and cons would be a good idea. So that we can measure how much the community would want this concept to be introduced, and if it's refused, the community would have a document where the "cons" are clearly written.

Robert Vanden Eynde

[Steven D'Aprano]

On Thu, Feb 15, 2018 at 10:13:37AM +0000, Jamie Willis wrote:

> I'm not sure it does indicate a need for refactoring, I'd argue it's quite
> a common pattern, at least in functional languages from which this
> construct arises.
>
> In fact, in those languages, there are laws that govern interactions with
> the comprehensions (though this comes from monads and monads perhaps don't
> quite apply to pythons model). These laws define behaviour that is expected
> equivalent by users;

Python is not a functional language, and the usual Pythonic solution to
an overly complex or inefficient functional expression is to refactor
into non-functional style.

In my experience, most Python users have never even heard of monads, and
those who have, few grok them. (I know I don't.)

> [x for x in xs] = xs
> [f(x) for x in [x]] = f(x)
> [g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in f(x)]

I think these should be:

[x for x in xs] = list(xs) # not to be confused with [x]

[f(x) for x in [x]] = [f(x)]
[g(y) for y in [f(x) for x in xs]] = [g(y) for x in xs for y in [f(x)]]


> Even though that last law isn't completely analogous to the given example
> from the OP,

It can be used though. He has:

[f(x) + g(f(x)) for x in xs]

which can be written as

[y + g(y) for x in xs for y in [f(x)]]


Here's an example:

# calls ord() twice for each x
py> [ord(x) + ord(x)**2 for x in "abc"]
[9506, 9702, 9900]

# calls ord() once for each x
py> [y + y**2 for x in "abc" for y in [ord(x)]]
[9506, 9702, 9900]

And one last version:

py> [y + y**2 for y in [ord(x) for x in "abc"]]
[9506, 9702, 9900]

In production, I'd change the last example to use a generator
comprehension.

> the transformation he wants to be able to do does arise from
> the laws. So it could be argued that not being able to flatten the
> comprehension down via law 3 is unexpected behaviour

Law 3 does apply, and I'm not sure what you mean by the statement that
we can't flatten the comprehension down.



--
Steve

[Martin Teichmann]

Hi list,

> But when it comes to something like
> [f(x) + g(f(x)) for x in range(10)]
> you find you have to sacrifice some readableness if you don't want two f(x)
> which might slow down your code.
>
> Someone may argue that one can write
> [y + g(y) for y in [f(x) for x in range(10)]]

personally I think that the biggest problem readability-wise is that "for" is a post-fix operator, which makes generators much harder to read. It's also very different from normal for loops, which have the "for" at the top. IMHO generators would be much easier to read with a prefix for, as in

    [for x in range(10): f(x) + g(f(x))]

also nested generators get nicer like that:

    [for y in (for x in range(10): f(x)):
          y + g(y)]

one could critique here that we shouldn't use colons in expressions, but that boat has sailed: we use them for lambdas. We do not write

     sq = x**2 lambda x

and I am not proposing that.

Also if/else could be written with colons, but I am not sure whether that's actually nicer:

    val = (if attr is None: 5 else: attr + 3)

but it certainly is in case of ifs in generators:

    [for x in range(10):
         if x % 3 != 2: x]

which could be problematic to parse if you compare that to

    [for x in range(10):
         if x % 3 == 2: x - 1
         else: x + 1]

one could even dig out the often-proposed always-rejected except-in-expression:

    [for x in range(10):
         try: f(x)
         except WhateverError: None]

or even a with:

    [for x in file_names:
         with open(x) as f:
             f.read()]


Greetings

Martin

[fhsxfhsx]

A generator can be a good idea, however, I wonder if it's really readable to have a `f_samples`.
And the structure is the same as in

[y + g(y) for y in [f(x) for x in range(10)]]

if you replace the list with a generator. And it's similar for other solutions you mentioned.
Well, I know that it can be quite different for everyone to determine whether a piece of code is readable or not, maybe it's wise to wait for more opinions.

There's another problem that, as you distinguished the `simple` and `more complex` case, the offered solutions seem very specific, I'm not sure if there's a universal solution for every case of temporary variables in comprehensions. If not, I think it's too high cost to reject a new syntax if you need to work out a new solution every time.

[fhsxfhsx]

Thank you Paul, what you said is enlightening and I agree on most part of it.

I'll propose two candidate syntaxs.
1. `with ... as ...`
    This syntax is more paralles as there would be `for` and `with` clause as well as `for` and `with` statement. However, the existing `with` statement is semantically different from this one, although similar.
2. `for ... is ...`
    This syntax is more uniform as the existing `for` clause make an iterator which is a special kind of variable. However, I'm afraid this syntax might be confused with `for ... in ...` as they differ only on one letter.

I like the latter one better. Other proposes are absolutely welcome.

And here is an example which appears quite often in my code where I think a new syntax can help a lot:
Suppose I have an list of goods showing by their ids in database, and I need to transform the ids into json including information from two tables, `Goods` and `GoodsCategory`, where the first table recording `id`, `name` and `category_id` indicating which category the goods belongs to, the second table recording `id`, `name` and `type` to the categories.
With the new syntax, I can write
[
  {
    'id': goods.id,
    'name': goods.name,
    'category': gc.name,
    'category_type': gc.type,
  }
  for goods_id in goods_id_list
  for goods is Goods.get_by_id(goods_id)
  for gc is GoodsCategory.get_by_id(goods.category_id)
]

And I cannot think of any good solutions as this one without it.
To generalize this case, for each element of the list, I need two temporary variables (`goods` and `gc` in my case), and each one was used twice.

And reply to the two past discussions you mentioned,
1.https://mail.python.org/pipermail/python-ideas/2011-April/009863.html
    This mail gave a solution to modify function `f` to keep the result. The weak point is obvious, you must modify the function `f`.
2.https://mail.python.org/pipermail/python-ideas/2012-January/013468.html
    This mail wrote
>The important thing is that you name the thing you care about before using it.
>I think this is a very natural way of writing: first you give the thing you
>care about a name, then you refer to it by name.
However, that's a problem every comprehension faces, not a problem drawn by the new syntax.

[fhsxfhsx]

You are right and actually I sometimes did the same thing in a temporary script such as in ipython.
Because in my opinion, it's not really elegant code as one may be puzzled for the list `[f(x)]`.
Well, although that's quite subjective.

And also I test the code and find another `for` clause can make time cost about 1.5 times in my computer, even when I optimize `[f(x)]` to a generator `(f(x), )`. Though that's not very big issue if you don't care about efficiency that much. But a temporary list or generator is redundant here. `[f(x)]` can be an alternative, but I think it is worth a new syntax.

[fhsxfhsx]

Hi Steve, Thank you for so detailed comments.
My comments also below interleaved with yours.

To be frank, I had not thought of this before.
However, in my opinion, when considering adding a new syntax, we care more about the marginal cost.
I mean, I think it is the functional-programming way which is tricky, but allowing a new syntax would not make things worse.
Well, that's just a guess, maybe only those who are puzzled with comprehensions can give us an answer.

>
>> But when it comes to something like
>> [f(x) + g(f(x)) for x in range(10)]
>> you find you have to sacrifice some readableness if you don't want two 
>> f(x) which might slow down your code.
>
>The usual comments about premature optimisation apply here.
>
>Setting a new comprehension variable is not likely to be free, and may even be 
>more costly than calling f(x) twice if f() is a cheap expression:
>
>    [x+1 + some_func(x+1) for x in range(10)]
>
>could be faster than
>
>    [y + some_func(y) for x in range(10) let y = x + 1]
>
>or whatever syntax we come up with.
>

It is true. But since there are still so many cases where a temporary variable is faster. 
Also, even without let-clause, one can write a for-loop with a temporary variable which slow down the code.
So, it seems that "setting a new comprehension variable may even be more costly" does not show any uselessness of temporary variables in comprehensions.

>
>> Someone may argue that one can write
>> [y + g(y) for y in [f(x) for x in range(10)]]
>
>Indeed. This would be the functional-programming solution, and I 
>personally think it is an excellent one. The only changes are that I'd 
>use a generator expression for the intermediate value, avoiding the need 
>to make a full list, and I would lay it out more nicely, using 
>whitespace to make the structure more clear:
>
>    result = [y + g(y) for y in 
>                 (f(x) for x in range(10))
>                 ]
>

In my opinion,

[

    y + g(y)
    for x in range(10)

    let y = f(x)
]
is better because it's more corresponding to a for-loop
for x in range(10):
    y = f(x)
    result.append(y + g(y))
In my opinion, comprehensions are not real functional-programming because there is not even a function. Though there're similarities, map and filter are real functional-programming. Since the similarity between for-clause in comprehensions and the for-loop, I think it's better to write comprehensions more close to for-loop.
I don't know but I guess maybe it can also help those who fear comprehensions better embrace them?

>
>> but it's not as clear as to show what `y` is in a subsequent clause, 
>> not to say there'll be another temporary list built in the process.
>
>There's no need to build the temporary list. Use a generator 
>comprehension. And I disagree that the value of y isn't as clear.
>
>An alternative is simply to refactor your list comprehension. Move the 
>calls to f() and g() into a helper function:
>
>def func(x):
>    y = f(x)
>    return y + g(y)
>
>and now you can write the extremely clear comprehension
>
>[func(x) for x in range(10)]
>
>that needs no extra variable.
>

I think it can be a goods idea if there's a name to `func` which is easy to understand, or `func` is put close to the comprehension and is in a obvious place.
But I feel it's not for the case I gave in another mail to Paul, https://mail.python.org/pipermail/python-ideas/2018-February/048997.html,
(I'm sorry that the example is quite long, and I don't hope to copy it here) 
To me, it can be confusing to have several `func` when I have several lists at the same time and have to transform them each in a similar but different way.

I think the assignment should be treated equally as for-clause and if-clause which means

[(y, y**2) for x in (1, 2, 3, 4) let y = x+1]

would be a right syntax.
And 
[(x, x**2) for x in (1, 2, 3, 4) let x = x+1]
would not cause an error because we can also write
[(x, x**2) for x in (1, 2, 3, 4) for x in (4, 3, 2, 1)]
now.

I didn't see any problem in 

[(w, w**2) for x in (1, 2, 3, 4) let y = x+1 
               for a in range(y) let z = a+1 if z > 2 
               for b in range(z) let w = z+1]

In my opinion, it would behave the same as
for x in (1, 2, 3, 4):
    y = x+1
    for a in range(y):

        z = a+1
        if z > 2:

            for b in range(z):
                w = z+1
                mylist.append((w, w**2))
According to my understanding, the present for-clause and if-clause does everything quite similar to this nested way,

[fhsxfhsx]

Thanks so much for the comments and the collect on this syntax!

Comments on *previous talk*
The list also mentioned some other previous proposals, so I myself search it and find that there're even more in the mailing list since 2008.
https://mail.python.org/pipermail/python-ideas/2008-August/001842.html

Comments on *previous talk: PEP*
The PEP seems to be about an explicit temporary namespace where any objects (including functions, classes, etc.) could be held.
However, I find that this syntax may not work for temporary variables in comprehensions.

We might expect this syntax work like
[?.y+2 for x in range(5) given: y = x+1]
But notice that the proposed `given` statement here appeared in comprehensions, where syntax changes in comprehensions are needed, and nothing about this is mentioned in the PEP.
What's more, the proposed syntax is a statement like `for` statement, `if` statement, rather than a for-clause or if-clause in comprehensions. In my opinion, it's not a good idea to have a statement in comprehensions. Instead, another given-clause should be added if one wants to write code like above, doing assignments in comprehensions, which can look like:
[?.y+2 for x in range(5) given y = x+1]
So the ? symbol seems useless here, so maybe
[y+2 for x in range(5) given y = x+1]
make it quite similar to the `where` syntax you proposed.

So, I agree with you that it is a good idea to have a new PEP, though for a different reason.
 
Comments on *Choice of syntax*
I gave two candidate syntaxs in https://mail.python.org/pipermail/python-ideas/2008-August/001842.html, one said `for ... in ...`, another said `with ... as ...`.
The first has the same problem as `for ... = ...` you proposed. And the biggest problem I think the second will face is the difference in semantic between the `with` statement and it.
When it comes to `where ... = ...`, there are one possible problem I can think of.
`where` is not now a keyword in Python. There are WHERE clauses in SQL, so in many modules including peewee and SQLAlchemy, `where` is an important method. The new syntax would cause quite incompatibilities.

Personally I agreed with you that postfix notation would have advantage over prefix notation. Other than `where`, `with` is quite readable in my opinion. So maybe `with ... = ...` can be another candidate?

Plus the `given ... = ...` I mentioned above, there are several more candidates now. Personally I perfer `with ... = ...`, because `with` is a python keyword so it would be good for backward compatibility.

*About comprehensions and expressions*
You gave `print(y+2 where y = x+1)` as an example, I think it should be clarified that it's not, or at least, does not look like a comprehension, but an expression. It should give an object `y+2` rather than a list or a generator. (otherwise what list can it give?)
There are for-clause and if-clause for comprehensions, and if-clause (aka ternary operators) for expressions. So, In my opinion, there should be additional discuss and examples to show that it's helpful to have such syntax.

For the where-clause in expressions, I think we could refer to how python handles if-clause.
The following setences are legal:
[x if x else 0 for x in mylist if x > 10]
The following illegal:
[x if x for x in mylist if x > 10]
[x if x else 0 for x in mylist if x > 10 else 10]
That's to say, the two kinds of if-clause, one is only used in expressions (aka `ternary operator`), the other is only used in comprehensions. They slightly differ in syntax.

The where-clause might work in similar ways. Then
[z+2 where z = y+1 for x in mylist where y = x+1]
means
[(z+2 where z=y+1) for x in mylist where y = x+1]
where the parenthesis (also the part before the first `for`) is a expression, the rest is comprehension clauses.
To be more accurate, the new syntax would be:

test: where_test ['if' where_test 'else' test] | lambdef
where_test: or_test | ( '(' or_test 'where' NAME '=' testlist_star_expr ')' )

Mandatory parenthesis in where_test is to resolve the ambiguity in code like
print(y+2 where y=x+1 if x>0 else x-1 if x>1 else 0)
It could be analysed like
print((y+2 where y=x+1 if x>0 else x-1) if x>1 else 0)
or
print(y+2 where y=(x+1 if x>0 else x-1 if x>1 else 0)).
I guess thektulu may have mandatory parenthesis for the same reason.

I haven't check the new syntax very carefully so there might be other ambiguities.

Another example is
print(y+2 if x>0 else y-2 where y=x+1)
with mandatory parenthesis, one must write
print((y+2 if x>0 else y-2 where y=x+1))
or
print(y+2 if x>0 else (y-2 where y=x+1))

However, it might still confuse many people. I wonder whether it's a good idea to have such syntax.

It would be much easier to add assignments in comprehensions.
comp_iter: comp_for | comp_if | comp_where
comp_where: 'where' NAME '=' testlist_star_expr [comp_iter]

Comments on *Goals of the new syntax*
I have a real-world example in https://mail.python.org/pipermail/python-ideas/2018-February/048997.html, it's to generate a big json, you seem to have a very goods feeling of it though you didn't give a real-world example.

[Steven D'Aprano]

On Sun, Feb 18, 2018 at 12:23:28AM +0800, fhsxfhsx wrote:
> Thank you Paul, what you said is enlightening and I agree on most part of it.
>
> I'll propose two candidate syntaxs.
> 1. `with ... as ...`
> This syntax is more paralles as there would be `for` and `with`
> clause as well as `for` and `with` statement. However, the
> existing `with` statement is semantically different from this one,
> although similar.

I don't think they are even a little bit similar. The existing `with`
statement is for controlling cleanup code (using a context manager).
This proposal doesn't have anything to do with context managers or
cleanup code. It's just a different way to spell "name = value" inside
comprehensions.

> 2. `for ... is ...`
> This syntax is more uniform as the existing `for` clause make an
> iterator which is a special kind of variable. However, I'm afraid
> this syntax might be confused with `for ... in ...` as they differ
> only on one letter.

Indeed.

And frankly, treated as English grammar, "for value is name" doesn't
make sense and is horribly ugly to my eyes:

result = [x for value in sequence for value+1 is x]

> And here is an example which appears quite often in my code where I
> think a new syntax can help a lot: Suppose I have an list of goods
> showing by their ids in database, and I need to transform the ids into
> json including information from two tables, `Goods` and
> `GoodsCategory`, where the first table recording `id`, `name` and
> `category_id` indicating which category the goods belongs to, the
> second table recording `id`, `name` and `type` to the categories.
>
> With the new syntax, I can write
> [
> {
> 'id': goods.id,
> 'name': goods.name,
> 'category': gc.name,
> 'category_type': gc.type,
> }
> for goods_id in goods_id_list
> for goods is Goods.get_by_id(goods_id)
> for gc is GoodsCategory.get_by_id(goods.category_id)
> ]
>
> And I cannot think of any good solutions as this one without it.

I can of a few, starting with the most simple: write a helper function.
Not every problem needs to be solved with new syntax.

def dict_from_id(goods_id):
goods = Goods.get_by_id(goods_id)
gc = GoodsCategory.get_by_id(goods.category_id)
return {'id': goods.id,

'name': goods.name,
'category': gc.name,
'category_type': gc.type
}

result = [dict_from_id(goods_id) for goods_id in goods_id_list]

That's much nicer to read, you can document and test the dict_from_id()
function, no new systax is required, it is easy to refactor, and I very
much doubt that adding one extra function call is going to be a
significant slowdown compared to the cost of two calls to get_by_id()
methods and constructing a dict.

(And if as you add more fields to the dict, the overhead of the function
call becomes an even smaller proportion.)

Or you can make this a method of the goods object, which is arguably a
better OO design. Let the goods object be responsible for creating the
dict.

result = [Goods.get_by_id(goods_id).make_dict() for goods_id in goods_id_list]
# or if you prefer
result = [goods.make_dict() for goods in map(Goods.get_by_id, goods_id_list)]

Here is a third solution: use a for-loop iterating over a single-item
tuple to get the effect of a local assignment to a temporary variable:

result = [{
# dict display truncated for brevity...
}
for goods_id in goods_id_list
for goods in (Goods.get_by_id(goods_id),)
for gc in (GoodsCategory.get_by_id(goods.category_id),)
]

If you don't like the look of single-item tuples (foo,) you can use
single-item lists instead [x] but they are a tiny bit slower to create.

Serhiy has suggested that the interpreter can optimize the single-item
loop to make it as fast as a bare assignment:

https://bugs.python.org/issue32856

I think this is a neat trick, although Yuri thinks it is an ugly hack
and doesn't want to encourage it. Neat or ugly, I think it is better
than "for value is name".


--
Steve

[Neil Girdhar]

You should give an actual motivating example.  I think none of these suggestions are more readable than just writing things out as a for loop.  You argue that you want to avoid appending to a result list.  In that case, I suggest writing your pattern as a generator function.

Best,

Neil

[Kyle Lahnakoski]

On 2018-02-17 11:23, fhsxfhsx wrote:

[
  {
    'id': goods.id,
    'name': goods.name,
    'category': gc.name,
    'category_type': gc.type,
  }
  for goods_id in goods_id_list
  for goods is Goods.get_by_id(goods_id)
  for gc is GoodsCategory.get_by_id(goods.category_id)
]

in the short term, it seems for...in [...] is good enough:

> [
>     {
>         'id': goods.id,
>         'name': goods.name,
>         'category': gc.name,
>         'category_type': gc.type,
>     }
>     for goods_id in goods_id_list

>     for goods in [Goods.get_by_id(goods_id)]
>     for gc in [GoodsCategory.get_by_id(goods.category_id)]
> ]

I personally would like to see with...as... syntax allowed in list comprehensions, despite `with` being limited to context managers to date.

> [
>     {
>         'id': goods.id,
>         'name': goods.name,
>         'category': gc.name,
>         'category_type': gc.type,
>     }
>     for goods_id in goods_id_list

>     with Goods.get_by_id(goods_id) as goods
>     with GoodsCategory.get_by_id(goods.category_id) as gc
> ]

..,or maybe `let` reads easier?

> [
>     {
>         'id': goods.id,
>         'name': goods.name,
>         'category': gc.name,
>         'category_type': gc.type,
>     }
>     for goods_id in goods_id_list

>     let goods = Goods.get_by_id(goods_id)
>     let gc = GoodsCategory.get_by_id(goods.category_id)
> ]

[Kyle Lahnakoski]

I believe list comprehensions are difficult to read because they are not
formatted properly. For me, list comprehension clauses are an
expression, followed by clauses executed in the order. Any list
comprehension with more than one clause should be one-line-per clause.

Examples inline:

On 2018-02-15 19:57, Steven D'Aprano wrote:
> Where should the assignment go? [(y, y**2) let y = x+1 for x in (1, 2,
> 3, 4)] [(y, y**2) for x in (1, 2, 3, 4) let y = x+1]

Since y is a function of x, it must follow the for clause:

> [
>     (y, y ** 2)

>     for x in (1, 2, 3, 4)
>     let y = x + 1
> ]

> How do they interact when you have multiple loops and if-clauses? [(w,

> w**2) for x in (1, 2, 3, 4) let y = x+1 for a in range(y) let z = a+1
> if z > 2 for b in range(z) let w = z+1]

They are applied in order:

> [

>     (w, w**2)
>     for x in (1, 2, 3, 4)
>     let y = x+1
>     for a in range(y)
>     let z = a+1
>     if z > 2
>     for b in range(z)
>     let w = z+1
> ]

which is a short form for:

> def stuff():
>     for x in (1, 2, 3, 4):
>         y = x+1
>         for a in range(y):

>             z = a+1
>             if z > 2:

>                 for b in range(z):
>                     w = z+1
>                     yield (w, w**2)
>
> list(stuff())

[Neil Girdhar]

Is it that much shorter that it's worth giving up the benefit of indentation? 

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

--

---
You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/KwZtO4rpAGE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-ideas...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kyle Lahnakoski]

On 2018-02-23 12:44, Neil Girdhar wrote:

On Fri, Feb 23, 2018 at 12:35 PM Kyle Lahnakoski <klahn...@mozilla.com> wrote:

> [
>     (w, w**2)
>     for x in (1, 2, 3, 4)
>     let y = x+1
>     for a in range(y)
>     let z = a+1
>     if z > 2
>     for b in range(z)
>     let w = z+1
> ]

which is a short form for:

> def stuff():
>     for x in (1, 2, 3, 4):
>         y = x+1
>         for a in range(y):
>             z = a+1
>             if z > 2:
>                 for b in range(z):
>                     w = z+1
>                     yield (w, w**2)
>
> list(stuff())

Is it that much shorter that it's worth giving up the benefit of indentation? 

Saving the indentation? Oh yes, for sure!  This code reads like a story, the indentation is superfluous to that story.  Should we add it to Python? I don't know; I quick scan through my own code, and I do not see much opportunity for list comprehensions of this complexity.  Either my data structures are not that complicated, or I have try/except blocks inside a loop, or I am using a real query language (like SQL).  pythonql seems to solve all these problems well enough (https://github.com/pythonql/pythonql).

[Neil Girdhar]

On Fri, Feb 23, 2018 at 1:42 PM Kyle Lahnakoski <klahn...@mozilla.com> wrote:

On 2018-02-23 12:44, Neil Girdhar wrote:

On Fri, Feb 23, 2018 at 12:35 PM Kyle Lahnakoski <klahn...@mozilla.com> wrote:

> [
>     (w, w**2)
>     for x in (1, 2, 3, 4)
>     let y = x+1
>     for a in range(y)
>     let z = a+1
>     if z > 2
>     for b in range(z)
>     let w = z+1
> ]

which is a short form for:

> def stuff():
>     for x in (1, 2, 3, 4):
>         y = x+1
>         for a in range(y):
>             z = a+1
>             if z > 2:
>                 for b in range(z):
>                     w = z+1
>                     yield (w, w**2)
>
> list(stuff())

Is it that much shorter that it's worth giving up the benefit of indentation? 

Saving the indentation? Oh yes, for sure!  This code reads like a story, the indentation is superfluous to that story.  Should we add it to Python? I don't know; I quick scan through my own code, and I do not see much opportunity for list comprehensions of this complexity. 

That's a good thing that you are not writing code like this.  I don't agree that the indentation is "superfluous".  It makes the code easy to read.  Anyway, Google's Python style guide also agrees that very long comprehensions like that are not worth it. (https://google.github.io/styleguide/pyguide.html?showone=List_Comprehensions#List_Comprehensions)

Either my data structures are not that complicated, or I have try/except blocks inside a loop, or I am using a real query language (like SQL).  pythonql seems to solve all these problems well enough (https://github.com/pythonql/pythonql). 

[Grégory Starck]

Le jeudi 15 février 2018 02:03:31 UTC-5, fhsxfhsx a écrit :

As far as I can see, a comprehension like
alist = [f(x) for x in range(10)]
is better than a for-loop
for x in range(10):
  alist.append(f(x))
because the previous one shows every element of the list explicitly so that we don't need to handle `append` mentally.

But when it comes to something like
[f(x) + g(f(x)) for x in range(10)]
you find you have to sacrifice some readableness if you don't want two f(x) which might slow down your code.

Someone may argue that one can write
[y + g(y) for y in [f(x) for x in range(10)]]
but it's not as clear as to show what `y` is in a subsequent clause, not to say there'll be another temporary list built in the process.
We can even replace every comprehension with map and filter, but that would face the same problems.

In a word, what I'm arguing is that we need a way to assign temporary variables in a comprehension.
In my opinion, code like
[y + g(y) for x in range(10) **some syntax for `y=f(x)` here**]
is more natural than any solution we now have.
And that's why I pro the new syntax, it's clear, explicit and readable, and is nothing beyond the functionality of the present comprehensions so it's not complicated.

And I hope the discussion could focus more on whether we should allow assigning temporary variables in comprehensions rather than how to solve the specific example I mentioned above.

what about:

items = [

    z * (x + y + b) / (i - j - a)

    with:

        x = f1(a - i + j)

        y = f2(i - j + b)

        z = f3(x + y - (i * j))

    for a in a_range

    with:

        i = f3(b)

        j = f4(b)

    for b in b_range

]

?

 

 

[Grégory Starck]

Le jeudi 1 mars 2018 20:00:51 UTC-5, Grégory Starck a écrit :

Le jeudi 15 février 2018 02:03:31 UTC-5, fhsxfhsx a écrit :

As far as I can see, a comprehension like

what about:

items = [

    z * (x + y + b) / (i - j - a)

    with:

        x = f1(a - i + j)

        y = f2(i - j + b)

        z = f3(x + y - (i * j))

    for a in a_range

    with:

        i = f3(b)

        j = f4(b)

    for b in b_range

]

?

or the opposite:

items = [

     z * (x + y + b) / (i - j - a)

    for a in a_range

    with:

        x = f1(a - i + j)

        y = f2(i - j + b)

        z = f3(x + y - (i * j))

    for b in b_range

    with:

        i = f3(b)

        j = f4(b)

]

and then you can compose with if too  : 

items = [

    z * (x + y + b) / (i - j - a)

    for a in a_range

    if a % 42 != 0

    with:

        x = f1(a - i + j)

        y = f2(i - j + b)

    if x - y > 0

    with:

        z = f3(x + y - (i * j))

    if z > 42

    for b in b_range

    with:

        i = f3(b)

        j = f4(b)

    if i * j < 42 or b == 42

]

or for more simpler forms:

items = [

    (y + f(y + z), x - z)

    for x in x_range

    if x % whatever != other

    with y, z = g(x), h(x)

]