Local variable definition in Python list comprehension

[James Tsai]

Hello,

I find it very useful if I am allowed to define new local variables in a list comprehension. For example, I wish to have something like
[(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or
[(x, y) for x in range(10) with y := x ** 2 if x + y < 80].

For now this functionality can be achieved by writing
[(x, y) for x in range(10) for y in [x ** 2] if x + y < 80].

Is it worthwhile to add a new feature like this in Python? If so, how can I propose this to PEP?

[Ben Bacarisse]

James Tsai <james...@gmail.com> writes:

> I find it very useful if I am allowed to define new local variables in
> a list comprehension. For example, I wish to have something like
> [(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or
> [(x, y) for x in range(10) with y := x ** 2 if x + y < 80].
>
> For now this functionality can be achieved by writing
> [(x, y) for x in range(10) for y in [x ** 2] if x + y < 80].

x and y are, to a first approximation, new local variables defined in a
list comprehension. I think you need to restate what it is you want.

> Is it worthwhile to add a new feature like this in Python? If so, how
> can I propose this to PEP?

To make any sort of case you'd need to give an example that does not
have a clearer way to write it already. Your working version is, to me,
clearer that the ones you want to be able to write.

--
Ben.

[Chris Angelico]

Not everything has to be a one-liner.

ChrisA

[Eryk Sun]

On 9/1/22, James Tsai <james...@gmail.com> wrote:
>
> I find it very useful if I am allowed to define new local variables in a
> list comprehension. For example, I wish to have something like
> [(x, y) for x in range(10) for y := x ** 2 if x + y < 80], or
> [(x, y) for x in range(10) with y := x ** 2 if x + y < 80].
>
> For now this functionality can be achieved by writing
> [(x, y) for x in range(10) for y in [x ** 2] if x + y < 80].

You can assign a local variable in the `if` expression. For example:

>>> [(x, y) for x in range(10) if x + (y := x**2) < 30]
[(0, 0), (1, 1), (2, 4), (3, 9), (4, 16)]

[James Tsai]

By local variable definition I mean binding a variable to a single value, so it doesn't include giving an iterable that a variable can take values iteratively, e.g. 'for x in range(10)'. Does it not worth introducing a specific syntax to do this, instead of creating a new list ad hoc to define the variable like 'for y in [1]'?

[James Tsai]

No but very often when I have written a neat list/dict/set comprehension, I find it very necessary to define local variable(s) to make it more clear and concise. Otherwise I have to break it down to several incrementally indented lines of for loops, if statements, and variable assignments, which I think look less nice.

[James Tsai]

Yeah this works great but like [(x, y) for x in range(10) for y in [x**2]] I written before, is kind of a hack. And if initially I do not need an "if" condition in the list comprehension, this becomes less convenient. I still can write
>>> [(x, y) for x in range(10) if (y := x**2) or True]

But I wonder if Python could have a specific syntax to support this.

[Chris Angelico]

But why would you need to assign to y in that example? If you're using
it more than once, you can use :=, and if you aren't, you don't need
to. But do be aware that := does not create a comprehension-local name
binding, but a nonlocal instead.

> No but very often when I have written a neat list/dict/set comprehension, I find it very necessary
> to define local variable(s) to make it more clear and concise. Otherwise I have to break it down
> to several incrementally indented lines of for loops, if statements, and variable assignments,
> which I think look less nice.

Well, if it's outgrown a list comp, write it on multiple lines. Like I
said, not everything has to be a one-liner.

ChrisA

[Peter J. Holzer]

In that case
[(x, x**2) for x in range(10)]
seems to be somewhat more readable.

I would also say that
[(x, x**2) for x in range(10) if x + x**2 < 80]
doesn't really seem worse than any of the variants with y in it.
(Yes, I get that your real duplicated expression is probably a bit more
complex than `x**2`, but by the time a temporary variable really
improves readability it's probably a good time to split that across
multiple lines, too.)

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | h...@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

[Cameron Simpson]

True, but a comprehension can be more expressive than a less
"functional" expression (series of statements).

James, can you provide (a) a real world example where you needed to
write a series of statements or loops and (b) a corresponding example of
how you would have preferred to have written that code, possibly
inventing some syntax or misusing ":=" as if it workeed they way you'd
like it to work?

Cheers,
Cameron Simpson <c...@cskk.id.au>

[Avi Gross]

Dumb question. Your y is purely a function of x. So create an f(x) where
you want your y. It probably can even be anonymous inline. I mean your
return values of (x, y) would be (x, f(x)) ...

On Thu, Sep 1, 2022, 5:04 PM Chris Angelico <ros...@gmail.com> wrote:

> On Fri, 2 Sept 2022 at 06:55, James Tsai <james...@gmail.com> wrote:
> >

> But why would you need to assign to y in that example? If you're using
> it more than once, you can use :=, and if you aren't, you don't need
> to. But do be aware that := does not create a comprehension-local name
> binding, but a nonlocal instead.
>

> > No but very often when I have written a neat list/dict/set
> comprehension, I find it very necessary
> > to define local variable(s) to make it more clear and concise. Otherwise
> I have to break it down
> > to several incrementally indented lines of for loops, if statements, and
> variable assignments,
> > which I think look less nice.
>
> Well, if it's outgrown a list comp, write it on multiple lines. Like I
> said, not everything has to be a one-liner.
>

> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[James Tsai]

Yeah, I think list comprehension is particularly useful to construct a deeply nested list/dict. For example, I am now using Plotly to visualize a cellular network including several base stations and users. Here is the function I have written:

def plot_network(area, base_stations, users):
bs_poses = np.array([bs.pos for bs in base_stations])
ue_poses = np.array([ue.pos for ue in users])
fig = px.scatter(x=bs_poses[:, 0], y=bs_poses[:, 1])
fig.add_scatter(x=ue_poses[:, 0], y=ue_poses[:, 1])
fig.update_layout(
xaxis=dict(range=[0, area[0]], nticks=5),
yaxis=dict(range=[0, area[1]], nticks=5),
shapes=[dict(
type="circle",
fillcolor="PaleTurquoise",
x0=x-r, y0=y-r, x1=x+r, y1=y+r,
hovertext=f"({x:.2f}, {y:.2f})",
opacity=0.3
) for bs in base_stations for x, y in [bs.pos]
for r in [bs.cell_radius]],
)
return fig

Simply put, I want to scatter the BSs and users, and additionally I want to draw a big circle around each BS to represent its cell coverage. I can choose to write 'x0=bs.pos[0]-bs.cell_radius, y0=...' instead, but it becomes less concise, and if x, y, or r is the return value of a function instead of a property, it becomes more computationally expensive to repeat calling the function as well. I also can create the list of 'shapes' by appending to a list, like

shapes = []
for bs in base_stations:
x, y = bs.pos
r = bs.cell_radius
shapes.append(dict(...))
fig.update_layout(
xaxis=dict(range=[0, area[0]], nticks=5),
yaxis=dict(range=[0, area[1]], nticks=5),
shapes=shapes
)

But in my opinion this is much less concise. I think it looks better to create the list within the nested structure. So I totally agree that list comprehension adds much expressiveness in Python. I only wonder whether it is a good idea to introduce a specific syntax for local variable assignment in list comprehensions, instead of using "for r in [bs.cell_radius]".
I am also surprised to know that the assignment operator ":=" in a list comprehension will assign a variable outside of the scope of the comprehension. I think it does not make sense since a list comprehension without a ":=" will never change name bindings outside itself.

[Dan Stromberg]

On Thu, Sep 1, 2022 at 9:16 AM Chris Angelico <ros...@gmail.com> wrote:

> On Fri, 2 Sept 2022 at 02:10, James Tsai <james...@gmail.com> wrote:
> >

> Not everything has to be a one-liner.
>
So true!

I like list comprehensions and generator expressions, but sometimes I end
up regretting their use when there's a bug, and I have to convert one to a
for loop + list.append in order to debug.