Creating lambdas inside generator expression

[Johannes Bauer]

Hi list,

I've just encounted something that I found extremely unintuitive and
would like your feedback. This bit me *hard*, causing me to question my
sanity for a moment. Consider this minimal example code (Py 3.10.4 on
Linux x64):


class Msg():
def hascode(self, value):
print("Check for", value)
return False

conds = [
lambda msg: msg.hascode("foo"),
lambda msg: msg.hascode("bar"),
]

msg = Msg()
print(conds[0](msg))
print(conds[1](msg))



It works perfectly and does exactly what it looks like. The output is:

Check for foo
False
Check for bar
False

But now consider what happens when we create the lambdas inside a list
comprehension (in my original I used a generator expresison, but the
result is the same). Can you guess what happens when we create conds
like this?

conds = [ lambda msg: msg.hascode(z) for z in ("foo", "bar") ]

I certainly could not. Here's what it outputs:

Check for bar
False
Check for bar
False

I.e., the iteration variable "z" somehow gets bound inside the lambda
not by its value, but by its reference. All checks therefore refence
only the last variable.

This totally blew my mind. I can understand why it's happening, but is
this the behavior we would expect? And how can I create lambdas inside a
generator expression and tell the expression to use the *value* and not
pass the "z" variable by reference?

Cheers,
Joe

[Johannes Bauer]

Aha!

conds = [ lambda msg, z = z: msg.hascode(z) for z in ("foo", "bar") ]

Is what I was looking for to explicitly use the value of z. What a
caveat, didn't see that coming.

Learning something new every day.

Cheers,
Joe


Am 29.06.22 um 11:50 schrieb Johannes Bauer:

[Antoon Pardon]

Or you could try this as an alternative:

conds = [ (lambda code: lambda msg: msg.hascode(code))(z) for z in

("foo", "bar") ]

Op 29/06/2022 om 12:43 schreef Johannes Bauer:

[Chris Angelico]

On Thu, 30 Jun 2022 at 02:49, Johannes Bauer <dfnson...@gmx.de> wrote:
> But now consider what happens when we create the lambdas inside a list
> comprehension (in my original I used a generator expresison, but the
> result is the same). Can you guess what happens when we create conds
> like this?
>
> conds = [ lambda msg: msg.hascode(z) for z in ("foo", "bar") ]
>
> I certainly could not. Here's what it outputs:
>
> Check for bar
> False
> Check for bar
> False
>
> I.e., the iteration variable "z" somehow gets bound inside the lambda
> not by its value, but by its reference. All checks therefore refence
> only the last variable.
>

Yep, that is the nature of closures. (Side point: This isn't actually
a generator expression, it's a list comprehension; current versions of
Python treat them broadly the same way, but there was previously a
difference in the way scoping worked.) What you're seeing is a
consequence of the way that closures work, and it is a very good thing
most of the time :)

The usual way to "snapshot" a variable is what you showed in your
followup: a default argument value.

def f(..., z=z):
... z has been snapshot

(As others have pointed out, this isn't unique to lambdas; any
function will behave that way.)

Antoon offered another variant, but written as a pair of lambda
functions, it's a little hard to see what's going on. Here's the same
technique written as a factory function:

def does_it_have(z):
return lambda msg: msg.hascode(z)

conds = [does_it_have(z) for z in ("foo", "bar")]

Written like this, it's clear that the variable z in the comprehension
is completely different from the one inside does_it_have(), and they
could have different names if you wanted to. This is a fairly clean
way to snapshot too, and has the advantage that it doesn't pretend
that the function takes an extra parameter.

ChrisA

[Peter Otten]

On 29/06/2022 23:17, Chris Angelico wrote:
> On Thu, 30 Jun 2022 at 02:49, Johannes Bauer <dfnson...@gmx.de> wrote:

>> But now consider what happens when we create the lambdas inside a list
>> comprehension (in my original I used a generator expresison, but the
>> result is the same). Can you guess what happens when we create conds
>> like this?
>>
>> conds = [ lambda msg: msg.hascode(z) for z in ("foo", "bar") ]
>>
>> I certainly could not. Here's what it outputs:
>>
>> Check for bar
>> False
>> Check for bar
>> False
>>
>> I.e., the iteration variable "z" somehow gets bound inside the lambda
>> not by its value, but by its reference. All checks therefore refence
>> only the last variable.
>>
>

> Yep, that is the nature of closures. (Side point: This isn't actually
> a generator expression, it's a list comprehension; current versions of
> Python treat them broadly the same way, but there was previously a
> difference in the way scoping worked.) What you're seeing is a
> consequence of the way that closures work, and it is a very good thing
> most of the time :)
>
> The usual way to "snapshot" a variable is what you showed in your
> followup: a default argument value.
>
> def f(..., z=z):
> ... z has been snapshot
>
> (As others have pointed out, this isn't unique to lambdas; any
> function will behave that way.)
>
> Antoon offered another variant, but written as a pair of lambda
> functions, it's a little hard to see what's going on. Here's the same
> technique written as a factory function:
>
> def does_it_have(z):
> return lambda msg: msg.hascode(z)
>
> conds = [does_it_have(z) for z in ("foo", "bar")]
>
> Written like this, it's clear that the variable z in the comprehension
> is completely different from the one inside does_it_have(), and they
> could have different names if you wanted to. This is a fairly clean
> way to snapshot too, and has the advantage that it doesn't pretend
> that the function takes an extra parameter.

While I'd go with Chris' suggestion there are two other options:
functools.partial() and operator.methodcaller().

Example:

>>> class Msg:
def __init__(self, msg):
self.msg = msg
def hascode(self, code):
return code in self.msg


>>> conds = [partial(lambda z, msg: msg.hascode(z), z) for z in ("foo",
"bar")]
>>> [cond(Msg("barbaz")) for cond in conds]
[False, True]
>>> conds = [methodcaller("hascode", z) for z in ("foo", "bar")]
>>> [cond(Msg("barbaz")) for cond in conds]
[False, True]