unintuitive for-loop behavior

[namenob...@gmail.com]

hello pythonistas

i've had a nodding acquaintance with python for some time, and all along i assumed that for-loops got a namespace of their own; now i'm reading up on the language and i find out that's not the case: the loop variable gets put into the enclosing namespace, overwriting any former value that was already there; of course i realize there are situations where one is interested in the value that a loop variable has after the loop has been exited, but this behavior seems grossly unintuitive to me: has there ever been any discussion of giving for-loops the option of running in namespaces of their own?

and it gets even worse; consider the following means of raising an exception:

#)-------------------------------------- begin code

def powerfunction(exponent):
return lambda base: (base ** exponent)

p1 = [powerfunction(exponent) for exponent in range(9)]
p2 = [lambda base: (base ** exponent) for exponent in range(9)]

assert p1[3](4) == p2[3](4)

#)---------------------------------------- end code

apparently the problem is that "exponent" gets evaluated when the relevant lambda function is run, not when it's defined, but no binding for "exponent" gets put into the enclosing namespace: it seems that python remembers this ghostly "exponent" for me on the theory that i don't WANT "p1" and "p2" to be the same; can anyone help to reconcile me to this semantics?

thanks if you can help
stm

[Brendan Abel]

Yes, loops don't have their own scope. Indeed, very few flow elements in
python -- if, with, try/except -- create a new scope. In that sense, it's
fairly consistent, but can be unexpected for people that have used
languages with many nested scopes.

The lambda behavior is a common gotcha - there are hundreds of questions on
StackOverflow that are caused by that misunderstanding

http://stackoverflow.com/questions/7368522/weird-behavior-lambda-inside-list-comprehension

The most common solution is to just provide the variable as a default
argument to the lambda function, which will bind it how you'd expect

[lambda base, exponent=exponent: (base ** exponent) for exponent in
range(9)]

> --
> https://mail.python.org/mailman/listinfo/python-list
>

[Chris Angelico]

On Fri, Sep 30, 2016 at 5:29 AM, <namenob...@gmail.com> wrote:
> i've had a nodding acquaintance with python for some time, and all along i assumed that for-loops got a namespace of their own; now i'm reading up on the language and i find out that's not the case: the loop variable gets put into the enclosing namespace, overwriting any former value that was already there; of course i realize there are situations where one is interested in the value that a loop variable has after the loop has been exited, but this behavior seems grossly unintuitive to me: has there ever been any discussion of giving for-loops the option of running in namespaces of their own?
>

No, there hasn't. In Python, there are no sub-function scopes - even
the couple of cases that look like subscopes (list comps and genexps)
are actually implemented as nested functions (fairly obvious in the
case of a genexp, more subtle with the list comp). C-like languages
can have infinitely nested scopes because they have variable
declarations:

int some_func() {
int foo;
for (...) {
if (whatever) {
foo = 1234;
} else {
foo = 2345;
}
use(foo);
}
}

The placement of 'int foo;' determines the scope of that name. Python
doesn't have that, so your options are:

1) The first assignment to a name determines its scope. That would
mean that the two assignments are in different scopes, and use(foo) is
not included. To solve this, you'd need a dummy assignment further up
(eg where "int foo" is) - not impossible but probably annoying.

2) Have a single and simple scoping rule, like "anything that's
assigned to is function-local".

Python chose the second option. Anything you assign to is local to the
function, but nothing narrower. There is no way [1] to have a local
name in a function AND use a global with the same name, even in
different parts of the function. Since a 'for' loop is assignment
("for X in Y" assigns to X), that name has to be function-scoped. You
could, in theory, have the name be unbound after the loop, but it's
usually not beneficial to do so, and can be detrimental.

ChrisA

[1] Well, you can use globals(), but that's not quite the same thing
as variable scope.

[Gregory Ewing]

namenob...@gmail.com wrote:
> can anyone help to reconcile me to this semantics?

Not really. Most people agree that it's not desirable
behaviour, but we've ended up here due to a convoluted
history, and there doesn't seem to be a good way to
fix it without breaking a lot of existing code.

Chris Angelico wrote:
> You could, in theory, have the name be unbound after the loop, but it's
> usually not beneficial to do so, and can be detrimental.

It also wouldn't help at all with this problem.

There *is* something that could be done, at least in
CPython: if the loop variable is referenced by a nested
function, then create a new cell for it each time
round the loop, instead of updating the contents of
the existing cell.

This would be backwards-compatible in almost all cases.
The last value would still be accessible after the
loop finishes, and other assignments to the loop
variable in the same function would work the same
way as always.

Guido doesn't like this idea, though, because it
depends on CPython implementation details, and other
Python implementations could have trouble matching
these semantics.

--
Greg

[Rustom Mody]

On Friday, September 30, 2016 at 10:23:31 AM UTC+5:30, Gregory Ewing wrote:

> namenobodywants wrote:
> > can anyone help to reconcile me to this semantics?
>
> Not really. Most people agree that it's not desirable
> behaviour, but we've ended up here due to a convoluted
> history, and there doesn't seem to be a good way to
> fix it without breaking a lot of existing code.

Thanks for an unusually helpful answer.

In my experience telling a beginner:
«This» is a misfeature/gotcha
And «this» is its workaround

greatly shortens the learning curve as compared to long-winded
justificatory explanations

[Steve D'Aprano]

On Fri, 30 Sep 2016 05:29 am, namenob...@gmail.com wrote:

> hello pythonistas
>
> i've had a nodding acquaintance with python for some time, and all along i
> assumed that for-loops got a namespace of their own;

Giving for-loops their own namespace is a grossly unintuitive and a very
weird thing to do. Modules, classes and functions are obviously namespaces.
Why should arbitrary syntactic structures create their own namespace?

It would be terribly inconvenient and surprising for if...else blocks to be
separate namespaces:

a = 1
if condition:
print(a) # UnboundLocalError: local 'a' referenced before assignment
a += 1


For-loops are no different. Making them their own namespace is a very
strange thing to do, it would mean you couldn't re-bind a value inside a
for-loop:

count = 0
for x in sequence:
count += 1
# raises UnboundLocalError: local 'count' referenced before assignment


unless you declared it nonlocal or global, depending on whether your for
loop was inside a function or not.

To me, "make for-loops be their own scope" sounds like a joke feature out of
joke languages like INTERCAL. I'm not aware of any sensible language that
does anything like this.

No, wait a minute, I tell a lie, I recall Chris Angelico mentioning that one
of his favourite languages, Pike or REXX, does it. I forget which.




--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[BartC]

On 30/09/2016 13:33, Steve D'Aprano wrote:
> On Fri, 30 Sep 2016 05:29 am, namenob...@gmail.com wrote:

>> i've had a nodding acquaintance with python for some time, and all along i
>> assumed that for-loops got a namespace of their own;

> It would be terribly inconvenient and surprising for if...else blocks to be
> separate namespaces:
>
> a = 1
> if condition:
> print(a) # UnboundLocalError: local 'a' referenced before assignment
> a += 1
>
>
> For-loops are no different. Making them their own namespace is a very
> strange thing to do, it would mean you couldn't re-bind a value inside a
> for-loop:
>
> count = 0
> for x in sequence:
> count += 1
> # raises UnboundLocalError: local 'count' referenced before assignment

It can make sense for 'x' to be local to this for-loop block (everything
else is at it works now), and independent of any x's in outer blocks. It
means being able to do stuff like this:

for x in a:
for x in b:
pass
pass # outer x still has its value

And any even more outer x's are not affected. With 'for', especially one
inside a list-comp, often you just need some throwaway index variable
without worrying if it will clash with an existing local.

But it's not useful enough I think to bother changing now. Or even when
the language was designed.

--
Bartc

[Chris Angelico]

On Fri, Sep 30, 2016 at 10:33 PM, Steve D'Aprano
<steve+...@pearwood.info> wrote:
> To me, "make for-loops be their own scope" sounds like a joke feature out of
> joke languages like INTERCAL. I'm not aware of any sensible language that
> does anything like this.
>
> No, wait a minute, I tell a lie, I recall Chris Angelico mentioning that one
> of his favourite languages, Pike or REXX, does it. I forget which.

In C-like languages (including Pike), you can legally define a
variable at any point, making it visible at that point and all inner
locations - effectively, every brace becomes a new subscope. It makes
sense ONLY because variables are declared, no matter where they are
(globals are declared at module scope, locals at some kind of inner
scope). So, it's not strictly true that Pike has for loops as their
own scope, just that C-like languages have a lot of nested scopes.

(REXX has nothing like this.)

ChrisA

[Grant Edwards]

On 2016-09-30, Steve D'Aprano <steve+...@pearwood.info> wrote:

> To me, "make for-loops be their own scope" sounds like a joke feature out of
> joke languages like INTERCAL. I'm not aware of any sensible language that
> does anything like this.

In C99 a for loop has its own namespac:

int main(void)
{
for (int i=0; i<5; ++i)
printf("i=%d\n",i);
}

If you try to access 'i' outside the for loop, it's an error, because
it doesn't exist in the file, global or 'main' namespace. It only
exists in the for-loop's namespace.

I think that's an absolutely brilliant feature, and I use it a _lot_
when writing C code. I'm a big fan of minimizing the lifetime/scope
of variables. I wish if/then/else did the same thing:

if ((r=some_function()) != R_SUCCESS)
printf("some_function() failed with status %d\n",r);

--
Grant Edwards grant.b.edwards Yow! I don't know WHY I
at said that ... I think it
gmail.com came from the FILLINGS in
my rear molars ...

[BartC]

C likes to make things a bit more complicated:

int A;
{ A; // this is the outer A
int A;
A; // this is the inner A
goto A;
A: // this is yet another A
}

Labels live in a different name-space from other names. I've left out
struct tags which also have their own name-space, and would allow five
different A's in that block (an outer and inner 'struct A') rather than
the three I've shown.

So a single pair of {-} can enclose three (sometimes five) versions of
the same identifier.

Personally I think function-scope is plenty. Only one possible top-level
'A' in a function instead of an unlimited number.

(Python allows extra 'A's inside nested functions and classes within
that function, but that's a bit different.)

--
Bartc

[Chris Angelico]

On Sat, Oct 1, 2016 at 12:36 AM, Grant Edwards
<grant.b...@gmail.com> wrote:

> On 2016-09-30, Steve D'Aprano <steve+...@pearwood.info> wrote:
>
>> To me, "make for-loops be their own scope" sounds like a joke feature out of
>> joke languages like INTERCAL. I'm not aware of any sensible language that
>> does anything like this.
>

> In C99 a for loop has its own namespac:
>
> int main(void)
> {
> for (int i=0; i<5; ++i)
> printf("i=%d\n",i);
> }
>
> If you try to access 'i' outside the for loop, it's an error, because
> it doesn't exist in the file, global or 'main' namespace. It only
> exists in the for-loop's namespace.

I believe that's the same semantics as C++ uses, and I agree, it's
very convenient. Among other things, it means that nested loops behave
more like they do in Python, with independent iterators:

int main(void)
{
for (int i=0; i<5; ++i)
{

printf("%d:", i);
for (int i=0; i<3; ++i)
printf(" %d", i);
printf("\n");
}
}

Now, granted, this is not something I would ever actually recommend
doing, and code review is absolutely justified in rejecting this...
but it's a lot better than pure function-scope variables, where you'd
get stuck in an infinite loop. Obviously the inner 'i' shadows the
outer 'i', but that's the only actual conflict between them.

> I think that's an absolutely brilliant feature, and I use it a _lot_
> when writing C code. I'm a big fan of minimizing the lifetime/scope
> of variables. I wish if/then/else did the same thing:
>
> if ((r=some_function()) != R_SUCCESS)
> printf("some_function() failed with status %d\n",r);

No particular reason not to. Ditto 'while' loops:

while (int r=some_function())
..

I agree with minimizing scope, but only where it's practical to do so.
In Python, it simply isn't. Like C, Python has infinitely nested
scopes; unlike C, Python requires explicit 'nonlocal' declarations in
order to assign to something in an outer scope, ergo the convenience
of not declaring local variables translates into extreme inconvenience
of working with overly-narrow scopes. To put it more simply: Scope is
cheap in C, but expensive in Python.

ChrisA

[Grant Edwards]

On 2016-09-30, Grant Edwards <grant.b...@gmail.com> wrote:
> On 2016-09-30, Steve D'Aprano <steve+...@pearwood.info> wrote:
>

>> To me, "make for-loops be their own scope" sounds like a joke
>> feature out of joke languages like INTERCAL. I'm not aware of any
>> sensible language that does anything like this.
>

> In C99 a for loop has its own namespac:

[...]

> I think that's an absolutely brilliant feature, and I use it a _lot_
> when writing C code. I'm a big fan of minimizing the lifetime/scope
> of variables. I wish if/then/else did the same thing:
>
> if ((r=some_function()) != R_SUCCESS)
> printf("some_function() failed with status %d\n",r);

The example of what I wished C did should have been this:

if ((int r=some_function()) != R_SUCCESS)

printf("some_function() failed with status %d\n",r);

--
Grant Edwards grant.b.edwards Yow! If I pull this SWITCH
at I'll be RITA HAYWORTH!!
gmail.com Or a SCIENTOLOGIST!

[Brendan Abel]

> a = 1
if condition:
print(a) # UnboundLocalError: local 'a' referenced before assignment
a += 1


For-loops are no different. Making them their own namespace is a very
strange thing to do, it would mean you couldn't re-bind a value inside a
for-loop:

count = 0
for x in sequence:
count += 1
# raises UnboundLocalError: local 'count' referenced before assignment

--------------

That's generally not how nested scopes work, you could still reference
objects in the outer scope from the inner scope, but the outer scope
couldn't reference objects in the inner scope

a = 1
if condition:

b = a + 2

print b # UnboundLocalError: local 'b' referenced before assignment


for x in sequence:
print x

print x # UnboundLocalError: local 'x' referenced before assignment.


----------

I wouldn't call either behavior intuitive or unintuitive. They're just
different behaviors of different languages.


On Fri, Sep 30, 2016 at 5:33 AM, Steve D'Aprano <steve+...@pearwood.info>
wrote:

> --
> https://mail.python.org/mailman/listinfo/python-list
>

[Chris Angelico]

On Sat, Oct 1, 2016 at 3:03 AM, Brendan Abel <007br...@gmail.com> wrote:
>> a = 1
> if condition:
> print(a) # UnboundLocalError: local 'a' referenced before assignment
> a += 1
>
>
> For-loops are no different. Making them their own namespace is a very
> strange thing to do, it would mean you couldn't re-bind a value inside a
> for-loop:
>
> count = 0
> for x in sequence:
> count += 1
> # raises UnboundLocalError: local 'count' referenced before assignment
>

> --------------
>
> That's generally not how nested scopes work, you could still reference
> objects in the outer scope from the inner scope, but the outer scope
> couldn't reference objects in the inner scope

The trouble is that, absent some sort of declaration, there's no way
for the compiler to know whether 'count' and 'a' are supposed to be
new variables in the inner scope, or the same variable as the one in
the next scope out. In Python, declarations are used when you rebind
nonlocal or global names, so these examples would have to be written
thus:

a = 1
if condition:

nonlocal a
print(a)
a += 1

count = 0
for x in sequence:

nonlocal count
count += 1

I don't think we want that. :)

ChrisA

[Steve D'Aprano]

What happens if it is *not* a misfeature? Gotchas are not always
misfeatures -- sometimes gotchas are gotchas because people's expectations
are simply wrong, and pandering to their confused expectations does not
actually help them.

I haven't made up my mind about *this* specific (mis)feature itself. I know
I don't want for-loops to introduce their own scope, but that doesn't mean
that the binding behaviour inside for-loops is necessarily the best way of
doing it. I'm still thinking about it.

But *in general*, people often don't think the logical consequences through
before deciding what behaviour is "obviously" right. They insist that
whatever behaviour would be convenient for them *now* is the one and only
correct way of doing things -- even if, an hour later, the *opposite*
behaviour is convenient and therefore the correct way of doing things.

Obviously late binding is the one and only correct way of setting function
parameter defaults -- until we need early binding, then it is the one and
only correct way of doing it.

Gotchas usually exist for a reason. Very few programming languages design
gotchas into them to deliberately trip people up[1], or even through
carelessness. More often, they're the unexpected consequences of a series
of sensible, or even unavoidable, decisions.






[1] "When I design my killer language, the identifiers `foo` and `bar` will
be reserved words, never used, and not even mentioned in the reference
manual. Any program using one will simply dump core without comment.
Multitudes will rejoice." - Tim Peters

[Steve D'Aprano]

On Fri, 30 Sep 2016 11:43 pm, BartC wrote:

> It can make sense for 'x' to be local to this for-loop block (everything
> else is at it works now), and independent of any x's in outer blocks. It
> means being able to do stuff like this:
>
> for x in a:
> for x in b:
> pass
> pass # outer x still has its value

If you're (generic you) not using the loop variables, as in your example,
why do you care if one overwrites the other? You're not using either of
them.

And if you are using them, then surely you need them to be different names
so you can use them *both*. So why does it matter if they are in different
scopes?

I can't think of an example of where I would want two nested for-loops,
where *both* loop variables need to be the same name, but I only want to
access the inner variable inside the inner loop and the outer variable in
the outer loop.

And if I was in that position, why couldn't I just relax the "must be the
same name" condition and give them different names? Its not like there is a
shortage of possible names.

> And any even more outer x's are not affected. With 'for', especially one
> inside a list-comp, often you just need some throwaway index variable
> without worrying if it will clash with an existing local.
>
> But it's not useful enough I think to bother changing now. Or even when
> the language was designed.

Indeed.

[Lawrence D’Oliveiro]

On Friday, September 30, 2016 at 8:29:18 AM UTC+13, namenob...@gmail.com wrote:
> ... the loop variable gets put into the enclosing namespace ...

Python is not a “block-structured” language, like C is. Except for list comprehensions, as you noticed. The latter behaviour only came in with Python 3.

[Gregory Ewing]

Steve D'Aprano wrote:
> Giving for-loops their own namespace is a grossly unintuitive and a very
> weird thing to do.
>

> It would be terribly inconvenient and surprising for if...else blocks to be
> separate namespaces:

There's an important difference between a for-loop and an
if-statement that's relevant here: a for-loop binds a name,
whereas an if-statement doesn't.

Whenever there's binding going on, it's necessary to decide
whether it should be creating a new binding or updating an
existing one.

This is actually a *different* issue from one of scope.
List comprehensions were changed so that the loop variable
lives in a different scope from the containing function.
However, they still have the same unintuitive behaviour
with regard to capture of the loop variable by a lambda.

>> l = [lambda: i for i in range(3)]
>>> for f in l: print(f())
...
2
2
2

Most people consider *this* behaviour to be far weirder
than anything that would result from giving for-loop
variables their own scope.

Even if you don't think it's weird, it's hard to argue
that it's *useful* in any significant number of cases.

> To me, "make for-loops be their own scope" sounds like a joke feature out of
> joke languages like INTERCAL.

Which is a straw man, since that's not actually what we're
talking about doing. It's neither necessary nor sufficient
to solve the problem.

What *is* necessary and sufficient is to make each iteration
of the for-loop create a new binding of the loop variable
(and not any other variable!).

> I'm not aware of any sensible language that
> does anything like this.

Scheme and Ruby come to mind as examples of languages in
which the equivalent of a for-loop results in each iteration
getting a new binding of the control variable. Although
you could argue that these languages are not "sensible". :-)

--
Greg

[Gregory Ewing]

Steve D'Aprano wrote:
> What happens if it is *not* a misfeature? Gotchas are not always
> misfeatures -- sometimes gotchas are gotchas because people's expectations
> are simply wrong, and pandering to their confused expectations does not
> actually help them.
>
> I haven't made up my mind about *this* specific (mis)feature itself.

Here's my analysis of it.

First, some definitions:

Current behaviour: The way for-loop variables currently
interact with nested functions.

Alternative behaviour: A nested function captures a
different instance of the for-loop variable on each
iteration. (But *nothing* else changes; in particular,
the body of the loop is *not* a separate scope.)

The current behaviour is completely useless (or at least it's
useful in in only a vanishingly small proportion of cases).
The alternative behaviour would sometimes be useful.

Nothing would need to change except in the rare case where
the loop variable is referenced by a nested function. You
would only pay for the feature if you used it.

There would be no need for ugly workarounds such as default
argument abuse or introducing another level of nested
function.

As a bonus, the alternative behaviour would be more in
line with people's intuitions.

There are some sharp contrasts here with the default
argument situation. There, the alternative behaviour would
be very wasteful in the majority of cases, and there is
a clean workaround for cases where a mutable default is
needed.

--
Greg

[Random832]

On Fri, Sep 30, 2016, at 20:46, Gregory Ewing wrote:
> What *is* necessary and sufficient is to make each iteration
> of the for-loop create a new binding of the loop variable
> (and not any other variable!).

I don't think that's true. I think this is logic that is excessively
tied to the toy examples that are used to illustrate the problem.

You don't think it's common [at least, as far as defining a lambda
inside a loop at all is common] to do something like this?

for a in collection:
b = some_calculation_of(a)
do_something_with(lambda: ... b ...)

There's a reason that C++'s solution to this problem is an explicit list
of what names are captured as references vs values inside the lambda.

I think what we really need is an explicit expression/block that gives a
list of names and says that all uses of that name in any function
defined inside the expression/block refer to the value that that
expression had when the expression/block was entered.

Suppose:

[final exponent: lambda base: (base ** exponent) for exponent in
range(9)]

for exponent in range: 9
final exponent:
def pwr(base):
return base ** exponent
result.append(pwr)

for a in collection:
b = some_calculation_of(a)
final b: do_something_with(lambda: ... b ...)

Maybe do stuff like "for final var in ..." for both loops and
comprehensions, or "final var = expr:", depending on if we can make the
intuitive-looking syntax well-defined. "final" is a placeholder for
whatever new keyword is chosen, though I will note that even though IIRC
Guido doesn't like this sort of thing, it would only appear next to
identifiers in the proposed syntax and two identifiers not joined by an
operator is meaningless.

An alternative would be, considering that the problem only applies to
nested functions, to expand the "keyword argument" hack into a
less-"hacky" and less verbose solution, e.g. "lambda base, $exponent:
base ** exponent" - creating, in effect, a keyword argument with the
name 'exponent' that can't actually be passed in to the function.

[Rustom Mody]

For the most part you've taken the words out of my mouth!
Some more details...

Yes one basic problem with comprehensions in python is that they are
defined by assignment not binding to the comprehension variable

>
> > I'm not aware of any sensible language that
> > does anything like this.
>
> Scheme and Ruby come to mind as examples of languages in
> which the equivalent of a for-loop results in each iteration
> getting a new binding of the control variable. Although
> you could argue that these languages are not "sensible". :-)

Python copied comprehensions from haskell and copied them wrong

Here are all the things (that I can think of) that are wrong:
1. Scope leakage from inside to outside the comprehension
2. Scope leakage from one value to the next
3. The name 'for' misleadingly associates for-loops and comprehensions
4. The for-loop based implementation strategy made into definitional semantics
5. The absence of simple binding inside comprehensions:
[f(newvar) for v in l newvar = rhs]

1 was considered sufficiently important to make a breaking change from python2 to 3
2 is what causes the lambda gotcha
3 is what makes noobs to take longer than necessary to grok them
4 is what causes a useless distinction between 1 and 2 — scope leakage is scope
leakage. The explanatory mechanisms of why/whither/what etc should at best be secondary
5. is workaroundable with a [... for newvar in [rhs]]
Possible and clunky

[Steve D'Aprano]

On Sat, 1 Oct 2016 10:46 am, Gregory Ewing wrote:

> Steve D'Aprano wrote:
>> Giving for-loops their own namespace is a grossly unintuitive and a very
>> weird thing to do.
>>
>> It would be terribly inconvenient and surprising for if...else blocks to
>> be separate namespaces:
>
> There's an important difference between a for-loop and an
> if-statement that's relevant here: a for-loop binds a name,
> whereas an if-statement doesn't.

That's incorrect: either can bind any number of names:

if condition:
a = 1
b = 2
else:
a = 222
b = 111


for i in seq:
x = i + 1
y = x**2
z = 3*y

> Whenever there's binding going on, it's necessary to decide
> whether it should be creating a new binding or updating an
> existing one.

Right.

> This is actually a *different* issue from one of scope.
> List comprehensions were changed so that the loop variable
> lives in a different scope from the containing function.
> However, they still have the same unintuitive behaviour
> with regard to capture of the loop variable by a lambda.
>
> >> l = [lambda: i for i in range(3)]
> >>> for f in l: print(f())
> ...
> 2
> 2
> 2
>
> Most people consider *this* behaviour to be far weirder
> than anything that would result from giving for-loop
> variables their own scope.
>
> Even if you don't think it's weird, it's hard to argue
> that it's *useful* in any significant number of cases.

Firstly, let's agree (or perhaps we don't?) that loop variables are the same
kind of variable as any other. It would be strange and confusing to have
different kinds of variables with different binding and lookup rules based
on where they came from.

The whole list comprehension and lambda syntax is a red herring. Let's write
it like this:

alist = []
for i in (0, 1, 2):
def func():
return i
alist.append(func)

for f in alist:
print(f())



And the output is the same:

2
2
2


Okay, that's inconvenient and not what I wanted. Obviously. But as they say
about computers, "this damn machine won't do what I want, only what I tell
it to do". It did *exactly what I told it to do*, and, yes, that is a good
thing. Let's remember that the variable i is no more special than any other
variable:

alist = []
x = 0
for i in (0, 1, 2):
def func():
return x
alist.append(func)

x = 999
# What do you predict the functions will return?
for f in alist:
print(f())



Are you surprised that each of the func()'s return the same value, the
current value of x instead of whatever accidental value of x happened to
exist when the function was defined? I should hope not. I would expect that
most people would agree that variable lookups should return the value of
the variable *as it is now*, not as it was when the function was defined.
Let's call that Rule 1, and I say it is fundamental to being able to reason
about code. If I say:

x = 1
spam(x)

then spam() MUST see the current value of x, namely 1, not some mysterious
value of x that happened to exist long ago in the mists of time when spam()
was first defined. And Rule 1 needs to apply to ALL variable look ups, not
just arguments to functions.


Closures, of course, are a special case of Rule 1, not an exception:

def make(arg):
x = arg
def func():
return x
return func

f = make(0)
print(f())


*Without* closures, that would lead to a NameError (unless there happened to
be a global called "x"): the local variables of make() no longer exist, so
you cannot refer to them. That was the behaviour in Python 1.5, for
example.

But *with* closures, the local variables still exist: the inner function
grabs the surrounding environment (or at least as much as it needs) and
keeps it alive, so that it can look up names in that surrounding scope.

What do you expect should happen here?

def make(arg):
x = arg
def func():
return x
x = 999
return func

f = make(0)
print(f())


By Rule 1, the only sensible behaviour is for f() to return 999, regardless
of whether it is convenient or not, regardless of whether that is what I
intended or not. The interpreter shouldn't try to guess what I mean, it
shouldn't cache variable look-ups, and it shouldn't try to give x special
behaviour different from all other variables just so this special case is
more convenient.

But wait, I hear you say, what about closures? They cache the value of the
surrounding variables don't they? Perhaps in a sense, but only a weak
sense. Each call to make() creates a *new* local environment (its a
function, each time you call a function it starts completely afresh) so
each of the inner functions returned is independent of the others, with its
own independent closure. And each inner function still looks up the current
value of *their own* closed-over x, not a snapshot of x as it was when the
inner function was defined.

Final step: let's unroll that original loop.

l = [lambda: i for i in range(3)]
for f in l: print(f())

becomes:

l = []
i = 0
l.append(lambda: i)
i = 1
l.append(lambda: i)
i = 2
l.append(lambda: i)

for f in l:
print(f())

Are you still surprised that it prints 2 2 2? Is this still "unintuitive"?

I say it isn't, and at the risk of coming across as smug and obnoxious I'm
going to say that anyone who *claims* to still be surprised that the three
lambda functions all print the same value for i, namely 2, is almost
certainly playing ignorant because they're unwilling to admit that the
actual behaviour of Python here is exactly what we should both expect and
desire, if only we think about it rather than hoping for some magical Do
What I Mean semantics.

Of course all three functions print the same value -- they're all looking up
the same variable, which can only have one value at a time. And the *last*
thing we would want would be for functions to magically take a snapshot of
variables' value at the time of function creation. (And no, closures do not
violate that rule -- they're a special case, not an exception.)

>> To me, "make for-loops be their own scope" sounds like a joke feature out
>> of joke languages like INTERCAL.
>
> Which is a straw man, since that's not actually what we're
> talking about doing.

Jeez, how is it a strawman when the OP **specifically and explicitly**
refers to making for-loops their own scope???

Quote:

all along i assumed that for-loops got a namespace of their own ...

has there ever been any discussion of giving for-loops the option of
running in namespaces of their own?

Donald Trump says that when elected president, he'll build a wall to keep
Mexicans out of the US. When people call him out on this and explain how
stupid that idea is, do you call it a strawman too?

A strawman is a weak argument that your opponent DID NOT MAKE, not any weak
argument. It is not a fallacy to directly challenge weak arguments. But
false accusations of strawman is a fallacy: poisoning the well.

> It's neither necessary nor sufficient
> to solve the problem.

Right. But that's not *my* problem.

> What *is* necessary and sufficient is to make each iteration
> of the for-loop create a new binding of the loop variable
> (and not any other variable!).

No, that's not right -- they're already new bindings. Variable bindings in
Python are not in-place modifications of a variable like in C, where the
variable is some bucket where you can modify the value in place. They are
values bound to a key in a namespace.

If you don't believe me, try this:


alist = []
for i in (0, 1, 2):
def func():
return i
alist.append(func)
if i == 2: continue
del i


Does it make *any* difference at all to the behaviour? Apart from being a
tiny bit slower, no, of course it does not.

>> I'm not aware of any sensible language that
>> does anything like this.
>
> Scheme and Ruby come to mind as examples of languages in
> which the equivalent of a for-loop results in each iteration
> getting a new binding of the control variable. Although
> you could argue that these languages are not "sensible". :-)

:-)

[Rustom Mody]

On Saturday, October 1, 2016 at 8:55:19 AM UTC+5:30, Random832 wrote:
> On Fri, Sep 30, 2016, at 20:46, Gregory Ewing wrote:
> > What *is* necessary and sufficient is to make each iteration
> > of the for-loop create a new binding of the loop variable
> > (and not any other variable!).
>
> I don't think that's true. I think this is logic that is excessively
> tied to the toy examples that are used to illustrate the problem.
>
> You don't think it's common [at least, as far as defining a lambda
> inside a loop at all is common] to do something like this?
>
> for a in collection:
> b = some_calculation_of(a)
> do_something_with(lambda: ... b ...)

Common? — Dunno
What I know — Yuck!

[Yeah… someone brought up on Lisp and APL and who finds C++ terrifying!]

[Steve D'Aprano]

Earlier, I wrote:

> On Sat, 1 Oct 2016 10:46 am, Gregory Ewing wrote:

[...]

>> Whenever there's binding going on, it's necessary to decide
>> whether it should be creating a new binding or updating an
>> existing one.
>
> Right.

I changed my mind -- I don't think that's correct.

I think Greg's suggestion only makes sense for languages where variables are
boxes with fixed locations, like C or Pascal. In that case, the difference
between creating a new binding and updating a new one is *possibly*
meaningful:

# create a new binding
x: address 1234 ----> [ box contains 999 ]
x: address 5678 ----> [ a different box, containing 888 ]

What happens to the old x? I have no idea, but the new x is a different box.
Maybe the old box remains there, for any existing code that refers to the
address of (old) x. Maybe the compiler is smart enough to add address 1234
to the free list of addresses ready to be used for the next variable. Maybe
its just lost and unavailable until this function exists.

# update an existing binding
x: address 1234 ----> [ box contains 999 ]
x: address 1234 ----> [ same box, now contains 888 ]


That's the normal behaviour of languages like C and Pascal. But its distinct
from the previous, hypothetical behaviour.

But Python doesn't work that way! Variables aren't modelled by boxes in
fixed locations, and there is no difference between "create a new binding"
and "update an existing one". They are indistinguishable. In both
cases, 'x' is a key in a namespace dict, which is associated with a value.
There's no difference between:

x = 999
del x
x = 888

and

x = 999
x = 888


If you consider the namespace dict as a hash table, e.g. something like this
(actual implementations may differ):

[ UNUSED, UNUSED, (key='y', value=23), (key='a', value=True),
UNUSED, (key='x', value=999), UNUSED ]

then binding 888 to 'x' must put the key in the same place in the dict,
since that's where hash('x') will point.

Subject to linear addressing, chaining, re-sizes, or other implementation
details of hash tables of course. But all else being equal, you cannot
distinguish between the "new binding" and "update existing binding"
cases -- in both cases, the same cell in the hash table gets affected,
because that's where hash('x') points. If you put it somewhere else, it
cannot be found.

So I think I was wrong to agree with Greg's statement. I think that for
languages like Python where variables are semantically name bindings in a
namespace rather than fixed addresses, there is no difference between
updating an existing binding and creating a new one.

In a language like Python, the only distinction we can make between name
bindings is, which namespace is the binding in? In other words, what is the
current block of code's scope?

[Chris Angelico]

> ...

> In a language like Python, the only distinction we can make between name
> bindings is, which namespace is the binding in? In other words, what is the
> current block of code's scope?

Yes, this is true; however, it's not difficult to create subscopes. I
put together a POC patch a while ago to make 'with' blocks able to put
their capture names into a separate subscope, transparently. I don't
think it's what Python wants, but it would be reasonably coherent
semantically ("with expr as name:" puts name, and name alone, into a
subscope - any other assignments are not), and wasn't difficult to
implement in CPython.

ChrisA

[Steve D'Aprano]

On Sat, 1 Oct 2016 02:39 am, Chris Angelico wrote:

> On Sat, Oct 1, 2016 at 12:36 AM, Grant Edwards
> <grant.b...@gmail.com> wrote:

>> In C99 a for loop has its own namespac:

[...]

> I believe that's the same semantics as C++ uses, and I agree, it's
> very convenient. Among other things, it means that nested loops behave
> more like they do in Python, with independent iterators:

*scratches head*

So let me see if I understand this...

You're suggesting that C99, where for-loops have their own namespaces, is
MORE like Python (where for-loops DON'T have their own namespace), than
C89, which, like Python, DOESN'T give for-loops their own namespace?

That's ... curious.

I'm not saying you're wrong: after spending three quarters of an hour trying
to fix a six line (including one blank line) C program because I
accidentally closed a comment with /* instead of */, I will believe
anything about C[1]. If you tell me that void causes birth defects and
printf is responsible for the police shootings of unarmed black men, I'll
believe every word of it.

> int main(void)
> {
> for (int i=0; i<5; ++i)
> {
> printf("%d:", i);
> for (int i=0; i<3; ++i)
> printf(" %d", i);
> printf("\n");
> }
> }
>
> Now, granted, this is not something I would ever actually recommend
> doing, and code review is absolutely justified in rejecting this...

So let me see if I understand your argument... for-loop namespaces are good,
because they let you write code that you personally wouldn't write and
would, in fact, reject in a code review.

O-kay.

> but it's a lot better than pure function-scope variables, where you'd
> get stuck in an infinite loop.

That's the risk that you take when you have a C-style for loop and you
modify the loop variable. There's nothing special about the inner loop in
that regard:


#include <stdio.h> /* for printf */

int main(void)
{
for (int i=0; i<5; ++i)
{
printf("%d:", i);

i = 0;
}
}


Solution: don't do that.







[1] Apparently useful error messages are one of those things C programmers
eschew, like type safety, memory safety, and correctness.

[Jussi Piitulainen]

Steve D'Aprano writes:

> In a language like Python, the only distinction we can make between
> name bindings is, which namespace is the binding in? In other words,
> what is the current block of code's scope?

Python already has nested scopes. A local scope for just those variables
introduced in for-loops could be carved out as in the following example.

The following example function has three parameters and two local
variables, one of which is only assigned inside the loop (and it also
references one global variable), in addition to the variable introduced
in the loop.

def eg(wev, a, b):
y = 0
for x in wev: # to be given a new semantics
f(a, x)
f(x, b)
y += 1
z = (y > 12)
return y, z

Replace the loop with a call to a thunk that refers to the variables in
the outer scopes, except for the one variable that the loop introduces -
that variable is to be local to the thunk. So the example function
should be equivalent to something like the following translation.

def eg(wev, a, b):
y = 0
def g31():
nonlocal y, z # the assigned variables
g32 = iter(wev)
while 1: # need to catch StopIteration
x = next(g32) # now local to g31
f(a, x)
f(x, b)
y += 1
z = (y > 12)
g31()
return y, z

A different translation would introduce each x in a different function
for each step in the iteration. That's just a variation on the overall
theme.

[Chris Angelico]

On Sat, Oct 1, 2016 at 6:35 PM, Steve D'Aprano
<steve+...@pearwood.info> wrote:
> On Sat, 1 Oct 2016 02:39 am, Chris Angelico wrote:
>
>> On Sat, Oct 1, 2016 at 12:36 AM, Grant Edwards
>> <grant.b...@gmail.com> wrote:
>
>>> In C99 a for loop has its own namespac:
> [...]
>
>> I believe that's the same semantics as C++ uses, and I agree, it's
>> very convenient. Among other things, it means that nested loops behave
>> more like they do in Python, with independent iterators:
>
> *scratches head*
>
> So let me see if I understand this...
>
> You're suggesting that C99, where for-loops have their own namespaces, is
> MORE like Python (where for-loops DON'T have their own namespace), than
> C89, which, like Python, DOESN'T give for-loops their own namespace?
>
> That's ... curious.

Well, when you put it like that, it does sound very curious!!

Here's how I read it:

# Python
def main():
for i in range(5):
print(i, end=": ")
for i in range(3):
print(i, end=" ")
print("")

/* Classic C */
int main() {
int i;
for (i=0; i<5; ++i) {
printf("%d:", i);
for (i=0; i<3; ++i)

printf(" %d", i);
printf("\n");
}

//C++ or C99
int main() {

for (int i=0; i<5; ++i) {
printf("%d:", i);
for (int i=0; i<3; ++i)
printf(" %d", i);
printf("\n");
}

The first and last have *independent iterators*. When you complete the
inner loop, the outer loop picks up where *it* left off, not where you
happen to have reassigned the iteration variable. The middle one is an
infinite loop.

> I'm not saying you're wrong: after spending three quarters of an hour trying
> to fix a six line (including one blank line) C program because I
> accidentally closed a comment with /* instead of */, I will believe
> anything about C[1]. If you tell me that void causes birth defects and
> printf is responsible for the police shootings of unarmed black men, I'll
> believe every word of it.

Did you turn all warnings on? Look for a "pedantic" mode. You'll get a
lot more information, assuming you're using a modern decent C
compiler. (GCC, for instance, will even warn about situations where it
looks like the braces and indentation don't match, thus bringing a
Python feature to C, more-or-less.)

>> int main(void)
>> {
>> for (int i=0; i<5; ++i)
>> {
>> printf("%d:", i);
>> for (int i=0; i<3; ++i)
>> printf(" %d", i);
>> printf("\n");
>> }
>> }
>>
>> Now, granted, this is not something I would ever actually recommend
>> doing, and code review is absolutely justified in rejecting this...
>
> So let me see if I understand your argument... for-loop namespaces are good,
> because they let you write code that you personally wouldn't write and
> would, in fact, reject in a code review.
>
> O-kay.

Yes. In the first place, I wouldn't necessarily reject this code; it's
a minor issue only, thanks to the namespacing and nested scoping. It
does cost a little in clarity, and I would point it out to a student,
but in less trivial examples than this, it's really not that big a
deal.

*Without* namespacing to keep them apart, this is a serious problem.
Thanks to namespacing, it's only a minor problem (you've shadowed one
variable with another). It's comparable to this Python code:

def send_message(from, to, subj, body):
str = "Mail: From " + from
str += "\nRecipient: To " + to
str += "\nSubject: " + subj
str += "\n\n" + body
send_raw_data(str)

Would you reject this in a code review? Maybe. I wouldn't yell at you
saying "shadowing is fine, you have no right to reject that". But it's
also not inherently a problem. The rules of name shadowing are
well-defined (in both languages) and designed to *permit*, not reject,
constructs like this.

>> but it's a lot better than pure function-scope variables, where you'd
>> get stuck in an infinite loop.
>
> That's the risk that you take when you have a C-style for loop and you
> modify the loop variable. There's nothing special about the inner loop in
> that regard:
>
>
> #include <stdio.h> /* for printf */
> int main(void)
> {
> for (int i=0; i<5; ++i)
> {
> printf("%d:", i);
> i = 0;
> }
> }
>
>
> Solution: don't do that.

Actually, I would call this version a feature. If you deliberately and
consciously reassign the iteration variable, it's usually because you
wanted to adjust the loop. Classic example: iterating through a series
of tokens, and sometimes needing to grab a second token. Python lets
you do this by explicitly calling iter() and next(); C lets you do
this by incrementing the loop counter (which works only with the
less-flexible looping style it has).

But when you didn't intend to reassign it, it's usually because you
had a local name that happened to collide. Sure, that can happen
anywhere, especially with abbreviated variable names (I once had the
amusing situation of trying to use "cursor" and "current" in the same
namespace, both abbreviated "cur"), but it's usually going to involve
some form of inner block construct. It mightn't be a 'for' loop - it
might be a 'while' loop, or it might not even be a loop at all, just
one branch of an 'if'. Unless, of course, you're customarily writing
thousand-line C programs with no control flow structures at all, in
which case I have a nice padded cell for you right over here...

> [1] Apparently useful error messages are one of those things C programmers
> eschew, like type safety, memory safety, and correctness.

Totally. Those error messages cost 0.0000001% in run time performance,
so they had to go.

In all seriousness, though... -Wall makes your C programming life a
lot more bearable.

ChrisA

[Gregory Ewing]

Random832 wrote:

> for a in collection:
> b = some_calculation_of(a)
> final b: do_something_with(lambda: ... b ...)

I would prefer something like

for a in collection:
let b = some_calculation_of(a):
do_something_with(lambda: ... b ...)

--
Greg

[Steve D'Aprano]

On Sat, 1 Oct 2016 01:44 pm, Rustom Mody wrote:

> Yes one basic problem with comprehensions in python is that they are
> defined by assignment not binding to the comprehension variable

¿Que Mr Fawlty?

I'm sorry, I don't understand you.

In Python, all assignments are name bindings. So you seem to be saying:

one basic problem with comprehensions in python is that they

are defined by binding (to the comprehension variable) not

binding to the comprehension variable

which confuses me. How do you think that comprehension variables are
different from all other variables?

And while we're at it, what do you mean by "comprehension variable"? Are you
talking about the variable that the comprehension is bound to:

comprehension_variable = [x+1 for x in sequence]

or are you talking about the loop variable inside the comprehension?

foo = [comprehension_variable+1 for comprehension_variable in sequence]


It is not clear what you mean. Although I suspect it makes no difference --
either way, the nature of the assignment is identical: its a name binding.

> Python copied comprehensions from haskell and copied them wrong
> Here are all the things (that I can think of) that are wrong:
> 1. Scope leakage from inside to outside the comprehension

The core devs agree that this was a mistake. It was rectified in generator
expressions, and for comprehensions in Python 3. This is no longer an
issue.

> 2. Scope leakage from one value to the next

I don't know what that means.

> 3. The name 'for' misleadingly associates for-loops and comprehensions

What is misleading about it?

> 4. The for-loop based implementation strategy made into definitional
> semantics
> 5. The absence of simple binding inside comprehensions:
> [f(newvar) for v in l newvar = rhs]

I don't know what this means.

> 1 was considered sufficiently important to make a breaking change from
> python2 to 3

Correct.

> 2 is what causes the lambda gotcha

I don't think so.

> 3 is what makes noobs to take longer than necessary to grok them

You haven't shown any evidence that beginners take longer than necessary to
grok list comprehensions. Perhaps they take exactly as long as necessary.

And you CERTAINLY haven't demonstrated that the average beginner would
understand Haskell's list comprehensions more easily than Python's.


https://wiki.haskell.org/List_comprehension

https://www.haskell.org/onlinereport/exps.html#sect3.11

> 4 is what causes a useless distinction between 1 and 2 — scope leakage is
> scope leakage.

What is scope leakage?

> The explanatory mechanisms of why/whither/what etc should
> at best be secondary 5. is workaroundable with a [... for newvar in [rhs]]
> Possible and clunky

I don't know what this means.

[Gregory Ewing]

Steve D'Aprano wrote:

> # create a new binding
> x: address 1234 ----> [ box contains 999 ]
> x: address 5678 ----> [ a different box, containing 888 ]

In the context of CPython and nested functions, replace
"box" with "cell".

When I said "creating a new binding" I meant that the
name x refers to different cells at different times.
When I said "updating an existing binding" I meant that
the name x still refers to the same cell, but that cell
refers to a different object.

In a wider context, replace "box" with "slot in a
stack frame" or "slot in a namespace dictionary".

> But Python doesn't work that way! Variables aren't modelled by boxes in
> fixed locations, and there is no difference between "create a new binding"
> and "update an existing one".

There is very much a distintion. Each time you invoke
a function, a new set of bindings is created for all of
its parameters and local names. Assigning to those names
within the function, on the other hand, updates existing
bindings.

--
Greg

[Rustom Mody]

On Saturday, October 1, 2016 at 5:02:30 PM UTC+5:30, Steve D'Aprano wrote:
> On Sat, 1 Oct 2016 01:44 pm, Rustom Mody wrote:
>
> > Yes one basic problem with comprehensions in python is that they are
> > defined by assignment not binding to the comprehension variable
>
> ¿Que Mr Fawlty?
>
> I'm sorry, I don't understand you.
>
> In Python, all assignments are name bindings. So you seem to be saying:
>
> one basic problem with comprehensions in python is that they
> are defined by binding (to the comprehension variable) not
> binding to the comprehension variable
>
> which confuses me. How do you think that comprehension variables are
> different from all other variables?

Sorry... I guess you are right [for once ;-) ]
I was making a *generic* distinction between binding-constructs that *create*
variables, eg function-defs, lambdas, excepts, withs etc and assignment
that *changes* a binding.
This distinction is standard in compiled languages like C where a
int x;
creates the x and
x=rhs;
modifies it

C++ preserves this distinction in user defined types making sure that
the constructor called for an assignment and for an initialization are distinct
even though they look very similar:
x = rhs;
vs
T x = rhs;

In functional languages, both the dynamic ones like lisp as well as the static
ones like Haskell are very strict and sacrosanct about this distinction.

However in python this does not work because assignment both assigns and
creates the variable.

Interestingly there is this thread running right now on haskell-cafe:
https://groups.google.com/forum/#!topic/haskell-cafe/6tqMdy9nGdc
which is inspired by python's “batteries included”
And there there is this comment:
«Python is even more imperative than C++ or Java, it's dynamically typed…»
which (I guess) is because of exactly such features

Long story short: My saying “binding” is meaningless in python.
I should have said function-binding; ie the mechanism by which functions bind
their arguments *anew*
And then define comprehensions not as now done in terms of for loops that
mutatingly extend the list being built up but as recursive functions that get (re)called for every new value of the comprehension variable passed and therefore
fresh-bound as parameter

>
> And while we're at it, what do you mean by "comprehension variable"? Are you
> talking about the variable that the comprehension is bound to:

>
> comprehension_variable = [x+1 for x in sequence]
>
> or are you talking about the loop variable inside the comprehension?
>
> foo = [comprehension_variable+1 for comprehension_variable in sequence]
>

Yeah by comprehension-variable I mean the one that sits left of the ‘in’ inside
The other obviously needn't exist

[Steve D'Aprano]

On Sat, 1 Oct 2016 09:33 pm, Gregory Ewing wrote:

> Steve D'Aprano wrote:
>
>> # create a new binding
>> x: address 1234 ----> [ box contains 999 ]
>> x: address 5678 ----> [ a different box, containing 888 ]
>
> In the context of CPython and nested functions, replace
> "box" with "cell".

Cells in Python are an implementation detail. It only applies to CPython (as
far as I know), certainly not to IronPython, and the interpreter takes care
to ensure that the user-visible behaviour of local variables in cells is
identical to the behaviour of name bindings in dicts.

The only user-visible semantic differences between variables in a dict and
variables in cells are:

- writing to locals() won't necessarily affect the actual locals;
- in Python 3 only, import * and exec in the local names space may be
prohibited.

Everything else (that I know of) is indistinguishable. Python takes care to
hide the differences, for example even though local variables have a cell
pre-allocated when the function is called, trying to access that cell
before a value is bound to the local gives a NameError (UnboundLocalError)
rather than accessing uninitialised memory, as a naive implementation might
have done.

In any case, I think this is going off on a rather wide tangent -- how is
this specifically relevant to the "unintuitive for-loop behavior" the OP
was surprised by?

> When I said "creating a new binding" I meant that the
> name x refers to different cells at different times.
> When I said "updating an existing binding" I meant that
> the name x still refers to the same cell, but that cell
> refers to a different object.

I don't believe that there is any test you can write in Python that will
distinguish those two cases. (Excluding introspection of implementation
details, such as byte-code, undocumented C-level structures, etc.) As far
as ordinary Python operations go, I don't see that there's any difference
between the two.

When you say:

x = 0
x = 1

inside a function, and the interpreter does the name binding twice, there's
no way of telling whether it writes to the same cell each time or not.
Apart from possible performance and memory use, what difference would it
make? It is still the same 'x' name binding, regardless of the
implementation details.

> In a wider context, replace "box" with "slot in a
> stack frame" or "slot in a namespace dictionary".
>
>> But Python doesn't work that way! Variables aren't modelled by boxes in
>> fixed locations, and there is no difference between "create a new
>> binding" and "update an existing one".
>
> There is very much a distintion. Each time you invoke
> a function, a new set of bindings is created for all of
> its parameters and local names. Assigning to those names
> within the function, on the other hand, updates existing
> bindings.

Certainly when you call a function, the local bindings need to be created.
Obviously they didn't exist prior to calling the function! I didn't think
that was the difference you were referring to, and I fail to see how it
could be relevant to the question of for-loop behaviour.

As I understood you, you were referring to assignments to *existing* names.
If a binding for x already exists, then and only then do you have a choice
between:

- update the existing binding for x;
- or create a new binding for x.


If there is no binding for x yet (such as before the function is called),
then you have no choice in the matter: you cannot possibly update what
doesn't exist.

[Jussi Piitulainen]

Rustom Mody writes:

> And then define comprehensions not as now done in terms of for loops
> that mutatingly extend the list being built up but as recursive
> functions that get (re)called for every new value of the comprehension
> variable passed and therefore fresh-bound as parameter

You'd get the magical semantics for comprehensions from the current
definition of comprehension semantics in terms of the loop, if the loop
semantics was magical. Let me demonstrate. (The b-word is unfortunately
not available, so I substitute "magical" instead.)

# delayd = [ lambda : c for c in "abracadabra" ]
# is roughly/almost equal to the following.

delayd = []
for c in "abracadabra":
delayd.append(lambda : c)

print('Python:', ''.join(f() for f in delayd))

# But that for-loop could have been roughly equal to the following,
# giving both the comprehension and the underlying for-loop a
# semantics that some people say they would prefer.

delayd = [] # reset the list, not part of the loop

g1 = iter("abracadabra")
try:
while True:
def g2(c):
delayd.append(lambda : c)
g2(next(g1))
except StopIteration:
pass

print('Magick:', ''.join(f() for f in delayd))

# Output from the above:
# Python: aaaaaaaaaaa
# Magick: abracadabra

[Jussi Piitulainen]

I'm not sure any more to what message this should be a followup, but
here is a demonstration of two different semantics of the for-loop
variable scope/update, this time with nested loops using the same loop
variable name. The first function, tabulate, uses Python semantics ("t"
for true, if you like); the second, fabulate, is a translation ("f" for
false, if you like) that uses the magical semantics where the loop
variable is not only local to the loop but also a different variable on
each iteration. The latter property makes no difference in this
demonstration, but the former does; there's also a spurious counter that
is not local to the nested loops, just to be sure that it works as
expected (it does).

A summary of sorts: it's possible to demonstrate the scope difference in
Python code, with no box in sight; boxes are irrelevant; the relevant
issue is what function and when the loop variable is associated with,
explicitly or implicitly.

def tabulate(m, n):
for i in range(m):
print(i, end = ': ')
c = 0
for i in range(n):
print(i, end = ', ' if i + 1 < n else ' : ')
c += 1
print(i, c)

def fabulate(m, n):
c = None # because c belong in this scope
g1 = iter(range(m))
try:
while True:
def g2(i):
nonlocal c
print(i, end = ': ')
c = 0
g3 = iter(range(n))
try:
while True:
def g4(i):
nonlocal c
print(i, end = ', ' if i + 1 < n else ' : ')
c += 1
g4(next(g3))
except StopIteration:
pass
print(i, c)
g2(next(g1))
except StopIteration:
pass

print('Python:')
tabulate(3, 4)
print()
print('Magick:')
fabulate(3, 4)

# Output from the above, each line being
# outer i: each inner i : i after inner loop, c
# where either c correctly counts inner steps but
# Python inner i clobbers outer i, Magick not.
#
# Python:
# 0: 0, 1, 2, 3 : 3 4
# 1: 0, 1, 2, 3 : 3 4
# 2: 0, 1, 2, 3 : 3 4
#
# Magick:
# 0: 0, 1, 2, 3 : 0 4
# 1: 0, 1, 2, 3 : 1 4
# 2: 0, 1, 2, 3 : 2 4

[Rustom Mody]

Hoo boy1
Thats some tour de force and makes my head spin

Point can be made more simply with map
ie if we *define*
[exp for cv in l]
as
map(lambda cv: exp, l)

the problem vanishes

Demo:

First a helper function for demoing:

def pam(fl,x):
return map(lambda f: f(x), fl)
# pam is the complement to map; map runs one fnc on a list of args
# pam runs a list of funcs on one arg

Trying to make a list of functions that add one, two and three to their arguments

fl = [lambda x: x + cv for cv in [1,2,3]]

Broken because of python's wrong LC semantics:
>>> pam(fl, 3)
[6, 6, 6]

Transform the LC into a map with the rule above:
fl_good = map((lambda cv :lambda x: x+cv), [1,2,3])

Works!
>>> pam(fl_good, 3)
[4, 5, 6]
>>>

Which is not very far from the standard workaround for this gotcha:
>>> fl_workaround = [lambda x, cv=cv: x+cv for cv in [1,2,3]]
>>> pam(fl_workaround, 3)
[4, 5, 6]
>>>

Maybe we could say the workaround is the map definition uncurried
And then re-comprehension-ified

[Steve D'Aprano]

On Sun, 2 Oct 2016 03:57 am, Rustom Mody wrote:

> Hoo boy1
> Thats some tour de force and makes my head spin

I certainly agree with the second part of your sentence.

> Point can be made more simply with map
> ie if we *define*
> [exp for cv in l]
> as
> map(lambda cv: exp, l)
>
> the problem vanishes

>
> Demo:
>
> First a helper function for demoing:
>
> def pam(fl,x):
> return map(lambda f: f(x), fl)
> # pam is the complement to map; map runs one fnc on a list of args
> # pam runs a list of funcs on one arg
>
> Trying to make a list of functions that add one, two and three to their
> arguments
>
> fl = [lambda x: x + cv for cv in [1,2,3]]
>
> Broken because of python's wrong LC semantics:
>>>> pam(fl, 3)
> [6, 6, 6]

Its not *broken*, its doing *exactly what you told it to do*. You said,
define a function that takes a single argument x, and return x + cv. Then
you delayed evaluating it until cv = 3, and passed the argument 3, so of
course it returns 6. That's exactly what you told it to calculate.

You seem to have the concept that lambda should be magical, and just
miraculously know how far back in time to look for the value of cv. And
then when it doesn't, you're angry that Python is "broken". But why should
it be magical? cv is just an ordinary variable, and like all variables,
looking it up returns the value it has at the time you do the look-up, not
some time in the past. Let's unroll the loop:

fl = []
cv = 1
def f(x): return x + cv
fl.append(f)
cv = 2
def f(x): return x + cv
fl.append(f)
cv = 3
def f(x): return x + cv
fl.append(f)

pam(fl, 3)

Are you still surprised that it returns [6, 6, 6]?

> Transform the LC into a map with the rule above:
> fl_good = map((lambda cv :lambda x: x+cv), [1,2,3])

This is equivalent to something completely different, using a closure over
cv, so of course it works:

def factory(cv):
def inner(x):
return x + cv
return inner

fl_good = []
fl_good.append(factory(1))
fl_good.append(factory(2))
fl_good.append(factory(3))


Each time you call factory(), you get a new scope, with its own independent
variable cv. The inner function captures that environment (a closure),
which includes that local variable cv. Each invocation of factory leads to
an inner function that sees a different local variable which is independent
of the others but happens to have the same name. Instead of three functions
all looking up a single cv variable, you have three functions looking up
three different cv variables.

This is essentially why closures exist.

> Which is not very far from the standard workaround for this gotcha:
>>>> fl_workaround = [lambda x, cv=cv: x+cv for cv in [1,2,3]]
>>>> pam(fl_workaround, 3)
> [4, 5, 6]
>>>>
>
> Maybe we could say the workaround is the map definition uncurried
> And then re-comprehension-ified

If your students think in terms of map, then fine, but I think it would
confuse more people than it would help. Your mileage may vary.

There are certainly a number of ways to get the desired behaviour. If you
prefer to work with map, go right ahead.

[Terry Reedy]

On 10/1/2016 8:24 AM, Rustom Mody wrote:

> Yeah by comprehension-variable I mean the one that sits left of the

> ‘in’ inside the conprehension.

In other words, a 'loop variable within a comprehension'. Keep in mind
that there may be multiple targets for the implicit (hidden) assignment,
so there may be multiple loop (comprehension) variables even without
nested loops.

--
Terry Jan Reedy

[Gregory Ewing]

Steve D'Aprano wrote:
> When you say:
>
> x = 0
> x = 1
>
> inside a function, and the interpreter does the name binding twice, there's
> no way of telling whether it writes to the same cell each time or not.

Yes, there is:

... x = 0
... f1 = lambda: x
... x = 1
... f2 = lambda: x
... print(f1(), f2())
...
>>> f()
1 1

This indicates that both assignments updated the same slot.
Otherwise the result would have been "0 1".

It's not currently possible to observe the other behaviour in
Python, because the only way to create new bindings for local
names is to enter a function. The change to for-loop semantics
I'm talking about would introduce another way.

> Certainly when you call a function, the local bindings need to be created.
> Obviously they didn't exist prior to calling the function! I didn't think
> that was the difference you were referring to, and I fail to see how it
> could be relevant to the question of for-loop behaviour.

My proposed change is (mostly) equivalent to turning the
loop body into a thunk and passing the loop variable in as
a parameter.

This is the way for-loops or their equivalent are actually
implemented in Scheme, Ruby, Smalltalk and many other similar
languages, which is why they don't have the same "gotcha".
(Incidentally, this is why some people describe Python's
behaviour here as "broken". They ask -- it works perfectly
well in these other languages, why is Python different?)

The trick with cells is a way to get the same effect in
CPython, without the overhead of an actual function call
on each iteration (and avoiding some of the other problems
that using a thunk would entail).

The cell trick isn't strictly necessary, though -- other
Python implementations could use a thunk if they had to.

--
Greg

[Steve D'Aprano]

On Sun, 2 Oct 2016 12:28 am, Jussi Piitulainen wrote:

> I'm not sure any more to what message this should be a followup, but
> here is a demonstration of two different semantics of the for-loop
> variable scope/update, this time with nested loops using the same loop
> variable name. The first function, tabulate, uses Python semantics ("t"
> for true, if you like); the second, fabulate, is a translation ("f" for
> false, if you like) that uses the magical semantics where the loop
> variable is not only local to the loop but also a different variable on
> each iteration.

I'm sorry, I don't understand what you mean by "Python semantics" versus "a
translation". A translation of what? In what way is it "magical semantics"?
I see nothing magical in your code: it is Python code.

Of course, it is complex, complicated, convoluted, obfuscated, hard to
understand, non-idiomatic Python code, but there's nothing magical in it
(unless you count nonlocal as magic). And it does nothing that can't be
done more simply: just change the tabulate inner loop variable to a
different name.


def fabulate2(m, n):
# The simple, Pythonic, non-complicated way.

for i in range(m):
print(i, end = ': ')
c = 0

for j in range(n):
print(j, end = ', ' if j + 1 < n else ' : ')

c += 1
print(i, c)

Your version of tabulate and fabulate:
py> tabulate(3, 4)

0: 0, 1, 2, 3 : 3 4

1: 0, 1, 2, 3 : 3 4

2: 0, 1, 2, 3 : 3 4

py> fabulate(3, 4)

0: 0, 1, 2, 3 : 0 4

1: 0, 1, 2, 3 : 1 4

2: 0, 1, 2, 3 : 2 4

My simple version of fabulate:

py> fabulate2(3, 4)

0: 0, 1, 2, 3 : 0 4

1: 0, 1, 2, 3 : 1 4

2: 0, 1, 2, 3 : 2 4

> The latter property makes no difference in this
> demonstration, but the former does; there's also a spurious counter that
> is not local to the nested loops, just to be sure that it works as
> expected (it does).
>
> A summary of sorts: it's possible to demonstrate the scope difference in
> Python code, with no box in sight; boxes are irrelevant; the relevant
> issue is what function and when the loop variable is associated with,
> explicitly or implicitly.

I don't know what "scope difference" you think you are demonstrating.
tabulate() has a single scope, fabulate() has multiple scopes because it
has inner functions that take i as argument, making them local to the inner
functions. Um, yeah, of course they are different. They're different
because you've written them differently. What's your point?

As far as I can see, all you have demonstrated is that it is possible to
write obfuscated code in Python. But we already knew that.

[Steve D'Aprano]

On Sun, 2 Oct 2016 11:44 am, Gregory Ewing wrote:

> Steve D'Aprano wrote:
>> When you say:
>>
>> x = 0
>> x = 1
>>
>> inside a function, and the interpreter does the name binding twice,
>> there's no way of telling whether it writes to the same cell each time or
>> not.
>
> Yes, there is:
>
> ... x = 0
> ... f1 = lambda: x
> ... x = 1
> ... f2 = lambda: x
> ... print(f1(), f2())
> ...
> >>> f()
> 1 1
>
> This indicates that both assignments updated the same slot.
> Otherwise the result would have been "0 1".

No it doesn't mean that at all. The result you see is compatible with *both*
the "update existing slot" behaviour and "create a new slot" behavior. The
*easiest* way to prove that is to categorically delete the existing "slot"
and re-create it:

x = 0
f1 = lambda: x
del x
assert 'x' not in locals()
x = 1
f2 = lambda: x
print(f1(), f2())


which will still print exactly the same results.

Objection: I predict that you're going to object that despite the `del x`
and the assertion, *if* this code is run inside a function, the "x slot"
actually does still exist. It's not "really" deleted, the interpreter just
makes sure that the high-level behaviour is the same as if it actually were
deleted.

Well yes, but that's exactly what I'm saying: that's not an objection, it
supports my argument! The way CPython handles local variables is an
implementation detail. The high-level semantics is *identical* between
CPython functions, where local variables live in a static array of "slots"
and re-binding always updates an existing slot, and IronPython, where they
don't. The only way you can tell them apart is by studying the
implementation.

In IronPython, you could have the following occur in a function locals, just
as it could happen CPython for globals:

- delete the name binding "x"
- which triggers a dictionary resize
- bind a value to x again
- because the dictionary is resized, the new "slot" for x is in a
completely different position of the dictionary to the old one

There is no user-visible difference between these two situations. Your code
works the same way whether it is executed inside a function or at the
global top level, whether functions use the CPython local variable
optimization or not.

> It's not currently possible to observe the other behaviour in
> Python, because the only way to create new bindings for local
> names is to enter a function.

Not at all. Delete a name, and the binding for that name is gone. Now assign
to that name again, and a new binding must be created, by definition, since
a moment ago it no longer existed.

x = 1
del x
assert 'x' not in locals()
x = 2

> The change to for-loop semantics
> I'm talking about would introduce another way.

I have lost track of how this is supposed to change for-loop semantics.

I'm especially confused because you seem to be arguing that by using an
implementation which CPython already uses, for-loops would behave
differently. So either I'm not reading you right or you're not explaining
yourself well.

>> Certainly when you call a function, the local bindings need to be
>> created. Obviously they didn't exist prior to calling the function! I
>> didn't think that was the difference you were referring to, and I fail to
>> see how it could be relevant to the question of for-loop behaviour.
>
> My proposed change is (mostly) equivalent to turning the
> loop body into a thunk and passing the loop variable in as
> a parameter.

A thunk is not really well-defined in Python, because it doesn't exist, and
therefore we don't know what properties it will have. But generally when
people talk about thunks, they mean something like a light-weight anonymous
function without any parameters:

https://en.wikipedia.org/wiki/Thunk


You say "passing the loop variable in as a parameter" -- this doesn't make
sense. Variables are not values in Python. You cannot pass in a
*variable* -- you can pass in a name (the string 'x') or the *value* bound
to the name, but there's no existing facility in Python to pass in a
variable. If there was, you could do the classic "pass by reference" test:

Write a *procedure* which takes as argument two variables
and swaps their contents

but there's no facility to do something like that in Python. So its hard to
talk about your hypothetical change except in hand-wavy terms:

"Something magically and undefined happens, which somehow gives the result I
want."

> This is the way for-loops or their equivalent are actually
> implemented in Scheme, Ruby, Smalltalk and many other similar
> languages, which is why they don't have the same "gotcha".

Instead, they presumably have some other gotcha -- "why don't for loops work
the same as unrolled loops?", perhaps.

In Python, the most obvious gotcha would be that if for-loops introduced
their own scope, you would have to declare any other variables in the outer
scope nonlocal. So this would work fine as top-level code:

x = 1
for i in range(10):
print(x)
x += 1

but inside a function it would raise UnboundLocalError, unless you changed
it to this:


def spam():
x = 1
for i in range(10):
nonlocal x
print(x)
x += 1


So we would be swapping a gotcha that only affects a small number of people,
using a fairly advanced concept (functional programming techniques), with
at least one trivial work-around, for a gotcha that would affect nearly
everyone nearly all the time. Yay for progress!

> (Incidentally, this is why some people describe Python's
> behaviour here as "broken". They ask -- it works perfectly
> well in these other languages, why is Python different?)

*shrug*

Define "it". Python works perfectly well too. It just works differently from
what some people expect, especially if they don't think about the meaning
of what they're doing and want the interpreter to DWIM.

> The trick with cells is a way to get the same effect in
> CPython, without the overhead of an actual function call
> on each iteration (and avoiding some of the other problems
> that using a thunk would entail).

"The trick with cells" -- what trick do you mean?

> The cell trick isn't strictly necessary, though -- other
> Python implementations could use a thunk if they had to.




--

[Chris Angelico]

On Sun, Oct 2, 2016 at 3:19 PM, Steve D'Aprano
<steve+...@pearwood.info> wrote:
> In IronPython, you could have the following occur in a function locals, just
> as it could happen CPython for globals:
>
> - delete the name binding "x"
> - which triggers a dictionary resize
> - bind a value to x again
> - because the dictionary is resized, the new "slot" for x is in a
> completely different position of the dictionary to the old one
>
> There is no user-visible difference between these two situations. Your code
> works the same way whether it is executed inside a function or at the
> global top level, whether functions use the CPython local variable
> optimization or not.

Hmm, interesting. I don't have IronPython here, but maybe you can tell
me what this does:

print(str)
str = "demo"
print(str)
del str
print(str)

and the same inside a function. In CPython, the presence of 'str =
"demo"' makes str function-local, ergo UnboundLocalError on the first
reference; but globals quietly shadow built-ins, so this will print
the class, demo, and the class again. If IronPython locals behave the
way CPython globals behave, that would most definitely be a
user-visible change to shadowing semantics.

ChrisA

[Rustom Mody]

You are explaining the mechanism behind the bug. Thanks. The bug remains.
My new car goes in reverse when I put it in first gear but only on full-moon
nights with the tank on reserve when the left light is blinking
The engineer explains the interesting software bug in the new PCB.
Interesting. But the bug remains

Likewise here:

[2 ** i for i in[1,2]] == [2**1, 2**2]

yet this fails

[lambda x: x + i for i in [1,2]] == [lambda x:x+1, lambda x:x=2]

is a bug for anyone including the OP of this thread

>
>
>
> > Transform the LC into a map with the rule above:
> > fl_good = map((lambda cv :lambda x: x+cv), [1,2,3])
>
>
> This is equivalent to something completely different, using a closure over
> cv, so of course it works:
>
> def factory(cv):
> def inner(x):
> return x + cv
> return inner
>
> fl_good = []
> fl_good.append(factory(1))
> fl_good.append(factory(2))
> fl_good.append(factory(3))
>
>
> Each time you call factory(), you get a new scope, with its own independent
> variable cv. The inner function captures that environment (a closure),
> which includes that local variable cv. Each invocation of factory leads to
> an inner function that sees a different local variable which is independent
> of the others but happens to have the same name. Instead of three functions
> all looking up a single cv variable, you have three functions looking up
> three different cv variables.
>
> This is essentially why closures exist.

Closures and lambdas are synonymous:
martin Fowler's article on lambda has this first sentence:

«a programming concept called Lambdas (also called Closures, Anonymous
Functions or Blocks)»
http://martinfowler.com/bliki/Lambda.html

So if you like you can say the bug here is that python lambdas fail to close
variables properly.
Which is a double-bug because the blame for closure-failure is falling
onto the lambda whereas it is the comprehension semantics in terms of loops
rather than in terms of closures/cells as Jussi/Greg are trying to show, that
is at fault

>
>
> > Which is not very far from the standard workaround for this gotcha:
> >>>> fl_workaround = [lambda x, cv=cv: x+cv for cv in [1,2,3]]
> >>>> pam(fl_workaround, 3)
> > [4, 5, 6]
> >>>>
> >
> > Maybe we could say the workaround is the map definition uncurried
> > And then re-comprehension-ified
>
> If your students think in terms of map, then fine, but I think it would
> confuse more people than it would help. Your mileage may vary.

You are as usual putting words in my mouth.
I did not talk of my students or that map is easier than comprehensions
I was taking the Haskell comprehension-semantics link (which BTW you posted!)
https://www.haskell.org/onlinereport/exps.html#sect3.11

Simplifying it to the simple case of only one for, no ifs
This allows the flatmap to become a map and the multiple cases to go
Thereby more succinctly highlighting the problem

[Jussi Piitulainen]

Steve D'Aprano writes:

> On Sun, 2 Oct 2016 12:28 am, Jussi Piitulainen wrote:
>
>> I'm not sure any more to what message this should be a followup, but
>> here is a demonstration of two different semantics of the for-loop
>> variable scope/update, this time with nested loops using the same
>> loop variable name. The first function, tabulate, uses Python
>> semantics ("t" for true, if you like); the second, fabulate, is a
>> translation ("f" for false, if you like) that uses the magical
>> semantics where the loop variable is not only local to the loop but
>> also a different variable on each iteration.
>
> I'm sorry, I don't understand what you mean by "Python semantics" versus "a
> translation". A translation of what? In what way is it "magical semantics"?
> I see nothing magical in your code: it is Python code.

"Python semantics" is the semantics that Python actually uses when it
updated the variables of a for-loop. I would like to call it "assignment
semantics".

The second function is an executable translation of the first function
under a different semantics. I would like to call it "binding semantics"
to distinguish it from "assignment semantics", but that doesn't work
when people insist that "binding" and "assignment" are the same thing in
Python, so I called it "magical semantics" instead.

The method of using equivalent code as an explanation is known in Python
documentation, as in the explanation of list comprehensions in terms of
list.append and for-loops. This is what I called "translation" above. In
another community it might be called "syntactic transformation" or
"macro expansion".

> Of course, it is complex, complicated, convoluted, obfuscated, hard to
> understand, non-idiomatic Python code, but there's nothing magical in
> it (unless you count nonlocal as magic). And it does nothing that
> can't be done more simply: just change the tabulate inner loop
> variable to a different name.

It was stated, in this thread, that it would have been impossible to
make Python for-loops behave the way the corresponding Python code in my
translations of the two nested for-loops behaves. I thought it would be
a good idea to *show* how the "impossible" thing can be done.

I could have just posted that yes, it *could* be done, there's nothing
so special about variables and scope in Python. But I pretty much know
that the response to *that* would have been yet another round of "You
don't understand that Python variables are different, they are not boxes
in fixes locations but name bindings, and no, it cannot be done."

> def fabulate2(m, n):
> # The simple, Pythonic, non-complicated way.
> for i in range(m):
> print(i, end = ': ')
> c = 0
> for j in range(n):
> print(j, end = ', ' if j + 1 < n else ' : ')
> c += 1
> print(i, c)

It misses only the point of the exercise.

The scope difference is the topic of this thread.

My point is that the for-loops could be translated/compiled/expanded
into the lower-level code so that the loop variables would be in their
own scopes.

> As far as I can see, all you have demonstrated is that it is possible to
> write obfuscated code in Python. But we already knew that.

The point is that the straightforward code *could* have the same meaning
as the "obfuscated" code. That for-loops *could* be magicking constructs
even in Python. (What word should I use instead of "magic"?)

The code might not seem so obfuscated to you if you thought of it not as
source code but as compiled code. Except it's still Python. What word
should I use instead of "translation"? Would "transformation" be
understood?

[Marko Rauhamaa]

Jussi Piitulainen <jussi.pi...@helsinki.fi>:

> I could have just posted that yes, it *could* be done, there's nothing
> so special about variables and scope in Python. But I pretty much know
> that the response to *that* would have been yet another round of "You
> don't understand that Python variables are different, they are not
> boxes in fixes locations but name bindings, and no, it cannot be
> done."

Your valiant effort to get a point across has been noted.


Marko

[Jussi Piitulainen]

Steve D'Aprano writes:

> On Sun, 2 Oct 2016 11:44 am, Gregory Ewing wrote:
>
>> Steve D'Aprano wrote:
>>
>>> Certainly when you call a function, the local bindings need to be
>>> created. Obviously they didn't exist prior to calling the function!
>>> I didn't think that was the difference you were referring to, and I
>>> fail to see how it could be relevant to the question of for-loop
>>> behaviour.
>>
>> My proposed change is (mostly) equivalent to turning the
>> loop body into a thunk and passing the loop variable in as
>> a parameter.
>
> A thunk is not really well-defined in Python, because it doesn't
> exist, and therefore we don't know what properties it will have. But
> generally when people talk about thunks, they mean something like a
> light-weight anonymous function without any parameters:
>
> https://en.wikipedia.org/wiki/Thunk

According to a story (probably told in that page), thunks got their name
from being "already thunk of". Scheme at least doesn't have any special
"light-weight" anonymous functions, it just has (lambda () ...).

> You say "passing the loop variable in as a parameter" -- this doesn't
> make sense. Variables are not values in Python. You cannot pass in a
> *variable* -- you can pass in a name (the string 'x') or the *value*
> bound to the name, but there's no existing facility in Python to pass
> in a variable.

He means it like this:

def g4159(i):
<loop body here>

g4159(next(g4158))

The loop variable becomes the parameter of a function. The values are
passed to that function, one by one.

[snip tangent]

> hard to talk about your hypothetical change except in hand-wavy terms:
>
> "Something magically and undefined happens, which somehow gives the
> result I want."

The following is intended to be more concrete. You, Steve, have already
declared (in a followup to me in this thread) that it is not magical
(but is obfuscated, convoluted, non-idiomatic, and so on, but then you
had not understood that the expansion is not intended to be actual
source code).

So a Python for-loop *could* behave (*could have been defined to*
behave) so that

for v in s:
c

is equivalent to

try:
gs = iter(s)
while True:
def gf(v):
<nonlocal declarations>
c
gf(next(gs))
except StopIteration:
pass

where gs and gf are new names and <nonlocal declarations> make it so
that only v becomes local to gf.

>> This is the way for-loops or their equivalent are actually
>> implemented in Scheme, Ruby, Smalltalk and many other similar
>> languages, which is why they don't have the same "gotcha".
>
> Instead, they presumably have some other gotcha -- "why don't for
> loops work the same as unrolled loops?", perhaps.

I don't think so, but then I only know Scheme where tail recursion is
its own reward.

On the other hand, in Scheme I can actually implement any number of
different looping constructs if I find them desirable. Instead of
trying to explain that something is, in principle, not only possible but
also quite straightforward, I could just show an actual implementation.

(I did that once, but then it was a beginner who thought that some
really basic thing would be impossible merely because they were such a
beginner.)

> In Python, the most obvious gotcha would be that if for-loops
> introduced their own scope, you would have to declare any other

> variables in the outer scope nonlocal. [snip]

The intention is that the for-loop would own the loop variables. Other
variables would stay in the outer scope.

[Jussi Piitulainen]

Marko Rauhamaa writes:

> Jussi Piitulainen wrote:
>> I could have just posted that yes, it *could* be done, there's
>> nothing so special about variables and scope in Python. But I pretty
>> much know that the response to *that* would have been yet another
>> round of "You don't understand that Python variables are different,
>> they are not boxes in fixes locations but name bindings, and no, it
>> cannot be done."
>
> Your valiant effort to get a point across has been noted.

Thank you. It's nice to be heard.

[Steve D'Aprano]

That's a nice catch!

But its not a difference between "update binding" versus "new binding" --
its a difference between LOAD_FAST and LOAD_whatever is used for things
besides locals.

steve@orac:~$ ipy
IronPython 2.6 Beta 2 DEBUG (2.6.0.20) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
>>> def func():
... print(str)
... str = 1
...
>>> func()
Traceback (most recent call last):
UnboundLocalError: Local variable 'str' referenced before assignment.

Although IronPython does the same thing as CPython here, I'm not 100% sure
that this behaviour would be considered language specification or a mere
guideline. If the author of an alternate implementation wanted to specify a
single name resolution procedure:

- look for local variable;
- look for nonlocal;
- look for global;
- look for builtin;
- fail

rather than two:

(1)
- look for local;
- fail;

(2)
- look for nonlocal;
- look for global;
- look for builtin;
- fail


I'm not entirely sure that Guido would say No. I think that "fail early if
the local doesn't exist" as CPython does would be permitted rather than
mandatory. But I could be wrong.



By the way, here's an example showing that IronPython does allowing writing
to locals to affect the local namespace:

>>> def func():
... locals()['x'] = 1
... print(x)
... if False:
... x = 9999
...
>>> func()
1

And *that* behaviour is most definitely allowed -- the fact that writing to
locals() isn't supported by CPython is most definitely an implementation-
specific limitation, not a feature.

[Chris Angelico]

On Sun, Oct 2, 2016 at 9:20 PM, Steve D'Aprano
<steve+...@pearwood.info> wrote:
> On Sun, 2 Oct 2016 04:06 pm, Chris Angelico wrote:
>> Hmm, interesting. I don't have IronPython here, but maybe you can tell
>> me what this does:
>>
>> print(str)
>> str = "demo"
>> print(str)
>> del str
>> print(str)
>>
>> and the same inside a function. In CPython, the presence of 'str =
>> "demo"' makes str function-local, ergo UnboundLocalError on the first
>> reference; but globals quietly shadow built-ins, so this will print
>> the class, demo, and the class again. If IronPython locals behave the
>> way CPython globals behave, that would most definitely be a
>> user-visible change to shadowing semantics.
>
> That's a nice catch!
>
> But its not a difference between "update binding" versus "new binding" --
> its a difference between LOAD_FAST and LOAD_whatever is used for things
> besides locals.

The only way to prove that something is a new binding is to
demonstrate that, when this binding is removed, a previous one becomes
visible. In Python, that only ever happens across namespaces, and in
CPython, the only way I can make it happen is with globals->builtins.

> steve@orac:~$ ipy
> IronPython 2.6 Beta 2 DEBUG (2.6.0.20) on .NET 2.0.50727.1433
> Type "help", "copyright", "credits" or "license" for more information.
>>>> def func():
> ... print(str)
> ... str = 1
> ...
>>>> func()
> Traceback (most recent call last):
> UnboundLocalError: Local variable 'str' referenced before assignment.

Beautiful, thank you.

> Although IronPython does the same thing as CPython here, I'm not 100% sure
> that this behaviour would be considered language specification or a mere
> guideline. If the author of an alternate implementation wanted to specify a
> single name resolution procedure:
>
> - look for local variable;
> - look for nonlocal;
> - look for global;
> - look for builtin;
> - fail
>
> rather than two:
>
> (1)
> - look for local;
> - fail;
>
> (2)
> - look for nonlocal;
> - look for global;
> - look for builtin;
> - fail
>
>
> I'm not entirely sure that Guido would say No. I think that "fail early if
> the local doesn't exist" as CPython does would be permitted rather than
> mandatory. But I could be wrong.

Interesting. The real question, then, is: Are function-local names
inherently special? I can't make anything else do this. Class
namespaces don't shadow globals or builtins, even during construction:

>>> class X:
... str = "test"
... def func(self):
... print(self)
... return str
... print("123 =", func(123))
... print("str =", func(str))
...
123
123 = <class 'str'>
test
str = <class 'str'>
>>> X().func()
<__main__.X object at 0x7f98643cc2b0>
<class 'str'>

The 'str' as a function parameter uses the class's namespace, but in
the function, no sir. (As it should be.) I don't know of any other
namespaces to test.

> By the way, here's an example showing that IronPython does allowing writing
> to locals to affect the local namespace:
>
>>>> def func():
> ... locals()['x'] = 1
> ... print(x)
> ... if False:
> ... x = 9999
> ...
>>>> func()
> 1
>
> And *that* behaviour is most definitely allowed -- the fact that writing to
> locals() isn't supported by CPython is most definitely an implementation-
> specific limitation, not a feature.

Yep. Would the same have happened if you'd omitted the "if False"
part, though - that is, if 'x' were not a local name due to
assignment, could it *become* local through mutation of locals()?

I would just grab IronPython and test, except that it isn't available
in the current Debian repositories:

https://archive.debian.net/search?searchon=names&keywords=ironpython

This most likely means I'll have to do some fiddling around with deps
to get it to install :(

ChrisA

[Steve D'Aprano]

On Sun, 2 Oct 2016 09:41 pm, Chris Angelico wrote:

> The only way to prove that something is a new binding is to
> demonstrate that, when this binding is removed, a previous one becomes
> visible. In Python, that only ever happens across namespaces, and in
> CPython, the only way I can make it happen is with globals->builtins.

It isn't necessary for a previous binding to become visible, because there
might not be any previous binding. There's no builtin 'x', but there is a
builtin 'str'. Hence:

x = str = 1
assert x == 1 and str == 1
del x, str
assert str # succeeds
assert x # NameError
x = str = 2 # create new bindings, or update existing ones?

Is it our conclusion that therefore Python creates a new binding for str
but not for x? Or that the evidence for x is "inconclusive"? Either answer
is illogical.

NameError should be sufficient to prove that the binding to 'x' is gone and
therefore any binding to 'x' is semantically a new binding, not an update.



[...]

>> I'm not entirely sure that Guido would say No. I think that "fail early
>> if the local doesn't exist" as CPython does would be permitted rather
>> than mandatory. But I could be wrong.
>
> Interesting. The real question, then, is: Are function-local names
> inherently special? I can't make anything else do this. Class
> namespaces don't shadow globals or builtins, even during construction:

Actually, they do:

py> class Example:
... str = 1
... x = str(999)
...

Traceback (most recent call last):

File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in Example
TypeError: 'int' object is not callable


Locals are certainly special. The language definition singles out binding[1]
to a variable inside a function as the trigger to treat it as a local, so
there are distinct "lookup name which is a local" and "lookup name which is
not a local" modes. Those distinct modes go back to at least Python 1.5.

"Local lookup" (LOAD_FAST) only looks for a local, and fails fast if it
isn't found, rather than searching globals and builtins; "not a local
lookup" never looks for a local (even if one exists -- see below).

>> By the way, here's an example showing that IronPython does allowing
>> writing to locals to affect the local namespace:
>>
>>>>> def func():
>> ... locals()['x'] = 1
>> ... print(x)
>> ... if False:
>> ... x = 9999
>> ...
>>>>> func()
>> 1
>>
>> And *that* behaviour is most definitely allowed -- the fact that writing
>> to locals() isn't supported by CPython is most definitely an
>> implementation- specific limitation, not a feature.
>
> Yep. Would the same have happened if you'd omitted the "if False"
> part, though - that is, if 'x' were not a local name due to
> assignment, could it *become* local through mutation of locals()?

Yes the local would have been created, but you wouldn't see it from an
ordinary lookup of name x. You would need to use

locals()['x']

to see that it exists. For x to return it, you have to convince the compiler
to use LOAD_FAST bytecode rather than LOAD_whatever.

IronPython doesn't use LOAD_* bytecodes, it compiles to whatever .Net's CLR
uses, but it follows the same semantics. Likewise Jython.

My feeling is that we're in a rather murky part of Python's behaviour here,
one where a clean name lookup model has been most definitely put aside in
favour of speed, but it isn't quite clear whether this is defined language
behaviour, reference behaviour that all implementations are required to
follow, or implementation-specific behaviour that may be varied but so far
hasn't been, at least not by any of the Big Four implementations (CPython,
PyPy, IronPython, Jython).





[1] For the purposes of deciding whether something is local or not,
unbinding (del) is considered a binding.

[Steve D'Aprano]

On Sun, 2 Oct 2016 05:35 pm, Jussi Piitulainen wrote:

[...]

>> I'm sorry, I don't understand what you mean by "Python semantics" versus
>> "a translation". A translation of what? In what way is it "magical
>> semantics"? I see nothing magical in your code: it is Python code.
>
> "Python semantics" is the semantics that Python actually uses when it
> updated the variables of a for-loop. I would like to call it "assignment
> semantics".
>
> The second function is an executable translation of the first function
> under a different semantics. I would like to call it "binding semantics"
> to distinguish it from "assignment semantics", but that doesn't work
> when people insist that "binding" and "assignment" are the same thing in
> Python, so I called it "magical semantics" instead.

Why shouldn't people say that binding and assignment are the same thing in
Python? What's the difference?



[...]

>> Of course, it is complex, complicated, convoluted, obfuscated, hard to
>> understand, non-idiomatic Python code, but there's nothing magical in
>> it (unless you count nonlocal as magic). And it does nothing that
>> can't be done more simply: just change the tabulate inner loop
>> variable to a different name.
>
> It was stated, in this thread, that it would have been impossible to
> make Python for-loops behave the way the corresponding Python code in my
> translations of the two nested for-loops behaves. I thought it would be
> a good idea to *show* how the "impossible" thing can be done.

I don't recall ever using the word "impossible", or seeing anyone else use
it. Obviously nothing is *literally* impossible unless it breaks the laws
of physics or requires more energy than available in the entire universe.

It goes without saying that if the Python core developers decide to change
the semantics of Python, add new features and functionality, add new
syntax, etc then such things are possible. They might be easy or they might
be hard -- in some cases *very* hard, requiring the Python VM to be rebuilt
from the ground up. But possible.

One should be careful about using the term "impossible", as well as taking
it literally.

> I could have just posted that yes, it *could* be done, there's nothing
> so special about variables and scope in Python. But I pretty much know
> that the response to *that* would have been yet another round of "You
> don't understand that Python variables are different, they are not boxes
> in fixes locations but name bindings, and no, it cannot be done."

I'm glad you know so much about what we would say before we say it.

>> def fabulate2(m, n):
>> # The simple, Pythonic, non-complicated way.
>> for i in range(m):
>> print(i, end = ': ')
>> c = 0
>> for j in range(n):
>> print(j, end = ', ' if j + 1 < n else ' : ')
>> c += 1
>> print(i, c)
>
> It misses only the point of the exercise.

Perhaps that's because your point wasn't clear.

>>> A summary of sorts: it's possible to demonstrate the scope difference in
>>> Python code, with no box in sight; boxes are irrelevant; the relevant
>>> issue is what function and when the loop variable is associated with,
>>> explicitly or implicitly.
>>
>> I don't know what "scope difference" you think you are demonstrating.
>> tabulate() has a single scope, fabulate() has multiple scopes because it
>> has inner functions that take i as argument, making them local to the
>> inner functions. Um, yeah, of course they are different. They're
>> different because you've written them differently. What's your point?
>
> The scope difference is the topic of this thread.
>
> My point is that the for-loops could be translated/compiled/expanded
> into the lower-level code so that the loop variables would be in their
> own scopes.

Well why didn't you say so?

Perhaps I missed it the first time, but the point of the exercise would have
been so much more clear if only you had introduced it was an explanation
like:

"Here is a proof of concept that shows that Python could give for loops
their own scope. If this Python code [nested for-loops] were translated
mechanically by the compiler [nested while loops with iterators] then each
loop would have their own scope."


And you are absolutely right: boxes are irrelevant. I think that boxes only
came into this with Greg's argument that there's a difference
between "creating a new binding" versus "updating an existing one".

> The code might not seem so obfuscated to you if you thought of it not as
> source code but as compiled code. Except it's still Python. What word
> should I use instead of "translation"? Would "transformation" be
> understood?

That might have helped.

[Steven D'Aprano]

On Monday 03 October 2016 12:42, Steve D'Aprano wrote:

> Yes the local would have been created, but you wouldn't see it from an
> ordinary lookup of name x. You would need to use
>
> locals()['x']
>
> to see that it exists. For x to return it, you have to convince the compiler
> to use LOAD_FAST bytecode rather than LOAD_whatever.

Sorry, that first sentence is misleading. The question is, can you create local
variables inside a function using locals()[name]? IronPython and Jython allow
it; CPython doesn't.

But if it did, then just as in the similar situation in IronPython and Jython,
you would still need to convince the compiler to use LOAD_FAST or equivalent in
order to see it. Hence:


x = y = 'global'
def example():
locals()['x'] = 'local'
locals()['y'] = 'local'
if False:
# convince the compiler to treat y as a local
del y
print(x) # => 'global' in everything
print(y) # => 'local' in Jython/IronPython, UnboundLocalError in CPython



--
Steven
git gets easier once you get the basic idea that branches are homeomorphic
endofunctors mapping submanifolds of a Hilbert space.

[Chris Angelico]

On Mon, Oct 3, 2016 at 3:09 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
> But if it did, then just as in the similar situation in IronPython and Jython,
> you would still need to convince the compiler to use LOAD_FAST or equivalent in
> order to see it. Hence:
>
>
> x = y = 'global'
> def example():
> locals()['x'] = 'local'
> locals()['y'] = 'local'
> if False:
> # convince the compiler to treat y as a local
> del y
> print(x) # => 'global' in everything
> print(y) # => 'local' in Jython/IronPython, UnboundLocalError in CPython

Or convince it to use LOAD_NAME. They're Python 2 implementations,
right? Here's CPython at work...

Python 2.7.12+ (default, Sep 1 2016, 20:27:38)
[GCC 6.2.0 20160822] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> x = y = 'global'
>>> def example():

... locals()['x'] = 'local'
... locals()['y'] = 'local'
... if False:
... exec("")
... print(x)
... print(y)
...
>>> example()
local
local

So apparently, locals() mutations CAN affect name lookups. Happens the
same way in Jython, too.

This is quite a rabbit-hole.

ChrisA

[Jussi Piitulainen]

Steve D'Aprano writes:

> On Sun, 2 Oct 2016 05:35 pm, Jussi Piitulainen wrote:
>
> [...]
>>> I'm sorry, I don't understand what you mean by "Python semantics" versus
>>> "a translation". A translation of what? In what way is it "magical
>>> semantics"? I see nothing magical in your code: it is Python code.
>>
>> "Python semantics" is the semantics that Python actually uses when it
>> updated the variables of a for-loop. I would like to call it "assignment
>> semantics".
>>
>> The second function is an executable translation of the first function
>> under a different semantics. I would like to call it "binding semantics"
>> to distinguish it from "assignment semantics", but that doesn't work
>> when people insist that "binding" and "assignment" are the same thing in
>> Python, so I called it "magical semantics" instead.
>
> Why shouldn't people say that binding and assignment are the same
> thing in Python? What's the difference?

Outside Python, (lambda x : f(x)) is said to "bind" x. It's different
from assigning a new value to x. It's similar to how quantifiers in
logic are said to bind their variables, so that variables bound to
different quantifiers are different variables.

[...]

>>> def fabulate2(m, n):
>>> # The simple, Pythonic, non-complicated way.
>>> for i in range(m):
>>> print(i, end = ': ')
>>> c = 0
>>> for j in range(n):
>>> print(j, end = ', ' if j + 1 < n else ' : ')
>>> c += 1
>>> print(i, c)
>>
>> It misses only the point of the exercise.
>
> Perhaps that's because your point wasn't clear.

Perhaps.

>>>> A summary of sorts: it's possible to demonstrate the scope
>>>> difference in Python code, with no box in sight; boxes are
>>>> irrelevant; the relevant issue is what function and when the loop
>>>> variable is associated with, explicitly or implicitly.
>>>
>>> I don't know what "scope difference" you think you are
>>> demonstrating. tabulate() has a single scope, fabulate() has
>>> multiple scopes because it has inner functions that take i as
>>> argument, making them local to the inner functions. Um, yeah, of
>>> course they are different. They're different because you've written
>>> them differently. What's your point?
>>
>> The scope difference is the topic of this thread.
>>
>> My point is that the for-loops could be translated/compiled/expanded
>> into the lower-level code so that the loop variables would be in
>> their own scopes.
>
> Well why didn't you say so?

I tried.

> Perhaps I missed it the first time, but the point of the exercise
> would have been so much more clear if only you had introduced it was
> an explanation like:
>
> "Here is a proof of concept that shows that Python could give for
> loops their own scope. If this Python code [nested for-loops] were
> translated mechanically by the compiler [nested while loops with
> iterators] then each loop would have their own scope."

Something like that but not quite that. The nonlocal declarations are
also important, but since they would not be visible in the code that the
programmer writes, it would be misleading to say that the loop has its
own scope.

In my original sketch perhaps "each loop variable is in its own scope",
and in the current version "the loop variable of each iteration step is
in its own scope"? Would those words work for you?

I would like to be able to say simply that the for-loop would "bind" its
variable, as opposed to assigning to it (and maybe update it by binding
as opposed to assignment), but this clashes with the way these terms are
used in the Python community.

> And you are absolutely right: boxes are irrelevant. I think that boxes
> only came into this with Greg's argument that there's a difference
> between "creating a new binding" versus "updating an existing one".

The frustrating thing is that Greg's posts make perfect sense to me, and
I agree with him about the difference.

How about "creating a new scope for the loop variable" versus "updating
the loop variable in the existing scope" then?

>> The code might not seem so obfuscated to you if you thought of it not
>> as source code but as compiled code. Except it's still Python. What
>> word should I use instead of "translation"? Would "transformation" be
>> understood?
>
> That might have helped.

Well. I think you understand now.

[Gregory Ewing]

Steve D'Aprano wrote:
> No it doesn't mean that at all. The result you see is compatible with *both*
> the "update existing slot" behaviour and "create a new slot" behavior.

We're getting sidetracked talking about slots here.
It's not really relevant. The point is that there is
only *one* binding for x here, and the second
assignment is updating that binding, not creating a
new binding. If it were otherwise, something different
would have been printed.

This is a high-level concept that has nothing to do
with implementation details. It's the *definition*
of what's meant by the terms "new binding" and
"existing binding".

> Objection: I predict that you're going to object that despite the `del x`
> and the assertion, *if* this code is run inside a function, the "x slot"
> actually does still exist. It's not "really" deleted, the interpreter just
> makes sure that the high-level behaviour is the same as if it actually were
> deleted.

Actually, it's the same as if it were *not* deleted.

Interestingly, if you run it inside a function, something different
again happens:

>>> def f():

... x = 0
... f1 = lambda: x

... del x
... assert 'x' not in locals()

... x = 1
... f2 = lambda: x
... print(f1(), f2())
...

SyntaxError: can not delete variable 'x' referenced in nested scope

So no, I'm not going to raise that objection. :-)

> In IronPython, you could have the following occur in a function locals, just
> as it could happen CPython for globals:
>
> - delete the name binding "x"
> - which triggers a dictionary resize
> - bind a value to x again
> - because the dictionary is resized, the new "slot" for x is in a
> completely different position of the dictionary to the old one

Which is all completely irrelevant. The important thing is
what's seen by closures created at different times within
that function.

I don't have an IronPython handy to try it, but if you do,
I'd be interested to know the result. (I'm fairly sure it
would still show same-binding behaviour, but you never
know.)

> Your code
> works the same way whether it is executed inside a function or at the
> global top level,

I've never claimed otherwise.

>>It's not currently possible to observe the other behaviour in
>>Python, because the only way to create new bindings for local
>>names is to enter a function.
>
> Not at all. Delete a name, and the binding for that name is gone. Now assign
> to that name again, and a new binding must be created,

No, you still don't understand what "new binding" means. If it
truly were creating a new binding, the observable behaviour
would be different, *by definition*.

> I'm especially confused because you seem to be arguing that by using an
> implementation which CPython already uses, for-loops would behave
> differently.

No, the implementation I'm talking about is definitely *not*
being used already.

> A thunk is not really well-defined in Python, because it doesn't exist, and
> therefore we don't know what properties it will have.

Just call it a function, then. Here's an example of the
transformation I'm talking about.

Before:

def f():
funcs = []
for x in range(3):
funcs.append(lambda: x)
for g in funcs:
print g()

After:

def f():
funcs = []
def _body(x):
funcs.append(lambda: x)
for _x in range(3):
_body(_x)
for g in funcs:
print g()

> In Python, the most obvious gotcha would be that if for-loops introduced
> their own scope, you would have to declare any other variables in the outer
> scope nonlocal.

I'm not suggesting they should. If the above transformation
were used as an actual implementation, the compiler would
have to introduce implicit 'nonlocal' declarations as needed
to keep the semantics of assignments within the loop body
the same.

>>(Incidentally, this is why some people describe Python's
>>behaviour here as "broken". They ask -- it works perfectly
>>well in these other languages, why is Python different?)
>

> Define "it". Python works perfectly well too.

So you think that useless behaviour is just as good as
useful behaviour?

> It just works differently from
> what some people expect, especially if they don't think about the meaning
> of what they're doing and want the interpreter to DWIM.

The point is that if they write the analogous thing in
any of those other languages, it *does* DWTM. So I don't
blame anyone for thinking that Python has some explaining
to do.

> "The trick with cells" -- what trick do you mean?

Each time around the loop, if the loop variable is in a
cell, instead of updating the contents of that cell (as
a normal assigment would do), create a new cell.

I don't know how to explain it any more clearly than
that. I can't write it in Python, because Python doesnt
currently have any way to express that operation. I
can't even do it with bytecode, because the bytecodes
that would be needed don't exist.

--
Greg

[Gregory Ewing]

Chris Angelico wrote:
> The only way to prove that something is a new binding is to
> demonstrate that, when this binding is removed, a previous one becomes
> visible.

Or capture them both with closures and show that each
closure sees a different version of the binding.

--
Greg

[Antoon Pardon]

Op 02-10-16 om 07:59 schreef Rustom Mody:

>
> You are explaining the mechanism behind the bug. Thanks. The bug remains.
> My new car goes in reverse when I put it in first gear but only on full-moon
> nights with the tank on reserve when the left light is blinking
> The engineer explains the interesting software bug in the new PCB.
> Interesting. But the bug remains
>
> Likewise here:
>
> [2 ** i for i in[1,2]] == [2**1, 2**2]
>
> yet this fails
>
> [lambda x: x + i for i in [1,2]] == [lambda x:x+1, lambda x:x=2]
>
> is a bug for anyone including the OP of this thread
>

I once suggested that a list comprehension like [<exp> for i in <iterator>] should
be implemented as [(lambda i:<exp>)(i) for i in <iterator>].

As far a I can see, it would behave as expected in all cases.

--
Antoon

[Jussi Piitulainen]

Antoon Pardon writes:

> Op 02-10-16 om 07:59 schreef Rustom Mody:
>>
>> You are explaining the mechanism behind the bug. Thanks. The bug
>> remains. My new car goes in reverse when I put it in first gear but
>> only on full-moon nights with the tank on reserve when the left light
>> is blinking The engineer explains the interesting software bug in the
>> new PCB. Interesting. But the bug remains
>>
>> Likewise here:
>>
>> [2 ** i for i in[1,2]] == [2**1, 2**2]
>>
>> yet this fails
>>
>> [lambda x: x + i for i in [1,2]] == [lambda x:x+1, lambda x:x=2]
>>
>> is a bug for anyone including the OP of this thread

(That x=2 should be x+2.)

(And that == for functions needs to be understood mathematically. In
Python, even (lambda : 1) == (lambda : 1) may (must?) be False.)

> I once suggested that a list comprehension like [<exp> for i in
> <iterator>] should be implemented as [(lambda i:<exp>)(i) for i in
> <iterator>].
>
> As far a I can see, it would behave as expected in all cases.

It's strictly circular as-is (the latter is an instance of the former),
and it needs spelled-out a bit in order to cover comprehensions of
multiple variables. I like it anyway.

On the other hand, fortunately, it's simple to *do* this in practice
when the need arises, without waiting for anyone to change the
language. (So is the default-parameter trick.)

[Gregory Ewing]

Steve D'Aprano wrote:
> x = str = 1
> assert x == 1 and str == 1
> del x, str
> assert str # succeeds
> assert x # NameError
> x = str = 2 # create new bindings, or update existing ones?
>
> Is it our conclusion that therefore Python creates a new binding for str
> but not for x? Or that the evidence for x is "inconclusive"? Either answer
> is illogical.

The assignment to 'str' must have created a new binding,
because the old one still exists, as evidenced by its
reappearance when we delete 'str'.

The situation as regards 'x' is inconclusive, but we can
find out by applying the closure test:

>>> x = 1

>>> f1 = lambda: x
>>> del x

>>> x = 2

>>> f2 = lambda: x
>>> print(f1(), f2())

(2, 2)

The second assignment to x must have updated the binding
established by the first one, because both closures see
the most recent value. (According to our definitions, this
is true *despite* the fact that we deleted x between times!)

However, things get a bit weird due to the dynamic nature
of the global and builtin namespaces:

>>> f1 = lambda: str
>>> str = 88
>>> f2 = lambda: str
>>> print(f1(), f2())
(88, 88)
>>> del str
>>> print(f1(), f2())
(<type 'str'>, <type 'str'>)

This is weird in a quantum-entanglement kind of way. It
appears as though the two closures have captured the
same binding, because at any given moment they both
see the same value. But *which* binding that is can
vary from one moment to the next!

I think the only way to make sense of this in terms of
the classical notions of bindings and lexical scoping
is to regard the global and builtin namespaces
together as a single lexical level, within which
dynamic lookup occurs.

By the way, Scheme implementations typically also treat
the global namespace in a more dynamic way than others.
However, in Scheme there is only a single global level,
so you don't get the same dynamic shadowing/unshadowing
behaviour.

> Locals are certainly special. The language definition singles out binding[1]
> to a variable inside a function as the trigger to treat it as a local,

As the above illustrates, there are also very real differences
between the way local and global bindings are captured
by closures. Whether those differences are considered
part of the language definition, I don't know. I suspect
Guido would regard them as artifacts of the implementation.

CPython seems to have punted on part of this by not allowing
you to delete a local name that's referenced by a nested
function, thus avoiding the issue of whether doing that should
be able to un-shadow a name further out.

--
Greg

[Gregory Ewing]

Rustom Mody wrote:
> My new car goes in reverse when I put it in first gear but only on full-moon
> nights with the tank on reserve when the left light is blinking

OT aside: When I went to take my current car (a manual)
for a test drive, I had to stop and ask the dealer how
to put it into reverse.

Turns out the only difference between first and reverse
on that model is whether you lift up a little ring on the
shaft of the gear lever prior to engagement.

Who came up with *that* brilliant piece of user interface
design I don't know. It seems specifically designed to
encourage velocity sign errors when starting off...

--
Greg

[Marko Rauhamaa]

Gregory Ewing <greg....@canterbury.ac.nz>:

> Turns out the only difference between first and reverse on that model
> is whether you lift up a little ring on the shaft of the gear lever
> prior to engagement.
>
> Who came up with *that* brilliant piece of user interface design I
> don't know. It seems specifically designed to encourage velocity sign
> errors when starting off...

Well, it could be worse. This layout is pretty traditional:

1 3 5
| | |
+--+--+
| | |
2 4 R


Marko

[alister]

that sounds like typical Vauxhall & i never found it an issue when i had
one

.
unlike my current car which supposedly requires extra force to move
across the gate to reverse. I constantly find that it selects reverse
without effort when not required, but goes into 1st when trying to select
reverse.

--
The Hollywood tradition I like best is called "sucking up to the stars."
-- Johnny Carson

[BartC]

Yes, you get a funny grinding sound when attempting to change from 5th
to '6th' at 70mph/110kph. Fortunately it doesn't actually go into reverse.

--
Bartc

[bream...@gmail.com]

On Monday, October 3, 2016 at 12:53:55 PM UTC+1, Marko Rauhamaa wrote:
> Gregory Ewing:

There was the shift on the steering column on the 2CV, but then there's also this.

<very old joke>
Traditional Italian military vehicle layout.

1 3 5
| | |
+--+--+
| | |
2 4 F

</very old joke>

Kindest regards.

Mark Lawrence.

[BartC]

On 03/10/2016 16:03, bream...@gmail.com wrote:
> On Monday, October 3, 2016 at 12:53:55 PM UTC+1, Marko Rauhamaa wrote:
>> Gregory Ewing:
>>
>>> Turns out the only difference between first and reverse on that model
>>> is whether you lift up a little ring on the shaft of the gear lever
>>> prior to engagement.
>>>
>>> Who came up with *that* brilliant piece of user interface design I
>>> don't know. It seems specifically designed to encourage velocity sign
>>> errors when starting off...
>>
>> Well, it could be worse. This layout is pretty traditional:
>>
>> 1 3 5
>> | | |
>> +--+--+
>> | | |
>> 2 4 R

>

> There was the shift on the steering column on the 2CV, but then there's also this.
>
> <very old joke>
> Traditional Italian military vehicle layout.
>
> 1 3 5
> | | |
> +--+--+
> | | |
> 2 4 F
>
> </very old joke>

I don't get it. Shouldn't it be:

R R R
| | |
+--+--+
| | |
R R R

?

--
Bartc

(It's OK, I'm Italian...)

[bream...@gmail.com]

On Monday, October 3, 2016 at 5:41:23 PM UTC+1, BartC wrote:

A very poor design, what happens if the enemy attack from the rear? :)

[sohca...@gmail.com]

My car is similar, but the R is actually to the left of 1. It looks like this:

R 1 3 5
+-+-+-+
2 4 6

However, you can't get to R unless you lift the ring around the stick upwards. once you go to the far left, the ring stays up until you slide it out of the far left notch. My previous car was the traditional:

1 3 5
+-+-+
2 4 R

Surprisingly, despite driving that previous car for 13 years, the switch was incredibly easy. I've never accidentally gone to sixth gear instead of reverse, or forgotten to shift into sixth on the highway. Also, accidentally going into first when I want to reverse has never happened. I was actually pretty surprised. I thought I'd mess it up constantly for the first couple months.

[Michael Torrie]

Yeah I'm not sure what's up with car engineers these days. The Chrysler
transmission shifter that springs back to the center position is another
one of those brain-dead things. There's no way to know if you've made
it all the way into park without looking at the dash. Click it up and
hold it to hit neutral, then reverse, then park. Then it springs back
to center position. This contributed to the death of that Star Trek
movie actor. He thought it was in park but it wasn't, and it rolled
over him.

[Michael Torrie]

On 10/03/2016 11:57 AM, sohca...@gmail.com wrote:
> Surprisingly, despite driving that previous car for 13 years, the switch was incredibly easy. I've never accidentally gone to sixth gear instead of reverse, or forgotten to shift into sixth on the highway. Also, accidentally going into first when I want to reverse has never happened. I was actually pretty surprised. I thought I'd mess it up constantly for the first couple months.

I drive a lot of trucks with different patterns. One is the traditional
one with reverse to the right and back. Several use left and up for
reverse (first or low is left and down). Never had bothered me and I
rarely select reverse when I want forward or vice versa. In fact all it
takes to determine which pattern I have without looking is to see which
side of the H has the spring-back on it.

What would throw me off would be left-side driving where the pattern is
the same but the stick is on the left hand side of me and the pattern
works its way towards me instead of away.

There is that old, but false, saying that the only intuitive interface
is the nipple. Turns out everything, even that, is learned, and humans
are pretty adaptable, even to non-obvious and non-intuitive things such
as how python's loops work, lambdas, comprehensions, etc. Anyone ever
driven a 5 and 4 transmission? :)

[Gregory Ewing]

Yes, my previous car had that layout. I never actually
tried to go from 5 to R at speed, so I don't know
what it would have sounded like, but I know from the
service manual that it had a mechanism to prevent it
from happening.

--
Greg

[Steve D'Aprano]

On Tue, 4 Oct 2016 05:48 am, Michael Torrie wrote:

> There is that old, but false, saying that the only intuitive interface
> is the nipple.  Turns out everything, even that, is learned

Citation required.

Of course many things are intuitive/instinctive, e.g. breathing, but as far
as *interfaces* go, I'd like to see the research that proves that babies
don't intuitively know how to suckle from a nipple.

[Steve D'Aprano]

On Mon, 3 Oct 2016 04:15 pm, Jussi Piitulainen wrote:

> Steve D'Aprano writes:
>> Why shouldn't people say that binding and assignment are the same
>> thing in Python? What's the difference?
>
> Outside Python, (lambda x : f(x)) is said to "bind" x. It's different
> from assigning a new value to x. It's similar to how quantifiers in
> logic are said to bind their variables, so that variables bound to
> different quantifiers are different variables.

o_O

Me: "How is binding different from assignment?"

You: "This is an example of binding: lambda x: f(x). Its different from
assigning to x. Clear now?"


What's "outside python"? Can you give some examples? Presumably you don't
actually mean "every single programming language apart from Python", even
if that's what it sounds like.

Anyway, I've done my own research, and what I've found is:

- Some people consider assignment and binding to be synonyms. This is
especially common in OOP languages (like Python), and procedural languages.

- Some people consider them to be different. This is especially common in
functional languages with an academic emphasis.

For those who consider them different, the distinction is usually along
these lines:

- Assignment associates a mutable label to a value. So if I say "x = 1",
that sets up a labelled box and puts 1 inside that box. Then if I say "x =
2", the box is updated to contain 2 instead of 1.

- Binding is more like setting up a label or identifier for a constant. For
example, in a legal contract, you might say words to the effect of:

John and Jane Smith, of 123 Evergreen Terrace, Metropolis ("the Client")

and then everywhere the contract refers to "the Client", you are supposed to
mentally substitute "John and Jane Smith, of ...". Its a fixed label for a
fixed value. Even if the value is mutable (John and Jane have a baby, and
the baby is now automatically covered by the contract, the value cannot be
replaced with a new value (you can't swap out John and Jane and replace
them with their neighbours Fred and Ethel, you have to cancel the contract
and write a new one).

So for these languages, bindings relate to a *fixed* association between a
label and value. Whether the *value* itself is mutable or immutable, the
association is fixed.

[Rustom Mody]

On Tuesday, October 4, 2016 at 8:11:41 AM UTC+5:30, Steve D'Aprano wrote:
> On Mon, 3 Oct 2016 04:15 pm, Jussi Piitulainen wrote:
>
> > Steve D'Aprano writes:
> >> Why shouldn't people say that binding and assignment are the same
> >> thing in Python? What's the difference?
> >
> > Outside Python, (lambda x : f(x)) is said to "bind" x. It's different
> > from assigning a new value to x. It's similar to how quantifiers in
> > logic are said to bind their variables, so that variables bound to
> > different quantifiers are different variables.
>
> o_O
>
> Me: "How is binding different from assignment?"
>
> You: "This is an example of binding: lambda x: f(x). Its different from
> assigning to x. Clear now?"
>
>
> What's "outside python"? Can you give some examples? Presumably you don't
> actually mean "every single programming language apart from Python", even
> if that's what it sounds like.
>
> Anyway, I've done my own research, and what I've found is:
>
> - Some people consider assignment and binding to be synonyms. This is
> especially common in OOP languages (like Python), and procedural languages.

Here’s C++
http://www.informit.com/articles/article.aspx?p=376876
roughly using initialization where Jussi/Greg/myself/etc are using binding

>
> - Some people consider them to be different. This is especially common in
> functional languages with an academic emphasis.
>
> For those who consider them different, the distinction is usually along
> these lines:
>
> - Assignment associates a mutable label to a value. So if I say "x = 1",
> that sets up a labelled box and puts 1 inside that box. Then if I say "x =
> 2", the box is updated to contain 2 instead of 1.
>
> - Binding is more like setting up a label or identifier for a constant. For
> example, in a legal contract, you might say words to the effect of:
>
> John and Jane Smith, of 123 Evergreen Terrace, Metropolis ("the Client")
>
> and then everywhere the contract refers to "the Client", you are supposed to
> mentally substitute "John and Jane Smith, of ...". Its a fixed label for a
> fixed value. Even if the value is mutable (John and Jane have a baby, and
> the baby is now automatically covered by the contract, the value cannot be
> replaced with a new value (you can't swap out John and Jane and replace
> them with their neighbours Fred and Ethel, you have to cancel the contract
> and write a new one).
>
> So for these languages, bindings relate to a *fixed* association between a
> label and value. Whether the *value* itself is mutable or immutable, the
> association is fixed.

Are you suggesting that python can be understood without this distinction?
Consider the ‘mutable default gotcha’ in which the mutable is assigned but not mutated

tl;dr
Many concepts are needed to understand programming languages
Some of them may be reified into the language; all cannot be.
“Not reified into the language” ≠ “Not a necessary concept for the language”
If you have variables+scopes you need (the concept of) binding
As understood by language-theorists, eg:
http://homepage.cs.uiowa.edu/~slonnegr/plf/Book/Chapter9.pdf

[Michael Torrie]

On 10/03/2016 08:21 PM, Steve D'Aprano wrote:
> On Tue, 4 Oct 2016 05:48 am, Michael Torrie wrote:
>
>> There is that old, but false, saying that the only intuitive interface
>> is the nipple. Turns out everything, even that, is learned
>
> Citation required.

Sure, just ask a nursing woman.

> Of course many things are intuitive/instinctive, e.g. breathing, but as far
> as *interfaces* go, I'd like to see the research that proves that babies
> don't intuitively know how to suckle from a nipple.

Research? Can't you just ask women about it? I've heard quite a few
women (with experience) comment on the learned nature of babies nursing
over the years whenever this statement would come up in such company.
Which it will when you're in with a large group of married computer nerds.

Sucking seems to be instinctive but the actual interface is learned
(albeit very quickly) by experience. Babies don't just see a nipple and
know what it's for. If your children ever did, they are remarkable
indeed. Anyway, it's certainly not "intuitive" if intuitive is defined
as understanding with only basic perception of an object. Similarly an
unmarked, red button you push to give you food is not intuitive. You
can't see the button and know what it does without experience, or
someone telling you.

[Gregory Ewing]

> On Mon, 3 Oct 2016 10:57:27 -0700 (PDT), sohca...@gmail.com declaimed
> the following:

>
>>My car is similar, but the R is actually to the left of 1. It looks like this:
>>
>>R 1 3 5
>>+-+-+-+
>> 2 4 6

Mine is actually like that too, but it feels like you're
doing the same thing in both cases -- push left as far
as it will go and then forward.

--
Greg

[Steven D'Aprano]

On Tuesday 04 October 2016 14:51, Michael Torrie wrote:

> On 10/03/2016 08:21 PM, Steve D'Aprano wrote:
>> On Tue, 4 Oct 2016 05:48 am, Michael Torrie wrote:
>>
>>> There is that old, but false, saying that the only intuitive interface
>>> is the nipple. Turns out everything, even that, is learned
>>
>> Citation required.
>
> Sure, just ask a nursing woman.

[...]

> Sucking seems to be instinctive but the actual interface is learned
> (albeit very quickly) by experience.

You say tomahto, I say tomarto. It sounds like we're using different language
to describe the same thing.

Babies do have an instinct to suck if you stick a nipple (or a finger) in their
mouth, or even if you merely touch them on the lips or cheek:

http://www.medicinenet.com/script/main/art.asp?articlekey=5392

https://en.wikipedia.org/wiki/Primitive_reflexes#Rooting_reflex

The rooting instinct, together with the sucking instinct, is present in all
healthy babies. I would expect that only the most severe development
abnormalities would prevent instincts as important for survival as these two.

Between the rooting and sucking instincts, I consider "the only intuitive
interface is the nipple" is correct. However, that doesn't necessarily mean
that babies will suckle well: there are all sorts of reasons why babies don't
breast-feed well, e.g.:

http://www.cyh.com/HealthTopics/HealthTopicDetails.aspx?p=114&np=302&id=1960

> Babies don't just see a nipple and know what it's for.

Of course the instinct isn't *visual* -- babies may not open their eyes for
many minutes after birth (one study found that about 3% of babies hadn't opened
their eyes within 20 minutes, the maximum time allotted) which may be long
after their first suckle.

Nevertheless, there are senses other than sight.

[dieter]

Steve D'Aprano <steve+...@pearwood.info> writes:
> On Mon, 3 Oct 2016 04:15 pm, Jussi Piitulainen wrote:
>> Steve D'Aprano writes:
>>> Why shouldn't people say that binding and assignment are the same
>>> thing in Python? What's the difference?
>>
>> Outside Python, (lambda x : f(x)) is said to "bind" x. It's different
>> from assigning a new value to x. It's similar to how quantifiers in
>> logic are said to bind their variables, so that variables bound to
>> different quantifiers are different variables.
>
> o_O
>
> Me: "How is binding different from assignment?"
>
> You: "This is an example of binding: lambda x: f(x). Its different from
> assigning to x. Clear now?"
>
>
> What's "outside python"?

The concept "assignment" comes from an operational semantics based
on some form of "RAM" machine. Such a machine has storage cells where
you can assign values to which remain there until overridden with
a new value.

The concept "binding" comes from a denotational semantics based on
some form of functions where parameter names are bound to values on
function "application" and do not change inside the function.


When you pass a Python variable as parameter to a function,
and this function modifies the value of the corresponding
"formal parameter", this does not change the variable's value.
This indicates, that the variable does not model directly the
corresponding storage cell.

On the other hand, you can assign a new value to a Python variable
which indicates that Python variables are neither simple bindings.


Of course, these are only semantic subtlenesses.

There are storage cells associated with Python variables and assignment
can change their content (only on function call, it is not the storage
cell that gets passed on into the function but the current value).

On the other hand, one can model Python variables as bindings
where language constructs (assignments) allow to change the binding.

Thus, at an appropriate level of abstraction, you can say that
"binding" and "assignment" are mostly equivalent.

[Marko Rauhamaa]

dieter <die...@handshake.de>:

> The concept "assignment" comes from an operational semantics based on
> some form of "RAM" machine. Such a machine has storage cells where you
> can assign values to which remain there until overridden with a new
> value.
>
> The concept "binding" comes from a denotational semantics based on
> some form of functions where parameter names are bound to values on
> function "application" and do not change inside the function.

I wonder if such pure-bred functional programming languages exist. Lisp,
Scheme and Python don't belong to them, at least. Object-oriented
programming is not really possible without assignment. Even Scheme's
"letrec" appeals to assignment semantics.

Ultimately, "binding" comes from outside the computer world. You talk
about "bound variables" and "free variables". The concepts are needed to
define the semantics of predicate logic and lambda calculus, for
example. The C Preprocessor (cpp) comes close to the classic
binding/transformation semantics of lambda calculus.

> Thus, at an appropriate level of abstraction, you can say that
> "binding" and "assignment" are mostly equivalent.

Yes, but you sound more erudite if you talk about binding.


Marko

[Rustom Mody]

On Tuesday, October 4, 2016 at 12:47:58 PM UTC+5:30, dieter wrote:
> Steve D'Aprano writes:
> > On Mon, 3 Oct 2016 04:15 pm, Jussi Piitulainen wrote:
> >> Steve D'Aprano writes:
> >>> Why shouldn't people say that binding and assignment are the same
> >>> thing in Python? What's the difference?
> >>
> >> Outside Python, (lambda x : f(x)) is said to "bind" x. It's different
> >> from assigning a new value to x. It's similar to how quantifiers in
> >> logic are said to bind their variables, so that variables bound to
> >> different quantifiers are different variables.
> >
> > o_O
> >
> > Me: "How is binding different from assignment?"
> >
> > You: "This is an example of binding: lambda x: f(x). Its different from
> > assigning to x. Clear now?"
> >
> >
> > What's "outside python"?
>
> The concept "assignment" comes from an operational semantics based
> on some form of "RAM" machine. Such a machine has storage cells where
> you can assign values to which remain there until overridden with
> a new value.
>
> The concept "binding" comes from a denotational semantics based on
> some form of functions where parameter names are bound to values on
> function "application" and do not change inside the function.
>

«Binding comes from semantics ie logicians/mathematicians
Assignment comes from RAM-machine ie engineers»

seems like an intelligent distinction…
Except for the “do not change inside the function”

Sure in pure functional languages they do not change because nothing changes.
However in scheme there is a strong difference between define/let that change
the ‘structure’ of the scope and set! that changes its contents.

>
> When you pass a Python variable as parameter to a function,
> and this function modifies the value of the corresponding
> "formal parameter", this does not change the variable's value.
> This indicates, that the variable does not model directly the
> corresponding storage cell.
>
> On the other hand, you can assign a new value to a Python variable
> which indicates that Python variables are neither simple bindings.
>
>
> Of course, these are only semantic subtlenesses.
>
> There are storage cells associated with Python variables and assignment
> can change their content (only on function call, it is not the storage
> cell that gets passed on into the function but the current value).
>
> On the other hand, one can model Python variables as bindings
> where language constructs (assignments) allow to change the binding.
>
> Thus, at an appropriate level of abstraction, you can say that
> "binding" and "assignment" are mostly equivalent.

Thats a bizarre statement -- Are not the contents of the scope and the shape of the scope different things?

No language (that I know) reifies this as completely as theoretically possible.
In particular scheme goes further but shies away from full first class
environments:
http://stackoverflow.com/questions/617325/why-doesnt-scheme-support-first-class-environments

However many scheme implementations provide it in some ad-hoc manner:
[renamed from ‘environment’ to ‘namespace’ presumably due to clashes with OS
env-variables]
https://docs.racket-lang.org/reference/syntax-model.html#%28part._namespace-model%29


Python goes some way towards this with locals/globals etc... not really first-class.

However at the conceptual level there is a fundamental difference between
the shape of the namespace/environment/scope and its contents.

[Rustom Mody]

On Tuesday, October 4, 2016 at 2:05:31 PM UTC+5:30, Marko Rauhamaa wrote:
> dieter :

You are scorning something basic: Dont like 'binding'?
Ok what about stack-frames? Activation-records?

Just had a conversation with a colleague yesterday in which he was
complaining about how hard students find to learn this stuff because of
confusions like hardware-stack and language-stack getting mixed up.

If you dont like abstractions you will have too many details
And likely confused terminology

[Marko Rauhamaa]

Rustom Mody <rusto...@gmail.com>:

> You are scorning something basic: Dont like 'binding'?
> Ok what about stack-frames? Activation-records?
>
> Just had a conversation with a colleague yesterday in which he was
> complaining about how hard students find to learn this stuff because
> of confusions like hardware-stack and language-stack getting mixed up.
>
> If you dont like abstractions you will have too many details
> And likely confused terminology

Well, I *did* introduce a data model for Python where the abstract
concepts are puppies, pegs and leashes. Instead of "binding", you'd hang
a leash on a labeled peg. Of course, you could also ask a puppy to hold
on to one or more leashes.


Marko

[Ben Bacarisse]

Marko Rauhamaa <ma...@pacujo.net> writes:

> dieter <die...@handshake.de>:
>> The concept "assignment" comes from an operational semantics based on
>> some form of "RAM" machine. Such a machine has storage cells where you
>> can assign values to which remain there until overridden with a new
>> value.
>>
>> The concept "binding" comes from a denotational semantics based on
>> some form of functions where parameter names are bound to values on
>> function "application" and do not change inside the function.
>
> I wonder if such pure-bred functional programming languages exist.

I'd put Haskell in that that class.

> Lisp,
> Scheme and Python don't belong to them, at least. Object-oriented
> programming is not really possible without assignment. Even Scheme's
> "letrec" appeals to assignment semantics.

Haskell defines let (it's version of multiple mutually recursive
bindings) in terms of the least fix point of a lambda function whose
(pattern) parameter binds the expressions in the definitions.

<snip>
--
Ben.

[Steve D'Aprano]

On Tue, 4 Oct 2016 09:32 pm, Rustom Mody wrote:

> Are not the contents of the scope and the shape of the scope different
> things?

What does "the shape of the scope" mean?

Scopes don't have a shape -- they aren't geometric objects. So I'm afraid I
don't understand what distinction you are trying to make.

[Steve D'Aprano]

On Tue, 4 Oct 2016 11:27 pm, Ben Bacarisse wrote:

> Haskell defines let (it's version of multiple mutually recursive
> bindings) in terms of the least fix point of a lambda function whose
> (pattern) parameter binds the expressions in the definitions.

It binds *the expression* itself? Not the value of the expression?

So Haskell uses some form of pass by name?

And (shamelessly using Python syntax) if I have a function:


def spam(x):
print(x)
print(x+1)


and then call it:

spam(time.sleep(60) or 1)


it will sleep for 60 seconds, print 1, then sleep for another 60 seconds,
then print 2. Is that right?

[Albert-Jan Roskam]

(Sorry for top-posting)

Yep, part of the baby's hardware. Also, the interface is not limited to visual and auditory information:
http://www.scientificamerican.com/article/pheromones-sex-lives/
________________________________
From: Python-list <python-list-bounces+sjeik_appie=hotma...@python.org> on behalf of Steven D'Aprano <steve+comp....@pearwood.info>
Sent: Tuesday, October 4, 2016 7:00:48 AM
To: pytho...@python.org
Subject: Re: Is that forwards first or backwards first? (Re: unintuitive for-loop behavior)

--
https://mail.python.org/mailman/listinfo/python-list

[Rustom Mody]

On Tuesday, October 4, 2016 at 10:06:00 PM UTC+5:30, Steve D'Aprano wrote:
> On Tue, 4 Oct 2016 09:32 pm, Rustom Mody wrote:
>
> > Are not the contents of the scope and the shape of the scope different
> > things?
>
>
> What does "the shape of the scope" mean?
>
> Scopes don't have a shape -- they aren't geometric objects. So I'm afraid I
> don't understand what distinction you are trying to make.

Ok I was speaking quasi metaphorically
If you have some non-metaphors please tell!

Take the basic 'content':
x = 1
y = 2
z = 3

A. Occuring exactly as above at module level
B. Occuring exactly as above inside a function
C. Occuring thus
x = 1
foo(2,3)

def foo(y,z):
...

D.
def foo():
x = 1
def bar():
y, z = 2,3
...

E.
def foo():
x = 1
bar(2,3)
...

def bar(y,z):
...


In A,B,C,D,E at some point there's x,y,z having values (contents) 1,2,3
How do you distinguish them?

I call it the shape of the scope (or environment or bindings or namespace or
...???)

You have a better descriptor for the idea?

[Rustom Mody]

On Tuesday, October 4, 2016 at 10:15:10 PM UTC+5:30, Steve D'Aprano wrote:
> On Tue, 4 Oct 2016 11:27 pm, Ben Bacarisse wrote:
>
> > Haskell defines let (it's version of multiple mutually recursive
> > bindings) in terms of the least fix point of a lambda function whose
> > (pattern) parameter binds the expressions in the definitions.
>
> It binds *the expression* itself? Not the value of the expression?

Lazy evaluation — one of the more touted features of Haskell — uses call-by-need
which is like memoized call-by-name:
https://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_need

>
> So Haskell uses some form of pass by name?
>
> And (shamelessly using Python syntax) if I have a function:
>
>
> def spam(x):
> print(x)
> print(x+1)
>
>
> and then call it:
>
> spam(time.sleep(60) or 1)
>
>
> it will sleep for 60 seconds, print 1, then sleep for another 60 seconds,
> then print 2. Is that right?

Strong type system would not allow mixing of effects and values like that

[Marko Rauhamaa]

Steve D'Aprano <steve+...@pearwood.info>:

> On Tue, 4 Oct 2016 11:27 pm, Ben Bacarisse wrote:
>
>> Haskell defines let (it's version of multiple mutually recursive
>> bindings) in terms of the least fix point of a lambda function whose
>> (pattern) parameter binds the expressions in the definitions.
>
> It binds *the expression* itself? Not the value of the expression?

Don't know about Haskell, but in a truly functional programming language
it wouldn't make a difference.

> And (shamelessly using Python syntax) if I have a function:
>
> def spam(x):
> print(x)
> print(x+1)
>
> and then call it:
>
> spam(time.sleep(60) or 1)
>
> it will sleep for 60 seconds, print 1, then sleep for another 60 seconds,
> then print 2. Is that right?

A truly functional programming language wouldn't have side effects (in
this case, it wouldn't sleep).


Marko

[Ben Bacarisse]

Steve D'Aprano <steve+...@pearwood.info> writes:

> On Tue, 4 Oct 2016 11:27 pm, Ben Bacarisse wrote:
>
>> Haskell defines let (it's version of multiple mutually recursive
>> bindings) in terms of the least fix point of a lambda function whose
>> (pattern) parameter binds the expressions in the definitions.
>
> It binds *the expression* itself? Not the value of the expression?

Yes and no. The result is as if the value is bound, but (a) Haskell is
lazy so in some sense it is the expression that is bound and (b) this is
a definition of the semantics, not Haskell itself. The least fixed
point operator has the effect of binding all the names in the expression
to the result of binding all the names in expression to the result of
binding all the names in the expression to the result of...

In effect I'm not entirely sure how to answer your question (but I
repeat that this is how a syntactic feature is defined, not how you
write Haskell).

> So Haskell uses some form of pass by name?

No, just lazy evaluation.

> And (shamelessly using Python syntax) if I have a function:
>
>
> def spam(x):
> print(x)
> print(x+1)
>
>
> and then call it:
>
> spam(time.sleep(60) or 1)
>
>
> it will sleep for 60 seconds, print 1, then sleep for another 60 seconds,
> then print 2. Is that right?

Because Haskell is a Lazy language, the argument is not evaluated at the
point of call -- it's only evaluated when x is needed. That part is
like call by name and "binding the expression and not the value". But
because it is also a purely function language, the result of an
evaluation must always be the same, so the expression, once evaluated is
not evaluated again, and in that sense it's not like call by name.

The question came up because (I paraphrase) "even Scheme uses
assignment to explain mutually recursive definitions". Haskell defines
them using the fixed point operator. It's not really about Haskell
programs so much as how the language features are defined.

--
Ben.

[Chris Angelico]

On Wed, Oct 5, 2016 at 6:10 AM, Marko Rauhamaa <ma...@pacujo.net> wrote:
> A truly functional programming language wouldn't have side effects (in
> this case, it wouldn't sleep).

That makes sense. When people are sleeping, they're non-functional,
ergo a programming language that can sleep is non-functional :)

Seriously though... this ties in with the other issues about *purely*
functional languages being rather impractical, and the purity
generally being sullied some by things like monads (which I still
don't understand, despite the explanations in another thread). So what
happens if you have a monad "print to the console" in place of Steve's
time.sleep example? Will you get one execution of it or two?

ChrisA

[Marko Rauhamaa]

Ben Bacarisse <ben.u...@bsb.me.uk>:

> The question came up because (I paraphrase) "even Scheme uses
> assignment to explain mutually recursive definitions". Haskell defines
> them using the fixed point operator. It's not really about Haskell
> programs so much as how the language features are defined.

BTW, here's the relevant definition (<URL: http://www.schemers.org/Docum
ents/Standards/R5RS/HTML/r5rs-Z-H-7.html#%_sec_4.2.2>):

library syntax: (letrec <bindings> <body>)

[...]

Semantics: The <variable>s are bound to fresh locations holding
undefined values, the <init>s are evaluated in the resulting
environment (in some unspecified order), each <variable> is assigned
to the result of the corresponding <init>, the <body> is evaluated in
the resulting environment, and the value(s) of the last expression in
<body> is(are) returned. Each binding of a <variable> has the entire
letrec expression as its region, making it possible to define
mutually recursive procedures.


Marko

[Steve D'Aprano]

On Wed, 5 Oct 2016 06:10 am, Marko Rauhamaa wrote:

> Steve D'Aprano <steve+...@pearwood.info>:
>
>> On Tue, 4 Oct 2016 11:27 pm, Ben Bacarisse wrote:
>>
>>> Haskell defines let (it's version of multiple mutually recursive
>>> bindings) in terms of the least fix point of a lambda function whose
>>> (pattern) parameter binds the expressions in the definitions.
>>
>> It binds *the expression* itself? Not the value of the expression?
>
> Don't know about Haskell, but in a truly functional programming language
> it wouldn't make a difference.

Unfortunately, there is no such thing as a "truly functional language",
because:

> A truly functional programming language wouldn't have side effects (in
> this case, it wouldn't sleep).

any real language[1] must perform computation, and computation has side-
effects: memory is used, power is consumed, the CPU warms up, and most
relevant to my example, calculations take time.

If you don't like my example of time.sleep(60) or 1, replace it by a very
long, very difficult, very time-consuming calculation that eventually
returns 1.

The point is, as Rustom explained, Haskell's evaluation strategy is
call-by-need which delays the evaluation of the expression but then caches
the result so it need only be evaluated once.



[1] That is, an interpreter or compiler for a programming language.

[Steve D'Aprano]

On Wed, 5 Oct 2016 04:12 am, Rustom Mody wrote:

> On Tuesday, October 4, 2016 at 10:06:00 PM UTC+5:30, Steve D'Aprano wrote:
>> On Tue, 4 Oct 2016 09:32 pm, Rustom Mody wrote:
>>
>> > Are not the contents of the scope and the shape of the scope different
>> > things?
>>
>>
>> What does "the shape of the scope" mean?
>>
>> Scopes don't have a shape -- they aren't geometric objects. So I'm afraid
>> I don't understand what distinction you are trying to make.
>
> Ok I was speaking quasi metaphorically
> If you have some non-metaphors please tell!

I think that by "shape" of the scope, you mean "some identifier or
description which identifies the scope" -- e.g. "globals", "builtins",
function foo, function bar, etc.

By which variables (name bindings) exist in which scopes.


A: all three names 'x', 'y' and 'z' exist in the module scope, i.e. are
globals.

B: all three names 'x', 'y' and 'z' exist in some (unknown) local function
scope.

C: 'x' is a global, 'y' and 'z' are locals of foo.

D: 'x' is a local of foo, 'y' and 'z' are locals of bar, where bar is nested
within foo.

E: 'x' is a local of foo, 'y' and 'z' are locals of bar, where foo and bar
exist in the same (global) scope.


Since the functions (unknown), foo, bar and other bar are themselves names,
to distinguish them you need to know which scope they come from. One of the
bars comes from foo's local namespace, the others are globals.

> I call it the shape of the scope (or environment or bindings or namespace
> or ...???)
>
> You have a better descriptor for the idea?

The name of the scope.

[Rustom Mody]

On Wednesday, October 5, 2016 at 6:22:45 AM UTC+5:30, Steve D'Aprano wrote:
> On Wed, 5 Oct 2016 04:12 am, Rustom Mody wrote:
>
> > On Tuesday, October 4, 2016 at 10:06:00 PM UTC+5:30, Steve D'Aprano wrote:
> >> On Tue, 4 Oct 2016 09:32 pm, Rustom Mody wrote:
> >>
> >> > Are not the contents of the scope and the shape of the scope different
> >> > things?
> >>
> >>
> >> What does "the shape of the scope" mean?
> >>
> >> Scopes don't have a shape -- they aren't geometric objects. So I'm afraid
> >> I don't understand what distinction you are trying to make.
> >
> > Ok I was speaking quasi metaphorically
> > If you have some non-metaphors please tell!
>
> I think that by "shape" of the scope, you mean "some identifier or
> description which identifies the scope" -- e.g. "globals", "builtins",
> function foo, function bar, etc.

No
I want to talk of a higher level of collectivity than (what you are calling)
*one* scope

Something analogous to:
a = [1,2,3]
b = [[1,2,3]]
c = [1,[2,3]]
d = [1,[2],3]

“a,b,c,d have the same stuff, shaped (or whatever verb you like) differently”

eg How would you explain that with
e = ["p", "q", "r"]

a and e are same
and
a and b are same
with the two ‘sames’ being different?

[Gregory Ewing]

Steve D'Aprano wrote:

> And (shamelessly using Python syntax) if I have a function:
>
> def spam(x):
> print(x)
> print(x+1)
>

> spam(time.sleep(60) or 1)

You can't write that in Haskell, because Haskell's
equivalent of print() is not a function (or at least
it's not a function that ever returns), and neither
is sleep().

It's hard to explain without going into a lot of
detail about monads, but if you tried to write that
in Haskell you would find that the type system,
together with the way the I/O "functions" are defined,
makes it impossible to write anything that uses the
return value of an expression that performs an I/O
operation. So "time.sleep(60) or 1" is unwriteable,
because the return value of the sleep() is not
accessible.

--
Greg

[Gregory Ewing]

Chris Angelico wrote:
> So what
> happens if you have a monad "print to the console" in place of Steve's
> time.sleep example? Will you get one execution of it or two?

Again, you wouldn't be able to write it that way in Haskell.

The things you *would* be able to write would all have the
property that, if there are two print operations being
done, they are either done by two different expressions or
by an expression being evaluated in two different
environments.

--
Greg

[Gregory Ewing]

Chris Angelico wrote:
> Seriously though... this ties in with the other issues about *purely*
> functional languages being rather impractical, and the purity
> generally being sullied some by things like monads (which I still
> don't understand, despite the explanations in another thread).

If you'd like to understand better, I could put together
an example that illustrates the basic idea behind monads
using Python. It's really not that hard; it only seems
hard because it's traditionally presented in a very
abstract and mathematical way.

In the process, I think I could also answer your earlier
question about why automatic currying is considered such
a good idea.

--
Greg

[Rustom Mody]

On Wednesday, October 5, 2016 at 11:34:31 AM UTC+5:30, Gregory Ewing wrote:
> Steve D'Aprano wrote:
>
> > And (shamelessly using Python syntax) if I have a function:
> >
> > def spam(x):
> > print(x)
> > print(x+1)
> >
> > spam(time.sleep(60) or 1)
>
> You can't write that in Haskell, because Haskell's
> equivalent of print() is not a function (or at least
> it's not a function that ever returns), and neither
> is sleep().

Its ironical:
- Function in Fortran was meant to emulate math-functions
- C took the same thing and did a ‘small little syntax elision’ viz
conflating function and subprogram (procedure in Pascal speak) all under
the heading of function. In Pascal it was certainly a publicised intent
to sharply distinguish functions — no side-effects — from procedures — no return value.
- Finally this abuse of notation (and notions) has become such a norm (normal!)
that people are surprised to find non-abusive languages!

[Chris Angelico]

Wasn't my question, but sure, would be happy to hear about that. How
do you handle variadic functions?

ChrisA

[Chris Angelico]

On Wed, Oct 5, 2016 at 5:19 PM, Rustom Mody <rusto...@gmail.com> wrote:
> Its ironical:
> - Function in Fortran was meant to emulate math-functions
> - C took the same thing and did a ‘small little syntax elision’ viz
> conflating function and subprogram (procedure in Pascal speak) all under
> the heading of function. In Pascal it was certainly a publicised intent
> to sharply distinguish functions — no side-effects — from procedures — no return value.
> - Finally this abuse of notation (and notions) has become such a norm (normal!)
> that people are surprised to find non-abusive languages!

So how do you handle something that, by its nature, has BOTH side
effects and a return value? For instance, writing to a socket has
three results:

1) Data is sent to the socket (side effect), possibly with consequent
changes to visible state
2) Status return saying how much was written
3) Possible error return in the event of failure.

#3 aligns fairly nicely with exceptions, although that doesn't easily
handle the possibility that some data was written prior to the error
occurring. #2 has to be some sort of return value, unless you simply
declare that writing anything less than the full data is an error, and
fold it into exception handling. And thanks to #1, it can't be a pure
function, because you aren't allowed to optimize it away (writing
"spam" to the socket twice is not the same as writing it once and
returning 4 to both callers).

The real world is not pure.

ChrisA