Nested scopes and lexical closures
Paul Rubin
However, I couldn't easily tell from the documentation whether that
means you can make lexical closures:
def counter():
value = 0
def incr():
value += 1
return value
return incr
This turns out to throw a runtime exception:
>>> a=counter()
>>> a()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/tmp/python-170562C", line 6, in incr
UnboundLocalError: local variable 'value' referenced before assignment
Of course that's disappointing and I'm wondering if there's some
intention to fix it in the future. Meanwhile I think the
documentation of nested scopes should be updated to make it clear that
closures don't work.
Alex Martelli
news:7xu1z5c...@ruckus.brouhaha.com...
> I see that Python 2.1.1 lets you import nested scopes from the future.
> However, I couldn't easily tell from the documentation whether that
> means you can make lexical closures:
Yes, but not re-bind variables bound in enclosing blocks.
> Of course that's disappointing and I'm wondering if there's some
> intention to fix it in the future.
Only Guido knows, but I suspect he doesn't mean to. But if you
can think of an utterly simple and obvious syntax that lets you
bind a variable locally even when it's already bound in an
enclosing block, AND also lets you re-bind the variable in the
enclosing block, he might be interested -- I think the inability
to find a syntax to express both in obvious ways was a decisive
factor.
I thought the Java rule might be better: when a variable named
X is bound in an enclosing block, an inner block CANNOT define
a variable named X -- neither bind a new one nor rebind the
one in the outer block. This would not change functionality
(you'd just have to change names of new variables in inner
blocks whose names conflict with outer-block variables), but
I thought it would avoid the slight confusion that comes with
the present situation, where one THINKS one's re-binding an
outer-block variable but is in fact binding a new local homonym
that shadows (hides) the outer one.
Java adopted this restrictive rule because, in their opinions
and/or studies, homonym variables between inner and outer
blocks turned out to be errors much more often than they
were deliberate choices -- and even when deliberate, the
resulting program was clearer if the inner variable had to
be renamed. Of course, Java and most other languages are
different from Python in that they distinguish definition and
assignment, which Python doesn't (we only have binding
and re-binding of variables, not 'assignment' as in most
other languages, although we use assignment-syntax).
Anyway, I don't think my musings of the time were very
much listened to. Some people can't stand my style, and,
alas, it appears our beloved BDFL belongs to that unlucky
minority (not too sure it IS a minority, but they're surely
unlucky for their unfortunate dislikes, aren't they:-).
> Meanwhile I think the
> documentation of nested scopes should be updated to make it clear that
> closures don't work.
Let's see, Python 2.1 reference manual, Appendix A.3 "Nested Scopes",
section A.3.1:
"""
When a name is used in a code block, it is resolved using the nearest
enclosing
scope. The set of all such scopes visible to a code block is called the
block's
environment.
If a name is bound in a block, it is a local variable of that block.
...
If a name binding operation occurs anywhere within a code block, all uses of
the name within the block are treated as references to the current block.
This
can lead to errors when a name is used within a block before it is bound.
The previous rule is a subtle. Python lacks declarations and allows name
binding operations to occur anywhere within a code block. The local
variables
of a code block can be determined by scanning the entire text of the block
for
name binding operations.
"""
It seems reasonably clear to me.
PEP 227 at http://python.sourceforge.net/peps/pep-0227.html is
I think even clearer:
"""
As a
consequence, it is not possible to rebind a name defined in an
enclosing scope. An assignment operation can only bind a name in
the current scope or in the global scope. The lack of
declarations and the inability to rebind names in enclosing scopes
are unusual for lexically scoped languages; there is typically a
mechanism to create name bindings (e.g. lambda and let in Scheme)
and a mechanism to change the bindings (set! in Scheme).
XXX Alex Martelli suggests comparison with Java, which does not
allow name bindings to hide earlier bindings.
"""
What "documentation of nested scopes" is so unclear and needs
to add copies or paraphrases of this paragraph? If you can supply
the URL's, I'm sure Fred Drake will be happy to fix them as soon
as somebody opens a bug report on the relevant docs.
Note that the fact that you can't re-bind names in enclosing
scopes doesn't mean "you can't make lexical closures" -- there
are plenty of functional languages that are single-assignment
and use immutable data (you can never re-bind any name
anywhere [nor are data mutable]), and yet they make plenty
of lexical closures and use them abundantly. Python's lexical
closures are more fluid than those, since you CAN still rebind
a local or module-global name, AND mutate (just not re-bind)
data that's bound to enclosing-scope names.
Alex
Tim Peters
> Only Guido knows, but I suspect he doesn't mean to.
Not if he doesn't "have to", no. Very few people asked for nested scopes
before lambda snuck into the language, but then the demands zoomed, due to
the unbearable inability of lambda expressions to "see" the vrbls in code
right next to them. *Re*binding isn't possible in a lambda anyway (all
rebinding contexts in Python are attached to stmts, and lambdas can't
contain stmts), so the hope is we'll be back to an easily overlooked trickle
of intemperate demands from unreformed Scheme-heads.
OTOH, Jeremy Hylton implemented nested scopes, and he's an MIT alum, so if
he's an unreformed Scheme-head in theory if not in practice. If a
compelling syntax had appeared, I bet he would have pushed hard for it.
Guido continues to believe that mutable state is much better modeled by
classes, so he's not going to change anything here without intense pressure.
> ...
> I thought the Java rule might be better: when a variable named
> X is bound in an enclosing block, an inner block CANNOT define
> a variable named X -- neither bind a new one nor rebind the
> one in the outer block. This would not change functionality
> (you'd just have to change names of new variables in inner
> blocks whose names conflict with outer-block variables), but
> I thought it would avoid the slight confusion that comes with
> the present situation, where one THINKS one's re-binding an
> outer-block variable but is in fact binding a new local homonym
> that shadows (hides) the outer one.
OTOH, Java allows locals to shadow class members, and that's a source of
bugs too -- and somtimes worse, because the class members' declarations may
be physically far removed. BTW, the most frequent blind spot in my C++ days
was using an editor to copy (e.g.)
int numFrobnicators;
from the class decl into a new method, *intending* to write e.g.
numFrobnicators = 0;
in the latter, but then leaving the "int " in by mistake:
int numFrobnicators = 0;
C++, like Java, is happy with that, and the resulting code is a disaster.
In Pythonland, the internal PythonLabs debate over shadowing rules
floundered over what to do about the builtins. In a real sense, they're in
an enclosing scope too, but billions (yes, I counted <wink>) of functions do
something like
str = first + last
"list" is also routinely rebound. So the idea hit a snag: making an
exception for builtins would have been at least a little ugly, but not
excepting them would have caused howls of protest.
> ...
> Anyway, I don't think my musings of the time were very
> much listened to.
I believe Guido was aware of them; I know I was.
> Some people can't stand my style, and, alas, it appears our beloved
> BDFL belongs to that unlucky minority
Guido loves you! But he has very little time for debate, while your
debating budget exceeds the ambitions of most third-world economies <wink>.
> (not too sure it IS a minority, but they're surely unlucky for
> their unfortunate dislikes, aren't they:-).
Indeed they are.
although-most-times-nesting-is-still-for-the-birds-ly y'rs - tim
Alex Martelli
news:mailman.998290590...@python.org...
...
> Guido continues to believe that mutable state is much better modeled by
> classes, so he's not going to change anything here without intense
pressure.
FWIW, I do agree that classes are a better tool for this. The best use
I've found so far for nested-scopes is a better way to express currying,
and that doesn't need any rebinding ability. My only issue with the
current rules is that they're somewhat surprising/unexpected. Maybe
a warning about local-shadowing-outer would do as well as a total
prohibition.
> > I thought the Java rule might be better: when a variable named
> > X is bound in an enclosing block, an inner block CANNOT define
> > a variable named X -- neither bind a new one nor rebind the
...
> OTOH, Java allows locals to shadow class members, and that's a source of
> bugs too -- and somtimes worse, because the class members' declarations
may
Good point -- Python, of course, finesses the issue thanks to the
wonderful decision that members are explicitly accessed AS members
(self.whatever, &c).
> In Pythonland, the internal PythonLabs debate over shadowing rules
> floundered over what to do about the builtins. In a real sense, they're
in
> an enclosing scope too, but billions (yes, I counted <wink>) of functions
do
> something like
>
> str = first + last
>
> "list" is also routinely rebound. So the idea hit a snag: making an
> exception for builtins would have been at least a little ugly, but not
> excepting them would have caused howls of protest.
I agree the issue is relevant, but builtins aren't "lexically" enclosing --
they're more of a "namespace of last resort". Breaking all code that
happens to bind a builtin name in local or module scope would have
been tantamount to adding a zillion new keywords at a stroke -- the
howls of protest would have been amply justified. Again, maybe,
warnings (which can be turned off) could be a good compromise.
Alex