Nested scopes: why is it weird?
Scott Long
weird things about it. Take a look:
>>> from __future__ import nested_scopes
>>> def a():
... def b():
... k = x
... print k
... x = 1
... b()
...
>>> a()
1
Right, this is what you would expect. But how about this?
>>> from __future__ import nested_scopes
>>> def a():
... def b():
... k = x
... x = 0
... print k
... x = 1
... b()
...
>>> a()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 7, in a
File "<stdin>", line 3, in b
UnboundLocalError: local variable 'x' referenced before assignment
Here is my explanation for this (tell me if I'm wrong):
During pass #1 of compilation, the statement "x = 0" in b() binds x as a
local variable local to b(). During execution, k = x executes *before* x
has been assigned, resulting in this flamage.
My first beef here is, why is the exception called UnboundLocalError?
The variable is certainly bound (into the local scope), it simply has
not been assigned. Why not call it UninitializedLocalError? I know it
isn't going to change, I'm just asking.
But my main concern is that this way of selecting a binding seems very
unintuitive. Shouldn't the first statement referencing x be the
statement that specifies the binding of x? In this case, shouldn't "k =
x" cause x to get bound as a free variable, not a local to b()? I'm not
claiming that this is a bug. I am curious as to why it was decided that
things are going to work this way. It makes assignment to a variable in
an enclosing scope impossible!
Regards,
Scott
Alex Martelli
Alex Martelli
> weird things about it. Take a look:
Not really, just a typical misunderstanding (which also existed
wrt global variables before nested scopes).
> Right, this is what you would expect. But how about this?
>
> >>> from __future__ import nested_scopes
> >>> def a():
> ... def b():
> ... k = x
> ... x = 0
The x name in procedure b is LOCAL -- the compiler knows it,
because x is re-bound in b's scope. (Incidentally, you can
never rebind a name from a lexically-nested local scope
in your scope; you CAN rebind a global name, if you use the
global statement at the start of a function).
> My first beef here is, why is the exception called UnboundLocalError?
Because x is a local name, and it is not yet bound when it is
first used in b's first instruction.
> The variable is certainly bound (into the local scope), it simply has
Nope. x is a local name, but not bound to any value, as yet.
Only the assignment statement (if it was executed) would bind
name x (to the value 0).
> not been assigned. Why not call it UninitializedLocalError? I know it
> isn't going to change, I'm just asking.
>
> But my main concern is that this way of selecting a binding seems very
> unintuitive. Shouldn't the first statement referencing x be the
> statement that specifies the binding of x? In this case, shouldn't "k =
*In general*, how could the compiler tell what the first statement
IS, that _references_ name x within a given scope? Assuming you
mean, "first to execute". The compiler _might_ assign some
arbitrary ordering, of course. But having:
if a>b:
y=x
else:
x=y
have *TOTALLY* different semantics from:
if b>=a:
x=y
else:
y=x
when it SEEMS they're totally equivalent, would be truly weird, so
I don't think you mean "first in some arbitrary order (such as,
longest-first, or lower-line-number-first, or whatever)".
> x" cause x to get bound as a free variable, not a local to b()? I'm not
> claiming that this is a bug. I am curious as to why it was decided that
> things are going to work this way. It makes assignment to a variable in
> an enclosing scope impossible!
Yes, the latter is exactly Guido's decision: he has always maintained
that the right (aka obvious) way for something to keep and update
state is to define a class, rather than stretch closures to let code
in them re-bind things "partways up" -- or so I hear, and I sure
see his point. But it's not any kind of necessary consequence of
Python's long-standing rules that all variables re-bound in a scope
are local to that scope unless explicitly mentioned in a global
statement -- it would have been trivial to say, for example, that
"global x in a" was THE way for b to inform the compiler that the
x name b uses and re-binds is the one in a, not one local to b itself
(somebody wanting to make closures really powerful/handy might have
stretched that to "global x in a as z":-).
Java just doesn't let a local variable 'shadow' one with the
same name from a lexically-outer scope, and that might have
been a nice choice too -- just forbid b to have a local variable
named x if a lexically-outer scope of b had a similarly named
local variable. Unfortunately, this restriction, which I have
always liked, and proposed when nested scope were PEP'd, was
not accepted (though it did get a mention in the PEP), so we'll
live forevermore with misunderstandings on this point.
Alex
Alex Martelli
> weird things about it. Take a look:
Not really, just a typical misunderstanding (which also existed
wrt global variables before nested scopes).
> Right, this is what you would expect. But how about this?
>
> >>> from __future__ import nested_scopes
> >>> def a():
> ... def b():
> ... k = x
> ... x = 0
The x name in procedure b is LOCAL -- the compiler knows it,
because x is re-bound in b's scope. (Incidentally, you can
never rebind a name from a lexically-nested local scope
in your scope; you CAN rebind a global name, if you use the
global statement at the start of a function).
> My first beef here is, why is the exception called UnboundLocalError?
Because x is a local name, and it is not yet bound when it is
first used in b's first instruction.
> The variable is certainly bound (into the local scope), it simply has
Nope. x is a local name, but not bound to any value, as yet.
Only the assignment statement (if it was executed) would bind
name x (to the value 0).
> not been assigned. Why not call it UninitializedLocalError? I know it
> isn't going to change, I'm just asking.
>
> But my main concern is that this way of selecting a binding seems very
> unintuitive. Shouldn't the first statement referencing x be the
> statement that specifies the binding of x? In this case, shouldn't "k =
*In general*, how could the compiler tell what the first statement
IS, that _references_ name x within a given scope? Assuming you
mean, "first to execute". The compiler _might_ assign some
arbitrary ordering, of course. But having:
if a>b:
y=x
else:
x=y
have *TOTALLY* different semantics from:
if b>=a:
x=y
else:
y=x
when it SEEMS they're totally equivalent, would be truly weird, so
I don't think you mean "first in some arbitrary order (such as,
longest-first, or lower-line-number-first, or whatever)".
> x" cause x to get bound as a free variable, not a local to b()? I'm not
> claiming that this is a bug. I am curious as to why it was decided that
> things are going to work this way. It makes assignment to a variable in
> an enclosing scope impossible!
Yes, the latter is exactly Guido's decision: he has always maintained
Fredrik Lundh
> During pass #1 of compilation, the statement "x = 0" in b() binds x as a
> local variable local to b(). During execution, k = x executes *before* x
> has been assigned, resulting in this flamage.
http://www.python.org/doc/current/ref/execframes.html
"Whether a name is local or global in a code block is determined
by static inspection of the source text for the code block: in the
absence of global statements, a name that is bound anywhere
in the code block is local in the entire code block; all other names
are considered global."
(where "global" should be read as "non-local" in 2.2 and later)
> My first beef here is, why is the exception called UnboundLocalError?
> The variable is certainly bound (into the local scope), it simply has
> not been assigned.
in Python, assignment binds names to objects. "x" hasn't been
bound to any object.
</F>
<!-- (the eff-bot guide to) the python standard library:
http://www.pythonware.com/people/fredrik/librarybook.htm
-->
Scott Long
> > >>> from __future__ import nested_scopes
> > >>> def a():
> > ... def b():
> > ... k = x
> > ... x = 0
>
> The x name in procedure b is LOCAL -- the compiler knows it,
> because x is re-bound in b's scope. (Incidentally, you can
> never rebind a name from a lexically-nested local scope
> in your scope; you CAN rebind a global name, if you use the
> global statement at the start of a function).
The spirit of Python seems to favor binding as late as possible
(sometimes it seems like the binding happens after my script has
finished running ;-)
In this spirit, why not place variables into scopes dynamically as well?
Instead of having the separate LOAD_FAST and LOAD_GLOBAL why not just
LOAD, and the first LOAD executed is what finally scopes the name?
Before raising an objection, read my other comments below (and feel free
to object at that point).
>
> > My first beef here is, why is the exception called UnboundLocalError?
>
> Because x is a local name, and it is not yet bound when it is
> first used in b's first instruction.
Ah, this is a semantic misunderstanding for me. In most statically
scoped languages, one speaks of binding a name to a *scope*, not to a
*value*.
> > But my main concern is that this way of selecting a binding seems very
> > unintuitive. Shouldn't the first statement referencing x be the
> > statement that specifies the binding of x? In this case, shouldn't "k =
>
> *In general*, how could the compiler tell what the first statement
> IS, that _references_ name x within a given scope? Assuming you
> mean, "first to execute". The compiler _might_ assign some
> arbitrary ordering, of course. But having:
> if a>b:
> y=x
> else:
> x=y
> have *TOTALLY* different semantics from:
> if b>=a:
> x=y
> else:
> y=x
> when it SEEMS they're totally equivalent, would be truly weird, so
> I don't think you mean "first in some arbitrary order (such as,
> longest-first, or lower-line-number-first, or whatever)".
The problem doesn't occur if you associate scopes with suites instead of
function defs (equivalent to block scopes in C/C++ for example). A suite
(or block) has a definite "first statement". Yes, I know that isn't how
it works. But my C++ brain keeps wanting it to work that way, so I get
confused a lot.
Well, time to shut up.
Scott
Neil Schemenauer
> In this spirit, why not place variables into scopes dynamically as well?
> Instead of having the separate LOAD_FAST and LOAD_GLOBAL why not just
> LOAD, and the first LOAD executed is what finally scopes the name?
> Before raising an objection, read my other comments below (and feel free
> to object at that point).
It would slow the interpreter down a lot. Also, I think it makes
programs harder to maintain. Static scoping is good.
Neil
Paul Prescod
>
>...
>
> Java just doesn't let a local variable 'shadow' one with the
> same name from a lexically-outer scope, and that might have
> been a nice choice too -- just forbid b to have a local variable
> named x if a lexically-outer scope of b had a similarly named
> local variable. Unfortunately, this restriction, which I have
> always liked, and proposed when nested scope were PEP'd, was
> not accepted (though it did get a mention in the PEP), so we'll
> live forevermore with misunderstandings on this point.
Why forevermore? PEP a change to the behaviour and then we can put in a
warning. Python's current behaviour is both confusing and dangerous and
I would love to see someone champion a change. This just shouldn't
silently do the wrong thing:
a = 5
def foo():
a = 6
It would be better to require one of the names to be changed.
--
Take a recipe. Leave a recipe.
Python Cookbook! http://www.ActiveState.com/pythoncookbook
Cliff Wells
> Why forevermore? PEP a change to the behaviour and then we can put in a
> warning. Python's current behaviour is both confusing and dangerous and
> I would love to see someone champion a change. This just shouldn't
> silently do the wrong thing:
>
> a = 5
>
> def foo():
> a = 6
>
> It would be better to require one of the names to be changed.
It silently does the right thing. Changing the value of a variable from an
enclosing scope is a _bad_ idea. This is what is called a "side effect" and
was one of the things structured programming was created to prevent. This is
bad:
>>> a = 5
>>> foo()
>>> a
6
If a function is going to alter a global variable or a variable from an
enclosing scope, then it should be passed as an argument or returned as a
value so that someone reading the code can reasonably expect that variable to
be changed. Better:
>>> a = 5
>>> a = foo()
>>> a
6
Note that _accessing_ a variable from an enclosing scope (or global, for that
matter), isn't nearly as bad and again, Python does the correct thing: it
allows you to see the variables from the enclosing scope, but the minute you
try to assign to them, they become local to the current scope (even if you do
the assignment _after_ you try to read the value - this IS confusing if you
don't know why)
Regards,
--
Cliff Wells
Software Engineer
Logiplex Corporation (www.logiplex.net)
(503) 978-6726 x308
(800) 735-0555 x308
Scott Long
> Note that _accessing_ a variable from an enclosing scope (or global, for that
> matter), isn't nearly as bad and again, Python does the correct thing: it
> allows you to see the variables from the enclosing scope, but the minute you
> try to assign to them, they become local to the current scope (even if you do
> the assignment _after_ you try to read the value - this IS confusing if you
> don't know why)
And that was my original confusion. As you stated, the only real reason
to have access to enclosing scopes is for reading outer-scope variables,
never for setting them. But it seems that if a nested function needs
access to an outer variable, you should just pass that variable as a
parameter.
It looks like the real reason for nested scopes was to get rid of the
lambda hack (passing variables through default arguments). Perhaps
nested scopes are over-general, and should ONLY apply to lambda
expressions? That's really the only place they are necessary...
Scott
Paul Prescod
>
>...
> > a = 5
> >
> > def foo():
> > a = 6
> >
> > It would be better to require one of the names to be changed.
>
> enclosing scope is a _bad_ idea. This is what is called a "side effect" and
> was one of the things structured programming was created to prevent.
Side effects are a fundamental part of Python programming. If they
weren't, Python wouldn't have the "global" keyword. I think it is great
that Python requires you to be explicit when you want to overwrite a
global variable. I think it should similarly reqiure you to be explicit
when you want to shadow one. Or it could simply disallow the shadowing
-- it isn't very useful anyhow. Just call the inner a "inner_a".
Brian Quinlan
> It silently does the right thing. Changing the value of a
> variable from an enclosing scope is a _bad_ idea. This is
> what is called a "side effect" and was one of the things
> structured programming was created to prevent.
I'm pretty sure that Paul was arguing that you should not be able to
shadow variables used in enclosing lexical scopes i.e. it would be an
error to rebind a name used in an enclosing lexical scope.
Cheers,
Brian
Cliff Wells
> Side effects are a fundamental part of Python programming. If they
> weren't, Python wouldn't have the "global" keyword.
Just because something is available doesn't make it fundamental.
> I think it is great
> that Python requires you to be explicit when you want to overwrite a
> global variable.
I agree.
> I think it should similarly reqiure you to be explicit
> when you want to shadow one. Or it could simply disallow the shadowing
> -- it isn't very useful anyhow. Just call the inner a "inner_a".
Actually, it is useful for things like lambda functions so you can avoid
passing half a dozen arguments.
Scott Long
> Side effects are a fundamental part of Python programming. If they
> weren't, Python wouldn't have the "global" keyword. I think it is great
> that Python requires you to be explicit when you want to overwrite a
> global variable. I think it should similarly reqiure you to be explicit
> when you want to shadow one. Or it could simply disallow the shadowing
> -- it isn't very useful anyhow. Just call the inner a "inner_a".
Maybe get an error for the following?
def a():
x = 0
def b():
x = 1
Something along the lines of "Local variable binding shadows variable
from an enclosing scope"? And then to force it, introduce 'local'
keyword:
def a():
x = 0
def b():
local x # No really, I want my variable to be called 'x'
x = 1
Chris Barker
> I think it should similarly reqiure you to be explicit
> when you want to shadow one. Or it could simply disallow the shadowing
> -- it isn't very useful anyhow. Just call the inner a "inner_a".
I think this would be a nightmare for code maintainance and simple
syntax. How often do you use a simple variable name like "i" for an
index? would you really want to use i in one scope, and then inner_i in
the next, and then inner_inner_i?
And as soon as you re-arranged your code a little, you would have to
re-name a bunch of stuff. I like the current scheme just fine.
-Chris
--
Christopher Barker,
Ph.D.
ChrisH...@home.net --- --- ---
http://members.home.net/barkerlohmann ---@@ -----@@ -----@@
------@@@ ------@@@ ------@@@
Oil Spill Modeling ------ @ ------ @ ------ @
Water Resources Engineering ------- --------- --------
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------
Cliff Wells
>
> I'm pretty sure that Paul was arguing that you should not be able to
> shadow variables used in enclosing lexical scopes i.e. it would be an
> error to rebind a name used in an enclosing lexical scope.
>
I don't know, I learned on Pascal way back when and it did the same thing
(basically), although it did allow you to alter the value of the variable
from the enclosing scope, so this approach makes sense to me (in fact, when I
moved to C, I missed nested function scoping as it can provide a nice
organizational method). I suppose this is simply one of those issues that is
mostly a matter of preference, so I simply disagree.
Cliff Wells
>
> It looks like the real reason for nested scopes was to get rid of the
> lambda hack (passing variables through default arguments). Perhaps
> nested scopes are over-general, and should ONLY apply to lambda
> expressions? That's really the only place they are necessary...
>
You're probably correct about the lambda hack. I detested that and nested
scopes make it much cleaner. As far as limiting nested scopes to lambda
functions, that doesn't seem too unreasonable.
Cliff Wells
> Paul Prescod wrote:
> > I think it should similarly reqiure you to be explicit
> > when you want to shadow one. Or it could simply disallow the shadowing
> > -- it isn't very useful anyhow. Just call the inner a "inner_a".
>
> I think this would be a nightmare for code maintainance and simple
> syntax. How often do you use a simple variable name like "i" for an
> index? would you really want to use i in one scope, and then inner_i in
> the next, and then inner_inner_i?
>
> And as soon as you re-arranged your code a little, you would have to
> re-name a bunch of stuff. I like the current scheme just fine.
>
> -Chris
ditto.
Alex Martelli
...
> In this spirit, why not place variables into scopes dynamically as well?
Because it doesn't work well. The original LISP had dynamic scoping, and
it was *by far* the biggest weakness of that otherwise-great language.
Ever since Scheme introduced lexical scoping, Lisp rushed to copy it on
that point, and Common Lisp now scopes lexically -- and works much better
because of that.
> The problem doesn't occur if you associate scopes with suites instead of
> function defs (equivalent to block scopes in C/C++ for example). A suite
> (or block) has a definite "first statement".
Yes, but that granularity would be utterly and totally useless for
determining local variables. If a variable was local to each block that
bound it, you could never modify any variable in a statement guarded by an
if without first artificially accessing the variable to show it's the
"outer" one you mean -- what a nightmare!
Fine-grained scoping can only work hand in hand with declarations, and
Python (thanks be) does not have them.
Alex
Paul Rubin
> Fine-grained scoping can only work hand in hand with declarations, and
> Python (thanks be) does not have them.
Python already has a "global" declaration. It seems reasonable to me
to also give it a "local" (or "my" as Perl would call it) declaration.
Paul Prescod
>
>...
>
> I think this would be a nightmare for code maintainance and simple
> syntax. How often do you use a simple variable name like "i" for an
> index? would you really want to use i in one scope, and then inner_i in
> the next, and then inner_inner_i?
How deeply do you nest scopes do you typically have in your Python
programs? Remember that class scopes don't count so you're talking about
a global for-loop *and* a function for-loop *and* a function within a
function for-loop all using the same index variable. This is incredibly
unlikely.
> And as soon as you re-arranged your code a little, you would have to
> re-name a bunch of stuff. I like the current scheme just fine.
It is not as if you can just move code around without looking at the
context anyhow. If you move the code around in the same file then you
are very unlikely to run into this problem because you will already have
avoided the names of global variables. If you move the code around to
another file, it is still unlikely but not quite so much. Nevertheless,
moving code between files often requires renaming variables if only
references to global variables and functions.