trouble with nested closures: one of my variables is missing...

Cameron Simpson

unread,

Oct 13, 2012, 11:59:24 PM10/13/12

to pytho...@python.org

I'm having some trouble with closures when defining a decorator.

TL;DR:
I have a function that makes a decorator, and only some of the names
from an outer scope appear in the inner closure's locals().
And I do not understand why at all.

Let me explain...

Environment: python 2.7.3 on MacOSX Mountain Lion from MacPorts.

Background: I have a decorator called "file_property" that watches a
file for changes and reloads the file at need. Otherwise it returns a
cached value. It has a lock and some sanity checks and works quite well.

Example method:

class myclass(object):
@file_property
def rules(self):
with open(self._rules_path) as rfp:
R = parse(rfp)
return R
C = myclass()

and using C.rules fetches parses the rule file as needed and caches the
value until the file next changes.

I naively wrote file_property with a bunch of default parameters:

def file_property(func, attr_name=None, unset_object=None, poll_rate=1):

and because I never overrode these failed to realise they're useless. Fine.

So, I'm rewriting file_property like this:

def file_property(func):
return make_file_property()(func)

i.e. it makes a "vanilla" file_property using the default internals.
Now I have make_file_property with the default parameters:

def make_file_property(attr_name=None, unset_object=None, poll_rate=1):

which I might use as:

class myclass(object):
@make_file_property(poll_rate=3)
def rules(self):
with open(self._rules_path) as rfp:
R = parse(rfp)
return R

The inner function is the same, but it won't reload the file more often
that once every 3 seconds.

However, I can't make my make_file_property function work. I've stripped
the code down and it does this:

[hg/css-mailfiler]fleet*1> python foo.py
make_file_property(attr_name=None, unset_object=None, poll_rate=1): locals()={'attr_name': None, 'poll_rate': 1, 'unset_object': None}
made_file_property(func=<function f at 0x10408b0c8>): locals()={'func': <function f at 0x10408b0c8>, 'unset_object': None}
Traceback (most recent call last):
File "foo.py", line 21, in <module>
def f(self, foo=1):
File "foo.py", line 4, in file_property
return make_file_property()(func)
File "foo.py", line 10, in made_file_property
if attr_name is None:
UnboundLocalError: local variable 'attr_name' referenced before assignment

Observe above that 'unset_object' is in locals(), but not 'attr_name'.
This surprises me.

The stripped back code (missing the internals of the file property
watcher) looks like this:

import sys

def file_property(func):
return make_file_property()(func)

def make_file_property(attr_name=None, unset_object=None, poll_rate=1):
print >>sys.stderr, "make_file_property(attr_name=%r, unset_object=%r, poll_rate=%r): locals()=%r" % (attr_name, unset_object, poll_rate,locals())
def made_file_property(func):
print >>sys.stderr, "made_file_property(func=%r): locals()=%r" % (func, locals())
if attr_name is None:
attr_name = '_' + func.__name__
lock_name = attr_name + '_lock'
def getprop(self):
with getattr(self, lock_name):
# innards removed here
pass
return getattr(self, attr_name, unset_object)
return property(getprop)
return made_file_property

@file_property
def f(self, foo=1):
print "foo=%r" % (foo,)

@make_file_property(attr_name="_blah")
def f2(self, foo=2):
print "foo=%r" % (foo,)

Can someone explain what I'm doing wrong, or tell me this is a python
bug?
--
Cameron Simpson <c...@zip.com.au>

Bolts get me through times of no courage better than courage gets me
through times of no bolts!
- Eric Hirst <er...@u.washington.edu>

Cameron Simpson

unread,

Oct 14, 2012, 5:54:02 PM10/14/12

to Chris Rebert, pytho...@python.org

On 13Oct2012 22:07, Chris Rebert <cl...@rebertia.com> wrote:

| On Saturday, October 13, 2012, Cameron Simpson wrote:
| > I'm having some trouble with closures when defining a decorator.

| <snip>

|
| > However, I can't make my make_file_property function work. I've stripped
| > the code down and it does this:

| <snip>

| You're missing a "nonlocal" declaration here.

| You assign to it, but there's no nonlocal declaration, so Python thinks
| it's a local var, hence your error.

But 'unset_object' is in locals(). Why one and not the other?
Obviously there's something about closures here I'm missing.

| Pardon my brevity and some lack of trimming; I'm on a smartphone and in a
| rush.

No worries. Thansk for the rpely.
--
Cameron Simpson <c...@zip.com.au>

A clean desk is the sign of a blank mind.

Ian Kelly

unread,

Oct 14, 2012, 8:32:01 PM10/14/12

to Python

On Sun, Oct 14, 2012 at 3:54 PM, Cameron Simpson <c...@zip.com.au> wrote:
> | You assign to it, but there's no nonlocal declaration, so Python thinks
> | it's a local var, hence your error.
>
> But 'unset_object' is in locals(). Why one and not the other?
> Obviously there's something about closures here I'm missing.

'unset_object' is in locals because it's a free variable and those are
included in locals(), and it has a value.
'attr_name' is not in locals because while it's a local variable, it
has not been assigned to yet. It has no value and an attempt to
reference it at that point would result in an UnboundLocalError.

Cameron Simpson

unread,

Oct 14, 2012, 9:08:40 PM10/14/12

to Ian Kelly, Python

Can you elaborate a bit on that? The only place in my code that
unset_object is set is as a default parameter in make_file_property
(snippet):

def make_file_property(attr_name=None, unset_object=None, poll_rate=1):
print >>sys.stderr, "make_file_property(attr_name=%r, unset_object=%r, poll_rate=%r): locals()=%r" % (attr_name, unset_object, poll_rate,locals())
def made_file_property(func):

print >>sys.stderr, "made_file_property(func=%r): locals()=%r" % (func, locals())
if attr_name is None:
attr_name = '_' + func.__name__

and attr_name is set there also.

Is attr_name omitted from locals() in made_file_property _because_ I
have an assignment statement?

If that's the case, should I be doing this (using distinct names for the
closure variable and the function local variable):

def make_file_property(attr_name=None, unset_object=None, poll_rate=1):
print >>sys.stderr, "make_file_property(attr_name=%r, unset_object=%r, poll_rate=%r): locals()=%r" % (attr_name, unset_object, poll_rate,locals())
def made_file_property(func):

print >>sys.stderr, "made_file_property(func=%r): locals()=%r" % (func, locals())
if attr_name is None:

my_attr_name = '_' + func.__name__
else:
my_attr_name = attr_name
lock_name = my_attr_name + '_lock'

def getprop(self):
with getattr(self, lock_name):

pass
return getattr(self, my_attr_name, unset_object)

i.e. deliberately _not_ assigning to attr_name as as to _avoid_ masking
the outer attr_name from the inner locals()?

BTW, doing that works. Is that The True Path for this situation?

If so, I think I now understand what's going on: Python has inspected
the inner function and not placed the outer 'attr_name' into locals()
_because_ the inner function seems to have its own local attr_name
in use, which should not be pre-tromped.

--
Cameron Simpson <c...@zip.com.au>

Nothing is so smiple that it can't get screwed up.

Ian Kelly

unread,

Oct 14, 2012, 9:27:24 PM10/14/12

to Python

On Sun, Oct 14, 2012 at 7:08 PM, Cameron Simpson <c...@zip.com.au> wrote:
> On 14Oct2012 18:32, Ian Kelly <ian.g...@gmail.com> wrote:
> | 'attr_name' is not in locals because while it's a local variable, it
> | has not been assigned to yet. It has no value and an attempt to
> | reference it at that point would result in an UnboundLocalError.
>
> Can you elaborate a bit on that? The only place in my code that
> unset_object is set is as a default parameter in make_file_property
> (snippet):
>
> def make_file_property(attr_name=None, unset_object=None, poll_rate=1):
> print >>sys.stderr, "make_file_property(attr_name=%r, unset_object=%r, poll_rate=%r): locals()=%r" % (attr_name, unset_object, poll_rate,locals())
> def made_file_property(func):
> print >>sys.stderr, "made_file_property(func=%r): locals()=%r" % (func, locals())
> if attr_name is None:
> attr_name = '_' + func.__name__
>
> and attr_name is set there also.
>
> Is attr_name omitted from locals() in made_file_property _because_ I
> have an assignment statement?

Yes. Syntactically, a variable is treated as local to a function if
it is assigned to somewhere in that function and there is no explicit
global or nonlocal declaration.

> If that's the case, should I be doing this (using distinct names for the
> closure variable and the function local variable):
>
> def make_file_property(attr_name=None, unset_object=None, poll_rate=1):
> print >>sys.stderr, "make_file_property(attr_name=%r, unset_object=%r, poll_rate=%r): locals()=%r" % (attr_name, unset_object, poll_rate,locals())
> def made_file_property(func):
> print >>sys.stderr, "made_file_property(func=%r): locals()=%r" % (func, locals())
> if attr_name is None:
> my_attr_name = '_' + func.__name__
> else:
> my_attr_name = attr_name
> lock_name = my_attr_name + '_lock'
> def getprop(self):
> with getattr(self, lock_name):
> pass
> return getattr(self, my_attr_name, unset_object)
>
> i.e. deliberately _not_ assigning to attr_name as as to _avoid_ masking
> the outer attr_name from the inner locals()?
>
> BTW, doing that works. Is that The True Path for this situation?

That's a perfectly good way to do it as long as you don't want to
actually change the value of the outer attr_name. If you did, then
you would either declare the variable as nonlocal (Python 3.x only) or
use a container (e.g. a 1-element list), which would allow you to
modify the contents of the container without actually assigning to the
variable.

> If so, I think I now understand what's going on: Python has inspected
> the inner function and not placed the outer 'attr_name' into locals()
> _because_ the inner function seems to have its own local attr_name
> in use, which should not be pre-tromped.

Exactly right.

Cameron Simpson

unread,

Oct 14, 2012, 9:39:18 PM10/14/12

to Ian Kelly, Python

On 14Oct2012 19:27, Ian Kelly <ian.g...@gmail.com> wrote:
| On Sun, Oct 14, 2012 at 7:08 PM, Cameron Simpson <c...@zip.com.au> wrote:
| > Is attr_name omitted from locals() in made_file_property _because_ I
| > have an assignment statement?
|
| Yes. Syntactically, a variable is treated as local to a function if
| it is assigned to somewhere in that function and there is no explicit
| global or nonlocal declaration.

Aha. Good.

| > If that's the case, should I be doing this (using distinct names for the
| > closure variable and the function local variable):
| >
| > def make_file_property(attr_name=None, unset_object=None, poll_rate=1):

[...]

| > if attr_name is None:
| > my_attr_name = '_' + func.__name__
| > else:
| > my_attr_name = attr_name

[...]

| > i.e. deliberately _not_ assigning to attr_name as as to _avoid_ masking
| > the outer attr_name from the inner locals()?
| >
| > BTW, doing that works. Is that The True Path for this situation?
|
| That's a perfectly good way to do it as long as you don't want to
| actually change the value of the outer attr_name.

Well, I don't need to - using a distinct local variable will do the job. I
just hadn't realised I needed the extra level of naming.

| If you did, then
| you would either declare the variable as nonlocal (Python 3.x only)

... which is why I couldn't find such in the 2.7.3 doco ...

| or
| use a container (e.g. a 1-element list), which would allow you to
| modify the contents of the container without actually assigning to the
| variable.

Ah. Yeah, tacky; I've done that kind of thing in the past on occasion but
using a distinct local name is much cleaner here, and probably usually.

| > If so, I think I now understand what's going on: Python has inspected
| > the inner function and not placed the outer 'attr_name' into locals()
| > _because_ the inner function seems to have its own local attr_name
| > in use, which should not be pre-tromped.
|
| Exactly right.

Thanks for the explaination. Now I know a New Thing.

Cheers,
--
Cameron Simpson <c...@zip.com.au>

"Vy can't ve chust climb?" - John Salathe