Where do nested functions live?
Steven D'Aprano
def foo():
def bar():
return "bar"
return "foo " + bar()
which works. Knowing how Python loves namespaces, I thought I could do
this:
>>> foo.bar()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'function' object has no attribute 'bar'
but it doesn't work as I expected.
where do nested functions live? How can you access them, for example, to
read their doc strings?
--
Steven.
Fredrik Lundh
> I defined a nested function:
>
> def foo():
> def bar():
> return "bar"
> return "foo " + bar()
>
> which works. Knowing how Python loves namespaces, I thought I could do
> this:
>
>>>> foo.bar()
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> AttributeError: 'function' object has no attribute 'bar'
>
> but it doesn't work as I expected.
>
> where do nested functions live?
in the local variable of an executing function, just like the variable
"bar" in the following function:
def foo():
bar = "who am I? where do I live?"
(yes, an inner function is *created* every time you execute the outer
function. but it's created from prefabricated parts, so that's not a
very expensive process).
</F>
Ben Finney
> I defined a nested function:
>
> def foo():
> def bar():
> return "bar"
> return "foo " + bar()
>
> which works. Knowing how Python loves namespaces, I thought I could
> do this:
>
> >>> foo.bar()
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> AttributeError: 'function' object has no attribute 'bar'
>
> but it doesn't work as I expected.
Functions don't get attributes automatically added to them the way
class do. The main exception is the '__doc__' attribute, referring to
the doc string value.
> where do nested functions live?
They live inside the scope of the function. Inaccessible from outside,
which is as it should be. Functions interact with the outside world
through a tightly-defined interface, defined by their input parameters
and their return value.
> How can you access them, for example, to read their doc strings?
If you want something that can be called *and* define its attributes,
you want something more complex than the default function type. Define
a class that has a '__call__' attribute, make an instance of that, and
you'll be able to access attributes and call it like a function.
--
\ "Writing a book is like washing an elephant: there no good |
`\ place to begin or end, and it's hard to keep track of what |
_o__) you've already covered." -- Anonymous |
Ben Finney
Steve Holden
def foo():
locvar = 23
return locvar
would you expect to be able to access foo.locvar?
It's exactly the same thing: the statement
def bar():
isn't executed until the foo() function is called, and its execution
binds the name bar in foo's local namespace to the function that is defined.
regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden
Fredrik Lundh
> If you want something that can be called *and* define its attributes,
> you want something more complex than the default function type. Define
> a class that has a '__call__' attribute, make an instance of that, and
> you'll be able to access attributes and call it like a function.
I turned Steven's question and portions of the answers into a Python FAQ
entry:
http://effbot.org/pyfaq/where-do-nested-functions-live.htm
Hope none of the contributors mind.
</F>
Steven D'Aprano
>> where do nested functions live?
>
> in the local variable of an executing function, just like the variable
> "bar" in the following function:
>
> def foo():
> bar = "who am I? where do I live?"
>
> (yes, an inner function is *created* every time you execute the outer
> function. but it's created from prefabricated parts, so that's not a
> very expensive process).
Does this mean I'm wasting my time writing doc strings for nested
functions? If there is no way of accessing them externally, should I make
them mere # comments?
--
Steven.
Marc 'BlackJack' Rintsch
D'Aprano wrote:
> Does this mean I'm wasting my time writing doc strings for nested
> functions? If there is no way of accessing them externally, should I make
> them mere # comments?
Whats the difference in "wasted time" between using """ or # as delimiters
for the explanation what the function is doing!? Or do you ask if you
should not document inner functions at all? Someone might read the source
and be very happy to find docs(trings) there.
And of course there are inner functions that are returned from the outer
one and on those objects it is possible to inspect and read the docs.
Ciao,
Marc 'BlackJack' Rintsch
Andrea Griffini
I'd add that while in some respect "def x" is like
an assigment to x ...
>>> def f():
global g
def g():
return "Yoo!"
>>> f()
>>> g()
'Yoo!'
in some other respect (unfortunately) it's not a regular assignment
>>> x = object()
>>> def x.g():
SyntaxError: invalid syntax
>>>
Andrea
Frederic Rentsch
> Steven D'Aprano wrote:
>
>
>> I defined a nested function:
>>
>> def foo():
>> def bar():
>> return "bar"
>> return "foo " + bar()
>>
>> which works. Knowing how Python loves namespaces, I thought I could do
>> this:
>>
>>
>>>>> foo.bar()
>>>>>
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in ?
>> AttributeError: 'function' object has no attribute 'bar'
>>
>> but it doesn't work as I expected.
>>
>> where do nested functions live?
>>
>
> "bar" in the following function:
>
> def foo():
> bar = "who am I? where do I live?"
>
> (yes, an inner function is *created* every time you execute the outer
> function. but it's created from prefabricated parts, so that's not a
> very expensive process).
>
>
>
If I may turn the issue around, I could see a need for an inner function
to be able to access the variables of the outer function, the same way a
function can access globals. Why? Because inner functions serve to
de-multiply code segments one would otherwise need to repeat or to
provide a code segment with a name suggestive of its function. In either
case the code segment moved to the inner function loses contact with its
environment, which rather mitigates its benefit.
If I have an inner function that operates on quite a few outer
variables it would be both convenient and surely more efficient, if I
could start the inner function with a declaration analogous to a
declaration of globals, listing the outer variables which I wish to
remain writable directly.
I guess I could put the outer variables into a list as argument to
the inner function. But while this relieves the inner function of
returning lots of values it burdens the outer function with handling the
list which it wouldn't otherwise need.
Frederic
Diez B. Roggisch
> to be able to access the variables of the outer function, the same way a
> function can access globals. Why? Because inner functions serve to
> de-multiply code segments one would otherwise need to repeat or to
> provide a code segment with a name suggestive of its function. In either
> case the code segment moved to the inner function loses contact with its
> environment, which rather mitigates its benefit.
Maybe I'm dense here, but where is your point? Python has nested lexical
scoping, and while some people complain about it's actual semantics, it
works very well:
def outer():
outer_var = 10
def inner():
return outer_var * 20
return inner
print outer()()
Diez
Frederic Rentsch
should be allowed to do so directly. Your example in its simplicity is
unproblematic. Let us consider a case where several outer variables need
to be changed:
weeks = days = hours = minutes = seconds = 0
mseconds = 0.0
(code)
# add interval in milliseconds
have_ms = ((((((((((weeks * 7) + days) * 24) + hours) * 60) +
minutes) * 60) + seconds) * 1000) + mseconds)
new_ms = have_ms + interval_ms
# reconvert
s = new_ms / 1000.0
s = int (s)
mseconds = new_ms - s * 1000
m, seconds = divmod (s, 60)
h, minutes = divmod (m, 60)
d, hours = divmod (h, 24)
weeks, days = divmod (d, 7)
(more code)
At some later point I need to increment my units some more and probably
will again a number of times. Clearly this has to go into a function. I
make it an inner function, because the scope of its service happens to
be local to the function in which it comes to live. It operates on
existing variables of what is now its containing function.
def increment_time (interval_ms):
have_ms = ((((((((((weeks * 7) + days) * 24) + hours) * 60) +
minutes) * 60) + seconds) * 1000) + mseconds)
new_ms = have_ms + interval_ms
# reconvert
s = new_ms / 1000.0
s = int (s)
ms -= s * 1000 # Was mseconds = new_ms - s * 1000
m, s = divmod (s, 60) # Was m, seconds = divmod (s, 60)
h, m = divmod (m, 60) # Was h, minutes = divmod (m, 60)
d, h = divmod (h, 24) # Was d, hours = divmod (h, 24)
w, d = divmod (d, 7) # Was weeks, days = divmod (d, 7)
return w, d, h, m, s, ms
Functionizing I must change the names of the outer variables. Assigning
to them would make them local, their outer namesakes would become
invisible and I'd have to pass them all as arguments. Simpler is
changing assignees names, retaining visibility and therefore not having
to pass arguments. In either case I have to return the result for
reassignment by the call.
weeks, days, hours, minutes, seconds, milliseconds = increment_time
(msec)
This is a little like a shop where the mechanics have to get their tools
and work pieces from the manager and hand them back to him when they're
done. The following two examples are illustrations of my point. They are
not proposals for 'improvement' of a language I would not presume to
improve.
def increment_time (interval_ms):
outer weeks, days, hours, minutes, seconds, mseconds # 'outer'
akin to 'global'
(...)
mseconds = new_ms - s * 1000 # Assignee remains outer
m, seconds = divmod (s, 60)
h, minutes = divmod (m, 60)
d, hours = divmod (h, 24)
weeks, days = divmod (d, 7) # No return necessary
The call would now be:
increment_time (msec) # No reassignment necessary
Hope this makes sense
Frederic
Fredrik Lundh
> At some later point I need to increment my units some more and probably
> will again a number of times. Clearly this has to go into a function.
since Python is an object-based language, clearly you could make your
counter into a self-contained object instead of writing endless amounts
of code and wasting CPU cycles by storing what's really a *single* state
in a whole bunch of separate variables.
in your specific example, you can even use an existing object:
t = datetime.datetime.now()
# increment
t += datetime.timedelta(milliseconds=msec)
print t.timetuple() # get the contents
if you're doing this so much that it's worth streamlining the timedelta
addition, you can wrap the datetime instance in a trivial class, and do
t += 1500 # milliseconds
when you need to increment the counter.
> This is a little like a shop where the mechanics have to get their
> tools and work pieces from the manager and hand them back to him when
> they're done.
that could of course be because when he was free to use whatever tool he
wanted, he always used a crowbar, because he hadn't really gotten around
to read that "tool kit for dummies" book.
</F>
Frederic Rentsch
> Frederic Rentsch wrote:
>
>
>> At some later point I need to increment my units some more and probably
>> will again a number of times. Clearly this has to go into a function.
>>
>
> since Python is an object-based language, clearly you could make your
> counter into a self-contained object instead of writing endless amounts
> of code and wasting CPU cycles by storing what's really a *single* state
> in a whole bunch of separate variables.
>
> in your specific example, you can even use an existing object:
>
> t = datetime.datetime.now()
>
> # increment
> t += datetime.timedelta(milliseconds=msec)
>
> print t.timetuple() # get the contents
>
> if you're doing this so much that it's worth streamlining the timedelta
> addition, you can wrap the datetime instance in a trivial class, and do
>
> t += 1500 # milliseconds
>
> when you need to increment the counter.
>
> > This is a little like a shop where the mechanics have to get their
> > tools and work pieces from the manager and hand them back to him when
> > they're done.
>
> that could of course be because when he was free to use whatever tool he
> wanted, he always used a crowbar, because he hadn't really gotten around
> to read that "tool kit for dummies" book.
>
employer.
> </F>
>
>
Rob Williscroft
li...@python.org in comp.lang.python:
> def increment_time (interval_ms):
> outer weeks, days, hours, minutes, seconds, mseconds # 'outer'
> akin to 'global'
> (...)
> mseconds = new_ms - s * 1000 # Assignee remains outer
> m, seconds = divmod (s, 60)
> h, minutes = divmod (m, 60)
> d, hours = divmod (h, 24)
> weeks, days = divmod (d, 7) # No return necessary
>
> The call would now be:
>
> increment_time (msec) # No reassignment necessary
>
>
> Hope this makes sense
Yes it does, but I prefer explicit in this case:
def whatever( new_ms ):
class namespace( object ):
pass
scope = namespace()
def inner():
scope.mseconds = new_ms - s * 1000
m, scope.seconds = divmod (s, 60)
h, scope.minutes = divmod (m, 60)
d, scope.hours = divmod (h, 24)
scope.weeks, scope.days = divmod (d, 7)
The only thing I find anoying is that I can't write:
scope = object()
Additionally if appropriate I can refactor further:
def whatever( new_ms ):
class namespace( object ):
def inner( scope ):
scope.mseconds = new_ms - s * 1000
m, scope.seconds = divmod (s, 60)
h, scope.minutes = divmod (m, 60)
d, scope.hours = divmod (h, 24)
scope.weeks, scope.days = divmod (d, 7)
scope = namespace()
scope.inner()
In short I think an "outer" keyword (or whatever it gets called)
will just add another way of doing something I can already do,
and potentially makes further refactoring harder.
Thats -2 import-this points already.
Frederic Rentsch
This is interesting. I am not too familiar with this way of using
objects. Actually it isn't all that different from a list, because a
list is also an object. But this way it's attribute names instead of
list indexes which is certainly easier to work with. Very good!
> The only thing I find anoying is that I can't write:
>
> scope = object()
>
> Additionally if appropriate I can refactor further:
>
> def whatever( new_ms ):
> class namespace( object ):
> def inner( scope ):
> scope.mseconds = new_ms - s * 1000
> m, scope.seconds = divmod (s, 60)
> h, scope.minutes = divmod (m, 60)
> d, scope.hours = divmod (h, 24)
> scope.weeks, scope.days = divmod (d, 7)
>
> scope = namespace()
> scope.inner()
>
> In short I think an "outer" keyword (or whatever it gets called)
> will just add another way of doing something I can already do,
> and potentially makes further refactoring harder.
>
>
Here I'm lost. What's the advantage of this? It looks more convoluted.
And speaking of convoluted, what about efficiency? There is much talk of
efficiency on this forum. I (crudely) benchmark your previous example
approximately three times slower than a simple inner function taking and
returning three parameters. It was actually the aspect of increased
efficiency that prompted me to play with the idea of allowing direct
outer writes.
> Thats -2 import-this points already.
>
>
Which ones are the two?
> Rob.
>
Frederic
Rob Williscroft
li...@python.org in comp.lang.python:
> Rob Williscroft wrote:
>> Frederic Rentsch wrote in news:mailman.1428.1162113628.11739.python-
>> li...@python.org in comp.lang.python:
>>
>>
>> def whatever( new_ms ):
>> class namespace( object ):
>> pass
>> scope = namespace()
>>
>> def inner():
>> scope.mseconds = new_ms - s * 1000
>> m, scope.seconds = divmod (s, 60)
>> h, scope.minutes = divmod (m, 60)
>> d, scope.hours = divmod (h, 24)
>> scope.weeks, scope.days = divmod (d, 7)
>>
>>
>
> This is interesting. I am not too familiar with this way of using
> objects. Actually it isn't all that different from a list, because a
> list is also an object. But this way it's attribute names instead of
> list indexes which is certainly easier to work with. Very good!
>
>> In short I think an "outer" keyword (or whatever it gets called)
>> will just add another way of doing something I can already do,
>> and potentially makes further refactoring harder.
>>
>>
>
> Here I'm lost. What's the advantage of this? It looks more convoluted.
I'll agree that having to explicitly define a namespace class first
does add some convolution.
But if you refer to having to prefix you "outer" variables with "scope."
then this would be the same as claiming that the explict use of self is
convoluted, which is a valid opinion, so fair enough, but I can't say
that I agree.
It should also be noted that although I have to declare and create a
scope object. My method doesn't require the attributes passed back from
the inner function be pre-declared, should I during coding realise
that I need to return another attribute I just assign it in the inner
function and use it in the outer function. I would count that as less
convoluted, YMMV.
> And speaking of convoluted, what about efficiency? There is much talk
> of efficiency on this forum. I (crudely) benchmark your previous
> example approximately three times slower than a simple inner function
> taking and returning three parameters. It was actually the aspect of
> increased efficiency that prompted me to play with the idea of
> allowing direct outer writes.
Did you have optimisation turned on ?
As I see it there should be no reason an optimiser couldn't transform
my code into the same code we might expect from your "outer keyword"
example, as the scope object's type and attributes are all contained
within (thus known to) the outer function defenition.
Wether CPython's optimiser does this or not, I don't know.
>
>> Thats -2 import-this points already.
>>
>>
>
> Which ones are the two?
Explicit is better than implicit.
There should be one-- and preferably only one --obvious way to do it.
Frederic Rentsch
I didn't mean to call into question. I didn't understand the advantage
of the added complexity of your second example over the first.
> It should also be noted that although I have to declare and create a
> scope object. My method doesn't require the attributes passed back from
> the inner function be pre-declared, should I during coding realise
> that I need to return another attribute I just assign it in the inner
> function and use it in the outer function. I would count that as less
> convoluted, YMMV.
>
That is certainly a very interesting aspect.
>
>
>> And speaking of convoluted, what about efficiency? There is much talk
>> of efficiency on this forum. I (crudely) benchmark your previous
>> example approximately three times slower than a simple inner function
>> taking and returning three parameters. It was actually the aspect of
>> increased efficiency that prompted me to play with the idea of
>> allowing direct outer writes.
>>
>
> Did you have optimisation turned on ?
>
>
No. I did a hundred thousand loops over each in IDLE using xrange.
> As I see it there should be no reason an optimiser couldn't transform
> my code into the same code we might expect from your "outer keyword"
> example, as the scope object's type and attributes are all contained
> within (thus known to) the outer function defenition.
>
>
I doubt that very much. The 'outer' keyword would give me the choice
between two alternatives. Such a choice can hardly be automated.
> Wether CPython's optimiser does this or not, I don't know.
>
>
>>> Thats -2 import-this points already.
>>>
>>>
>>>
>> Which ones are the two?
>>
>
> Explicit is better than implicit.
> There should be one-- and preferably only one --obvious way to do it.
>
> Rob.
>
Frederic
Rob Williscroft
li...@python.org in comp.lang.python:
> Rob Williscroft wrote:
>> Frederic Rentsch wrote in news:mailman.1536.1162292996.11739.python-
>>> Rob Williscroft wrote:
>>>> Frederic Rentsch wrote in news:mailman.1428.1162113628.11739.python-
[snip]
>>>>
>>> Here I'm lost. What's the advantage of this? It looks more convoluted.
>>>
>>
>> I'll agree that having to explicitly define a namespace class first
>> does add some convolution.
>>
>> But if you refer to having to prefix you "outer" variables with
>> "scope." then this would be the same as claiming that the explict use
>> of self is convoluted, which is a valid opinion, so fair enough, but
>> I can't say that I agree.
>>
>>
>
> I didn't mean to call into question. I didn't understand the advantage
> of the added complexity of your second example over the first.
>
Ok, I missed your point, as for the second example it was an attempt
to show that further refactoring from a function with local functions
that are sharing some state via the scope object, to a class with
methods that share state via the instance, is a matter of editing
a few lines.
This is useful when a function grows too large (for some value of
"too large"). As an added bonus you get to use the same thechniques
with both styles of coding.
[snip]
Carl Banks
Ben Finney wrote:
> "Steven D'Aprano" <st...@REMOVE.THIS.cybersource.com.au> writes:
>
> > I defined a nested function:
> >
> > def foo():
> > def bar():
> > return "bar"
> > return "foo " + bar()
> >
> > which works. Knowing how Python loves namespaces, I thought I could
> > do this:
> >
> > >>> foo.bar()
> > Traceback (most recent call last):
> > File "<stdin>", line 1, in ?
> > AttributeError: 'function' object has no attribute 'bar'
> >
> > but it doesn't work as I expected.
>
> Functions don't get attributes automatically added to them the way
> class do. The main exception is the '__doc__' attribute, referring to
> the doc string value.
>
> > where do nested functions live?
>
> They live inside the scope of the function. Inaccessible from outside,
Not so fast. You can get at the nested function by peeking inside code
objects (all bets off for Pythons other than CPython).
import new
def extract_nested_function(func,name):
codetype = type(func.func_code)
for obj in func.func_code.co_consts:
if isinstance(obj,codetype):
if obj.co_name == name:
return new.function(obj,func.func_globals)
raise ValueError("function with name %s not found in %r"
% (name,func))
Not exactly something I'd recommend for ordinary usage, but it can be
done.
Carl Banks
Fredrik Lundh
> Not so fast. You can get at the nested function by peeking inside code
> objects (all bets off for Pythons other than CPython).
>
> import new
> def extract_nested_function(func,name):
> codetype = type(func.func_code)
> for obj in func.func_code.co_consts:
> if isinstance(obj,codetype):
> if obj.co_name == name:
> return new.function(obj,func.func_globals)
> raise ValueError("function with name %s not found in %r"
> % (name,func))
that doesn't give you the actual nested function, though; just another
function object created from the same code object. and doing this right
is a bit harder than you make it look; your code fails in somewhat
confusing ways on straightforward things like:
def outer():
def inner(arg="hello"):
print arg
inner()
extract_nested_function(outer, "inner")()
and
def outer():
arg = "hello"
def inner():
print arg
inner()
extract_nested_function(outer, "inner")()
</F>
Frederic Rentsch
Rob,
Thanks a lot for your input. I'll have to digest that. Another question
I had and forgot was this: Since we have a class that goes out of scope
when the function returns, and we don't need more than one instance, why
bother to make an instance? Why not use the class object itself?
def whatever( new_ms ):
class scope ( object ):
def inner():
scope.mseconds = new_ms - s * 1000
m, scope.seconds = divmod (s, 60)
h, scope.minutes = divmod (m, 60)
d, scope.hours = divmod (h, 24)
scope.weeks, scope.days = divmod (d, 7)
Frederic
Steve Holden
>
> Thanks a lot for your input. I'll have to digest that. Another question
> I had and forgot was this: Since we have a class that goes out of scope
> when the function returns, and we don't need more than one instance, why
> bother to make an instance? Why not use the class object itself?
>
>
> class scope ( object ):
>
> def inner():
> scope.mseconds = new_ms - s * 1000
> m, scope.seconds = divmod (s, 60)
> h, scope.minutes = divmod (m, 60)
> d, scope.hours = divmod (h, 24)
> scope.weeks, scope.days = divmod (d, 7)
>
class scope(object): pass
to avoid syntax errors, I suspect. There doesn't seem to be any reason
why you couldn't use a class instead of an instance. And, of course,
either might give you problems in the case of a recursive inner function.
Rob Williscroft
li...@python.org in comp.lang.python:
> Since we have a class that goes out of scope
> when the function returns, and we don't need more than one instance, why
> bother to make an instance? Why not use the class object itself?
Because you hadn't yet pointed this out to me :-).
Thanks
Rob Williscroft
news:mailman.1615.1162411...@python.org in
comp.lang.python:
> Since we have a class that goes out of scope
>> when the function returns, and we don't need more than one instance,
>> why bother to make an instance? Why not use the class object itself?
>>
>> def whatever( new_ms ):
>>
>> class scope ( object ):
>>
>> def inner():
>> scope.mseconds = new_ms - s * 1000
>> m, scope.seconds = divmod (s, 60)
>> h, scope.minutes = divmod (m, 60)
>> d, scope.hours = divmod (h, 24)
>> scope.weeks, scope.days = divmod (d, 7)
>>
> That will need to be
>
> class scope(object): pass
>
> to avoid syntax errors, I suspect. There doesn't seem to be any reason
> why you couldn't use a class instead of an instance. And, of course,
> either might give you problems in the case of a recursive inner
> function.
What problems would it have that a recursive global function
that modified a global object, or a recursive method that
modified its instance doesn't have ?