C-style static variables in Python?
kj
When coding C I have often found static local variables useful for
doing once-only run-time initializations. For example:
int foo(int x, int y, int z) {
static int first_time = TRUE;
static Mongo *mongo;
if (first_time) {
mongo = heavy_lifting_at_runtime();
first_time = FALSE;
}
return frobnicate(mongo, x, y, z);
}
In this case, the static variable mongo is initialized only once
(at most).
What I like most about this is that it obviates the need for a
global variable to hold the persistent value (I avoid globals like
the plague, especially in Python). It also nicely encapsulates
the logic that determines whether initialization is required.
The best way I've found to achieve a similar effect in (procedural)
Python defines the function as a closure. For example, here's a
function that keeps track of (and prints out) how many times it
has been called:
>>> def make_spam():
... counter = [0]
... def _():
... counter[0] += 1
... print counter[0]
... return _
...
>>> spam = make_spam()
>>> spam()
1
>>> spam()
2
>>> spam()
3
(Too bad that one can't stick the whole def inside parentheses and
call the function right there, like one can do with JavaScript.)
Another approach would be to stuff the static values in the function's
__dict__. This is less satisfactory than the closure approach
because the "pseudo-static" variable is accessible from outside
the function, but the code is arguably a little more straightforward,
and one does not end up with the now useless one-time closure-generating
function kicking around. Here's another version of the function
above:
>>> def spam():
... d = spam.__dict__
... if not 's' in spam.__dict__:
... spam.s = 1
... print spam.s
... spam.s += 1
...
>>> spam()
1
>>> spam()
2
>>> spam()
3
Besides the external accessibility issue, I don't like explictly
coding the name of the function within the function. Is there any
way to have the function access its own __dict__ without having to
explicitly code its name in its body? E.g., is there some generic
special variable that, within a function, refers to the function
object itself?
I'm sure that there are many other ways to skin this cat, especially
if one starts definining fancy callable classes and whatnot. But
is there a better *simple* way to achieve C-style static locals in
Python that does not require a lot of extra machinery?
TIA!
~K
Chris Rebert
> When coding C I have often found static local variables useful for
> doing once-only run-time initializations.
Nope. It's been proposed in that past
(http://www.python.org/dev/peps/pep-3130/), but that proposal was
rejected.
> I'm sure that there are many other ways to skin this cat, especially
> if one starts definining fancy callable classes and whatnot. But
> is there a better *simple* way to achieve C-style static locals in
> Python that does not require a lot of extra machinery?
You can abuse the default argument value mechanism:
def spam(s_cell=[1]):
s = s_cell[0]
print s
s_cell[0] += 1
It's a bit less ugly when the value itself is mutable, which isn't the
case here with the integer.
Personally, I hate such abuse with a passion; I think a global
variable is clearest.
Cheers,
Chris
--
http://blog.rebertia.com
Steve Holden
>> doing once-only run-time initializations.
> (http://www.python.org/dev/peps/pep-3130/), but that proposal was
> rejected.
>
>> if one starts definining fancy callable classes and whatnot. But
>> is there a better *simple* way to achieve C-style static locals in
>> Python that does not require a lot of extra machinery?
>
>
> def spam(s_cell=[1]):
> s = s_cell[0]
> print s
> s_cell[0] += 1
>
> It's a bit less ugly when the value itself is mutable, which isn't the
> case here with the integer.
>
> Personally, I hate such abuse with a passion; I think a global
> variable is clearest.
But the real problem is that the OP is insisting on using purely
procedural Python when the problem is screaming for an object-oriented
answer.
If the function were instead a method then the instance namespace would
be the logical place to store the required data.
regards
Steve
regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS: http://holdenweb.eventbrite.com/
Terry Reedy
>
>
> When coding C I have often found static local variables useful for
> doing once-only run-time initializations. For example:
>
> int foo(int x, int y, int z) {
>
> static int first_time = TRUE;
> static Mongo *mongo;
> if (first_time) {
> mongo = heavy_lifting_at_runtime();
> first_time = FALSE;
> }
>
> return frobnicate(mongo, x, y, z);
Global var or class or closure such as below (obviously untested ;=):
make_foo()
mongo = heavy_lifting_at_runtime();
def _(x,y,z):
return frobnicate(mongo, x, y, z)
return _
foo = make_foo
del make_foo # to make sure it is *never* called again ;
Now you only have foo with a hard-to-access private object and no
first_time checks when you call it.
Terry Jan Reedy
Patrick Maupin
> Chris Rebert wrote:
> > Personally, I hate such abuse with a passion; I think a global
> > variable is clearest.
>
> But the real problem is that the OP is insisting on using purely
> procedural Python when the problem is screaming for an object-oriented
> answer.
>
> If the function were instead a method then the instance namespace would
> be the logical place to store the required data.
In some situations I will use either the default parameter
initialization Chris mentioned, or the closure mechanism that the OP
presented, but only on code that I am optimizing for speed (local
variable lookups, even in nested functions, being much faster than
global or instance lookups). If it doesn't need to go fast, globals
or instance variables are the way to go.
Pat
Steve Holden
> On 4/1/2010 6:34 PM, kj wrote:
>>
>>
>> doing once-only run-time initializations. For example:
>>
>> int foo(int x, int y, int z) {
>>
>> static int first_time = TRUE;
>> static Mongo *mongo;
>> if (first_time) {
>> mongo = heavy_lifting_at_runtime();
>> first_time = FALSE;
>> }
>>
>> return frobnicate(mongo, x, y, z);
>
>
> make_foo()
> mongo = heavy_lifting_at_runtime();
> def _(x,y,z):
> foo = make_foo
I suspect you mean
foo = make_foo()
> del make_foo # to make sure it is *never* called again ;
>
> Now you only have foo with a hard-to-access private object and no
> first_time checks when you call it.
>
> Terry Jan Reedy
>
I don't think I'd ever want to use such an obscure technique in a
program. You might want to consider using functools.wraps to make sure
that the foo function looks right.
Alf P. Steinbach
> When coding C I have often found static local variables useful for
> doing once-only run-time initializations. For example:
>
> int foo(int x, int y, int z) {
>
> static int first_time = TRUE;
> static Mongo *mongo;
> if (first_time) {
> mongo = heavy_lifting_at_runtime();
> first_time = FALSE;
> }
>
> return frobnicate(mongo, x, y, z);
> }
>
> In this case, the static variable mongo is initialized only once
> (at most).
>
> What I like most about this is that it obviates the need for a
> global variable to hold the persistent value (I avoid globals like
> the plague, especially in Python). It also nicely encapsulates
> the logic that determines whether initialization is required.
In C++ you just write
int foo( int x, int y, int z )
{
static Mongo* const mongo = heavy_lifting_at_runtime();
return frobnicate( mongo, x, y, z );
}
> The best way I've found to achieve a similar effect in (procedural)
> Python defines the function as a closure. For example, here's a
> function that keeps track of (and prints out) how many times it
> has been called:
>
>>>> def make_spam():
> ... counter = [0]
> ... def _():
> ... counter[0] += 1
> ... print counter[0]
> ... return _
> ...
>>>> spam = make_spam()
>>>> spam()
> 1
>>>> spam()
> 2
>>>> spam()
> 3
>
> (Too bad that one can't stick the whole def inside parentheses and
> call the function right there, like one can do with JavaScript.)
Off the cuff, Py3:
class Spam:
def __init__( self ):
self._counter = 0
def __call__( self ):
self._counter += 1
print( counter )
spam = Spam()
spam()
spam()
spam()
[snip]
> I'm sure that there are many other ways to skin this cat, especially
> if one starts definining fancy callable classes and whatnot.
As I see it it's the closure that's fancy, and the class that's simple and direct.
> But
> is there a better *simple* way to achieve C-style static locals in
> Python that does not require a lot of extra machinery?
If you often need this functionality you might consider a general decorator that
supplies the function with a self argument, e.g. like this:
<example>
#Py3
class Object: pass
def static_initialization( init_func ):
def self_aware( f ):
def wrapped( *args, **kwargs ):
return f( f, *args, **kwargs )
init_func( f )
return wrapped
o = Object()
o.body = self_aware
return o
# Example usage:
@static_initialization
def spam( self ):
self.counter = 0
@spam.body
def spam( self ):
self.counter += 1
print( self.counter )
spam()
spam()
spam()
</example>
But as mentioned, a class is (at least IMHO) simpler and more direct.
Cheers & hth.,
- Alf (department of disingenious solutions)
Paul Rubin
> When coding C I have often found static local variables useful for
> doing once-only run-time initializations. For example:
>
> int foo(int x, int y, int z) {
> static int first_time = TRUE;
> static Mongo *mongo;
Here are some cheesy ways.
1. Put an attribute onto the function name:
def foo(x, y, z):
if foo.first_time:
foo.mongo = heavy_lifting_at_runtime()
foo.first_time = False
...
foo.first_time = True
2. Use a mutable keyword parameter:
def foo(x, y, z, wrapped_mongo=[]):
if len(wrapped_mongo) == 0:
wrapped_mongo.append(heavy_lifting_at_runtime())
mongo = wrapped_mongo[0]
...
3. Streamline the first method a little:
def foo(x, y, z):
if len(foo.wrapped_mongo == 0):
foo.wrapped_mongo.append(heavy_lifting_at_runtime())
mongo = foo.wrapped_mongo[0]
...
foo.wrapped_mongo = []
All of these of course don't give as good encapsulation as one might
like.
kj
>But the real problem is that the OP is insisting on using purely
>procedural Python when the problem is screaming for an object-oriented
>answer.
My initial reaction to this comment was something like "What? switch
from procedural to OO just to be able to do some one-time initialization
of function-private data???" But then, again, since Python allows
easy mixing of both programming styles, I suppose one could refactor this:
<procedural>
def spam(x, y, z):
try:
mongo = spam.mongo
except AttributeError:
mongo = spam.mongo = heavy_lifting_at_runtime()
return frobnicate(x, y, z, mongo)
ham = spam(3, 4, 5)
</procedural>
into this:
<OO>
class _Spam(object):
@classmethod
def _(cls, x, y, z):
try:
mongo = cls.mongo
except AttributeError:
mongo = cls.mongo = heavy_lifting_at_runtime()
return frobnicate(x, y, z, mongo)
ham = _Spam._(1, 2, 3)
</OO>
Is this really more natural or more readable? Hmmm.
In any case, the first solution does rely on the fact that functions
are objects, and therefore can have attributes, so even the
"procedural" version relies on Python's OO model.
Other responses advocated for global variables. I avoid them in
general, and doubly so in Python, because I find Python's shenanigans
with globals mystifying (this business of becoming silently local
if assigned to); it's one rare instance in which Python out-Perls
Perl. And yes, I know that the language includes ways to deal with
this (with the global keyword, etc.) but I find the whole scheme
is so much "cutting against the grain".
Thanks for all the replies. There are a lot of good ideas there.
I'm particular, I'm thankful for the pointers to PEP 3130 (initial
reaction: maybe I should learn Dutch) and to functools.wraps, and
for the code snippets.
~K
Paul McGuire
> When coding C I have often found static local variables useful for
> doing once-only run-time initializations. For example:
>
Here is a decorator to make a function self-aware, giving it a "this"
variable that points to itself, which you could then initialize from
outside with static flags or values:
from functools import wraps
def self_aware(fn):
@wraps(fn)
def fn_(*args):
return fn(*args)
fn_.__globals__["this"] = fn_
return fn_
@self_aware
def foo():
this.counter += 1
print this.counter
foo.counter = 0
foo()
foo()
foo()
Prints:
1
2
3
-- Paul
Mel
> In <mailman.1437.1270163...@python.org> Steve Holden
> <st...@holdenweb.com> writes:
>
>>But the real problem is that the OP is insisting on using purely
>>procedural Python when the problem is screaming for an object-oriented
>>answer.
>
> My initial reaction to this comment was something like "What? switch
> from procedural to OO just to be able to do some one-time initialization
> of function-private data???"
Yeah, actually. If the subject had been "Python-style object attributes in
C?" somebody might have suggested C static variables. An example I wrote
lately
volatile static int random_bit ()
{
static unsigned short lfsr = 0xACE1u; // seeded LFSR
// taps: 16 14 13 11; characteristic polynomial: x^16 + x^14 + x^13 +
x^11 + 1
lfsr = (lfsr >> 1) ^ (-(lfsr & 1u) & 0xB400u);
return lfsr & 1;
} // random_bit
(excuse is: this was written for cheap execution in an 8-bit processor.)
This does OK -- but fails the instant I decide that my program needs more
than one pseudo-random bit stream. Then I have the choice of writing
several different random_bit functions, or extending random_bit to take a
pointer to a seeded LFSR provided by the individual caller.
Refactoring the Python function to a Python class, as you mention later,
solves the static-access problem, but that solution is just as vulnerable to
the need-more-than-just-the-one problem as my C function.
Mel.
Duncan Booth
> I suppose one could refactor this:
>
><procedural>
> def spam(x, y, z):
> try:
> mongo = spam.mongo
> except AttributeError:
> mongo = spam.mongo = heavy_lifting_at_runtime()
> return frobnicate(x, y, z, mongo)
>
> ham = spam(3, 4, 5)
></procedural>
>
> into this:
>
><OO>
> class _Spam(object):
> @classmethod
> def _(cls, x, y, z):
> try:
> mongo = cls.mongo
> except AttributeError:
> mongo = cls.mongo = heavy_lifting_at_runtime()
> return frobnicate(x, y, z, mongo)
>
> ham = _Spam._(1, 2, 3)
></OO>
>
>
> Is this really more natural or more readable? Hmmm.
No, but that's because it is needlessly obfuscated. What's with the weird _
method? Why use a class method? Why not just create an instance?
class Spam(object):
mongo = None
def __call__(self, x, y, z):
if self.mongo is None:
self.mongo = heavy_lifting_at_runtime()
return frobnicate(x, y, z, self.mongo)
spam = Spam()
ham = spam(1, 2, 3)
That's natural and readable.
There's also another good reason why the class is better than the static
variable: you can construct multiple different instances with different
calls to 'heavy_lifting_at_runtime'. e.g. You could write a unit test where
mongo is initialised to mock_heavy_lifting_at_runtime().
Ethan Furman
> <OO>
> class _Spam(object):
> @classmethod
> def _(cls, x, y, z):
> try:
> mongo = cls.mongo
> except AttributeError:
> mongo = cls.mongo = heavy_lifting_at_runtime()
> return frobnicate(x, y, z, mongo)
>
> ham = _Spam._(1, 2, 3)
> </OO>
>
>
> Is this really more natural or more readable? Hmmm.
For this type of situation, my preference would be:
class spam(object):
def __call__(self, x, y, z):
try:
mongo = self.mongo
except AttributeError:
mongo = self.mongo = heavy_lifting_at_runtime()
return frobnicate(x, y, z, mongo)
spam = spam()
No extra objects, out-of-place underscores, etc.
~Ethan~
Patrick Maupin
> For this type of situation, my preference would be:
>
> class spam(object):
> def __call__(self, x, y, z):
> try:
> mongo = self.mongo
> except AttributeError:
> mongo = self.mongo = heavy_lifting_at_runtime()
> return frobnicate(x, y, z, mongo)
> spam = spam()
>
> No extra objects, out-of-place underscores, etc.
>
> ~Ethan~
Well, I'm not a big fan of unnecessary try/except, so I would at least
change it to:
class spam(object):
def __getattr__(self, name):
if name != 'mongo':
raise AttributeError
self.mongo = heavy_lifting_at_runtime()
return self.mongo
def __call__(self, x, y, z):
return frobnicate(x, y, z, self.mongo)
spam = spam()
Regards,
Pat
Steven D'Aprano
> Other responses advocated for global variables. I avoid them in
> general,
In general this is wise, but remember that because Python globals are not
globally global, but local to a single module, they're safer than globals
in other languages. Still, it's better to avoid them when possible.
> and doubly so in Python, because I find Python's shenanigans
> with globals mystifying (this business of becoming silently local if
> assigned to);
Globals don't become local when assigned to. You can shadow a global with
a local of the same name, but the global remains untouched:
>>> myglobal = 42
>>> def test():
... myglobal = 0 # shadow the global with a new local
...
>>> test()
>>> myglobal
42
I find this behaviour perfectly natural, and desirable: it means I can
assign to locals without worrying whether or not I'm about to stomp all
over a global and destroy it. The alternative behaviour would be
disastrous:
>>> def f(x): return x+1
...
>>> def test():
... f = 'spam'
...
>>> test()
>>> f(2) # this doesn't happen
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'str' object is not callable
--
Steven
Ethan Furman
Sounds like a personal preference issue, rather than a necessary /
unnecessary issue -- after all, if you call that function a thousand
times, only once is mongo not defined... clearly the exception. ;)
~Ethan~
Patrick Maupin
Well, I think the whole discussion has basically been about personal
preference. OTOH, but if you call the function a few million times,
you might find the cost of try/except to be something that you would
rather not incur -- it might become a performance issue rather than a
personal choice issue. On the other OTHER hand, if you call the
function a few billion times, performance weighs more heavily in favor
of the closure approach rather than the object approach, since local
variable lookup is so much cheaper.
Regards,
Pat
Ethan Furman
[snippage]
> Well, I think the whole discussion has basically been about personal
> preference. OTOH, but if you call the function a few million times,
> you might find the cost of try/except to be something that you would
> rather not incur -- it might become a performance issue rather than a
> personal choice issue. On the other OTHER hand, if you call the
> function a few billion times, performance weighs more heavily in favor
> of the closure approach rather than the object approach, since local
> variable lookup is so much cheaper.
Indeed. I was surprised to find your __getattr__ approach faster than
the try/except approach (about 20% on my machine). I'll have to think
about that for future situations like this.
My main point, though, was using __call__, and not some weird _ method. ;)
~Ethan~
Patrick Maupin
> My main point, though, was using __call__, and not some weird _ method. ;)
Yes, __call__ is good. In general, not naming things that don't need
to be named is good (but if you have too many of them to keep track
of, then, obviously, they need to be named, hence named tuples).
But I didn't need to address that, since you already did :-)
kj
>class Spam(object):
> mongo = None
> def __call__(self, x, y, z):
> if self.mongo is None:
> self.mongo = heavy_lifting_at_runtime()
> return frobnicate(x, y, z, self.mongo)
>spam = Spam()
>ham = spam(1, 2, 3)
I really like this. Thanks.
>That's natural and readable.
From reading this thread, and the "(a==b) ? 'Yes' : 'No'" one, the
inescapable conclusion is that "readability" (like beauty) is very
much in the eye of the beholder, or, in this case, in the eye of
Guido.
~K
Steven D'Aprano
> On Apr 2, 2:38 pm, Ethan Furman <et...@stoneleaf.us> wrote:
[...]
>> Sounds like a personal preference issue, rather than a necessary /
>> unnecessary issue -- after all, if you call that function a thousand
>> times, only once is mongo not defined... clearly the exception. ;)
>>
>> ~Ethan~
>
> Well, I think the whole discussion has basically been about personal
> preference. OTOH, but if you call the function a few million times, you
> might find the cost of try/except to be something that you would rather
> not incur -- it might become a performance issue rather than a personal
> choice issue.
The cost of a try...except is *very* low -- about the same as a pass
statement:
>>> from timeit import Timer
>>> t1 = Timer("pass", "")
>>> t2 = Timer("try:\n pass\nexcept Exception:\n pass", "")
>>> min(t2.repeat())/min(t1.repeat())
1.9227982449955801
Actually catching the exception, on the other hand, is quite expensive:
>>> t1 = Timer("len('')", "")
>>> t2 = Timer("try:\n len(0)\nexcept Exception:\n pass", "")
>>> min(t2.repeat())/min(t1.repeat())
10.598482743564809
The heuristic I use is, if I expect the try block to raise an exception
more than about one time in ten, I change to an explicit test. In this
case, since the exception should only be raised once, and then never
again, I would use a try...except block.
--
Steven
Patrick Maupin
cybersource.com.au> wrote:
> On Fri, 02 Apr 2010 12:39:16 -0700, Patrick Maupin wrote:
> > On Apr 2, 2:38 pm, Ethan Furman <et...@stoneleaf.us> wrote:
> [...]
> >> Sounds like a personal preference issue, rather than a necessary /
> >> unnecessary issue -- after all, if you call that function a thousand
> >> times, only once is mongo not defined... clearly the exception. ;)
>
> >> ~Ethan~
>
> > Well, I think the whole discussion has basically been about personal
> > preference. OTOH, but if you call the function a few million times, you
> > might find the cost of try/except to be something that you would rather
> > not incur -- it might become a performance issue rather than a personal
> > choice issue.
>
> The cost of a try...except is *very* low -- about the same as a pass
> statement:
Agreed. In the example above, if frobnicate() is a null function, the
try/except adds about 5% to execution time on my machine. If I were
really worried about execution time, I would use a closure *for this
particular example* as I mentioned elsewhere. However, the cost of
the try/except is not zero, and when I have something I prefer looking
at (the __getattr__ doesn't clutter up the main-line execution with
conditionals for stuff that only gets used once at initialization)
that is always known to be cheaper in execution, that's what I use. I
suppose some people might not like looking at the __getattr__, but
this is a memoization technique I use quite often, so I find it
idiomatic.
Regards,
Pat
Terry Reedy
> In<Xns9D4EC021DC...@127.0.0.1> Duncan Booth<duncan...@invalid.invalid> writes:
>
>> class Spam(object):
>> mongo = None
>> def __call__(self, x, y, z):
>> if self.mongo is None:
>> self.mongo = heavy_lifting_at_runtime()
>> return frobnicate(x, y, z, self.mongo)
Unless one wants the intialization of mongo delayed in case spam is
never called, it can go in __init__ instead.
Terry Reedy
> On Apr 1, 5:34 pm, kj<no.em...@please.post> wrote:
>> When coding C I have often found static local variables useful for
>> doing once-only run-time initializations. For example:
>>
>
> Here is a decorator to make a function self-aware, giving it a "this"
> variable that points to itself, which you could then initialize from
> outside with static flags or values:
>
> from functools import wraps
>
> def self_aware(fn):
> @wraps(fn)
> def fn_(*args):
> return fn(*args)
> fn_.__globals__["this"] = fn_
> return fn_
In 3.1, at least, the wrapper is not needed.
def self_aware(fn):
fn.__globals__["this"] = fn
return fn
Acts the same
> @self_aware
> def foo():
> this.counter += 1
> print this.counter
>
> foo.counter = 0
Explicit and separate initialization is a pain. This should be in a
closure or class.
> foo()
> foo()
> foo()
> Prints:
> 1
> 2
> 3
However, either way, the __globals__ attribute *is* the globals dict,
not a copy, so one has
>>> this
<function foo at 0x00F5F5D0>
Wrapping a second function would overwrite the global binding.
Terry Jan Reedy
Ethan Furman
>> In<Xns9D4EC021DC...@127.0.0.1> Duncan
>> Booth<duncan...@invalid.invalid> writes:
>>
>>> class Spam(object):
>>> mongo = None
>>> def __call__(self, x, y, z):
>>> if self.mongo is None:
>>> self.mongo = heavy_lifting_at_runtime()
>>> return frobnicate(x, y, z, self.mongo)
>
>
> never called, it can go in __init__ instead.
As a matter of fact, I have an object that is usually not called during
it's modules use, so I put in __getattr__. Sped the modules load time
back up to pert near instantaneous. :)
~Ethan~
Ethan Furman
That was my reasoning as well, but when I timed it for one million runs
(so 1 instantiation, 999,999 simple calls), the __getattr__ time was .5
seconds, the try...execpt block was .6; at ten million it was 5 and 6.
At those rates, personal preference takes over, at least for me.
~Ethan~
Stephen Hansen
I prefer:
class Spam(object):
def __init__(self):
self._mondo = None
def _get_mondo(self):
if self._mondo is None:
self._mondo = heavy_lifting_at_runtime()
return self._mondo
mondo = property(_get_mondo)
def __call__(self, x, y, z):
return frobnicate(x,y,z, self.mondo)
I don't know if properties are really faster or slower then a
__getattr__, but I find them a lot cleaner if I want to delay some
calculation until needed like that.
--
--S
... p.s: change the ".invalid" to ".com" in email address to reply privately.
Patrick Maupin
>
> I don't know if properties are really faster or slower then a
> __getattr__, but I find them a lot cleaner if I want to delay some
> calculation until needed like that.
Well, the relative speed of properties vs. __getattr__ can become
irrelevant in at least two ways:
1) If the __getattr__ only calculates the value one time and then
stuffs it into the instance dictionary, now you are really comparing
the relative speed of properties vs. lookup of an attribute in the
instance dict. If you're at all concerned about speed, I think there
is a clear winner here.
2) There is a single __getattr__ function, vs. one property for every
attribute that needs a property. In cases where you can somehow
easily compute the attribute names as well as the attribute values,
__getattr__ can be a *lot* less code than defining dozens of
properties.
But you're absolutely right that, in many cases, property is the best
way to go for readability (especially if the property is read-only and
you're using a recent enough python to use decorators).
Regards,
Pat
Steven D'Aprano
>> The heuristic I use is, if I expect the try block to raise an exception
>> more than about one time in ten, I change to an explicit test. In this
>> case, since the exception should only be raised once, and then never
>> again, I would use a try...except block.
>
> That was my reasoning as well, but when I timed it for one million runs
> (so 1 instantiation, 999,999 simple calls), the __getattr__ time was .5
> seconds, the try...execpt block was .6; at ten million it was 5 and 6.
Care to share your timing code? Not that I don't trust your results, but
timings are very sensitive to the exact thing you do, and I'd like to see
what that is.
--
Steven
Ethan Furman
Happy to do so -- if I made a mistake I'd like to know about it and learn.
It'll have to wait two days 'til I get back to work, though... I'll post
it asap.
~Ethan~
Stephen Hansen
> On Apr 2, 10:11 pm, Stephen Hansen <apt.shan...@gmail.invalid> wrote:
>>
>> I don't know if properties are really faster or slower then a
>> __getattr__, but I find them a lot cleaner if I want to delay some
>> calculation until needed like that.
>
> Well, the relative speed of properties vs. __getattr__ can become
> irrelevant in at least two ways:
>
> 1) If the __getattr__ only calculates the value one time and then
> stuffs it into the instance dictionary, now you are really comparing
> the relative speed of properties vs. lookup of an attribute in the
> instance dict. If you're at all concerned about speed, I think there
> is a clear winner here.
I concede it would probably be notably faster, but there's a big
difference between "at all concerned about speed" and "optimizing a
profiled bottleneck".
The speed difference between direct attribute lookup and properties may
be notable, but that doesn't make a clear winner here. Now that I have
(with either method) optimized the expensive value-calculation
operation such that it only happens on-demand and once, I now have to
weigh further optimization.
Is the difference in speed between a standard attribute lookup and a
property fetch worth losing the clarity the property brings over the
__getattr__ solution, especially considering the __getattr__ creates a
fuzzy 'sometimes this code is responsible, othertimes the dict is'
situation that someone may down the road miss in maintenance?
For me, usually not-- unless profiling pushes me to reconsider. But
everyone makes these calls differently.
> 2) There is a single __getattr__ function, vs. one property for every
> attribute that needs a property. In cases where you can somehow
> easily compute the attribute names as well as the attribute values,
> __getattr__ can be a *lot* less code than defining dozens of
> properties.
I don't really mind a lot of properties, if they're simple. Then again,
I often prefer regular ol' attributes where possible :) However, if I'm
doing a dispatching sort of mechanism, or a situation where the "name"
isn't something static, set in stone or pre-defined-- then certainly,
__getattr__ is a fine solution. I don't mind it where its the clearest
way to accomplish a goal.
John Nagle
> When coding C I have often found static local variables useful for
> doing once-only run-time initializations.
If you want functions with state, use an object. That's what they're
for. Don't muck with the internal representation of functions.
John Nagle
Patrick Maupin
> If you want functions with state, use an object. That's what they're
> for. Don't muck with the internal representation of functions.
>
While "Don't muck with the internal representation of functions" is
excellent advice over 99% of the time, it is also true that it is
often possible, sometimes even encouraged, to have "functions with
state."
This is done without "mucking" and without explicitly declaring a
class or a class instance. See, e.g. closures and generator
functions.
Regards,
Pat
Lee Harr
> Another approach would be to stuff the static values in the function's
> __dict__.
That's how I did it when I wanted something similar.
I created this decorator:
def static(**kw):
'''
Used to create a decorator function that will add an
attribute to a function and initialize it.
>>> @static(foo=5)
... def bar():
... print bar.foo
... bar.foo += 1
...
>>> bar()
5
>>> bar()
6
'''
def decorator(f):
f.__dict__.update(kw)
return f
return decorator
_________________________________________________________________
Hotmail: Trusted email with Microsoft’s powerful SPAM protection.
https://signup.live.com/signup.aspx?id=60969
Ethan Furman
>
> It'll have to wait two days 'til I get back to work, though... I'll post
> it asap.
Well, so much for asap, but here's what I used (with one correction: in
the 'if' code I had forgotten to actually reference the missing
attribute, so the __getattr__ look up never happened; now the
try...except block is /slightly/ faster, as opposed to 20% slower).
class spam_except(object):
def __call__(self, x, y, z):
try:
mongo = self.mongo
except AttributeError:
mongo = self.mongo = 1
return
class spam_if(object):
def __getattr__(self, name):
if name != 'mongo':
raise AttributeError
self.mongo = 1
return self.mongo
def __call__(self, x, y, z):
self.mongo # didn't have this line before. d'oh!
return
--> timeit.Timer('spammer(1,2,3)','from spam import spam_except;
spammer=spam_except()').timeit()
0.65764130543749388
--> timeit.Timer('spammer(1,2,3)','from spam import spam_if;
spammer=spam_if()').timeit()
0.66972877235545525
~Ethan~
Patrick Maupin
(Posted some code with a timeit...)
Well, I'm not going to debug this, but with the *original* thing you
posted, and the thing I posted, with a call and everything (more
realistic scenario), the exception version seems slower on my machine:
#!/usr/bin/env python
import timeit
def frobnicate(a,b,c,d):
pass
def heavy_lifting_at_runtime():
print 'heavy'
class spam_except(object):
def __call__(self, x, y, z):
try:
mongo = self.mongo
except AttributeError:
mongo = self.mongo = heavy_lifting_at_runtime()
return frobnicate(x, y, z, mongo)
se = spam_except()
class spam_if(object):
def __getattr__(self, name):
if name != 'mongo':
raise AttributeError
self.mongo = heavy_lifting_at_runtime()
return self.mongo
def __call__(self, x, y, z):
return frobnicate(x, y, z, self.mongo)
si = spam_if()
tse = timeit.Timer('se(1,2,3)', "from __main__ import se")
tsi = timeit.Timer('si(1,2,3)', "from __main__ import si")
for i in range(5):
ve = tse.timeit(10000000)
vi = tsi.timeit(10000000)
print ve, vi, '%.1f' % ((ve-vi) / vi * 100)
------
heavy
heavy
5.45695090294 5.10844397545 6.8
5.43381404877 5.01345705986 8.4
5.42474508286 5.02641201019 7.9
5.40713405609 5.04178905487 7.2
5.38063693047 4.96194696426 8.4
The output indicates that the exception one is, on average, around
7.5% slower.
Regards,
Pat
Albert van der Horst
Steve Holden <st...@holdenweb.com> wrote:
>Terry Reedy wrote:
>> On 4/1/2010 6:34 PM, kj wrote:
>>>
>>>
>>> When coding C I have often found static local variables useful for
>>>
>>> int foo(int x, int y, int z) {
>>>
>>> static int first_time = TRUE;
>>> static Mongo *mongo;
>>> if (first_time) {
>>> mongo = heavy_lifting_at_runtime();
>>> first_time = FALSE;
>>> }
>>>
>>> return frobnicate(mongo, x, y, z);
>>
>> Global var or class or closure such as below (obviously untested ;=):
>>
>> make_foo()
>> mongo = heavy_lifting_at_runtime();
>> def _(x,y,z):
>> return frobnicate(mongo, x, y, z)
>> return _
>> foo = make_foo
>
>I suspect you mean
>
>foo = make_foo()
>
>> del make_foo # to make sure it is *never* called again ;
>>
>> Now you only have foo with a hard-to-access private object and no
>> first_time checks when you call it.
>>
>> Terry Jan Reedy
>>
>I don't think I'd ever want to use such an obscure technique in a
>program. You might want to consider using functools.wraps to make sure
>that the foo function looks right.
Imagine that heavy_lifting is only ever used here and uses 4 Gbyte of core.
Suddenly deleting those function objects seems the right thing to do,
instead of an obscure technique.
(I'm not sure the Python compiler could take advantage of this,
I know I could in my Forth compiler, under circumstances.)
>
>regards
> Steve
>--
>Steve Holden +1 571 484 6266 +1 800 494 3119
Groetjes Albert
--
--
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst