using "private" parameters as static storage?
Joe Strout
have static storage within a method -- i.e. storage that is persistent
between calls, but not visible outside the method. I frequently use
this for such things as caching, or for keeping track of how many
objects a factory function has created, and so on.
Today it occurred to me to use a mutable object as the default value
of a parameter. A simple example:
def spam(_count=[0]):
_count[0] += 1
return "spam " * _count[0]
>>> spam()
'spam '
>>> spam()
'spam spam '
This appears to work fine, but it feels a little unclean, having stuff
in the method signature that is only meant for internal use. Naming
the parameter with an underscore "_count" makes me feel a little
better about it. But then, adding something to the module namespace
just for use by one function seems unclean too.
What are your opinions on this idiom? Is there another solution
people generally prefer?
Ooh, for a change I had another thought BEFORE hitting Send rather
than after. Here's another trick:
def spam2():
if not hasattr(spam2,'count'):spam2.count=0
spam2.count += 1
return "spam2 " * spam2.count
This doesn't expose any uncleanliness outside the function at all.
The drawback is that the name of the function has to appear several
times within itself, so if I rename the function, I have to remember
to change those references too. But then, if I renamed a function,
I'd have to change all the callers anyway. So maybe this is better.
What do y'all think?
Best,
- Joe
Chris Mellon
Static storage is a way of preserving state. Objects are a way of
encapsulating state and behavior. Use an object.
Matimus
> One thing I miss as I move from REALbasic to Python is the ability to
> have static storage within a method -- i.e. storage that is persistent
> between calls, but not visible outside the method. I frequently use
> this for such things as caching, or for keeping track of how many
> objects a factory function has created, and so on.
>
> Today it occurred to me to use a mutable object as the default value
> of a parameter. A simple example:
>
> def spam(_count=[0]):
> _count[0] += 1
> return "spam " * _count[0]
>
> >>> spam()
> 'spam '
> >>> spam()
> 'spam spam '
>
Don't Do this, it is confusing and there are definitely (many) better
facilities in python for handling saved state.
> Ooh, for a change I had another thought BEFORE hitting Send rather
> than after. Here's another trick:
>
> def spam2():
> if not hasattr(spam2,'count'):spam2.count=0
> spam2.count += 1
> return "spam2 " * spam2.count
This is definitely preferred over the first. However the preferred
method is just to use a class. Preserving state is what classes are
for.
>>> class Spam(object):
... def __init__(self):
... self._count = 0
... def spam(self):
... self._count += 1
... return " ".join("spam" for _ in xrange(self._count))
...
>>> x = Spam()
>>> print x.spam()
spam
>>> print x.spam()
spam spam
>>> print x.spam()
spam spam spam
It also gives you the ability to have two compleately separate
instances of the same state machine.
>>> y = Spam()
>>> print y.spam()
spam
>>> print y.spam()
spam spam
>>> print y.spam()
spam spam spam
>>>
You can use it like a function if you need to for convenience or
backwards compatibility.:
>>> spam = Spam().spam
>>> print spam()
spam
>>> print spam()
spam spam
>>> print spam()
spam spam spam
Or:
>>> class Spam(object):
... def __init__(self):
... self._count = 0
...
... def spam(self):
... self._count += 1
... return " ".join("spam" for _ in xrange(self._count))
...
... __call__ = spam
...
>>> spam = Spam()
>>> print spam()
spam
>>> print spam()
spam spam
>>> print spam()
spam spam spam
Matt
J. Cliff Dyer
Preserving state is what *objects* are for. Even the builtins have
state to be preserved (list.__len__, func.func_code, for example).
Classes are for creating custom objects.
> >>> class Spam(object):
> ... def __init__(self):
> ... self._count = 0
> ... def spam(self):
> ... self._count += 1
> ... return " ".join("spam" for _ in xrange(self._count))
> ...
Oh of course. This is a much cleaner way to return the response than
the one I used. (FYI, I used: `return ("spam " * self._count).rstrip()`
and I didn't like the rstrip even when I was doing it. Dunno why I
didn't think of it.)
> >>> class Spam(object):
> ... def __init__(self):
> ... self._count = 0
> ...
> ... def spam(self):
> ... self._count += 1
> ... return " ".join("spam" for _ in xrange(self._count))
> ...
> ... __call__ = spam
> ...
Interesting. I hadn't thought of making __call__ a synonym for an
existing method. I think I like that, but I'm not quite sure. There's
something that nags at me about having two ways to do the same thing,
but I like giving the method a more descriptive name than __call__.
Cheers,
Cliff
Steve Holden
> One thing I miss as I move from REALbasic to Python is the ability to
> have static storage within a method -- i.e. storage that is persistent
> between calls, but not visible outside the method. I frequently use
> this for such things as caching, or for keeping track of how many
> objects a factory function has created, and so on.
>
such information is either class variables or instance variables.
> Today it occurred to me to use a mutable object as the default value of
> a parameter. A simple example:
>
> def spam(_count=[0]):
> _count[0] += 1
> return "spam " * _count[0]
>
>>>> spam()
> 'spam '
>>>> spam()
> 'spam spam '
>
> This appears to work fine, but it feels a little unclean, having stuff
> in the method signature that is only meant for internal use. Naming the
> parameter with an underscore "_count" makes me feel a little better
> about it. But then, adding something to the module namespace just for
> use by one function seems unclean too.
>
It's a bad smell.
> What are your opinions on this idiom? Is there another solution people
> generally prefer?
>
> Ooh, for a change I had another thought BEFORE hitting Send rather than
> after. Here's another trick:
>
> def spam2():
> if not hasattr(spam2,'count'):spam2.count=0
> spam2.count += 1
> return "spam2 " * spam2.count
>
> This doesn't expose any uncleanliness outside the function at all. The
> drawback is that the name of the function has to appear several times
> within itself, so if I rename the function, I have to remember to change
> those references too. But then, if I renamed a function, I'd have to
> change all the callers anyway. So maybe this is better. What do y'all
> think?
>
I think you'd be much better off creating an instance of a class and
using that.
regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/
J. Cliff Dyer
On Thu, 2008-11-13 at 11:19 -0600, Chris Mellon wrote:
> On Thu, Nov 13, 2008 at 11:16 AM, Joe Strout <j...@strout.net> wrote:
> > One thing I miss as I move from REALbasic to Python is the ability to have
> > static storage within a method -- i.e. storage that is persistent between
> > calls, but not visible outside the method. I frequently use this for such
> > things as caching, or for keeping track of how many objects a factory
> > function has created, and so on.
> >
> > parameter. A simple example:
> >
> > def spam(_count=[0]):
> > _count[0] += 1
> > return "spam " * _count[0]
> >
> >>>> spam()
> > 'spam '
> >>>> spam()
> > 'spam spam '
> >
> > This appears to work fine, but it feels a little unclean, having stuff in
> > the method signature that is only meant for internal use. Naming the
> > parameter with an underscore "_count" makes me feel a little better about
> > it. But then, adding something to the module namespace just for use by one
> > function seems unclean too.
> >
> > generally prefer?
> >
> > Ooh, for a change I had another thought BEFORE hitting Send rather than
> > after. Here's another trick:
> >
> > def spam2():
> > if not hasattr(spam2,'count'):spam2.count=0
> > spam2.count += 1
> > return "spam2 " * spam2.count
> >
> > This doesn't expose any uncleanliness outside the function at all. The
> > drawback is that the name of the function has to appear several times within
> > itself, so if I rename the function, I have to remember to change those
> > references too. But then, if I renamed a function, I'd have to change all
> > the callers anyway. So maybe this is better. What do y'all think?
> >
>
> encapsulating state and behavior. Use an object.
He is using an object. Specifically, he's using a function object.
Though perhaps you meant put it into a class.
Here are a few essays into the matter
>>> def foo():
... foo._count += 1
... return ("spam " * foo.count).rstrip()
...
>>> foo._count=0
>>> foo()
'spam'
>>> foo()
'spam spam'
>>>
Simple and straightforward, and _count is still encapsulated in the
function, but it's kind of ugly, because when the function has been
defined, it is not functional.
Attempt #2 -- put it in a decorator
>>> def add_counter(f):
... f._count = 0
... return f
...
>>> @add_counter
... def foo():
... foo._count += 1
... return ("spam " * foo._count).rstrip()
>>> foo()
'spam'
>>> foo()
'spam spam'
>>>
Now it's complete as soon as the function is defined, but the decorator
is tightly bound to the function, in its use of _count. I find that
kind of ugly. This is the first decorator I've written that doesn't
define a function inside itself.
Try three. Let's put it in a class:
>>> class Foo(object):
... def __init__(self, counter_start=0):
... self._count = counter_start
... def __call__(self):
... self._count += 1
... return ("spam " * self._count).rstrip()
...
>>> foo = Foo()
>>> foo()
'spam'
>>> foo()
'spam spam'
>>>
Essentially, this has the same behavior as the other two. But it looks
cleaner, and you don't have to worry about coupling separate functions,
or refering to a function by name within itself (because you have self
to work with). But the "object" semantics are essentially the same, and
state is just as legitimately preserved on a function object as a Foo
object.
Cheers,
Cliff
ru...@yahoo.com
Not exclusively, generators also preserve state.
def _spam():
count = 1
while 1:
yield "spam " * count
count += 1
spam = _spam.next()
Jean-Paul Calderone
>On Nov 13, 11:32 am, "J. Cliff Dyer" <j...@sdf.lonestar.org> wrote:
>> On Thu, 2008-11-13 at 09:38 -0800, Matimus wrote:
>> > Preserving state is what classes are for.
>>
>> Preserving state is what *objects* are for.
>
>Not exclusively, generators also preserve state.
>
>def _spam():
> count = 1
> while 1:
> yield "spam " * count
> count += 1
>spam = _spam.next()
>
Surprise - generators are objects.
Jean-Paul
Alan Baljeu
Alan Baljeu
__________________________________________________________________
Instant Messaging, free SMS, sharing photos and more... Try the new Yahoo! Canada Messenger at http://ca.beta.messenger.yahoo.com/
ru...@yahoo.com
No surprise. Everything in Python is an object. I was talking at the
language level, not the implementation level.
Chris Rebert
> When I call unittest.main(), it invokes sys.exit(). I would like to run tests without exiting. How can I?
There's probably a better way that stops it from trying to exit in the
first place, but here's a quick kludge:
try:
unittest.main()
except SystemExit:
pass
sys.exit() does its job by noting the return code and then raising the
SystemExit exception, which you are free to catch.
Cheers,
Chris
--
Follow the path of the Iguana...
http://rebertia.com
>
>
> Alan Baljeu
>
>
> __________________________________________________________________
> Instant Messaging, free SMS, sharing photos and more... Try the new Yahoo! Canada Messenger at http://ca.beta.messenger.yahoo.com/
Steve Holden
> Joe Strout wrote:
>> One thing I miss as I move from REALbasic to Python is the ability to
>> have static storage within a method -- i.e. storage that is persistent
>> between calls, but not visible outside the method. I frequently use
>> this for such things as caching, or for keeping track of how many
>> objects a factory function has created, and so on.
>>
> I think you'd be much better off creating an instance of a class and
> using that.
>
whatever state you need in the function's local namespace.
Aaron Brady
Worse yet, if you define a duplicate object at the same scope with the
same name later, it breaks all your references within the function to
itself.
One way around it, which I like the idea of but I'll be honest, I've
never used, is getting a function a 'self' parameter. You could make
it a dictionary or a blank container object, or just the function
itself.
@self_param
def spam( self ):
self._count[0] += 1 #<--- how to initialize?
return "spam " * self._count[0]
Only problem is, how do you initialize _count?
Perhaps 'self_param' can take some initializers, and just initialize
them off of **kwargs in the construction.
@self_param( _count= [] )
def spam( self ):
self._count[0] += 1
return "spam " * self._count[0]
Looks really pretty (imo), but untested.
Paul McGuire
>
> - Show quoted text -
Initialization does not have to be in the body of the method.
>>> def spam():
... spam._count[0] += 1 #<--- how to initialize? see below
... return "spam " * spam._count[0]
...
>>> spam._count = [2] # just initialize it, and not necessarily to 0
>>> spam()
'spam spam spam '
>>> spam()
'spam spam spam spam '
>>> spam()
'spam spam spam spam spam '
>>>
-- Paul
Paul Boddie
> One thing I miss as I move from REALbasic to Python is the ability to
> have static storage within a method -- i.e. storage that is persistent
> between calls, but not visible outside the method. I frequently use
> this for such things as caching, or for keeping track of how many
> objects a factory function has created, and so on.
Why not use a module global? It isn't hidden, but it is quite a clean
approach. Modifying your example...
spam_count = 0
def spam():
global spam_count
spam_count += 1
return "spam " * spam_count
[...]
> This doesn't expose any uncleanliness outside the function at all.
I wouldn't be too worried about that. Although namespaces can become
somewhat crowded with all these extra names, there can be benefits in
exposing such names, too, but if you find this too annoying, it could
be advisable to collect these entries and put them in another
namespace, maybe a class or a module.
Paul
Aaron Brady
That is actually susceptible to a subtle kind of bug:
>>> def spam( ):
... spam._count[0] += 1
... return "spam " * spam._count[0]
...
>>> spam._count=[2]
>>> spam()
'spam spam spam '
>>> f= spam
>>> f()
'spam spam spam spam '
>>> spam= 'spam and eggs'
>>> f()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in spam
AttributeError: 'str' object has no attribute '_count'
It would be worse if you assigned 'spam' to another function! Of
course one option is 'just don't do that', which is alright. Adding
the self parameter is just a second option, if you need the function
to change names. Though the decorator is a nice place for the
initialization.
Ben Finney
> One thing I miss as I move from REALbasic to Python is the ability
> to have static storage within a method -- i.e. storage that is
> persistent between calls
This is precisely what classes are for: allowing functionality and
state to exist in a single object.
> but not visible outside the method.
Bind the state to a name with a single leading underscore (‘_foo’),
which is the convention for “not part of the public interface”.
Don't look for hard access restrictions, though, because they don't
really exist in Python.
--
\ “I hope that after I die, people will say of me: ‘That guy sure |
`\ owed me a lot of money’.” —Jack Handey |
_o__) |
Ben Finney
Ben Finney
> When I call unittest.main(), it invokes sys.exit(). I would like to
> run tests without exiting. How can I?
Use a ‘TestRunner’ instance, instead of ‘main()’.
--
\ “If you go flying back through time and you see somebody else |
`\ flying forward into the future, it's probably best to avoid eye |
_o__) contact.” —Jack Handey |
Ben Finney
Steve Holden
> Joe Strout <j...@strout.net> writes:
>
>> One thing I miss as I move from REALbasic to Python is the ability
>> to have static storage within a method -- i.e. storage that is
>> persistent between calls
>
> This is precisely what classes are for: allowing functionality and
> state to exist in a single object.
>
>> but not visible outside the method.
>
> Bind the state to a name with a single leading underscore (‘_foo’),
> which is the convention for “not part of the public interface”.
> Don't look for hard access restrictions, though, because they don't
> really exist in Python.
>
Joe Strout
> Static storage is a way of preserving state. Objects are a way of
> encapsulating state and behavior. Use an object.
Argh. I've been back in the Python community for about a month, and
I've been continually amazed at how every single "how do I do X" or
"what do you think of this method of doing X" question is answered by
people on high horses claiming "you shouldn't do X".
I know very well about state and objects. I'll be happy to whip out
my software engineering credentials and measure them against yours if
that's how you like to play. I understand very well when data should
be stored as instance data, and when it should be instead tucked away
as static data within a method. If you don't understand that, or are
happy without having the choice, and have no answer to the question I
was asking, then that's fine. I don't always have anything useful to
contribute when somebody asks a question either. But in that case, I
resist the urge to reply anyway.
Maybe we should define some standard tags people could add to the top
of their email: "Helpful-Answer" for helpful answers, and "Unhelpful-
Preaching" for the other kind. Then those of us not interested in one
sort or the other could set up an email filter.
Best,
- Joe
P.S. I'm sorry, Chris, I don't mean to rip your head off in
particular. You were just the straw that broke the camels back; there
have been plenty of others adding to the frustration. I'll try to
just ignore such responses in the future... though it is a little
disturbing to think how many newbies are probably driven away by this
sort of thing.
Joe Strout
> This is a pretty bizarre requirement, IMHO. The normal place to keep
> such information is either class variables or instance variables.
Holy cow, I thought it was just Chris, but there were half a dozen
similar responses after that.
I'm starting to see a pattern here... any time Python lacks a feature,
the Python community's party line is "You don't need that feature!"
I understand embracing the language rather than fighting against it,
but that can be taken too far -- if somebody expresses a need, and is
earnestly asking for input on a real programming problem, I'd think
the nice thing would be to explore the programming problem with them,
rather than arguing with them that they don't need what they claim
they need. Especially if they're not somebody who just stumbled out
of college, but has been writing software professionally for decades,
and may have a better idea than anybody else what their needs are.
Hypothetically speaking, is it possible that there could be any
language feature Python doesn't have, which might be useful to anyone
in any situation? If so, how would you recognize such a rare beast?
Thanks,
- Joe
Ben Finney
> On Nov 13, 2008, at 10:19 AM, Chris Mellon wrote:
>
> > Static storage is a way of preserving state. Objects are a way of
> > encapsulating state and behavior. Use an object.
>
> Argh. I've been back in the Python community for about a month, and
> I've been continually amazed at how every single "how do I do X" or
> "what do you think of this method of doing X" question is answered
> by people on high horses claiming "you shouldn't do X".
You seem to be reading the responses differently from me. I don't see
that the above response fits the pattern you describe at all.
Instead, it looks like you're falling foul of one of the classic
mistakes in the “How to ask questions the smart way” document:
you've got a goal, but you're assuming that you need to use a specific
tool to get there. Instead, you're being shown that your assumption is
false: there's a better tool available to achieve your goal.
--
\ “Why, I'd horse-whip you if I had a horse.” —Groucho Marx |
`\ |
_o__) |
Ben Finney
Joe Strout
> Here are a few essays into the matter
>
>>>> def foo():
> ... foo._count += 1
> ... return ("spam " * foo.count).rstrip()
>
> Simple and straightforward, and _count is still encapsulated in the
> function, but it's kind of ugly, because when the function has been
> defined, it is not functional.
Right, though the hasattr() check I put in my version of this approach
takes care of that.
> Attempt #2 -- put it in a decorator
>
>>>> def add_counter(f):
> ... f._count = 0
> ... return f
> ...
>>>> @add_counter
> ... def foo():
> ... foo._count += 1
> ... return ("spam " * foo._count).rstrip()
Now THAT is something that hadn't occurred to me! I haven't really
studied decorators yet (and I'm pretty sure Python didn't have them
when I was last into it a decade ago). This is an interesting solution.
> Now it's complete as soon as the function is defined, but the
> decorator
> is tightly bound to the function, in its use of _count. I find that
> kind of ugly.
True. And we have two functions where we really only wanted one --
that's probably worse than just having a function plus a module-level
variable. Unless, perhaps, we could make a generic "has_cache"
decorator that can be reused in any method that needs some static
storage.
> Try three. Let's put it in a class:
>
>>>> class Foo(object):
> ... def __init__(self, counter_start=0):
> ... self._count = counter_start
> ... def __call__(self):
> ... self._count += 1
> ... return ("spam " * self._count).rstrip()
At first I thought this suggestion was just to add an instance
variable to whatever class my method was already in, which violates
encapsulation at the level I'm after. But now I see this is is
something more subtle: you've made a callable object. We'd presumably
combine this with a Singleton pattern to get the desired semantics.
This is clever, but an awful lot of code for such a simple need.
> Essentially, this has the same behavior as the other two. But it
> looks
> cleaner, and you don't have to worry about coupling separate
> functions,
> or refering to a function by name within itself (because you have self
> to work with).
True. Pity there isn't a way for a function to get a reference to
itself except by name. Still, when you rename a method, you're going
to have to update all the callers anyway -- updating a couple of extra
references within the method is not that much extra effort.
All of these approaches may turn out to be more trouble than they're
worth; a module variable isn't THAT ugly. But it's nice to have some
options.
At any rate, you (and rurpy) have taken the time to consider the
question I was actually asking, instead of arguing with me whether I
should be asking it. I really appreciate that. Your suggestions are
helpful and have taught me some new Python tricks. Thanks very much!
Best,
- Joe
alex23
> Argh. I've been back in the Python community for about a month, and
> I've been continually amazed at how every single "how do I do X" or
> "what do you think of this method of doing X" question is answered by
> people on high horses claiming "you shouldn't do X".
If someone asked you how to hammer together a cabinet with a saw,
wouldn't -you- respond with "don't use a saw"?
Suggesting the right tool for the job is hardly getting on a high
horse...
Luis Zarrabeitia
Quoting Joe Strout <j...@strout.net>:
> On Nov 13, 2008, at 10:19 AM, Chris Mellon wrote:
>
> > Static storage is a way of preserving state. Objects are a way of
> > encapsulating state and behavior. Use an object.
>
> Argh. I've been back in the Python community for about a month, and
> I've been continually amazed at how every single "how do I do X" or
> "what do you think of this method of doing X" question is answered by
> people on high horses claiming "you shouldn't do X".
>
> I know very well about state and objects. I'll be happy to whip out
> my software engineering credentials and measure them against yours if
> that's how you like to play.
I'm sure your credentials are bigger than mine. But he is right. A lot of
languages have ditched the "concept" of a static variable on a method (how do
you parse that sentence, btw?) in favour of using encapsulation. Of the top of
my head, there is Java, C# and Python, and I think even PHP and Perl. They call
them "private variables", including the
name-mangled-publicly-accessible-__python's variables.
I only know one family of languages that support it: C/C++ (LISP, maybe? I don't
remember - my list is very sketchy), but here comes into play the fact that your
credentials are bigger. But that may be the reason why you refuse to follow yet
another model.
Python doesn't have the concept of 'static variables', as in "a variable that
remembers its value between function calls". Instance attributes (or class
atributes, or metaclass attributes - your choice, depending on the problem)
exist for that.
So, if you want to simulate your C++ static variables in python, you ha ve a few
choices (more than in other languages, btw):
* Global variable. Ouch. We really don't want you going there :D
* Instance variable. Maybe a __private instance variable, to avoid polluting
your descendant's namespaces.
* Class/Metaclass variables (if you need them - that's rare).
* Closures
* Default arguments (too fragile for my taste)
* Instance variables _of the function object_.
For the last three: (untested code follows)
* With closures:
===
def myfunc():
pseudo_static_list = []
def real_function(args):
pseudo_static_list.append(args)
print pseudo_static_list
return real_function
myfunc = myfunc()
===
[or, until python removes the apply function]
===
@apply
def myfunc():
# same definition as before
return real_function
===
Caveat: you cannot assign, as in python 2.5 at least, to the closure variable -
doing so would create a local variable instead.
* With default argument:
===
def myfunc(normal_args, _hidden_arg=[]):
_hidden_arg.append(normal_args)
print _hidden arg
===
(You can't shouldn't to _hidden_arg either, and you risk someone invoking the
function with that argument)
* With function's instance members:
===
def myfunc(args):
myfunc.counter=1 # well, you should use some logic to see if it's not
# initialized
myfunc.counter+=1
print myfunc.counter
===
Instance attributes beats them all, though.
> I understand very well when data should
> be stored as instance data, and when it should be instead tucked away
> as static data within a method.
OT: Please enlighthen me. I didn't grew with C, and even when I saw C++,
instance variables made a lot more sense to me that 'static' variables.
> If you don't understand that, or are
> happy without having the choice, and have no answer to the question I
> was asking, then that's fine.
I believe he had an answer... You didn't like it, though. I hope mine was more
palatable to you.
--
Luis Zarrabeitia
Facultad de Matemática y Computación, UH
http://profesores.matcom.uh.cu/~kyrie
George Sakkis
> Steve wrote:
> > This is a pretty bizarre requirement, IMHO. The normal place to keep
> > such information is either class variables or instance variables.
>
> Holy cow, I thought it was just Chris, but there were half a dozen
> similar responses after that.
>
> I'm starting to see a pattern here... any time Python lacks a feature,
> the Python community's party line is "You don't need that feature!"
>
> I understand embracing the language rather than fighting against it,
> but that can be taken too far -- if somebody expresses a need, and is
> earnestly asking for input on a real programming problem, I'd think
> the nice thing would be to explore the programming problem with them,
> rather than arguing with them that they don't need what they claim
> they need.
The burden of proof is on you to show that none of the several ways to
solve your problem in Python is good enough. So far your argument is
"I really miss that I can't do it exactly like in my pet-language".
> Hypothetically speaking, is it possible that there could be any
> language feature Python doesn't have, which might be useful to anyone
> in any situation?
Sure; true multithreading, macros, non-crippled lambda, optional
static typing are some reasonable features people miss in Python. The
topic of this thread just isn't one of them.
George
Arnaud Delobelle
> One way around it, which I like the idea of but I'll be honest, I've
> never used, is getting a function a 'self' parameter. You could make
> it a dictionary or a blank container object, or just the function
> itself.
>
> @self_param
> def spam( self ):
> self._count[0] += 1 #<--- how to initialize?
> return "spam " * self._count[0]
>
> Only problem is, how do you initialize _count?
>
> Perhaps 'self_param' can take some initializers, and just initialize
> them off of **kwargs in the construction.
>
> @self_param( _count= [] )
> def spam( self ):
> self._count[0] += 1
> return "spam " * self._count[0]
>
> Looks really pretty (imo), but untested.
Rummaging through my ~/python/junk/ I found the almost exact same:
class NS(object):
def __init__(self, dict):
self.__dict__.update(dict)
def static(**vars):
ns = NS(vars)
def deco(f):
return lambda *args, **kwargs: f(ns, *args, **kwargs)
return deco
@static(ncalls=0, history=[])
def foo(ns, x):
ns.ncalls += 1
ns.history.append(x)
print "Number of calls: %s\nHistory:%s" % (ns.ncalls, ns.history)
>>> foo(3)
Number of calls: 1
History:[3]
>>> foo(5)
Number of calls: 2
History:[3, 5]
>>> foo('spam')
Number of calls: 3
History:[3, 5, 'spam']
>>>
--
Arnaud
Steven D'Aprano
> One thing I miss as I move from REALbasic to Python is the ability to
> have static storage within a method -- i.e. storage that is persistent
> between calls, but not visible outside the method. I frequently use
> this for such things as caching, or for keeping track of how many
> objects a factory function has created, and so on.
>
> Today it occurred to me to use a mutable object as the default value of
> a parameter. A simple example:
>
> def spam(_count=[0]):
> _count[0] += 1
> return "spam " * _count[0]
This is a common trick, often used for things like caching. One major
advantage is that you are exposing the cache as an *optional* part of the
interface, which makes testing easier. For example, instead of a test
that looks something like this:
cache = get_access_to_secret_cache() # somehow
modify(cache)
result = function(arg)
restore(cache)
assert something_about(result)
you can simple do this:
result = function(arg, _cache=mycache)
assert something_about(result)
Periodically people complain that Python's mutable default argument
behaviour is a problem, and ask for it to be removed. I agree that it is
a Gotcha that trips up newbies, but it is far to useful to give up, and
simple caching is one such reason.
> Ooh, for a change I had another thought BEFORE hitting Send rather than
> after. Here's another trick:
>
> def spam2():
> if not hasattr(spam2,'count'):spam2.count=0 spam2.count += 1
> return "spam2 " * spam2.count
>
> This doesn't expose any uncleanliness outside the function at all. The
> drawback is that the name of the function has to appear several times
> within itself, so if I rename the function, I have to remember to change
> those references too. But then, if I renamed a function, I'd have to
> change all the callers anyway. So maybe this is better. What do y'all
> think?
I've used this myself, but to me it feels more icky than the semi-private
argument trick above.
--
Steven
Steven D'Aprano
Yes, technically, but the value of count is not stored as an instance
attribute: you can't access spam.count. Instead, it is stored in the
internals of the generator.
Generators are a special case of closures:
http://en.wikipedia.org/wiki/Closure_(computer_science)
Note that in some languages, closures are used to implement objects,
rather than the other way around.
--
Steven
Steven D'Aprano
> Joe Strout wrote:
>> One thing I miss as I move from REALbasic to Python is the ability to
>> have static storage within a method -- i.e. storage that is persistent
>> between calls, but not visible outside the method. I frequently use
>> this for such things as caching, or for keeping track of how many
>> objects a factory function has created, and so on.
>>
> This is a pretty bizarre requirement, IMHO. The normal place to keep
> such information is either class variables or instance variables.
*jaw drops*
You know, there were one or two programs written before the invention of
object-oriented programming techniques. Some of them required persistent
storage between function calls, and global variables have obvious
disadvantages. That's why the C language uses static variables.
http://c.ittoolbox.com/documents/popular-q-and-a/difference-between-
static-global-variable-1596
http://en.wikipedia.org/wiki/Static_variable
--
Steven
Joe Strout
> A lot of languages have ditched the "concept" of a static variable
> on a method (how do
> you parse that sentence, btw?) in favour of using encapsulation.
A static variable IS encapsulation. Encapsulation happens at many
levels: module, class, instance, and (in languages that support it)
method. A static local variable is simply the finest level of
encapsulation. (Well, actually you could go one finer in some
languages and have block-level static scope.)
> * With closures:
>
> ===
> def myfunc():
> pseudo_static_list = []
> def real_function(args):
> pseudo_static_list.append(args)
> print pseudo_static_list
> return real_function
> myfunc = myfunc()
That's interesting -- I'll need to study it further before I really
understand it, so thanks very much. (That's why I ask these
questions; I'm hoping there is some Python feature I haven't yet fully
grokked which applies to the problem at hand.)
> Caveat: you cannot assign, as in python 2.5 at least, to the closure
> variable -
> doing so would create a local variable instead.
Makes sense.
> * With default argument:
> ===
> def myfunc(normal_args, _hidden_arg=[]):
> _hidden_arg.append(normal_args)
> print _hidden arg
> ===
> (You can't shouldn't to _hidden_arg either, and you risk someone
> invoking the
> function with that argument)
Right, that was my first idea. Though it's in the "gentleman's
agreement" spirit of Python.
> * With function's instance members:
> ===
> def myfunc(args):
> myfunc.counter=1 # well, you should use some logic to see if it's
> not
> # initialized
> myfunc.counter+=1
> print myfunc.counter
That was my second idea.
> Instance attributes beats them all, though.
No, just adding an instance attribute to the class the method is
already on does not give you the same semantics at all, unless the
class happens to be a singleton. Otherwise, each instance would get
its own cache (or count or whatever -- there are several different use
cases I've seen for this), rather than a shared one. Sometimes that's
acceptable, but often it's not.
As noted before, the obvious solution in this case is to use a module-
or class-level variable, prefixed with an underscore. That's not
horrible, but I wanted to explore possible alternatives.
>> I understand very well when data should be stored as instance data,
>> and when it
>> should be instead tucked away as static data within a method.
>
> OT: Please enlighthen me. I didn't grew with C, and even when I saw C
> ++,
> instance variables made a lot more sense to me that 'static'
> variables.
Maybe a couple examples will help:
1. You have an instance method whose job it is to map something to
something else (e.g. zip codes to city names), say by looking it up in
a big file or database. Per the "lazy initialization" strategy, you
don't want to look stuff up before it's needed, but you find that
looking it up every time causes big performance problems (because
whatever zip codes are in use at the time get used a lot, and the file
or DB access is slow). Moreover, the object is lightweight and you're
going to have a lot of them coming and going, so just caching the
result on the instance won't help much. You instead need a cache that
sticks around for the life of the app, like a class or module
attribute. But because the existence of that cache is an
implementation detail internal to this method, and shouldn't be
anybody else's business, you want to tuck it away inside that method.
2. You have an object which, among other things, can create other
objects. Part of its job is to assign the newly created objects ID
numbers which must be unique across the entire application (maybe
they're wx identifiers or used for serialization or whatever). So it
needs to keep track of what the next available ID is. But again, if
you consider that to be implementation detail internal to the factory
method, then it should be encapsulated inside that method. And
storing it on the factory-object instance won't do, because you need
the IDs to be unique even across instances.
>> If you don't understand that, or are happy without having the
>> choice, and have no answer to the question I
>> was asking, then that's fine.
>
> I believe he had an answer... You didn't like it, though.
If he had an answer to the question I was asking, he didn't share it.
I asked "what's the best way to do this?" Several responses ignored
that question, and instead said, "Don't do that."
If you ask me how to cook an omelet, I might say "don't eat omelets;
pancakes are much tastier, you should eat them instead." But that
doesn't tell you how to cook an omelet, does it?
> I hope mine was more palatable to you.
Yours was tasty indeed, thanks very much!
Cheers,
- Joe
Steven D'Aprano
> On 13 Nov, 18:16, Joe Strout <j...@strout.net> wrote:
>> One thing I miss as I move from REALbasic to Python is the ability to
>> have static storage within a method -- i.e. storage that is persistent
>> between calls, but not visible outside the method. I frequently use
>> this for such things as caching, or for keeping track of how many
>> objects a factory function has created, and so on.
>
> Why not use a module global? It isn't hidden, but it is quite a clean
> approach. Modifying your example...
For some definition of "clean".
http://archive.eiffel.com/doc/manuals/technology/bmarticles/joop/
globals.html
http://weblogs.asp.net/wallen/archive/2003/05/08/6750.aspx
Python globals aren't quite as bad, because they are merely global to a
module and not global to your entire application. Nevertheless, your
example is one of the *worst* usages for globals. See below.
> spam_count = 0
>
> def spam():
> global spam_count
> spam_count += 1
> return "spam " * spam_count
>
> [...]
>
>> This doesn't expose any uncleanliness outside the function at all.
Nonsense. Any other function in the code can write to spam_count and
cause all sorts of havoc. Any function that needs to temporarily modify
spam_count needs to be careful to wrap it with a save and a restore:
n = spam_count
spam_count = 47
s = spam()
spam_count = n
This is precisely one of the anti-patterns that global variables
encourage, and one of the reasons why globals are rightly considered
harmful. Don't Do This.
--
Steven
Steven D'Aprano
> Instead, it looks like you're falling foul of one of the classic
> mistakes in the “How to ask questions the smart way” document: you've
> got a goal, but you're assuming that you need to use a specific tool to
> get there. Instead, you're being shown that your assumption is false:
> there's a better tool available to achieve your goal.
For the sake of the argument I will accept your characterization of Joe's
question. Even given that, you have still made an error: Joe hasn't been
shown that his assumption is wrong, he's merely had a bunch of people
declare, without evidence or even reason, that he is wrong. If anyone has
given any reason for avoiding the idiom that doesn't boil down to "I
don't like it!" (I need a smiley for pursed lips), I haven't read it.
I for one do not think he is wrong, and I have given reasons for
preferring the idiom:
def parrot(public_args, _cache={}) # or similar
for implementing caches and similar internal storage. It's quick, it's
easy, it's lightweight, and it exposes the cache to those who need it,
e.g. for testing.
If anyone has a *reason* why this idiom is harmful, please say so.
Repeating "That's not the way to do it!!! Use an object!!!" is not a
reason.
Without such a reason, then the decision whether or not to use a functor
(class with a __call__ method) in Python is a matter of personal taste.
--
Steven
Steven D'Aprano
> Pity there isn't a way for a function to get a reference to itself
> except by name. Still, when you rename a method, you're going to have
> to update all the callers anyway -- updating a couple of extra
> references within the method is not that much extra effort.
I have come across a situation where this was a problem.
I was writing a bunch of different versions of the classic factorial and
Fibonacci functions, some iterative, some recursive, some with caches,
and some without. What I wanted to do was do something like this (for
factorial):
def iterative_without_cache(n):
product = 1
for i in xrange(1, n+1):
product *= i
return product
iterative_with_cache = decorate_with_cache(iterative_without_cache)
which works. But it doesn't work for the recursive version:
def recursive_without_cache(n):
if n <= 1:
return 1
else:
return n*recursive_without_cache(n-1)
recursive_with_cache = decorate_with_cache(recursive_without_cache)
for obvious reasons. Solution: the copy-and-paste anti-pattern. Good
enough for test code, not for real work.
Now multiply the above by about a dozen slightly different recursive
versions of the Fibonacci function, and you will see why it was a problem.
--
Steven
Steven D'Aprano
If somebody asked me how to hammer together a cabinet with a regular
carpenter's hammer, and I answered "Don't do that, anything less than an
object-oriented Uber-Hammer is wrong", then they might be justified in
thinking that I was full of it, particularly if I was unable to explain
just why using a regular hammer was harmful.
--
Steven
Aaron Brady
cybersource.com.au> wrote:
> On Fri, 14 Nov 2008 13:35:02 +1100, Ben Finney wrote:
> > Instead, it looks like you're falling foul of one of the classic
> > mistakes in the “How to ask questions the smart way” document: you've
> > got a goal, but you're assuming that you need to use a specific tool to
> > get there. Instead, you're being shown that your assumption is false:
> > there's a better tool available to achieve your goal.
>
> For the sake of the argument I will accept your characterization of Joe's
> question. Even given that, you have still made an error: Joe hasn't been
> shown that his assumption is wrong, he's merely had a bunch of people
> declare, without evidence or even reason, that he is wrong. If anyone has
> given any reason for avoiding the idiom that doesn't boil down to "I
> don't like it!" (I need a smiley for pursed lips), I haven't read it.
>
> I for one do not think he is wrong, and I have given reasons for
> preferring the idiom:
>
> def parrot(public_args, _cache={}) # or similar
>
> for implementing caches and similar internal storage. It's quick, it's
> easy, it's lightweight, and it exposes the cache to those who need it,
> e.g. for testing.
Does it follow that variables should be exposed too to those who need
it, e.g. for testing?
> If anyone has a *reason* why this idiom is harmful, please say so.
Or forever hold his peace?
I've alluded to them before. The reasons are *NOT STRONG*. They are
mild, and do not (*NOT*) constitute a conclusive case. (Not.) The
idiom is not harmful, but here is what whatever case there is against
it is.
1. It's an argument list, not an argument-and-static-variable list.
2. Mistakes by the caller.
3. BIG gotcha.
4. Not obvious from the syntax. Much of Python is; this isn't, it's
arbitrarily assigned.
5. Arnaud's 'static' decorator nearly as convenient and much clearer.
6. Signature can get really long with a few arguments and a few
statics. Split into two statements, one being a decorator.
Even combined, these don't warrant the effort to remove the feature.
P.S. At least it's not globals! Ha ha ha.
Peter Otten
You can have Python do the copying for you:
def make_recursive(deco):
@deco
def recursive(n):
if n <= 1:
return 1
return n * recursive(n-1)
return recursive
def cached(f, _cache={}):
def g(*args):
try:
result = _cache[args]
except KeyError:
result = _cache[args] = f(*args)
return result
return g
r1 = make_recursive(lambda f: f)
r2 = make_recursive(cached)
Peter
Bruno Desthuilliers
> One thing I miss as I move from REALbasic to Python is the ability to
> have static storage within a method
s/method/function/
> -- i.e. storage that is persistent
> between calls, but not visible outside the method. I frequently use
> this for such things as caching, or for keeping track of how many
> objects a factory function has created, and so on.
>
> Today it occurred to me to use a mutable object as the default value of
> a parameter. A simple example:
>
> def spam(_count=[0]):
> _count[0] += 1
> return "spam " * _count[0]
>
> >>> spam()
> 'spam '
> >>> spam()
> 'spam spam '
>
> This appears to work fine, but it feels a little unclean, having stuff
> in the method signature that is only meant for internal use.
It's indeed a hack. But it's a very common one, and
> Naming the
> parameter with an underscore "_count"
is also a pretty idiomatic way to warn that it's implementation-only.
> makes me feel a little better
> about it.
> But then, adding something to the module namespace just for
> use by one function seems unclean too.
> What are your opinions on this idiom? Is there another solution people
> generally prefer?
If it's really in a *method*, you can always use a class (or instance,
depending on concrete use case) attribute. Else, if you really insist on
cleanliness, you can define your own callable:
class Spammer(object):
def __init__(self):
self._count = 0
def __call__(self):
self._count += 1
return "spam " * self._count
spam = Spammer()
But this might be a little overkill for most concrete use case.
NB : you'll also have to implement __get__ if you want Spammer instances
to be used as methods.
> Ooh, for a change I had another thought BEFORE hitting Send rather than
> after. Here's another trick:
>
> def spam2():
> if not hasattr(spam2,'count'):spam2.count=0
> spam2.count += 1
> return "spam2 " * spam2.count
>
> This doesn't expose any uncleanliness outside the function at all. The
> drawback is that the name of the function has to appear several times
> within itself, so if I rename the function, I have to remember to change
> those references too.
There's another drawback:
old_spam2 = spam2
def spam2():
print "yadda yadda", old_spam2()
Remember that Python functions are ordinary objects...
Luis Zarrabeitia
Quoting Joe Strout <j...@strout.net>:
> Hi Luis,
>
> > A lot of languages have ditched the "concept" of a static variable
> > on a method (how do
> > you parse that sentence, btw?) in favour of using encapsulation.
>
> A static variable IS encapsulation. Encapsulation happens at many
> levels: module, class, instance, and (in languages that support it)
> method. A static local variable is simply the finest level of
> encapsulation. (Well, actually you could go one finer in some
> languages and have block-level static scope.)
But, at least in C++, its really confusing. The 'static' word is reserved on C++
for more than one meaning, but I'll agree that it is a C++ problem and not a
problem with the concept. I wish I remembered how and if LISP did it, though.
Being LISP, I'd guess their implementation is also based on closures.
> > * With closures:
> >
> > ===
> > def myfunc():
> > pseudo_static_list = []
> > def real_function(args):
> > pseudo_static_list.append(args)
> > print pseudo_static_list
> > return real_function
> > myfunc = myfunc()
>
> That's interesting -- I'll need to study it further before I really
> understand it, so thanks very much. (That's why I ask these
> questions; I'm hoping there is some Python feature I haven't yet fully
> grokked which applies to the problem at hand.)
Well, I love closures :D. And decorators.
> No, just adding an instance attribute to the class the method is
> already on does not give you the same semantics at all, unless the
> class happens to be a singleton. Otherwise, each instance would get
> its own cache (or count or whatever -- there are several different use
> cases I've seen for this), rather than a shared one. Sometimes that's
> acceptable, but often it's not.
Not necesarily.
One of the things I dislike about 'static' is precisely that it would define the
variable's lifetime as 'forever, everywhere'. But as you say, that may not be
the only answer. You could need 'as long as _this function_ lives' (python gives
you closures for that), 'as long as this object lives' (instance attributes),
'as long as this object's class lives' (class attribute), 'forever' (a global,
or a class attribute of a fixed class, that would be like a global but with a
namespace).
Btw, you can yank a function out of an object and put it on another, if you need
your counters to survive that, use closures, if you need your counters to die in
that case, you'll have to stick with class or instance attributes.
> As noted before, the obvious solution in this case is to use a module-
> or class-level variable, prefixed with an underscore. That's not
> horrible, but I wanted to explore possible alternatives.
Fair enough.
> >> I understand very well when data should be stored as instance data,
> >> and when it
> >> should be instead tucked away as static data within a method.
> >
> > OT: Please enlighthen me. I didn't grew with C, and even when I saw C
> > ++,
> > instance variables made a lot more sense to me that 'static'
> > variables.
>
> Maybe a couple examples will help:
Thank you.
> > I hope mine was more palatable to you.
>
> Yours was tasty indeed, thanks very much!
/me smiles.
Bruno Desthuilliers
> On Nov 13, 9:16 am, Joe Strout <j...@strout.net> wrote:
(snip)
>> def spam2():
>> if not hasattr(spam2,'count'):spam2.count=0
>> spam2.count += 1
>> return "spam2 " * spam2.count
>
>
I beg to disagree. This solution stores "count" on *whatever* the name
"spam2" resolves at runtime.
(snip)
Chris Mellon
Honestly given his attitude I don't think there's any reason to give
him the benefit of the doubt. Python doesn't have static storage - it
uses objects for the same purpose. Since anyone with the credentials
he's mentioned at least 3 times not only knew the history of OO
programming and how it addresses the same problems that people used
static storage to address, but also could have read one friggin python
tutorial and gotten all the answers he needed, I'm pretty much forced
to decide that he's either a troll or lazy.
Especially since in his email he gave no reason why "just use a
fucking object" isn't a good solution to the problem, and he still
hasn't.
This isn't an UberSpecialTypeFactoryInstance answer, if you'll forgive
me using an answer from a future email. There's a simple, obvious,
straightforward solution to the need to maintain state between
function calls.The entire idea of objects in programming was developed
to fill *exactly this need*.
And finally! You will note that in his email he asked for peoples
opinions and suggestions on how they solve this problem. "I'd use an
object" is a totally reasonable answer to the question he asked. Joe
seems to be pissed off that we didn't answer the question he didn't
ask, and we didn't respect articificial constraints that he didn't
fully express, and apparently that we don't all love RealBasic, which
he talks about a lot. So fuck that.
Joe Strout
> Aaron Brady <casti...@gmail.com> writes:
>
>> One way around it, which I like the idea of but I'll be honest, I've
>> never used, is getting a function a 'self' parameter. You could make
>> it a dictionary or a blank container object, or just the function
>> itself.
>> ...
> Rummaging through my ~/python/junk/ I found the almost exact same:
>
> class NS(object):
> def __init__(self, dict):
> self.__dict__.update(dict)
>
> def static(**vars):
> ns = NS(vars)
> def deco(f):
> return lambda *args, **kwargs: f(ns, *args, **kwargs)
> return deco
>
> @static(ncalls=0, history=[])
> def foo(ns, x):
> ns.ncalls += 1
> ns.history.append(x)
> print "Number of calls: %s\nHistory:%s" % (ns.ncalls, ns.history)
Thanks, Arnaud (and Aaron), that's really clever. I was thinking this
morning that something like this might be possible: a decorator that
adds the static storage with some standard name. I really like how
you've set this so that the static data is initialized right in the
decorator; that makes the intent very clear and hard to screw up.
My only regret with this one is the need to add "ns" to the parameter
list. That makes it lok like part of the signature, when really (in
intent) it is not. If we have to add something to the parameter list,
we may as well do this:
def foo(x, _ns=NS(ncalls=0, history=[])):
...
and skip the decorator. Then at least it's clear that this parameter
isn't really intended for callers. On the other hand, I guess the
decorator actually changes the signature as seen by the calling code,
which is a good thing. Rearranging the parameter order in the
decorator a bit, we could make this so that the intent is clear to the
reader, as well as enforced by the interpreter.
Neat stuff, thank you!
Best,
- Joe
Joe Strout
>> def spam(_count=[0]):
>> _count[0] += 1
>> return "spam " * _count[0]
>
> This is a common trick, often used for things like caching. One major
> advantage is that you are exposing the cache as an *optional* part
> of the
> interface, which makes testing easier. For example, instead of a test
> that looks something like this:
>
> cache = get_access_to_secret_cache() # somehow
> modify(cache)
> result = function(arg)
> restore(cache)
> assert something_about(result)
>
> you can simple do this:
>
> result = function(arg, _cache=mycache)
> assert something_about(result)
That's a very good point. I'd been working under the assumption that
any outside mucking with the cache was a Bad Thing, but for testing,
it can be darned helpful. And this is a very safe form of mucking; it
doesn't actually affect the "real" cache at all, but just substitutes
a temporary one.
Thanks also for pointing out that this is a common trick -- after all
the attacks on the very idea last night, I was wondering if I were off
alone in the woods again.
>> def spam2():
>> if not hasattr(spam2,'count'):spam2.count=0 spam2.count += 1
>> return "spam2 " * spam2.count
>>
>> This doesn't expose any uncleanliness outside the function at all.
>> The
>> drawback is that the name of the function has to appear several times
>> within itself, so if I rename the function, I have to remember to
>> change
>> those references too. But then, if I renamed a function, I'd have to
>> change all the callers anyway. So maybe this is better. What do
>> y'all
>> think?
>
> I've used this myself, but to me it feels more icky than the semi-
> private
> argument trick above.
Thanks for the feedback.
Best,
- Joe
Arnaud Delobelle
Check the wraps() function in the functools module. I think I wrote
this decorator before it came into the standard library.
--
Arnaud
Aaron Brady
> def static(**vars):
> ns = NS(vars)
> def deco(f):
> return lambda *args, **kwargs: f(ns, *args, **kwargs)
> return deco
>
> @static(ncalls=0, history=[])
> def foo(ns, x):
> ns.ncalls += 1
> ns.history.append(x)
> print "Number of calls: %s\nHistory:%s" % (ns.ncalls, ns.history)
One could even add 'ns' as an attribute of 'f', so that the statics
were visible from the outside: 'foo.ns.ncalls'.
jhermann
> try:
> unittest.main()
> except SystemExit:
> pass
You most probably want this instead:
try:
unittest.main()
except SystemExit, exc:
# only exit if tests failed
if exc.code:
raise