__getattr__, __setattr__ and pickle
mwojc
My class with implemented __getattr__ and __setattr__ methods cannot be
pickled because of the Error:
======================================================================
ERROR: testPickle (__main__.TestDeffnet2WithBiases)
----------------------------------------------------------------------
Traceback (most recent call last):
File "deffnet.py", line 246, in testPickle
cPickle.dump(self.denet, file)
TypeError: 'NoneType' object is not callable
----------------------------------------------------------------------
Is there an obvious reason i don't know, which prevents pickling with those
methods (if i comment them out the pickling test passes)?
I'm using Python 2.4.4 on Gentoo Linux. The mentioned methods goes as
follows:
def __setattr__(self, name, value):
if name == 'weights':
j = 0
for net in self.nets:
w1 = self.wmarks[j]
w2 = self.wmarks[j+1]
net.weights = value[w1:w2]
j += 1
else:
self.__dict__[name] = value
def __getattr__(self, name):
if name == 'weights':
j = 0
for net in self.nets:
w1 = self.wmarks[j]
w2 = self.wmarks[j+1]
self._weights[w1:w2] = net.weights
j += 1
return self._weights
Greetings,
--
Marek
Bruno Desthuilliers
__getattr__ should raise an AttributeError when name != 'weight' instead
of (implicitely) returning None. pickle looks for a couple special
method in your object[1], and it looks like it doesn't bother to check
if what it found was really callable.
[1] cf http://docs.python.org/lib/pickle-inst.html
FWIW, you'd be better using a property instead of __getattr__ /
__setattr__ if possible. And while we're at it, you dont need to
manually take care of your index in the for loop - you can use
enumerate(iterable) instead:
for j, net in enumerate(self.nets):
w1 = self.wmarks[j]
w2 = self.wmarks[j+1]
self._weights[w1:w2] = net.weights
return self._weights
HTH
> Greetings,
mwojc
Yes, raising AttributeError helped. Thanks!
>
> FWIW, you'd be better using a property instead of __getattr__ /
> __setattr__ if possible.
You're probably right again, in this case it's better to use property.
> And while we're at it, you dont need to
> manually take care of your index in the for loop - you can use
> enumerate(iterable) instead:
>
> for j, net in enumerate(self.nets):
> w1 = self.wmarks[j]
> w2 = self.wmarks[j+1]
> self._weights[w1:w2] = net.weights
> return self._weights
>
Sometimes i use manual handling of index because i'm convinced that
enumeration is a bit slower than this. But i'm not really sure about it...
Thanks again.
Greetings,
--
Marek
Bruno Desthuilliers
> Bruno Desthuilliers wrote:
(snip)
>> FWIW, you'd be better using a property instead of __getattr__ /
>> __setattr__ if possible.
>
> You're probably right again, in this case it's better to use property.
Since you seem to have concerns wrt/ execution time, properties might be
a little bit faster than __getattr__/__setattr__ hooks - cf below...
>> And while we're at it, you dont need to
>> manually take care of your index in the for loop - you can use
>> enumerate(iterable) instead:
>>
>> for j, net in enumerate(self.nets):
>> w1 = self.wmarks[j]
>> w2 = self.wmarks[j+1]
>> self._weights[w1:w2] = net.weights
>> return self._weights
>>
> Sometimes i use manual handling of index because i'm convinced that
> enumeration is a bit slower than this. But i'm not really sure about it...
It's easy to try out:
bruno@bibi ~ $ python
Python 2.5.1 (r251:54863, Apr 6 2008, 17:20:35)
[GCC 4.1.2 (Gentoo 4.1.2 p1.0.2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def manual(cnt):
... j = 0
... for i in xrange(cnt):
... x = i
... j += 1
...
>>> def auto(cnt):
... for j, i in enumerate(xrange(cnt)):
... x = i
...
>>> from timeit import Timer
>>> tm = Timer("manual(10000)", "from __main__ import manual")
>>> ta = Timer("auto(10000)", "from __main__ import auto")
>>> tm.timeit(1000)
3.3354489803314209
>>> tm.timeit(1000)
3.3376359939575195
>>> tm.timeit(1000)
3.3400180339813232
>>> ta.timeit(1000)
2.8350770473480225
>>> ta.timeit(1000)
2.8400650024414062
>>> ta.timeit(1000)
2.8361449241638184
>>>
Looks like enum is a bit faster by a mostly constant factor here.
And while we're at it:
>>> class Prop(object):
... @apply
... def prop():
... def fget(self): return self._prop
... def fset(self, val): self._prop = val
... return property(**locals())
... def __init__(self, val): self.prop=val
...
>>> class Hook(object):
... def __getattr__(self, name):
... if name == 'prop':
... return self._prop
... raise AttributeError("yadda")
... def __setattr__(self, name, val):
... if name == 'prop':
... self.__dict__['_prop'] = val
... else:
... # XXX : INCORRECT IMPLEMENTATION, DONT DO THIS !
... self.__dict__[name] = val
... # correct implementation:
... # super(Hook, self).__setattr__(name, value)
... def __init__(self, val): self.prop=val
...
>>> def testprop(cnt):
... p = Prop('test')
... for i in xrange(cnt):
... p.prop = i
... x = p.prop
...
>>> def testhook(cnt):
... h = Hook('test')
... for i in xrange(cnt):
... h.prop = i
... x = h.prop
...
>>> tp = Timer("testprop(1000)", "from __main__ import testprop")
>>> th = Timer("testhook(1000)", "from __main__ import testhook")
>>> tp.timeit(1000)
3.0640909671783447
>>> tp.timeit(1000)
3.0650019645690918
>>> th.timeit(1000)
7.0889511108398438
>>> th.timeit(1000)
7.0815410614013672
Looks like properties are significatively faster than the
__getattr__/__setattr__ hook too... Which is probably explained by the
following facts:
Looking for binding descriptors (like read/write properties) is the
first very stage of the attribute resolution algorithm (since they must
be looked up before the instance's __dict__ to avoid setting an instance
attribute which would then shadow the property).
OTHO, __getattr__ is always looked for last - which means it takes
longer to resolve __getattr__ than to resolve a read access to a
read/write property.
wrt/ __setattr__ - which is always called for attribute assignement,
whatever -, overloading it (instead of relying on the optimized builtin
implementation) is not only tricky (BTW, your implementation is broken -
try to add a read/write property to your class and enjoy...[1]), but
also cause a penalty for *all* attributes 'write' access.
[1] oh, yes, should I mention it ? The correct implementation is even
slower:
>>> class Hook(object):
... def __getattr__(self, name):
... if name == 'prop':
... return self._prop
... raise AttributeError("yadda")
... def __setattr__(self, name, val):
... if name == 'prop':
... self.__dict__['_prop'] = val
... else:
... #self.__dict__[name] = val
... super(Hook, self).__setattr__(name, value)
... def __init__(self, val): self.prop=val
...
>>> th.timeit(1000)
7.1943540573120117
>>> th.timeit(1000)
7.1930480003356934
>>>
HTH
mwojc
> >>> class Prop(object):
> ... @apply
> ... def prop():
> ... def fget(self): return self._prop
> ... def fset(self, val): self._prop = val
> ... return property(**locals())
> ... def __init__(self, val): self.prop=val
> ...
> >>> class Hook(object):
> ... def __getattr__(self, name):
> ... if name == 'prop':
> ... return self._prop
> ... raise AttributeError("yadda")
> ... def __setattr__(self, name, val):
> ... if name == 'prop':
> ... self.__dict__['_prop'] = val
> ... else:
> ... # XXX : INCORRECT IMPLEMENTATION, DONT DO THIS !
> ... self.__dict__[name] = val
> ... # correct implementation:
> ... # super(Hook, self).__setattr__(name, value)
> ... def __init__(self, val): self.prop=val
Hi!
Finally i ended up with all the suggestions you gave me. The speed is
important to me besause i get/set my attribute a lot... The only doubts i
have are with the @apply decorator, because 'apply' seems to be depreciated
since python 2.3... And another thing: why writing for all
attributes 'self.__dict__[name] = val' in __setattr__ is incorrect?
Thanks a lot!
Greetings,
--
Marek
Michele Simionato
> Hi!
> My class with implemented __getattr__ and __setattr__ methods cannot be
> pickled because of the Error:
Another option is to define __getstate__ on your class:
def __getstate__(self): return vars(self)
M.S.
Bruno Desthuilliers
As far as I can tell, it should stay until Python 3.0. But you don't
have to write your properties that way anyway - it's just something I
personnaly find to be a convenient shortcut, period. FWIW, the above
code is just a copy/paste from an interactive session, not an example of
"clean" Python coding.
> And another thing: why writing for all
> attributes 'self.__dict__[name] = val' in __setattr__ is incorrect?
Because __setattr__ is *always* called. With your above solution,
read/write properties setters won't be called.
Python 2.5.1 (r251:54863, Mar 7 2008, 03:41:45)
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo(object):
@apply
def bar():
def fget(self):
print "bar get"
return self._bar
def fset(self, val):
print "bar set"
self._bar = val
def __getattr__(self, name):
print "__getattr__", name
if name == 'baaz':
return self._baaz
raise AttributeError("yadda")
def __setattr__(self, name, val):
print "__setattr__", name
self.__dict__[name] = val
>>> foo = Foo('bar', 'baaz')
__setattr__ bar
__setattr__ baaz
>>> foo.baaz
'baaz'
>>> foo.bar
'bar'
>>> foo.__dict__
{'baaz': 'baaz', 'bar': 'bar'}