Storage as super class

[Paolo Valleri]

Hi, I've been working on a way to use Storage as a super class for some classes we have in pydal (see https://github.com/ilvalle/pydal/tree/storage).

Beside performance, by using Storage we can avoid some code repetitions. It seems to work well as a super class for classes without properties.

Now, I've this issue

import copy

from gluon.storage import Storage

class AAA(Storage):

    @property

    def f(self):

        return 'ff'

        

a=AAA()

print a.f

b = copy.copy(a)

print b.f

I got as result 

'ff'
None

My understanding is that:

a.f is bound to the method AAA.f

b.f is just an unexisting obj in the storage dict (storage return None instead of raising an exception).

I've tried few hacks by overriding __copy__,  __setstate__ and so on, but I haven't understood how to fix it.

Any clue?

[Giovanni Barillari]

The main issue with @property decorator is that it inject a different object inside the class rather than the original method.

You will be able to copy the Storage class and its attributes with a dir() on the other object and a for with some exclusions on the attributes/methods names, but in case of @property I think you won't see the real method but only the return value.

If this will work only with internal code, we can re-write a @property decorator to keep track of the real objects, but if it has to be compatible with users' subclassing with @property it can be hard.

Which objects inside pydal now use @property?

/Giovanni

[Paolo Valleri]

Hi Giovanni,

Table has a few methods marked as @property. DAL only one.

In the weekend I fixed the '@property issue' by commenting the methods __getnewargs__ and __copy__ in Storage.

Regarding performance, I've noticed that accessing attribute instances is slower when the class inherits from storage rather than when it inherits from object (almost 6x slower on a win7). Given that, I've created an experimental storage that has the same performance of a 'normal class' when accessing an attribute. see https://github.com/ilvalle/pydal/blob/storage/pydal/helpers/classes.py#L448. The basic idea is to build 'basicStorage' on top of self.__dict__

Beside Row, basicStorage2 can be used as a super class for Table too.

 Paolo

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Niphlod]

beware that Storage was yet super-optimized and we have a FastStorage too in web2py, but it suffers from memory leaks....

[Paolo Valleri]

I know, I remember that optimizations. The proposed version actually follows a similar idea of FastStorage. 

In particular, attributes are handled as in a common class, items are mapped into self.__dict__, no more dict as super class (unfortunately I had to redefine many common methods). A further optimization can be the introduction of lambda instead of explicit method def.

 Paolo

[Niphlod]

On Monday, April 20, 2015 at 8:50:34 PM UTC+2, Paolo Valleri wrote:

I know, I remember that optimizations. The proposed version actually follows a similar idea of FastStorage. 

In particular, attributes are handled as in a common class, items are mapped into self.__dict__, no more dict as super class (unfortunately I had to redefine many common methods). A further optimization can be the introduction of lambda instead of explicit method def.

 Paolo

beware: if you follow the path of FastStorage and your code leaks, I'll happily stay with things as they are.

[Paolo Valleri]

I fully agree with you. 

I started working on it in order to aggregate the same code we have in Row, Table, DAL etc in a simple (super) class. To some extent, that class is actually Storage, optimizing it is like a 'side' effect, not mandatory off course. How should I check I don't have memory leaks?

 Paolo

--

[Niphlod]

On Monday, April 20, 2015 at 9:56:34 PM UTC+2, Paolo Valleri wrote:

I fully agree with you. 

I started working on it in order to aggregate the same code we have in Row, Table, DAL etc in a simple (super) class. To some extent, that class is actually Storage, optimizing it is like a 'side' effect, not mandatory off course. How should I check I don't have memory leaks?

sadly only profiling, profiling profiling. And to some extent keeping an app active and NOT seeing ram usage increase.

[Paolo Valleri]

I first profiling on a RH python 2.4 (just an old version that doesn't contain the fix)

I tried the following:

for n in xrange(1000000):
   import gc
   gc.collect()
   ad = FastStorage()
   ad['x'] = n
   ad.y = ad.x
   print n, ad.x, ad.y

The memory used by FastStorage grows while it is constant with basicStorage2

In the bug report is mentioned that the problem should raise when self.__dict__ is self

 Paolo

--