major change in DAL

[Massimo DiPierro]

There is a major and important change in trunk about DAL and I need help with testing:

BEFORE:

>>> a = DAL('sqlite://filename.db')
>>> b = DAL('sqlite://filename.db')
>>> a is b
False

NOW
>>> a = DAL('sqlite://filename.db')
>>> b = DAL('sqlite://filename.db')
>>> a is b
True

Now DAL is a singleton. You can still have multiple connections to different databases but not multiple connections to the same database (which does not make sense anyway). This means you can also do:

>>> a = DAL('sqlite://filename.db')
>>> a.define_table('person',Field('name'))
>>> b = DAL('sqlite://filename.db')
>>> b.person.name

and also

>>> x = pickle.dumps(a)
>>> b = pickle.loads(x)

Yes. Now any DAL object (db) is fully packable as long as unpicked after there is an existing connection to the same database. This has some major and useful consequences. For example you can do:

db = DAL(uri)
….
rows = db(query).select()
x = pickle.dumps(rows)

db =DAL(uri)
y = pickle.loads(rows)
y.first().delete_record()

In other words you can pickle ANY rows object and when unpicked it looks the same as the original including all the special methods.
Is this 100% backward compatible? I think yes except that:

1) now if you cache, pickle and unpickle Rows and Row you get the original object back, not some approximation of it.

2) Given

db.define_table('person',Field('name'))
db.define_table('thing',Field('name'),Field('owner','reference person'))
id = db.person.insert(name='Tim')
db.thing.insert(name='chair',owner=id)
rows = db(db.person).select()

rows.first().thing used to belong of class Set now rows.first().thing is class LazySet. This is the only place class LazySet exists. It has the same methods as class Set. As before you can still do:

print rows.first().thing.select()
print rows.first().thing(db.thing.name.startswith('c')).select()
print rows.first().thing.delect()

This is rare corner case. We did not change the behavior, only the name of an internal class.

Now any Rows object can be stored in a session. Since the session is loaded before the db=DAL() is called the db in the session is created as DAL('<lazy>') by the unpickler. This is a little messy and works until one tries to access a session object that stored a set of Rows created with a db using a different uri then the current one. For example session.rows=db().select() breaks if you change your db password. Should this be a concern?

The logic is quite complex and I may have missed something. Please help me test it and check it.

Massimo

[Bruno Rocha]

Excelent!

I will try memcache and redis to cache Rows, it will be very useful for me.

Thank you again!

Bruno Cezar Rocha - @rochacbruno
rocha...@gmail.com | Mobile: +55 (11) 99210-8821
www.CursoDePython.com.br | www.rochacbruno.com.br

Get a signature like this. Click here.  

[Massimo DiPierro]

Wait…  something is messed up. :-(

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 

[Massimo DiPierro]

This should now work.

[Massimo DiPierro]

I still think the logic is convoluted and can be simplified. I think it may be possible to make all DAL objects serializable and automatically re-connect when de-serialized of a singleton db object is defined within the same thread…. working on it.

Ideas and help testing are welcome.

[Massimo Di Pierro]

I think I simplified it quite a bit: I will try explain it, mostly to clarify it to myself (again):

uri = 'postgres://user:passwd@hostname/db'

>>> a = DAL(uri,pool_size=10)

>>> b = DAL(uri,pool_size=10)

>>> a is b

True

>>> a.close() # close both a and b

>>> a = DAL(uri,pool_size=10) # new singleton

>>> a is b

False

>>> pickle.dump(a,open('db.pickle','w'))

>>> c = pickle.load(open('db.pickle','r'))

>>> a is c

True

DAL instances are singletons linked by a singleton_code which is a function of the uri. They are serializable. The serialization only stores the singleton_code. Serialized DAL objects can be stores in session and retrieved before the actual connection is stablished:

New process...

>>> c = pickle.load(open('db.pickle','r'))

>>> # c is unusable here

>>> a = DAL(uri,pool_size=10)

>>> a is c

True

>>> # c is usable here

Each singleton DAL can be '<lazy>' or normal. A '<lazy'> one is one that is deserialized (like c) but does not have an adapter (until a is defined in the example).

Table, Field, Set and Query are still not serializable because they contain lambdas but DAL, Row and Rows are now almost always serializable.

Each DAL singleton object also has another state: connected or not-connected:

>>> a = DAL(uri,pool_size=10,do_connect=False) # defaults to true

>>> # a is NOT connected to db

>>> a._adapter.reconnect() # does connection pooling

>>> # a is now connected to db

>>> a._adapter.close('commit')

>>> # a is NOT connected to db

>>> a._adapter.reconnect() # does connection pooling

>>> # a is now connected to db

This makes it work great with gluino in such a way that is thread safe and tables are defined once and for all, not at every request. Look into wrapper.py and the tornado examples.

Hope this makes some sense.

Massimo

[Massimo Di Pierro]

On second thought... this works but gluino is still broken.

[Bruno Rocha]

I will test this with redis and memcache tomorrow morning...

[szimszon]

If I try to start a new task I got this now:
2012-10-02 11:57:01,293 - root - INFO - new task 14 "e" web2print/appadmin.download_task
2012-10-02 11:57:01,293 - root - DEBUG -  new task allocated: web2print/appadmin.download_task
2012-10-02 11:57:01,296 - root - DEBUG -    task starting
2012-10-02 11:57:01,298 - root - DEBUG -     task started
Traceback (most recent call last):
  File "/home/gyszabolcs/fejlesztes/web2py/gluon/restricted.py", line 209, in restricted
    exec ccode in environment
  File "applications/web2print/models/db.py", line 134, in <module>
    label = T( 'Státusz' ) ),
  File "/home/gyszabolcs/fejlesztes/web2py/gluon/dal.py", line 7034, in define_table
    raise SyntaxError, 'invalid table name: %s' % tablename
SyntaxError: invalid table name: webconfig


All my tablename is invalid. But the webapp is running...

[szimszon]

http://code.google.com/p/web2py/issues/detail?id=1062

[Massimo DiPierro]

I think this is now fixed but behavior has changed again

>>> a = DAL()

>>> b = DAL()

>>> a is b 

False 

DAL objects are not usually singleton. Behaves as it used to.

Yet

>>> a =DAL()

>>> s = pickle.dumps(a)

>>> a.close()

>>> b = pickle.loads(s) # b is a zombie DAL. It is not connected and has no attributes…. until

>>> a = DAL() # connection again

>>> a is b

True

>>> c = pickle.loads(s)

>>> a is c

True

Unpickled DAL objects behave like singleton and point to the last open connection with the same URI. So in some cases they are singletons.

Moreover you can force redefinition of tables:

>>> db = DAL()

>>> db.define_table('person')

>>> db.define_table('person',Field('name'),redefine=True)

I believe this fixes all standing problems.