new dal
176 views
Skip to first unread message
mdipierro
Dec 4, 2010, 12:32:28 PM12/4/10
to web2py-users
The good news...
I have finished the new DAL for RDBMs. If you now replace gluon/sql.py
with gluon/dal.py as long as you do not use GAE, everything should
work fine. All tests pass.
The new dal should have all the features as the old dal but no more.
It just comes with a better more modular design that will allow adding
new features.
It defines the following classes:
class ConnectionPool(object):
class BaseAdapter(ConnectionPool):
# adapters
class SQLiteAdapter(BaseAdapter):
class JDBCSQLiteAdapter(SQLiteAdapter):
class MySQLAdapter(BaseAdapter):
class PostgreSQLAdapter(BaseAdapter):
class JDBCPostgreSQLAdapter(PostgreSQLAdapter):
class OracleAdapter(BaseAdapter):
class MSSQLAdapter(BaseAdapter):
class MSSQLAdapter2(MSSQLAdapter):
class FireBirdAdapter(BaseAdapter):
class FireBirdEmbeddedAdapter(FireBirdAdapter):
class InformixAdapter(BaseAdapter):
class DB2Adapter(BaseAdapter):
class IngresAdapter(BaseAdapter):
class IngresUnicodeAdapter(IngresAdapter):
# exposed by DAL
class Row(dict):
class SQLCallableList(list):
class DAL(dict):
class SQLALL(object):
class Reference(int):
class Table(dict):
class Expression(object):
class SQLCustomType:
class Field(Expression):
class Query(object):
class Set(object):
class Rows(object):
TODO:
1)
test dal.py
test dal.py
test dal.py
2)
break gluon/dal.py into
gluon/dal/__init__.py
gluon/dal/pooling.py
gluon/dal/adapter_base.py
gluon/dal/adapter_*.py
3)
reimplement gluon/contrib/gql.py as
gluon/dal/adapter_nosql.py
gluon/dal/adapter_gae.py
4)
implement
gluon/dal/adapter_mongodb.py
etc.
Can you help with 1) and 2)?
Massimo
I have finished the new DAL for RDBMs. If you now replace gluon/sql.py
with gluon/dal.py as long as you do not use GAE, everything should
work fine. All tests pass.
The new dal should have all the features as the old dal but no more.
It just comes with a better more modular design that will allow adding
new features.
It defines the following classes:
class ConnectionPool(object):
class BaseAdapter(ConnectionPool):
# adapters
class SQLiteAdapter(BaseAdapter):
class JDBCSQLiteAdapter(SQLiteAdapter):
class MySQLAdapter(BaseAdapter):
class PostgreSQLAdapter(BaseAdapter):
class JDBCPostgreSQLAdapter(PostgreSQLAdapter):
class OracleAdapter(BaseAdapter):
class MSSQLAdapter(BaseAdapter):
class MSSQLAdapter2(MSSQLAdapter):
class FireBirdAdapter(BaseAdapter):
class FireBirdEmbeddedAdapter(FireBirdAdapter):
class InformixAdapter(BaseAdapter):
class DB2Adapter(BaseAdapter):
class IngresAdapter(BaseAdapter):
class IngresUnicodeAdapter(IngresAdapter):
# exposed by DAL
class Row(dict):
class SQLCallableList(list):
class DAL(dict):
class SQLALL(object):
class Reference(int):
class Table(dict):
class Expression(object):
class SQLCustomType:
class Field(Expression):
class Query(object):
class Set(object):
class Rows(object):
TODO:
1)
test dal.py
test dal.py
test dal.py
2)
break gluon/dal.py into
gluon/dal/__init__.py
gluon/dal/pooling.py
gluon/dal/adapter_base.py
gluon/dal/adapter_*.py
3)
reimplement gluon/contrib/gql.py as
gluon/dal/adapter_nosql.py
gluon/dal/adapter_gae.py
4)
implement
gluon/dal/adapter_mongodb.py
etc.
Can you help with 1) and 2)?
Massimo
ron_m
Dec 4, 2010, 1:31:15 PM12/4/10
to web2py-users
Describe what I did
Copied the trunk hg area to a work area
Renamed sql.py to sql-save.py
Copied dal.py to sql.py in order to replace sql.py with dal.py
Copied application I am working on to applications directory
Backed up the database
Ran the server as python web2py.py
Got this stack trace
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/main.py", line 446, in
wsgibase
BaseAdapter.close_all_instances(BaseAdapter.commit)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 161, in
close_all_instances
if instance._pool_size:
AttributeError: 'SQLiteAdapter' object has no attribute '_pool_size'
I don't use SQLite as a database but the driver exists through default
install, the db line invoking SQLite is commented out.
Copied the trunk hg area to a work area
Renamed sql.py to sql-save.py
Copied dal.py to sql.py in order to replace sql.py with dal.py
Copied application I am working on to applications directory
Backed up the database
Ran the server as python web2py.py
Got this stack trace
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/main.py", line 446, in
wsgibase
BaseAdapter.close_all_instances(BaseAdapter.commit)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 161, in
close_all_instances
if instance._pool_size:
AttributeError: 'SQLiteAdapter' object has no attribute '_pool_size'
I don't use SQLite as a database but the driver exists through default
install, the db line invoking SQLite is commented out.
ron_m
Dec 4, 2010, 1:38:26 PM12/4/10
to web2py-users
Sorry should have mentioned that was in the welcome app fetching the
index page as the default browser load occured fomr the TK server
interface. In my own application I get
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/restricted.py", line 188,
in restricted
exec ccode in environment
File "/home/camcentral/Dev/web2py_hg/applications/ccims/models/
A_db.py", line 19, in <module>
db = DAL('postgres://ccims:oss3838@localhost/cc_ims') # CamCentral
IMS configuration database
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 2408, in
__init__
raise RuntimeError, "Failure to connect, tried 5 times"
RuntimeError: Failure to connect, tried 5 times
Maybe I did something wrong getting dal.py activated?
index page as the default browser load occured fomr the TK server
interface. In my own application I get
Traceback (most recent call last):
in restricted
exec ccode in environment
File "/home/camcentral/Dev/web2py_hg/applications/ccims/models/
A_db.py", line 19, in <module>
db = DAL('postgres://ccims:oss3838@localhost/cc_ims') # CamCentral
IMS configuration database
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 2408, in
__init__
raise RuntimeError, "Failure to connect, tried 5 times"
RuntimeError: Failure to connect, tried 5 times
Maybe I did something wrong getting dal.py activated?
mdipierro
Dec 4, 2010, 5:00:27 PM12/4/10
to web2py-users
can you please try again?
Massimo
Massimo
mr.freeze
Dec 4, 2010, 9:30:14 PM12/4/10
to web2py-users
I was getting that error. After updating it goes away but now I get
the below. I am running it against web2pyslices.com source on a sqlite
db with Python 2.5:
Traceback (most recent call last):
File "C:\Users\nathan\workspace\web2py\gluon\restricted.py", line
menu.py", line 115, in <module>
response.tagcloud =
db().select(db.tags.ALL,orderby=~db.tags.count,limitby=(0,15),cache=(cache.disk,
600))
File "C:\Users\nathan\workspace\web2py\gluon\sql.py", line 3319, in
select
shutil.copyfileobj(file, dest_file)
File "C:\Users\nathan\workspace\web2py\gluon\cache.py", line 282, in
__call__
storage = shelve.open(self.shelve_name)
File "C:\Python25\lib\shelve.py", line 225, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "C:\Python25\lib\shelve.py", line 209, in __init__
Shelf.__init__(self, anydbm.open(filename, flag), protocol,
writeback)
File "C:\Python25\lib\anydbm.py", line 83, in open
return mod.open(file, flag, mode)
File "C:\Python25\lib\dbhash.py", line 16, in open
return bsddb.hashopen(file, flag, mode)
File "C:\Python25\lib\bsddb\__init__.py", line 310, in hashopen
d.open(file, db.DB_HASH, flags, mode)
DBPermissionsError: (1, 'Operation not permitted')
the below. I am running it against web2pyslices.com source on a sqlite
db with Python 2.5:
Traceback (most recent call last):
188, in restricted
exec ccode in environment
File "C:/Users/nathan/workspace/web2py/applications/main/models/
exec ccode in environment
menu.py", line 115, in <module>
response.tagcloud =
db().select(db.tags.ALL,orderby=~db.tags.count,limitby=(0,15),cache=(cache.disk,
600))
File "C:\Users\nathan\workspace\web2py\gluon\sql.py", line 3319, in
select
shutil.copyfileobj(file, dest_file)
File "C:\Users\nathan\workspace\web2py\gluon\cache.py", line 282, in
__call__
storage = shelve.open(self.shelve_name)
File "C:\Python25\lib\shelve.py", line 225, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "C:\Python25\lib\shelve.py", line 209, in __init__
Shelf.__init__(self, anydbm.open(filename, flag), protocol,
writeback)
File "C:\Python25\lib\anydbm.py", line 83, in open
return mod.open(file, flag, mode)
File "C:\Python25\lib\dbhash.py", line 16, in open
return bsddb.hashopen(file, flag, mode)
File "C:\Python25\lib\bsddb\__init__.py", line 310, in hashopen
d.open(file, db.DB_HASH, flags, mode)
DBPermissionsError: (1, 'Operation not permitted')
mdipierro
Dec 4, 2010, 10:57:35 PM12/4/10
to web2py-users
Can you try delete everything in app/cache/
The error seems in opening the cache file.
The error seems in opening the cache file.
mr.freeze
Dec 4, 2010, 11:26:14 PM12/4/10
to web2py-users
I get this now:
exec ccode in environment
File "C:/web2py/applications/main/models/db.py", line 22, in
<module>
db = SQLDB(dal_connection,pool_size=10)
File "C:\web2py\gluon\sql.py", line 2405, in __init__
Traceback (most recent call last):
File "C:\web2py\gluon\restricted.py", line 188, in restricted
exec ccode in environment
File "C:/web2py/applications/main/models/db.py", line 22, in
<module>
db = SQLDB(dal_connection,pool_size=10)
File "C:\web2py\gluon\sql.py", line 2405, in __init__
raise RuntimeError, "Failure to connect, tried 5 times"
RuntimeError: Failure to connect, tried 5 times
RuntimeError: Failure to connect, tried 5 times
mr.freeze
Dec 4, 2010, 11:33:40 PM12/4/10
to web2py-users
Looking closer, there is a syntax error on line 189: COnnectionPool
After I fixed that, I get:
BaseAdapter.close_all_instances(BaseAdapter.rollback)
File "C:\web2py\gluon\sql.py", line 163, in close_all_instances
pool = ConnectionPool._pools[instance._uri]
AttributeError: 'SQLiteAdapter' object has no attribute '_uri'
After I fixed that, I get:
Traceback (most recent call last):
File "C:\web2py\gluon\main.py", line 488, in wsgibase
BaseAdapter.close_all_instances(BaseAdapter.rollback)
File "C:\web2py\gluon\sql.py", line 163, in close_all_instances
pool = ConnectionPool._pools[instance._uri]
AttributeError: 'SQLiteAdapter' object has no attribute '_uri'
mdipierro
Dec 5, 2010, 12:02:57 AM12/5/10
to web2py-users
One more test please. I do not have mysql installed here so I did not
test pooling. This helps a lot. thanks.
test pooling. This helps a lot. thanks.
ron_m
Dec 5, 2010, 2:43:05 AM12/5/10
to web2py-users
I was away until 10pm, just did an hg update and recopied trunk to a
new work area plus copied dal.py to sql.py in gluon.
The welcome app gets this traceback on the index page after auto start
browser from TK interface.
web2py™ Version 1.89.5 (2010-11-21 16:03:13)
Python Python 2.6.5: /usr/bin/python
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/main.py", line 446, in
wsgibase
BaseAdapter.close_all_instances(BaseAdapter.commit)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 169, in
close_all_instances
instance._connection.close()
AttributeError: 'SQLiteAdapter' object has no attribute '_connection'
My own application gets
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/restricted.py", line 188,
of sql.py to install the new dal?
The hg update was compared using meld to the work area from this
morning and I see quite a few changes to dal.py. Just re-ran meld
after I recopied the new trunk to the work area and I don't see the
differences so I am very sure I got the latest as of tonight.
new work area plus copied dal.py to sql.py in gluon.
The welcome app gets this traceback on the index page after auto start
browser from TK interface.
web2py™ Version 1.89.5 (2010-11-21 16:03:13)
Python Python 2.6.5: /usr/bin/python
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/main.py", line 446, in
wsgibase
BaseAdapter.close_all_instances(BaseAdapter.commit)
close_all_instances
instance._connection.close()
AttributeError: 'SQLiteAdapter' object has no attribute '_connection'
My own application gets
Traceback (most recent call last):
in restricted
exec ccode in environment
exec ccode in environment
File "/home/camcentral/Dev/web2py_hg/applications/ccims/models/
A_db.py", line 19, in <module>
db = DAL('postgres://ccims:oss3838@localhost/cc_ims') # CamCentral
IMS configuration database
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 2402, in
A_db.py", line 19, in <module>
db = DAL('postgres://ccims:oss3838@localhost/cc_ims') # CamCentral
IMS configuration database
__init__
raise RuntimeError, "Failure to connect, tried 5 times"
RuntimeError: Failure to connect, tried 5 times
Did I do something wrong in the test install copying dal.py over top
raise RuntimeError, "Failure to connect, tried 5 times"
RuntimeError: Failure to connect, tried 5 times
of sql.py to install the new dal?
The hg update was compared using meld to the work area from this
morning and I see quite a few changes to dal.py. Just re-ran meld
after I recopied the new trunk to the work area and I don't see the
differences so I am very sure I got the latest as of tonight.
mr.freeze
Dec 5, 2010, 11:12:54 AM12/5/10
to web2py-users
mr.freeze
Dec 5, 2010, 11:20:39 AM12/5/10
to web2py-users
Working with mysql also. I tested with migrate=False. Nice work!
mdipierro
Dec 5, 2010, 11:36:04 AM12/5/10
to web2py-users
can you try migrate=True (add a dummy field and the remove it) with
mysql?
mysql?
mr.freeze
Dec 5, 2010, 11:37:37 AM12/5/10
to web2py-users
mr.freeze
Dec 5, 2010, 11:45:18 AM12/5/10
to web2py-users
I have a global variable migrate_db which is passed to all
define_table functions. I switched it to false and get this:
<module>
readable=False, default=""),migrate=migrate_db)
File "C:\web2py\gluon\sql.py", line 1406, in define_table
obj = str(obj)
File "C:\web2py\gluon\sql.py", line 1858, in _create
'boolean': 'CHAR(1)',
File "C:\web2py\gluon\sql.py", line 1024, in <lambda>
'string': 'CHAR(%(length)s)',
File "C:\Python26\lib\site-packages\MySQLdb\cursors.py", line 166,
in execute
self.errorhandler(self, exc, value)
File "C:\Python26\lib\site-packages\MySQLdb\connections.py", line
35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1050, "Table 'auth_user' already exists")
define_table functions. I switched it to false and get this:
Traceback (most recent call last):
File "C:\web2py\gluon\restricted.py", line 188, in restricted
exec ccode in environment
File "C:/web2py/applications/main/models/db.py", line 34, in
File "C:\web2py\gluon\restricted.py", line 188, in restricted
exec ccode in environment
<module>
readable=False, default=""),migrate=migrate_db)
File "C:\web2py\gluon\sql.py", line 1406, in define_table
obj = str(obj)
File "C:\web2py\gluon\sql.py", line 1858, in _create
'boolean': 'CHAR(1)',
File "C:\web2py\gluon\sql.py", line 1024, in <lambda>
'string': 'CHAR(%(length)s)',
File "C:\Python26\lib\site-packages\MySQLdb\cursors.py", line 166,
in execute
self.errorhandler(self, exc, value)
File "C:\Python26\lib\site-packages\MySQLdb\connections.py", line
35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1050, "Table 'auth_user' already exists")
mr.freeze
Dec 5, 2010, 12:47:20 PM12/5/10
to web2py-users
I seem to remember this failing before though. Something about
my .TABLE files being out of sync. What is the procedure to re-sync
them?
my .TABLE files being out of sync. What is the procedure to re-sync
them?
mdipierro
Dec 5, 2010, 1:41:05 PM12/5/10
to web2py-users
for the troublesome table (auth_user) set migrate=False,
fake_migrate=True reload, then migrate=True and remove fake_migrate.
It should fix your broken .table.
Should work with old and with new dal.
fake_migrate=True reload, then migrate=True and remove fake_migrate.
It should fix your broken .table.
Should work with old and with new dal.
mr.freeze
Dec 5, 2010, 1:53:45 PM12/5/10
to web2py-users
I reverted to the old DAL and did the steps you outlined. Same result.
Should I delete the .table files first?
Should I delete the .table files first?
mdipierro
Dec 5, 2010, 1:56:56 PM12/5/10
to web2py-users
It is the same. try without delete them.
mr.freeze
Dec 5, 2010, 2:05:17 PM12/5/10
to web2py-users
I tried and the file modification times of the .table files do not
change. I deleted them (safely backed up) and they are not re-created.
change. I deleted them (safely backed up) and they are not re-created.
mdipierro
Dec 5, 2010, 2:24:35 PM12/5/10
to web2py-users
can you try migrate=True, fake_migrate=True?
mr.freeze
Dec 5, 2010, 2:38:18 PM12/5/10
to web2py-users
Auth.define_tables doesn't have a fake_migrate attribute so I added
one (want a patch?). Once I did, my .table files were re-created when
migrate=True and fake_migrate=True. It still works once I set
fake_migrate to False. I will test it against the new DAL now.
one (want a patch?). Once I did, my .table files were re-created when
migrate=True and fake_migrate=True. It still works once I set
fake_migrate to False. I will test it against the new DAL now.
mr.freeze
Dec 5, 2010, 2:43:29 PM12/5/10
to web2py-users
Okay, migrate=True works with the new DAL too. On a whim, I deleted
my .table files and tried migrate=True and fake_migrate=True with the
new DAL. Everything worked. The .table files were re-created.
my .table files and tried migrate=True and fake_migrate=True with the
new DAL. Everything worked. The .table files were re-created.
mdipierro
Dec 5, 2010, 2:56:47 PM12/5/10
to web2py-users
mdipierro
Dec 5, 2010, 10:11:19 PM12/5/10
to web2py-users
started work integrating with gluon/contrib/gql.py do
please keep testing dal.py
now it no longer requires web2py and you can do
% python
>>> from dal import DAL, Field
>>> db=DAL('sqlite://file.sqlite')
>>> db.define_table('person',Field('name'))
etc etc.
you ONLY need dal.py
> ...
>
> read more »
please keep testing dal.py
now it no longer requires web2py and you can do
% python
>>> from dal import DAL, Field
>>> db=DAL('sqlite://file.sqlite')
>>> db.define_table('person',Field('name'))
etc etc.
you ONLY need dal.py
>
> read more »
ron_m
Dec 6, 2010, 12:13:56 PM12/6/10
to web2py-users
I did an update of trunk using hg and copied dal.py over sql.py in
gluon and start the web2py.py server and still get
Traceback (most recent call last):
close_all_instances
self.connection.close()
NameError: global name 'self' is not defined
main is calling BaseAdapter.close_all_instances() as a class method so
the self reference at line 276 blows up because there is no instance
of a class. I see from the latest post we can test dal.py standalone,
are we not to be testing as part of web2py yet because of integration
issues?
Ron
> ...
>
> read more »
gluon and start the web2py.py server and still get
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/main.py", line 446, in
wsgibase
BaseAdapter.close_all_instances(BaseAdapter.commit)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 276, in
wsgibase
BaseAdapter.close_all_instances(BaseAdapter.commit)
close_all_instances
self.connection.close()
NameError: global name 'self' is not defined
main is calling BaseAdapter.close_all_instances() as a class method so
the self reference at line 276 blows up because there is no instance
of a class. I see from the latest post we can test dal.py standalone,
are we not to be testing as part of web2py yet because of integration
issues?
Ron
>
> read more »
mdipierro
Dec 6, 2010, 12:25:39 PM12/6/10
to web2py-users
please try replace
@staticmethod
def close_all_instances(action):
""" to close cleanly databases in a multithreaded environment
"""
if not hasattr(thread,'instances'):
return
while thread.instances:
instance = thread.instances.pop()
action(instance)
# ## if you want pools, recycle this
connection
really = True
if instance.pool_size:
sql_locker.acquire()
pool = ConnectionPool.pools[self.uri]
if len(pool) < instance.pool_size:
pool.append(self.connection)
really = False
sql_locker.release()
if really:
self.connection.close()
return
with
@staticmethod
def close_all_instances(action):
""" to close cleanly databases in a multithreaded environment
"""
if not hasattr(thread,'instances'):
return
while thread.instances:
instance = thread.instances.pop()
action(instance)
# ## if you want pools, recycle this
connection
really = True
if instance.pool_size:
sql_locker.acquire()
pool = ConnectionPool.pools[self.uri]
if len(pool) < instance.pool_size:
pool.append(instance.connection)
really = False
sql_locker.release()
if really:
instance.connection.close()
return
I think this is global replace error.
@staticmethod
def close_all_instances(action):
""" to close cleanly databases in a multithreaded environment
"""
if not hasattr(thread,'instances'):
return
while thread.instances:
instance = thread.instances.pop()
action(instance)
# ## if you want pools, recycle this
connection
really = True
if instance.pool_size:
sql_locker.acquire()
pool = ConnectionPool.pools[self.uri]
if len(pool) < instance.pool_size:
pool.append(self.connection)
really = False
sql_locker.release()
if really:
self.connection.close()
return
with
@staticmethod
def close_all_instances(action):
""" to close cleanly databases in a multithreaded environment
"""
if not hasattr(thread,'instances'):
return
while thread.instances:
instance = thread.instances.pop()
action(instance)
# ## if you want pools, recycle this
connection
really = True
if instance.pool_size:
sql_locker.acquire()
pool = ConnectionPool.pools[self.uri]
if len(pool) < instance.pool_size:
pool.append(instance.connection)
really = False
sql_locker.release()
if really:
instance.connection.close()
return
I think this is global replace error.
Jonathan Lundell
Dec 6, 2010, 12:27:33 PM12/6/10
to web...@googlegroups.com
On Dec 6, 2010, at 9:13 AM, ron_m wrote:
>
> I did an update of trunk using hg and copied dal.py over sql.py in
> gluon and start the web2py.py server and still get
>
> Traceback (most recent call last):
> File "/home/camcentral/Dev/web2py_hg/gluon/main.py", line 446, in
> wsgibase
> BaseAdapter.close_all_instances(BaseAdapter.commit)
> File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 276, in
> close_all_instances
> self.connection.close()
> NameError: global name 'self' is not defined
>
> main is calling BaseAdapter.close_all_instances() as a class method so
> the self reference at line 276 blows up because there is no instance
> of a class. I see from the latest post we can test dal.py standalone,
> are we not to be testing as part of web2py yet because of integration
> issues?
>
> I did an update of trunk using hg and copied dal.py over sql.py in
> gluon and start the web2py.py server and still get
>
> Traceback (most recent call last):
> File "/home/camcentral/Dev/web2py_hg/gluon/main.py", line 446, in
> wsgibase
> BaseAdapter.close_all_instances(BaseAdapter.commit)
> File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 276, in
> close_all_instances
> self.connection.close()
> NameError: global name 'self' is not defined
>
> main is calling BaseAdapter.close_all_instances() as a class method so
> the self reference at line 276 blows up because there is no instance
> of a class. I see from the latest post we can test dal.py standalone,
> are we not to be testing as part of web2py yet because of integration
> issues?
Judging from the parallel code in sql.py, you might try replacing 'self' with 'instance' in dal.ConnectionPool.close_all_instances (3 places).
ron_m
Dec 6, 2010, 2:12:20 PM12/6/10
to web2py-users
Ok that worked for starting the welcome app. As Jonathan suggests the
self.uri in ConnectionPool.pools[self.uri] needs to be instance.uri as
well.
I will look to see why the application I am working on won't connect
to postgresql - timeout after 5 tries and get back if I find the
cause.
Thanks
Ron
> ...
>
> read more »
self.uri in ConnectionPool.pools[self.uri] needs to be instance.uri as
well.
I will look to see why the application I am working on won't connect
to postgresql - timeout after 5 tries and get back if I find the
cause.
Thanks
Ron
>
> read more »
mdipierro
Dec 6, 2010, 3:55:27 PM12/6/10
to web2py-users
Important news!
1) The new DAL (dal.py) passes all the tests I have.
2) The new DAL has integrated GAE support (datastore) without need for
contrib/gql.py
3) The new DAL is better because more customizable, smaller (10%), and
more readable
4) The new DAL is one single file and it does not depend on web2py.
You can use it with other frameworks out of the box without tweaking.
TODO:
- test test test and more tests
- benckmark old dal vs new dal
- replace old dal with new dal in stable web2py
I may keep working on this make more corrections/improvements.
Massimo
> ...
>
> read more »
1) The new DAL (dal.py) passes all the tests I have.
2) The new DAL has integrated GAE support (datastore) without need for
contrib/gql.py
3) The new DAL is better because more customizable, smaller (10%), and
more readable
4) The new DAL is one single file and it does not depend on web2py.
You can use it with other frameworks out of the box without tweaking.
TODO:
- test test test and more tests
- benckmark old dal vs new dal
- replace old dal with new dal in stable web2py
I may keep working on this make more corrections/improvements.
Massimo
>
> read more »
Jonathan Lundell
Dec 6, 2010, 4:10:48 PM12/6/10
to web...@googlegroups.com
On Dec 6, 2010, at 12:55 PM, mdipierro wrote:
>
> 1) The new DAL (dal.py) passes all the tests I have.
> 2) The new DAL has integrated GAE support (datastore) without need for
> contrib/gql.py
> 3) The new DAL is better because more customizable, smaller (10%), and
> more readable
> 4) The new DAL is one single file and it does not depend on web2py.
>
> 1) The new DAL (dal.py) passes all the tests I have.
> 2) The new DAL has integrated GAE support (datastore) without need for
> contrib/gql.py
> 3) The new DAL is better because more customizable, smaller (10%), and
> more readable
> 4) The new DAL is one single file and it does not depend on web2py.
Have you decided to keep it as a single file, or break it into a package?
mdipierro
Dec 6, 2010, 4:15:43 PM12/6/10
to web2py-users
I am not sure. There are pros and cons in both cases.
What do you think.
For me it is easier to manage to this way. I tried to break it but it
would require even more re-factoring and I was not sure it was worth
it.
Massimo
What do you think.
For me it is easier to manage to this way. I tried to break it but it
would require even more re-factoring and I was not sure it was worth
it.
Massimo
ron_m
Dec 6, 2010, 5:19:14 PM12/6/10
to web2py-users
Ok I found the cannot connect to PostgreSQL database problem. I have a
password on the user id to access the db in my application so in
class PostgreSQLAdapter(BaseAdapter): function __init__(...)
The re.compile line element for the database password is 'passwd'
which does not match the
password = m.group('password') line for the extraction of this
element of the uri.
The SyntaxError exception is raised but covered up by the 5 times try
for loop in DAL.__init__() higher up the call stack and after 5 1
second sleeps DAL says it cannot connect.
Looking around at the other drivers the m = re.compile line should
have the 'passwd' element of the RE changed to 'password' to be
consistent.
I tested the app after changing and get a lot further into the app but
now see a new fault - will work on that.
Ron
> ...
>
> read more »
password on the user id to access the db in my application so in
class PostgreSQLAdapter(BaseAdapter): function __init__(...)
The re.compile line element for the database password is 'passwd'
which does not match the
password = m.group('password') line for the extraction of this
element of the uri.
The SyntaxError exception is raised but covered up by the 5 times try
for loop in DAL.__init__() higher up the call stack and after 5 1
second sleeps DAL says it cannot connect.
Looking around at the other drivers the m = re.compile line should
have the 'passwd' element of the RE changed to 'password' to be
consistent.
I tested the app after changing and get a lot further into the app but
now see a new fault - will work on that.
Ron
>
> read more »
mdipierro
Dec 6, 2010, 5:31:37 PM12/6/10
to web2py-users
Fixed in trunk. Thanks for testing
> ...
>
> read more »
>
> read more »
ron_m
Dec 6, 2010, 5:38:02 PM12/6/10
to web2py-users
Put dal.py over sql.py in trunk
Start server python web2py.py
Get welcome app in browser
Hit login link get ticket.
I also get this in my own application trying to login.
Field does not have attribute _tablename
Ron
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/restricted.py", line 188,
controllers/default.py", line 56, in <module>
File "/home/camcentral/Dev/web2py_hg/gluon/globals.py", line 95, in
<lambda>
self._caller = lambda f: f()
File "/home/camcentral/Dev/web2py_hg/applications/welcome/
controllers/default.py", line 33, in user
return dict(form=auth())
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1025, in
__call__
return self.login()
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1395, in
login
formstyle=self.settings.formstyle
File "/home/camcentral/Dev/web2py_hg/gluon/sqlhtml.py", line 810, in
__init__
inp = self.widgets.string.widget(field, default)
File "/home/camcentral/Dev/web2py_hg/gluon/sqlhtml.py", line 103, in
widget
attr = StringWidget._attributes(field, default, **attributes)
File "/home/camcentral/Dev/web2py_hg/gluon/sqlhtml.py", line 62, in
_attributes
_id = '%s_%s' % (field._tablename, field.name),
AttributeError: 'Field' object has no attribute '_tablename'
Start server python web2py.py
Get welcome app in browser
Hit login link get ticket.
I also get this in my own application trying to login.
Field does not have attribute _tablename
Ron
Traceback (most recent call last):
in restricted
exec ccode in environment
File "/home/camcentral/Dev/web2py_hg/applications/welcome/
exec ccode in environment
controllers/default.py", line 56, in <module>
File "/home/camcentral/Dev/web2py_hg/gluon/globals.py", line 95, in
<lambda>
self._caller = lambda f: f()
File "/home/camcentral/Dev/web2py_hg/applications/welcome/
controllers/default.py", line 33, in user
return dict(form=auth())
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1025, in
__call__
return self.login()
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1395, in
login
formstyle=self.settings.formstyle
File "/home/camcentral/Dev/web2py_hg/gluon/sqlhtml.py", line 810, in
__init__
inp = self.widgets.string.widget(field, default)
File "/home/camcentral/Dev/web2py_hg/gluon/sqlhtml.py", line 103, in
widget
attr = StringWidget._attributes(field, default, **attributes)
File "/home/camcentral/Dev/web2py_hg/gluon/sqlhtml.py", line 62, in
_attributes
_id = '%s_%s' % (field._tablename, field.name),
AttributeError: 'Field' object has no attribute '_tablename'
mdipierro
Dec 6, 2010, 5:38:24 PM12/6/10
to web2py-users
With the new DAL it should take very little to create a DAL interface
to mongodb, couchdb, etc.
I can work on this if
- you help me prioritize
- you can give me access to a machine that already has your favorite
nosql db installed ready for testing
to mongodb, couchdb, etc.
I can work on this if
- you help me prioritize
- you can give me access to a machine that already has your favorite
nosql db installed ready for testing
mdipierro
Dec 6, 2010, 7:08:02 PM12/6/10
to web2py-users
For now, for testing purposes I am going to copy dal.py over sql.py
and post a nightly build using the new dal.
Long term, is this ia good solution. I would like to keep both sql.py
and dal.py and import DAL, Field from the latter but this would break
existing "from gluon.sql import *". In order not to break it, the
current approach is the only approach (replace the old dal with the
new dal and call it sql.py).
Any opinion on this matter?
Massimo
and post a nightly build using the new dal.
Long term, is this ia good solution. I would like to keep both sql.py
and dal.py and import DAL, Field from the latter but this would break
existing "from gluon.sql import *". In order not to break it, the
current approach is the only approach (replace the old dal with the
new dal and call it sql.py).
Any opinion on this matter?
Massimo
mdipierro
Dec 6, 2010, 7:19:43 PM12/6/10
to web2py-users
both trunk and the nightly built in the web2py download page use the
new dal as opposed to the old one.
Please give it a try. Specifically if you are using something other
than sqlite.
please report any test you make. It is critical to know if this works
fine because I am planning to remove completely the old dal as soon as
possible.
Massimo
new dal as opposed to the old one.
Please give it a try. Specifically if you are using something other
than sqlite.
please report any test you make. It is critical to know if this works
fine because I am planning to remove completely the old dal as soon as
possible.
Massimo
villas
Dec 6, 2010, 7:24:09 PM12/6/10
to web2py-users
Providing that it's backwards compatible, you don't need to ask.
IMO test, upgrade, then never look back!
-D
IMO test, upgrade, then never look back!
-D
mdipierro
Dec 6, 2010, 7:35:46 PM12/6/10
to web2py-users
Here is some more info about the new DAL
http://web2py.com/examples/static/new_dal_help.pdf
hope it helps read the source code.
http://web2py.com/examples/static/new_dal_help.pdf
hope it helps read the source code.
Vasile Ermicioi
Dec 6, 2010, 7:36:05 PM12/6/10
to web...@googlegroups.com
I am testing new dal (alone, without web2py) and also tested one old idea -
I use pymysql instead of MySQLdb, and it works fine with python (2.7) and jython (2.5rc2)
why not integrating it on web2py, I think that makes web2py what it claims to be "requires no installation"
also it seems to have a good performance http://code.google.com/p/pymysql/wiki/Performance
just replace 2 lines:
import MySQLdb
with
import pymysql
and
charset=charset: MySQLdb.Connection(db=db,
with
charset=charset: pymysql.connect(db=db,
mdipierro
Dec 6, 2010, 7:38:14 PM12/6/10
to web2py-users
That is the point. I think it is fully backward compatible but the
source is so different that it is possible something breaks. Nothing
that cannot be fixed but I'd rather fix it sooner than later.
source is so different that it is possible something breaks. Nothing
that cannot be fixed but I'd rather fix it sooner than later.
mdipierro
Dec 6, 2010, 7:39:34 PM12/6/10
to web2py-users
No objection. We will default to the new one if it is installed of the
latter otherwise
On Dec 6, 6:36 pm, Vasile Ermicioi <elff...@gmail.com> wrote:
> I am testing new dal (alone, without web2py) and also tested one old idea -
> I use pymysql instead of MySQLdb, and it works fine with python (2.7) and
> jython (2.5rc2)
>
> why not integrating it on web2py, I think that makes web2py what it claims
> to be "requires no installation"
> also it seems to have a good performancehttp://code.google.com/p/pymysql/wiki/Performance
latter otherwise
On Dec 6, 6:36 pm, Vasile Ermicioi <elff...@gmail.com> wrote:
> I am testing new dal (alone, without web2py) and also tested one old idea -
> I use pymysql instead of MySQLdb, and it works fine with python (2.7) and
> jython (2.5rc2)
>
> why not integrating it on web2py, I think that makes web2py what it claims
> to be "requires no installation"
Vasile Ermicioi
Dec 6, 2010, 7:45:46 PM12/6/10
to web...@googlegroups.com
I propose to include a copy in site-packages and distribute it with web2py,
so no installation will be required, it is quite small < 100kb
mdipierro
Dec 6, 2010, 8:26:05 PM12/6/10
to web2py-users
I like that it is pure python. It would go in contrib not in site-
packages.
Anybody opposes?
Can you help us keep it in sync with the official repo?
packages.
Anybody opposes?
Can you help us keep it in sync with the official repo?
ron_m
Dec 6, 2010, 8:31:41 PM12/6/10
to web2py-users
Another patch
Hit login form getting a ticket.
sqlhtml.py
line 62 and line 216
field._tablename should be field.tablename
Ron
Hit login form getting a ticket.
sqlhtml.py
line 62 and line 216
field._tablename should be field.tablename
Ron
mdipierro
Dec 6, 2010, 9:04:09 PM12/6/10
to web2py-users
fixed in trunk. Thanks.
ron_m
Dec 6, 2010, 9:45:14 PM12/6/10
to web2py-users
That above patch prevents the old sql.py DAL from working with
sqlhtml.py because old dal does have a field attribute
field._tablename which sqlhtml.py was looking for before the patch.
I get the login form posted, fill it in but submit on the form to
complete the login gets a ticket. I tried to follow it but got lost, I
need a multi-threaded debugger such as winpdb. The one on Komodo IDE
6.0 doesn't seem to follow spawned threads OR I don't know how to use
it properly yet, Here is the traceback after the login form submit
(PostgreSQL database). It is inserting into the auth_event table the
fact that I logged in when it fails.
I went back to 1.89.5 and the application runs fine so the database
isn't damaged.
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/restricted.py", line 188,
default.py", line 70, in <module>
default.py", line 44, in user
login
self.log_event(log % self.user)
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1268, in
log_event
origin=origin, user_id=user_id)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 3540, in
insert
return self._db._adapter.insert(self, fields)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 673, in
insert
id = self.lastrowid(table)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 1500, in
lastrowid
self.execute("select currval('%s')" % table._sequence_name)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 1017, in
execute
return self.log_execute(*a, **b)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 1014, in
log_execute
return self.cursor.execute(*a,**b)
ProgrammingError: relation "none" does not exist
LINE 1: select currval('None')
sqlhtml.py because old dal does have a field attribute
field._tablename which sqlhtml.py was looking for before the patch.
I get the login form posted, fill it in but submit on the form to
complete the login gets a ticket. I tried to follow it but got lost, I
need a multi-threaded debugger such as winpdb. The one on Komodo IDE
6.0 doesn't seem to follow spawned threads OR I don't know how to use
it properly yet, Here is the traceback after the login form submit
(PostgreSQL database). It is inserting into the auth_event table the
fact that I logged in when it fails.
I went back to 1.89.5 and the application runs fine so the database
isn't damaged.
Traceback (most recent call last):
in restricted
exec ccode in environment
File "/home/camcentral/Dev/web2py_hg/applications/ccims/controllers/
exec ccode in environment
default.py", line 70, in <module>
File "/home/camcentral/Dev/web2py_hg/gluon/globals.py", line 95, in
<lambda>
self._caller = lambda f: f()
File "/home/camcentral/Dev/web2py_hg/applications/ccims/controllers/
<lambda>
self._caller = lambda f: f()
default.py", line 44, in user
return dict(form=auth())
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1025, in
__call__
return self.login()
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1516, in
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1025, in
__call__
return self.login()
login
self.log_event(log % self.user)
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1268, in
log_event
origin=origin, user_id=user_id)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 3540, in
insert
return self._db._adapter.insert(self, fields)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 673, in
insert
id = self.lastrowid(table)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 1500, in
lastrowid
self.execute("select currval('%s')" % table._sequence_name)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 1017, in
execute
return self.log_execute(*a, **b)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 1014, in
log_execute
return self.cursor.execute(*a,**b)
ProgrammingError: relation "none" does not exist
LINE 1: select currval('None')
ron_m
Dec 6, 2010, 9:58:01 PM12/6/10
to web2py-users
I pulled in the latest trunk version of sqlhtml.py and there was an
additional change at line 973 over and above the 2 lines I mentioned
before at 62 and 216
self.table._tablename became self.table.tablename which now causes a
new ticket rendering the login form.
I guess that line needs to be put back to use table._tablename unless
you are planning something else. :-)
additional change at line 973 over and above the 2 lines I mentioned
before at 62 and 216
self.table._tablename became self.table.tablename which now causes a
new ticket rendering the login form.
I guess that line needs to be put back to use table._tablename unless
you are planning something else. :-)
mdipierro
Dec 6, 2010, 10:08:02 PM12/6/10
to web2py-users
I think I fixed the problem with sequence name. Please check.
ron_m
Dec 6, 2010, 10:39:36 PM12/6/10
to web2py-users
Massimo, you nailed it for the sequence name problems. I tested almost
all of my app and got one more ticket.
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/restricted.py", line 188,
in restricted
exec ccode in environment
File "/home/camcentral/Dev/web2py_hg/applications/ccims/views/live/
menu.html", line 118, in <module>
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 4171, in
select
return self.db._adapter.select(self.query,fields,attributes)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 951, in
select
key = self._uri + '/' + query
AttributeError: 'PostgreSQLAdapter' object has no attribute '_uri'
Changed line 951 self._uri to self.uri and I can't make the
application break any more so far. I believe it was my one cached item
in the application.
Ron
all of my app and got one more ticket.
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/restricted.py", line 188,
in restricted
exec ccode in environment
menu.html", line 118, in <module>
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 4171, in
select
return self.db._adapter.select(self.query,fields,attributes)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 951, in
select
key = self._uri + '/' + query
AttributeError: 'PostgreSQLAdapter' object has no attribute '_uri'
Changed line 951 self._uri to self.uri and I can't make the
application break any more so far. I believe it was my one cached item
in the application.
Ron
VP
Dec 6, 2010, 10:59:57 PM12/6/10
to web2py-users
On Dec 6, 6:35 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> Here is some more info about the new DAL
> http://web2py.com/examples/static/new_dal_help.pdf
>
just curious.
ron_m
Dec 6, 2010, 11:29:30 PM12/6/10
to web2py-users
I still have MySQL installed so copied the app, changed database URL,
deleted all files under the applications databases directory, created
empty db, let app create empty tables by running it once to login
screen. I then did the all data from tables at once export from
PostgreSQL (db.export_to_csv_file()) and imported the data to MySQL
using db.import_from_csv_file().
Happy to say it all just worked.
Ron
G. Clifford Williams
Dec 7, 2010, 1:19:59 AM12/7/10
to web...@googlegroups.com
I can donate a couple of FreeBSD jails with various databases setup if you let me know what you want.
--G. Clifford Williams
Sent from some mobile device
--G. Clifford Williams
Sent from some mobile device
mdipierro
Dec 7, 2010, 1:23:33 AM12/7/10
to web2py-users
I got CouchDB working. Denes is working on MongoDB. I think we are
setup for December. In January, you could provide a Cassandra
server. ;-)
massimo
On Dec 7, 12:19 am, "G. Clifford Williams" <g...@notadiscussion.com>
wrote:
setup for December. In January, you could provide a Cassandra
server. ;-)
massimo
On Dec 7, 12:19 am, "G. Clifford Williams" <g...@notadiscussion.com>
wrote:
> I can donate a couple of FreeBSD jails with various databases setup if you let me know what you want.
>
> --G. Clifford Williams
> Sent from some mobile device
>
>
> --G. Clifford Williams
> Sent from some mobile device
>
G. Clifford Williams
Dec 7, 2010, 1:46:34 AM12/7/10
to web...@googlegroups.com
You got it.
BTW, anyone else interested in ZODB support? It's not the new fangled toy that these NoSQL databases are but it's still awesome.
BTW, anyone else interested in ZODB support? It's not the new fangled toy that these NoSQL databases are but it's still awesome.
--G. Clifford Williams
Sent from some mobile device
mdipierro
Dec 7, 2010, 1:53:08 AM12/7/10
to web2py-users
I am but it is hard to fit in the DAL paradigm. I can think of NoSQL
of something a relational database without JOINS and transactions but
ZODB is something more complex. If you have any idea of it could work,
let us know.
Massimo
On Dec 7, 12:46 am, "G. Clifford Williams" <g...@notadiscussion.com>
wrote:
of something a relational database without JOINS and transactions but
ZODB is something more complex. If you have any idea of it could work,
let us know.
Massimo
On Dec 7, 12:46 am, "G. Clifford Williams" <g...@notadiscussion.com>
wrote:
> You got it.
>
> BTW, anyone else interested in ZODB support? It's not the new fangled toy that these NoSQL databases are but it's still awesome.
>
> --G. Clifford Williams
> Sent from some mobile device
>
>
> BTW, anyone else interested in ZODB support? It's not the new fangled toy that these NoSQL databases are but it's still awesome.
>
> --G. Clifford Williams
> Sent from some mobile device
>
ron_m
Dec 7, 2010, 3:23:44 AM12/7/10
to web2py-users
New problem with latest hg pull, update. After filling login form and
submit get a ticket during the insert into auth_event different cause
than before.
The def insert at line 3610 is building a new_fields list but the
db._adapter.insert() called on the return statement at 3621 is
expecting a dict because of the **fields parameter at line 663
At line 3603 the def _insert calls self.db._adapter._insert with
new_field which is not defined. I didn't get a ticket, just noticed it
in the code.
The self._uri should be self.uri problem I mentioned in an earlier
thread is now at line 950 instead of 951
key = self._uri + '/' + query should be key = self.uri + '/' +
query
Do you want me to keep testing. There seems to be a lot of code in
flight that I am not part of developing so maybe I am more of an
annoyance than a help? A lot has changed and at one point it was
running flawless for me. It is a credit to the development that so
much has been added and changed and much of it still works. I am very
interested in either couchdb or mongodb for storing the config portion
of the application I am working on.
Ron
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/restricted.py", line 188,
in restricted
exec ccode in environment
File "/home/camcentral/Dev/web2py_hg/applications/ccims/controllers/
default.py", line 70, in <module>
File "/home/camcentral/Dev/web2py_hg/gluon/globals.py", line 95, in
<lambda>
self._caller = lambda f: f()
File "/home/camcentral/Dev/web2py_hg/applications/ccims/controllers/
default.py", line 44, in user
return dict(form=auth())
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1025, in
__call__
return self.login()
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1516, in
login
self.log_event(log % self.user)
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1268, in
log_event
origin=origin, user_id=user_id)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 3621, in
insert
return self._db._adapter.insert(self, new_fields)
default.py", line 70, in <module>
File "/home/camcentral/Dev/web2py_hg/gluon/globals.py", line 95, in
<lambda>
self._caller = lambda f: f()
File "/home/camcentral/Dev/web2py_hg/applications/ccims/controllers/
default.py", line 44, in user
return dict(form=auth())
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1025, in
__call__
return self.login()
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1516, in
login
self.log_event(log % self.user)
File "/home/camcentral/Dev/web2py_hg/gluon/tools.py", line 1268, in
log_event
origin=origin, user_id=user_id)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 3621, in
insert
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 663, in
insert
query = table._insert(**fields)
TypeError: _insert() argument after ** must be a mapping, not list
Vidul
Dec 7, 2010, 3:46:05 AM12/7/10
to web2py-users
(changeset: 1322:a4378725f3cb):
db.tester.insert(name='name')
sql.pyc in insert(self, **fields)
3619 elif field.required:
3620 raise SyntaxError,'Table: missing required
field: %s'%field
-> 3621 return self._db._adapter.insert(self, new_fields)
3622
3623 def _truncate(self, mode = None):
sql.pyc in insert(self, table, fields)
661
662 def insert(self,table,fields):
--> 663 query = table._insert(**fields)
664 try:
665 self.execute(query)
TypeError: _insert() argument after ** must be a mapping, not list
Probably this error is related to the newest DAL changes.
On Dec 4, 7:32 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> The good news...
>
> I have finished the new DAL for RDBMs. If you now replace gluon/sql.py
> with gluon/dal.py as long as you do not use GAE, everything should
> work fine. All tests pass.
>
> The new dal should have all the features as the old dal but no more.
> It just comes with a better more modular design that will allow adding
> new features.
>
> It defines the following classes:
>
> class ConnectionPool(object):
> class BaseAdapter(ConnectionPool):
>
> # adapters
> class SQLiteAdapter(BaseAdapter):
> class JDBCSQLiteAdapter(SQLiteAdapter):
> class MySQLAdapter(BaseAdapter):
> class PostgreSQLAdapter(BaseAdapter):
> class JDBCPostgreSQLAdapter(PostgreSQLAdapter):
> class OracleAdapter(BaseAdapter):
> class MSSQLAdapter(BaseAdapter):
> class MSSQLAdapter2(MSSQLAdapter):
> class FireBirdAdapter(BaseAdapter):
> class FireBirdEmbeddedAdapter(FireBirdAdapter):
> class InformixAdapter(BaseAdapter):
> class DB2Adapter(BaseAdapter):
> class IngresAdapter(BaseAdapter):
> class IngresUnicodeAdapter(IngresAdapter):
>
> # exposed by DAL
> class Row(dict):
> class SQLCallableList(list):
> class DAL(dict):
> class SQLALL(object):
> class Reference(int):
> class Table(dict):
> class Expression(object):
> class SQLCustomType:
> class Field(Expression):
> class Query(object):
> class Set(object):
> class Rows(object):
>
> TODO:
>
> 1)
> test dal.py
> test dal.py
> test dal.py
>
> 2)
> break gluon/dal.py into
> gluon/dal/__init__.py
> gluon/dal/pooling.py
> gluon/dal/adapter_base.py
> gluon/dal/adapter_*.py
>
> 3)
> reimplement gluon/contrib/gql.py as
> gluon/dal/adapter_nosql.py
> gluon/dal/adapter_gae.py
>
> 4)
> implement
> gluon/dal/adapter_mongodb.py
> etc.
>
> Can you help with 1) and 2)?
>
> Massimo
db.tester.insert(name='name')
sql.pyc in insert(self, **fields)
3619 elif field.required:
3620 raise SyntaxError,'Table: missing required
field: %s'%field
-> 3621 return self._db._adapter.insert(self, new_fields)
3622
3623 def _truncate(self, mode = None):
sql.pyc in insert(self, table, fields)
661
662 def insert(self,table,fields):
--> 663 query = table._insert(**fields)
664 try:
665 self.execute(query)
TypeError: _insert() argument after ** must be a mapping, not list
On Dec 4, 7:32 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> The good news...
>
> I have finished the new DAL for RDBMs. If you now replace gluon/sql.py
> with gluon/dal.py as long as you do not use GAE, everything should
> work fine. All tests pass.
>
> The new dal should have all the features as the old dal but no more.
> It just comes with a better more modular design that will allow adding
> new features.
>
> It defines the following classes:
>
> class ConnectionPool(object):
> class BaseAdapter(ConnectionPool):
>
> # adapters
> class SQLiteAdapter(BaseAdapter):
> class JDBCSQLiteAdapter(SQLiteAdapter):
> class MySQLAdapter(BaseAdapter):
> class PostgreSQLAdapter(BaseAdapter):
> class JDBCPostgreSQLAdapter(PostgreSQLAdapter):
> class OracleAdapter(BaseAdapter):
> class MSSQLAdapter(BaseAdapter):
> class MSSQLAdapter2(MSSQLAdapter):
> class FireBirdAdapter(BaseAdapter):
> class FireBirdEmbeddedAdapter(FireBirdAdapter):
> class InformixAdapter(BaseAdapter):
> class DB2Adapter(BaseAdapter):
> class IngresAdapter(BaseAdapter):
> class IngresUnicodeAdapter(IngresAdapter):
>
> # exposed by DAL
> class Row(dict):
> class SQLCallableList(list):
> class DAL(dict):
> class SQLALL(object):
> class Reference(int):
> class Table(dict):
> class Expression(object):
> class SQLCustomType:
> class Field(Expression):
> class Query(object):
> class Set(object):
> class Rows(object):
>
> TODO:
>
> 1)
> test dal.py
> test dal.py
> test dal.py
>
> 2)
> break gluon/dal.py into
> gluon/dal/__init__.py
> gluon/dal/pooling.py
> gluon/dal/adapter_base.py
> gluon/dal/adapter_*.py
>
> 3)
> reimplement gluon/contrib/gql.py as
> gluon/dal/adapter_nosql.py
> gluon/dal/adapter_gae.py
>
> 4)
> implement
> gluon/dal/adapter_mongodb.py
> etc.
>
> Can you help with 1) and 2)?
>
> Massimo
mdipierro
Dec 7, 2010, 10:08:10 AM12/7/10
to web2py-users
fixed. sorry.
ron_m
Dec 7, 2010, 12:06:20 PM12/7/10
to web2py-users
Perfect now except this ticket as far as my app is concerned testing
against new dal
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/restricted.py", line 188,
in restricted
exec ccode in environment
select
return self.db._adapter.select(self.query,fields,attributes)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 950, in
select
and it works for me.
I will keep testing as you move forward if you like, not a problem for
me to spend some time on it. I have both PostgreSQL and MySQL
installed with a live database modeling the production system I am
building.
Ron
against new dal
Traceback (most recent call last):
File "/home/camcentral/Dev/web2py_hg/gluon/restricted.py", line 188,
in restricted
exec ccode in environment
File "/home/camcentral/Dev/web2py_hg/applications/ccims/views/live/
menu.html", line 118, in <module>
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 4255, in
menu.html", line 118, in <module>
select
return self.db._adapter.select(self.query,fields,attributes)
File "/home/camcentral/Dev/web2py_hg/gluon/sql.py", line 950, in
select
key = self._uri + '/' + query
AttributeError: 'PostgreSQLAdapter' object has no attribute '_uri'
Line 950 replace self._uri with self.uri
and it works for me.
I will keep testing as you move forward if you like, not a problem for
me to spend some time on it. I have both PostgreSQL and MySQL
installed with a live database modeling the production system I am
building.
Ron
mdipierro
Dec 7, 2010, 12:33:15 PM12/7/10
to web2py-users
fixed in trunk
mdipierro
Dec 7, 2010, 12:39:25 PM12/7/10
to web2py-users
I run some benchmarks. The time to do insert and select is the same.
The difference it no measurable because dominated by db IO. Anyway the
code is below and the output is:
old dal:
7.98740386963e-05 (sec)
0.00134269499779
new dal:
8.04572105408e-05
0.00139242005348
Notice all the time is in the IO. The time to actually parse the
complex query is negligible.
code:
import os
os.system('rm *.sqlite test* *.table sql.log')
#from dal import
*
from sql_old import *
db=DAL('sqlite://test.sqlite')
db.define_table('person',Field('name'))
db(db.person).delete()
db.commit()
import time
def t(f,n=1000):
t0=time.time()
for i in range(n): f()
return (time.time()-t0)/n
def a():
db.person.insert(name='max')
def b():
db((db.person.name=='max')&(db.person.name.like('max
%'))&(db.person.name.startswith('m'))).select(limitby=(0,20))
print t(a)
print t(b)
The difference it no measurable because dominated by db IO. Anyway the
code is below and the output is:
old dal:
7.98740386963e-05 (sec)
0.00134269499779
new dal:
8.04572105408e-05
0.00139242005348
Notice all the time is in the IO. The time to actually parse the
complex query is negligible.
code:
import os
os.system('rm *.sqlite test* *.table sql.log')
#from dal import
*
from sql_old import *
db=DAL('sqlite://test.sqlite')
db.define_table('person',Field('name'))
db(db.person).delete()
db.commit()
import time
def t(f,n=1000):
t0=time.time()
for i in range(n): f()
return (time.time()-t0)/n
def a():
db.person.insert(name='max')
def b():
db((db.person.name=='max')&(db.person.name.like('max
%'))&(db.person.name.startswith('m'))).select(limitby=(0,20))
print t(a)
print t(b)
Bruno Rocha
Dec 7, 2010, 3:09:31 PM12/7/10
to web...@googlegroups.com
MongoHQ can be useful for testing, they offer a free limited instance
G. Clifford Williams
Dec 7, 2010, 4:45:08 PM12/7/10
to web...@googlegroups.com
Is VoltDB supported?
Sent from some mobile device
mdipierro
Dec 7, 2010, 7:01:49 PM12/7/10
to web2py-users
No. Do not much about it.
On Dec 7, 3:45 pm, "G. Clifford Williams" <g...@notadiscussion.com>
wrote:
On Dec 7, 3:45 pm, "G. Clifford Williams" <g...@notadiscussion.com>
wrote:
> Is VoltDB supported?
>
> Sent from some mobile device
>
>
> Sent from some mobile device
>
> On Dec 7, 2010, at 2:09 PM, Bruno Rocha <rochacbr...@gmail.com> wrote:
>
> > MongoHQ can be useful for testing, they offer a free limited instance
>
> >https://mongohq.com/pricing
>
> > 2010/12/7 mdipierro <mdipie...@cs.depaul.edu>
>
> > MongoHQ can be useful for testing, they offer a free limited instance
>
> >https://mongohq.com/pricing
>
mdipierro
Dec 8, 2010, 12:21:30 AM12/8/10
to web2py-users
I contacted the developer. They added a license which was missing.
The replacement is now done in trunk.
Vasile, can you please test it?
Massimo
The replacement is now done in trunk.
Vasile, can you please test it?
Massimo
ron_m
Dec 8, 2010, 9:32:43 AM12/8/10
to web2py-users
This is very interesting. I believe this would change the GPL license
reach for MySQL. The regular Python driver is a wrapper around the C
MySQL client library so as a user of that driver you have embedded a
part of MySQL into your application. Using the pure Python driver
means you are now using wire protocol only to get to the MySQL server.
This becomes more like using Apache or Linux for that matter. Now that
I use PostgreSQL I prefer that database but sometimes I run into
situations dictating "Must use database X".
reach for MySQL. The regular Python driver is a wrapper around the C
MySQL client library so as a user of that driver you have embedded a
part of MySQL into your application. Using the pure Python driver
means you are now using wire protocol only to get to the MySQL server.
This becomes more like using Apache or Linux for that matter. Now that
I use PostgreSQL I prefer that database but sometimes I run into
situations dictating "Must use database X".
ron_m
Dec 8, 2010, 10:02:30 AM12/8/10
to web2py-users
I pulled latest version from trunk and tested my app using MySQL and
got a ticket. Easy fix
In sql.py change line 1403 pymysqldb should be pymysql to match the
name from the import
self.pool_connection(lambda db=db,
user=credential_decoder(user),
password=credential_decoder(password),
host=host,
port=port,
1403 ---> charset=charset: pymysql.connect(db=db,
user=user,
passwd=password,
host=host,
port=port,
charset=charset,
))
got a ticket. Easy fix
In sql.py change line 1403 pymysqldb should be pymysql to match the
name from the import
self.pool_connection(lambda db=db,
user=credential_decoder(user),
password=credential_decoder(password),
host=host,
port=port,
1403 ---> charset=charset: pymysql.connect(db=db,
user=user,
passwd=password,
host=host,
port=port,
charset=charset,
))
mdipierro
Dec 8, 2010, 10:07:17 AM12/8/10
to web2py-users
fixed Thanks.
Brian M
Dec 9, 2010, 12:26:55 AM12/9/10
to web...@googlegroups.com
Massimo, executesql() isn't working correctly with the new DAL in trunk. The fix looks to be easy, simply replace self._cursor with self._adaptor.cursor this fixes things for SQLite and I would assume the other DBs as well - I can try testing MS SQL tomorrow.
Fixed code is below, should be right around line 3345...
def executesql(self, query, placeholders=None, as_dict=False):
"""
<snip>docstring removed for brevity</snip>
"""
if placeholders:
self._adapter.execute(query, placeholders)
else:
self._adapter.execute(query)
if as_dict:
if not hasattr(self._adapter.cursor,'description'):
raise RuntimeError, "database does not support executesql(...,as_dict=True)"
# Non-DAL legacy db query, converts cursor results to dict.
# sequence of 7-item sequences. each sequence tells about a column.
# first item is always the field name according to Python Database API specs
columns = self._adapter.cursor.description
# reduce the column info down to just the field names
fields = [f[0] for f in columns]
# will hold our finished resultset in a list
data = self._adapter.cursor.fetchall()
# convert the list for each row into a dictionary so it's
# easier to work with. row['field_name'] rather than row[0]
return [dict(zip(fields,row)) for row in data]
# see if any results returned from database
try:
return self._adapter.cursor.fetchall()
except:
return None
~Brian M
Fixed code is below, should be right around line 3345...
def executesql(self, query, placeholders=None, as_dict=False):
"""
<snip>docstring removed for brevity</snip>
"""
if placeholders:
self._adapter.execute(query, placeholders)
else:
self._adapter.execute(query)
if as_dict:
if not hasattr(self._adapter.cursor,'description'):
raise RuntimeError, "database does not support executesql(...,as_dict=True)"
# Non-DAL legacy db query, converts cursor results to dict.
# sequence of 7-item sequences. each sequence tells about a column.
# first item is always the field name according to Python Database API specs
columns = self._adapter.cursor.description
# reduce the column info down to just the field names
fields = [f[0] for f in columns]
# will hold our finished resultset in a list
data = self._adapter.cursor.fetchall()
# convert the list for each row into a dictionary so it's
# easier to work with. row['field_name'] rather than row[0]
return [dict(zip(fields,row)) for row in data]
# see if any results returned from database
try:
return self._adapter.cursor.fetchall()
except:
return None
~Brian M
mdipierro
Dec 9, 2010, 12:29:57 AM12/9/10
to web2py-users
Your fix is in trunk. Please check it.
Mariano Reingart
Dec 9, 2010, 10:22:38 PM12/9/10
to web...@googlegroups.com
There seems to be a problem with options widget and new dal:
http://code.google.com/p/web2py/issues/detail?id=136
Widget seems to be generating string fields (i.e. 'tipo_cbte.desc,
tipo_cbte.cod'), and dal is specting Expressions.
Best regards,
Mariano Reingart
http://www.sistemasagiles.com.ar
http://reingart.blogspot.com
mart
Dec 10, 2010, 12:03:35 PM12/10/10
to web2py-users
apologies, I may have posted this in the wrong thread. Reposting, just
in case:
when I made the switch, this (just below) was working great. With the
latest code it doesn't. I was able to switch my import statements
"from blablabla.sql import DAL" to "from blablabla.dal import DAL".
Has the syntax changed?
Thanks,
Mart :)
tbl = tableParam
for mTbl in db.tables:
if tbl in mTbl:
if mTbl.startswith(tbl):
rows = db(db[mTbl].name==name).select()
for row in rows:
if row.name == name and row.value > 0:
id = row.id
if id != None:
db(db[tbl].name==name).update(name=name,
value=value)
else:
db[tbl].insert(name=name,
value=value)
in case:
when I made the switch, this (just below) was working great. With the
latest code it doesn't. I was able to switch my import statements
"from blablabla.sql import DAL" to "from blablabla.dal import DAL".
Has the syntax changed?
Thanks,
Mart :)
tbl = tableParam
for mTbl in db.tables:
if tbl in mTbl:
if mTbl.startswith(tbl):
rows = db(db[mTbl].name==name).select()
for row in rows:
if row.name == name and row.value > 0:
id = row.id
if id != None:
db(db[tbl].name==name).update(name=name,
value=value)
else:
db[tbl].insert(name=name,
value=value)
Brian M
Dec 10, 2010, 11:47:48 PM12/10/10
to web...@googlegroups.com
Thanks Massimo, executesql() seems to be working well now for SQLite & MS SQL
Reply all
Reply to author
Forward
0 new messages