I'd like to add some features to TG's pagination and would like some feedback.
The first is SQLAlchemy support. paginate.py uses SQLObject's SelectResults to order the result set. SA has similar functionality in an extension called SelectResults which can be used as a near drop in replacement.
The second is session support. The URI's get long and complex, so if sessions are in use, only the change information has to be passed in the URI and everything else (like the data query params) can be stored in a session. I've currently got it as an option in the decorator params. @paginate(..., use_session=True). One thing I'm not sure about yet is how to keep session data for the DataGrid widget from overwriting another paginating DataGrid instance's session data. Right now I'm using the id (id(func)) of the func passed into the decorator's "decorated" function.
Third. All field sorting. Instead of only sorting by one field, each sorting selection will build on the last.
My current implementation of these features does not change the default behavior. The extra sorting capability is only present when using sessions. Also it includes this bug fix: http://trac.turbogears.org/turbogears/ticket/1044 which I submitted yesterday.
I'd like to hear opinions on these additions a well as requests for other features.
"Randall" <rand...@tnr.cc> writes: > The first is SQLAlchemy support. paginate.py uses SQLObject's > SelectResults to order the result set. SA has similar functionality in > an extension called SelectResults which can be used as a near drop in > replacement.
This will be gladly welcome! I was going to look at this as soon as I could, but I confess that it might take a while for that...
> The second is session support. The URI's get long and complex, so if > sessions are in use, only the change information has to be passed in > the URI and everything else (like the data query params) can be stored > in a session. I've currently got it as an option in the decorator > params. @paginate(..., use_session=True). One thing I'm not sure > about yet is how to keep session data for the DataGrid widget from > overwriting another paginating DataGrid instance's session data. Right > now I'm using the id (id(func)) of the func passed into the decorator's > "decorated" function.
Hmmm... This isn't nice either. I can open two tabs on the same grid to work with it -- e.g. making comparisons -- and it would probably be a problem.
Using this kind of static data when one can have multiple accesses to something doesn't sound like it will work to me. I can't say anything, though, since I haven't yet used sessions here (don't have the need for it).
> Third. All field sorting. Instead of only sorting by one field, each > sorting selection will build on the last.
What do you mean? Sorting by multiple fields? Or clicking a column to sort data? And what happens with data on screen: will it all change or just the current set being viewed? (I mean, if I am seeing records 1-10 sorted by firstname and somehow when sorting by lastname I get 14-5, will it show me 14-5 or will it sort 1-10 using the selected column?)
Jorge Godoy wrote: > > The second is session support. The URI's get long and complex, so if > > sessions are in use, only the change information has to be passed in > > the URI and everything else (like the data query params) can be stored > > in a session. I've currently got it as an option in the decorator > > params. @paginate(..., use_session=True). One thing I'm not sure > > about yet is how to keep session data for the DataGrid widget from > > overwriting another paginating DataGrid instance's session data. Right > > now I'm using the id (id(func)) of the func passed into the decorator's > > "decorated" function.
> Hmmm... This isn't nice either. I can open two tabs on the same grid to work > with it -- e.g. making comparisons -- and it would probably be a problem.
Yea, you're right. Multiple windows or tabs of the same browser use the same session.
> > Third. All field sorting. Instead of only sorting by one field, each > > sorting selection will build on the last.
> What do you mean? Sorting by multiple fields? Or clicking a column to sort > data? And what happens with data on screen: will it all change or just the > current set being viewed? (I mean, if I am seeing records 1-10 sorted by > firstname and somehow when sorting by lastname I get 14-5, will it show me > 14-5 or will it sort 1-10 using the selected column?)
It works like the current implementation, but remembers previous sorts. So if I sort by age, then again by name, the data will be sorted by age then name. The current implementation sorts all data and goes to page one, which seems like reasonable behaviour to me. So from your example, it would show records 1-10. It's a minor change. Do you find it would be better to sort only the currently viewed set? Maybe both would be good.
"Randall" <rand...@tnr.cc> writes: > Yea, you're right. Multiple windows or tabs of the same browser use > the same session.
So this is a problem on your implementation using sessions :-( Too bad... I was liking the idea...
> It works like the current implementation, but remembers previous sorts. > So if I sort by age, then again by name, the data will be sorted by > age then name. The current implementation sorts all data and goes to > page one, which seems like reasonable behaviour to me. So from your > example, it would show records 1-10. It's a minor change. Do you find > it would be better to sort only the currently viewed set? Maybe both > would be good.
Having both would be good. I don't know which one is better, it depends on the application and what users expect from it.
> > Yea, you're right. Multiple windows or tabs of the same browser use > > the same session.
> So this is a problem on your implementation using sessions :-( Too bad... I > was liking the idea...
> > It works like the current implementation, but remembers previous sorts. > > So if I sort by age, then again by name, the data will be sorted by > > age then name. The current implementation sorts all data and goes to > > page one, which seems like reasonable behaviour to me. So from your > > example, it would show records 1-10. It's a minor change. Do you find > > it would be better to sort only the currently viewed set? Maybe both > > would be good.
> Having both would be good. I don't know which one is better, it depends on > the application and what users expect from it.
The current implementation uses the most prevalent approach from what I've seen. The change I'm proposing will go unnoticed by most users, but is quite nice when needed. For an example, look at this grid on a website I built (non-TG).
"Randall" <rand...@tnr.cc> writes: > The current implementation uses the most prevalent approach from what > I've seen. The change I'm proposing will go unnoticed by most users, > but is quite nice when needed. For an example, look at this grid on a > website I built (non-TG).
> If you click on Price, then Rooms, the listings are sorted by rooms > then price. If you don't require the feature, it doesn't have any > negative impact.
I liked that approach. You do what I asked: change the records that appear on the grid instead of just sorting the visible records. This is what makes most sense to me in most cases. But, as I said, some people expect it different... :-)
> I'd like to add some features to TG's pagination and would like some > feedback.
> The first is SQLAlchemy support. paginate.py uses SQLObject's > SelectResults to order the result set. SA has similar > functionality in > an extension called SelectResults which can be used as a near drop in > replacement.
Yeah,we need this. A few months back, we were trying to reconcile the differences between this paging implementation and that of WebHelpers. I think this is a fine solution for now. Pagination also needs docs so that we can declare it as a feature :)
Traceback (most recent call last): File "/usr/local/bin/tg-admin", line 7, in ? sys.exit( File "/usr/lib/python2.4/site-packages/TurboGears-0.9a9-py2.4.egg/turbogears/com mand/base.py", line 351, in main command.run() File "/usr/lib/python2.4/site-packages/TurboGears-0.9a9-py2.4.egg/turbogears/com mand/base.py", line 297, in run root = SecureObject(toolbox.Toolbox(),from_any_host(self.hostlist), exclude=['noaccess']) File "/usr/lib/python2.4/site-packages/TurboGears-0.9a9-py2.4.egg/turbogears/too lbox/base.py", line 95, in __init__ self.toolbox = self.get_tools() File "/usr/lib/python2.4/site-packages/TurboGears-0.9a9-py2.4.egg/turbogears/too lbox/base.py", line 107, in get_tools setattr(self, i.name, tool()) File "/usr/lib/python2.4/site-packages/TurboGears-0.9a9-py2.4.egg/turbogears/too lbox/catwalk/__init__.py", line 112, in __init__ self.model = turbogears.util.get_model() File "/usr/lib/python2.4/site-packages/TurboGears-0.9a9-py2.4.egg/turbogears/uti l.py", line 118, in get_model package = __import__(package_name, {}, {}, ["model"]) File "/home/jose/turbosicer/sicer/model/__init__.py", line 6, in ? from sicer.model import tables File "/home/jose/turbosicer/sicer/model/tables.py", line 7, in ? database.bind_meta_data() File "/usr/lib/python2.4/site-packages/TurboGears-0.9a9-py2.4.egg/turbogears/dat abase.py", line 49, in bind_meta_data get_engine() File "/usr/lib/python2.4/site-packages/TurboGears-0.9a9-py2.4.egg/turbogears/dat abase.py", line 32, in get_engine raise KeyError("No sqlalchemy database config found!") KeyError: 'No sqlalchemy database config found!
I'm currently using PaginateDataGrid to display sortable/pageable SQLAlchemy result sets (which are just lists by default). Had to modify paginate.pyand paginate_datagrid.kid to get it working. Not sure if my solution if the "right" way, but it works. Diffs are attached if anyone cares to look.
A feature I'd like to see added: add a sort_value kw arg to the DataGrid.Column constructor. There are lots of times where the value I want to sort on for a column and the value I actually want to display for that column are different.
On 9/13/06, Kevin Dangoor <dang...@gmail.com> wrote:
> > I'd like to add some features to TG's pagination and would like some > > feedback.
> > The first is SQLAlchemy support. paginate.py uses SQLObject's > > SelectResults to order the result set. SA has similar > > functionality in > > an extension called SelectResults which can be used as a near drop in > > replacement.
> Yeah,we need this. A few months back, we were trying to reconcile the > differences between this paging implementation and that of > WebHelpers. I think this is a fine solution for now. Pagination also > needs docs so that we can declare it as a feature :)
Tim Van Steenburgh wrote: > I'm currently using PaginateDataGrid to display sortable/pageable SQLAlchemy > result sets (which are just lists by default). Had to modify > paginate.pyand paginate_datagrid.kid to get it working. Not sure if > my solution if the > "right" way, but it works. Diffs are attached if anyone cares to look.
Using SelectResults for SQLAlchemy has the advantage of only pulling records from the database that will be displayed and also the sorting is done in the database. If you've got a large dataset (several pages) this can make a difference. In addition, it keeps the code simpler in paginate.py because SQLObject's and SQLAlchemy's SelectResults have the same API. Creating a SelectResults instance is simple.
from sqlalchemy.ext.selectresults import SelectResults sr = SelectResults(query)
If I read your code right, it looks like you added sorting for lists, which is another feature that is needed I think.
There are a few areas I'd like to have someone look over.
1. Line 130 is where I made a fix for the handling of submitted form data. It is passed via URI, but must first be converted from a dict in the case of CompoundWidget data. More info in the docstring. I just wonder if there is a better way to do this.
2. The ordering data is stored as a dict and passed in the URI. Check how I convert it from string to dict in line 261. I'm trying to keep is secure since it is untrusted input.
3. Line 248 - I sure would like to be more explicit when checking for an SQLObject column. InstanceType is quite generic. Any ideas.
I did not make any API changes (that I know of). I don't think there are any existing tests for this module, so I can't verify I didn't break something. I'll have to write tests from scratch, but I wanted to get some feedback on it ASAP.
Is database.py script using a different configuration file instead of dev.cfg ? my dev.cfg is like this one --------------- [global] sqlalchemy.dburi="postgres://pippo:pluto@localhost/mydb" server.environment="development" autoreload.package="gs" [/static] staticFilter.on = True staticFilter.dir = "static" --------------- but database.py doesn't find the token 'sqlalchemy.dburi' and I can't understand why. =-O
26 def get_engine(): 27 "Retreives the engine based on the current configuration" 28 global _engine 29 if not _engine: 30 dburi = config.get("sqlalchemy.dburi") 31 if not dburi: 32 raise KeyError("No sqlalchemy database config found!") 33 _engine = sqlalchemy.create_engine(dburi) 34 metadata.connect(_engine) 35 elif not metadata.is_bound(): 36 metadata.connect(_engine) 37 return _engine
> Is database.py script using a different configuration file instead > of dev.cfg ? > my dev.cfg is like this one > --------------- > [global] > sqlalchemy.dburi="postgres://pippo:pluto@localhost/mydb" > server.environment="development" > autoreload.package="gs" > [/static] > staticFilter.on = True > staticFilter.dir = "static" > --------------- > but database.py doesn't find the token 'sqlalchemy.dburi' and I > can't understand why. =-O
> 26 def get_engine(): > 27 "Retreives the engine based on the current configuration" > 28 global _engine > 29 if not _engine: > 30 dburi = config.get("sqlalchemy.dburi") > 31 if not dburi: > 32 raise KeyError("No sqlalchemy database config > found!") > 33 _engine = sqlalchemy.create_engine(dburi) > 34 metadata.connect(_engine) > 35 elif not metadata.is_bound(): > 36 metadata.connect(_engine) > 37 return _engine
> Jose Soares wrote:
>> Hi all,
>> I'm using sqlalchemy and I have in my dev.cfg the line:
> Is database.py script using a different configuration file instead of dev.cfg ? > my dev.cfg is like this one > --------------- > [global] > sqlalchemy.dburi="postgres://pippo:pluto@localhost/mydb" > server.environment="development" > autoreload.package="gs" > [/static] > staticFilter.on = True > staticFilter.dir = "static" > --------------- > but database.py doesn't find the token 'sqlalchemy.dburi' and I can't understand why. =-O
> 26 def get_engine(): > 27 "Retreives the engine based on the current configuration" > 28 global _engine > 29 if not _engine: > 30 dburi = config.get("sqlalchemy.dburi") > 31 if not dburi: > 32 raise KeyError("No sqlalchemy database config found!") > 33 _engine = sqlalchemy.create_engine(dburi) > 34 metadata.connect(_engine) > 35 elif not metadata.is_bound(): > 36 metadata.connect(_engine) > 37 return _engine
> Jose Soares wrote:
> >Hi all,
> >I'm using sqlalchemy and I have in my dev.cfg the line:
actually. with 1.0b1 you can't even start the toolbox if you are using both SA and identity. forgot to the it to get the sqlalchemy.dburi and friends from dev.cfg at startup. should be fixed now.
On 9/18/06, Kevin Dangoor <dang...@gmail.com> wrote:
Can anyone tell me where things are at with this patch? I have been working on some documentation for paginate ( http://docs.turbogears.org/1.0/PaginateDecorator ) and would like to make sure it stays current.
Jorge Godoy indicated earlier in this thread that he would accept it, though it hasn't been applied yet. The changes I made, if accepted, would only impact your documentation in two ways. First, the addition of SQLAlchemy and second, removal of the 'reversed' attribute.
In the patched version, fields are cummulatively sorted and the sort info is stored in a dict named tg.paginate.ordering of the form: {'fieldname': [0, True]} where the 0 indicates the field position (for multiple sorted fields) and True indicates Ascending sort order (False would be Descending).
SQLAlchemy support uses sqlalchemy.ext.selectresults to mimic the behaviour of the SQLObject SelectResults. It expects a SelectResults instance just as SO does.
"Randall" <rand...@tnr.cc> writes: > Very nice documentation.
> Jorge Godoy indicated earlier in this thread that he would accept it, > though it hasn't been applied yet. The changes I made, if accepted, > would only impact your documentation in two ways. First, the addition > of SQLAlchemy and second, removal of the 'reversed' attribute.
I haven't done anything because I had no time -- yet. And I can't test it -- also "yet". But I'll have to take a look at a stopped pet project soon, and there I'll try it.
If anybody else uses it and had no problems, please say it here.
> If anybody else uses it and had no problems, please say it here.
> I just don't want to break anything. :-)
Agreed. The absence of tg.paginate.reversed could break someone's app, but I hesitate to put it in because it would be cruft. On the other hand, API changes shouldn't occur during beta. Suggestions?
That brings me to another subject, the interface to ordering. The ordering attribute is what tracks the ordering of all the fields. For three fields, it may look like this: {'fieldone':[0, True], 'fieldtwo':[1, False], 'fieldthree':[2, True]} The values in the dictionary contain the field order and a boolean for Ascending order. This structure was designed for easy internal usage, not as a public API. Should the field ordering be exposed in some other fashion?
For SQLAlchemy support, it accepts a SelectResults instance which is very similar to SQLObject's SelectResults. I'm wondering if maybe it should accept a query instance instead and create the SelectResults instance internally. That might save the developer a little trouble. So it would go from this:
from sqlalchemy.ext.selectresults import SelectResults paginate_input = SelectResults(tg_session.query(MySAClass))
to this:
paginate_input = tg_session.query(MySAClass)
Note that I'm using tg_session (turbogears.databse.session) because I'm not using ActiveMapper classes with a default session.
What do you think? I'm starting to prefer the latter. That way if SelectResults becomes a problem later (not maintained, feature limited, etc.), paginate can use a different approach internally without breaking the API.
So I'm leaning towards:
1. Making a compatible reversed attribute. 2. Requiring only a SA query (not a SelectResults instance).
