Turbogears now based on Pylons!
Spider
Jonathan LaCour writing at http://cleverdevil.org/ says :
"This past weekend, Mark Ramm and I held a sprint to experiment on
implementing a TurboGears like API on top of the excellent Pylons web
framework. The sprint was hugely successful, and as a result,
TurboGears 2.0 will be based upon our work."
He links to a longer article by Mark Ramm at http://compoundthinking.com/blog/
which describes the work that they did.
I think this reflects very positively on the WSGI-based approach of
Pylons.
Would the Pylons core developers care to comment?
rikl...@gmail.com
Django 2.0 too ? ;) :D
Jonathan LaCour
> I think this reflects very positively on the WSGI-based approach of
> Pylons.
We do too! TurboGears started a bit before WSGI really took hold, and
as we all got further along, it seemed clear to most of us that we
really needed to latch onto WSGI and build it into the foundation. The
more we looked, the more we realized that Pylons was so very similar to
TurboGears, with a few differences, except it already had WSGI at its
core. Putting WSGI at the core of TurboGears after the fact would be
much harder than just making a thin-layer on top of Pylons.
Plus, now we get all the great benefits of Pylons itself, and get to
participate in the great community that is growing around WSGI and
Pylons.
Very exciting stuff!
--
Jonathan LaCour
http://cleverdevil.org
anderbubble
almost a year now, and had been frustrated with the stagnation in
development and the myriad of Python web frameworks. I'm glad to see
two of the top contenders combining, rather than duplicating, their
efforts.
Keep up the good work, both of you! Best of luck.
~jon
On Jun 27, 10:10 am, Jonathan LaCour <jonathan-li...@cleverdevil.org>
wrote:
Mark Ramm
> almost a year now, and had been frustrated with the stagnation in
> development and the myriad of Python web frameworks. I'm glad to see
> two of the top contenders combining, rather than duplicating, their
> efforts.
Pylons really is a good tool for building web framework type stuff.
We were able to get a lot done very quickly, and we're very excited
about the benefits of closer collaboration, and we hope that we're not
the only ones who reallize how usefull it is to have a solid, stable,
core and to innovate on top of that.
I'm looking forward to our new pylons powered future.
--Mark
Mike Orr
Reminds me of ZDjangoGears and the joke that ultimately became Parrot.
This is great news for Python web interpoerability! Pylons users
will benefit from having access to a TG-style dispatcher and
decorators for validation, JSON, etc., without having to make a
wholesale choice between one framework or the other. I do think
Django is on a dead-end road with its "not invented here" structure,
and eventually it will have to become more interoperable and
component-sharing somehow. But that's up to the Django developers to
decide.
Mark and John, the two things I'm most concerned about are the
SQLAlchemy and templating interfaces, both of which are in flux in
Pylons. I just want to make sure we're on the same page as far as
their direction goes. Have you seen my work on SAContext and
Smorgasbord, and do they look like something TG can build upon? What
compatibility issues do you see, and how much compatibility with TG 1
are you willing to break?
SAContext has been performing well and I think it's close to
stability. It hasn't been officially accepted into Pylons yet but
there's consensus that pylons.templating is inadequate and I haven't
seen any other proposal. It's proven trivial to drop SAContext into
an existing Pylons application (one variable definition and two
search-and-replaces) so I expect the same will be true for TG
applications. I'm not sure whether the module will be called
pylons.database or pylons.database.sqlalchemy or something like that,
but that'll be just a matter of changing imports.
Templating is a bigger issue so I'll start a separate thread for that.
--
Mike Orr <slugg...@gmail.com>
Mark Ramm
> an existing Pylons application (one variable definition and two
> search-and-replaces) so I expect the same will be true for TG
> applications.
Yea, Jonathan has stuff to talk to you about. I think SAContext
could be pretty easily modified to meet his needs. (He wants to share
models between Pylons and other command line tools), but I think
that's a use case we should try to address if we can.
> I'm not sure whether the module will be called
> pylons.database or pylons.database.sqlalchemy or something like that,
> but that'll be just a matter of changing imports.
I had this crazy idea this morning when talking to Jonathan. The idea
would be to create a more generic database interface go in (perhaps on
top of SA Context.)
I'd like there to be an entry point for database "engines" (SA, SO,
DejaVu, or whatever) that offer a very minimal set of functions in a
database agnostic way.
* connect
* start_transaction
* end_transaction
* rollback
This would make it possible for Evelind or others to write a DejaVu
engine for Pylons/TG that just works when installed. Actual database
manipulations would happen in application code in whatever way users
want and would not touch any of these functions. But we can call
connect to initalize the engine, and we can call the other methods
from a new generic "transaction middleware" component that we need to
write for TG on pylons.
But Jonathan has more details on the actual use case for this than I
do. So, hopefully he'll chime in on this too.
Ian Bicking
> I'd like there to be an entry point for database "engines" (SA, SO,
> DejaVu, or whatever) that offer a very minimal set of functions in a
> database agnostic way.
>
> * connect
> * start_transaction
> * end_transaction
> * rollback
Personally I'd like to see something a bit different:
* Way for a library to get "the" transaction manager.
* Interface for that transaction manager, maybe copied from Zope.
* Single convention for how to specify a database connection (ideally
string-based).
* Probably a way to get "the" configuration, so you can lazily get a
connection based on the configured connection.
The frameworks would setup the transaction manager and configuration,
and database frameworks would basically consume this information.
--
Ian Bicking | ia...@colorstudy.com | http://blog.ianbicking.org
| Write code, do good | http://topp.openplans.org/careers
Jonathan LaCour
> Personally I'd like to see something a bit different:
>
> * Way for a library to get "the" transaction manager.
> * Interface for that transaction manager, maybe copied from Zope.
> * Single convention for how to specify a database connection (ideally
> string-based).
> * Probably a way to get "the" configuration, so you can lazily get
> a connection based on the configured connection.
>
> The frameworks would setup the transaction manager and configuration,
> and database frameworks would basically consume this information.
This would be pretty much ideal, and would satisfy my use cases very
well. It would also allow framework authors to build in support for any
ORM / database layer in an agnostic way, sort of like the Buffet API
does for templating systems.
Then, we could also create a simple WSGI middleware for Pylons that
gives it TurboGears-like automatic transaction start/commit/rollback
on a per request basis. Only, we could make it configurable so that
it didn't happen on read-only operations. All of this would be ORM /
database layer agnostic, which would be very nice indeed.
The big issue for me with SAContext right now is that it uses a
BoundMetaData and expects you to know your dburi "up-front" as it
were. I don't like having my model tied to my configuration system.
I'd rather have my model fetch its configuration from some third party
source, such as this new package we are discussing, so that I can
create desktop applications, command line utilities, etc. that share my
pylons application's model, without having to ship pylons or a pylons
configuration file. The last thing that Ian requests here would let me
do that to a certain extent:
from db_buffet_api import config
from sacontext import SAContext
sac = SAContext(dburi=config.dburi)
So, I am a big +1 on such a beast.
Noah Gift
--
Jonathan LaCour
http://cleverdevil.org
Ian Bicking
> Ian Bicking wrote:
>
>> Personally I'd like to see something a bit different:
>>
>> * Way for a library to get "the" transaction manager.
>> * Interface for that transaction manager, maybe copied from Zope.
>> * Single convention for how to specify a database connection (ideally
>> string-based).
>> * Probably a way to get "the" configuration, so you can lazily get
>> a connection based on the configured connection.
>>
>> The frameworks would setup the transaction manager and configuration,
>> and database frameworks would basically consume this information.
>
> This would be pretty much ideal, and would satisfy my use cases very
> well. It would also allow framework authors to build in support for any
> ORM / database layer in an agnostic way, sort of like the Buffet API
> does for templating systems.
>
> Then, we could also create a simple WSGI middleware for Pylons that
> gives it TurboGears-like automatic transaction start/commit/rollback
> on a per request basis. Only, we could make it configurable so that
> it didn't happen on read-only operations. All of this would be ORM /
> database layer agnostic, which would be very nice indeed.
The way I see this working is something like (vaguely):
def transaction_middleware(app):
def wrapper(environ, start_response):
manager = TransactionManager()
environ['transaction.manager'] = manager
try:
app_iter = app(environ, start_response)
except:
manager.rollback()
raise
else:
manager.commit()
The manager is basically a container of *actual* transactions, and
calling rollback or commit on it gets passed on to all the transactions
in the manager.
If you don't do anything that needs a transaction (e.g., read-only), you
shouldn't put your transaction in the manager.
> The big issue for me with SAContext right now is that it uses a
> BoundMetaData and expects you to know your dburi "up-front" as it
> were. I don't like having my model tied to my configuration system.
> I'd rather have my model fetch its configuration from some third party
> source, such as this new package we are discussing, so that I can
> create desktop applications, command line utilities, etc. that share my
> pylons application's model, without having to ship pylons or a pylons
> configuration file. The last thing that Ian requests here would let me
> do that to a certain extent:
>
> from db_buffet_api import config
> from sacontext import SAContext
>
> sac = SAContext(dburi=config.dburi)
>
> So, I am a big +1 on such a beast.
I'd kind of like a way of getting the "current" WSGI environment. Then
one possible implementation of this config-getter is:
def get_dburi():
environ = get_environ()
return environ['paste.config']['dburi']
Or it could get it out of another key, of course. And in a non-web
context you have a different get_dburi() implementation. The only
annoying part is actually figuring out how you get that function, and
how you provide that function.
The way they do it in Zope, which is similar in ways to Paste's
StackedObjectProxy and paste.registry, is basically something like:
dburi = getUtility('get_dburi')()
Except in Zope they use an interface instead of 'get_dburi'. But I
think we should use a string.
We might want to look at PEAK's contextual stuff, as this is basically
addressing the same problem. I believe Phillip recently extracted that
from PEAK. (And all this config stuff is exactly the same issue as
getting the current transaction manager.) Personally I'm not terribly
comfortable with paste.registry and StackedObjectProxy, as I feel it
pretends to be more transparent than it really is; I prefer something
more explicit like getUtility().
Mike Orr
> Yea, Jonathan has stuff to talk to you about. I think SAContext
> could be pretty easily modified to meet his needs. (He wants to share
> models between Pylons and other command line tools), but I think
> that's a use case we should try to address if we can.
That's a good idea. I also will increasingly need to access my web
applications' data from command-line tools, such as usage-report
generators and quasi-static webpages.
> > I'm not sure whether the module will be called
> > pylons.database or pylons.database.sqlalchemy or something like that,
> > but that'll be just a matter of changing imports.
>
> I had this crazy idea this morning when talking to Jonathan. The idea
> would be to create a more generic database interface go in (perhaps on
> top of SA Context.)
It sounds a bit abstract for me to work on, but let me know if it will
require changes to SAContext. I envision SAContext can be a "plugin"
to this system. I also use Durus so I'd like to get it working with
pylons.templating autoconfiguration at some point. I have an idea
even crazier than yours: a SQLAlchemy engine for Durus, which may
support only some SQLAlchemy features but would at least explore the
possibility of managing a SQL and non-SQL database from a common front
end, for possible use in interoperability or database-copying
scenarios. But it's still just an idea.
> I'd like there to be an entry point for database "engines" (SA, SO,
> DejaVu, or whatever) that offer a very minimal set of functions in a
> database agnostic way.
>
> * connect
> * start_transaction
> * end_transaction
> * rollback
>
> This would make it possible for Evelind or others to write a DejaVu
> engine for Pylons/TG that just works when installed. Actual database
> manipulations would happen in application code in whatever way users
> want and would not touch any of these functions. But we can call
> connect to initalize the engine, and we can call the other methods
> from a new generic "transaction middleware" component that we need to
> write for TG on pylons.
I hadn't heard of DejaVu until now. One issue with SQLAlchemy is
connect/rollback are buried in the SQL and ORM APIs, so I'm not sure
how useful it would be to shoehorn them into this API which is "out of
band" from the controller code. Is this a full-request transaction
the controller is expected to use? Can we assume that every
controller has no more than one transaction, and that rollback always
corresponds to certain HTTP status codes?
--
Mike Orr <slugg...@gmail.com>
Jonathan LaCour
> The way I see this working is something like (vaguely):
>
> def transaction_middleware(app):
> def wrapper(environ, start_response):
> manager = TransactionManager()
> environ['transaction.manager'] = manager
> try:
> app_iter = app(environ, start_response)
> except:
> manager.rollback()
> raise
> else:
> manager.commit()
>
> The manager is basically a container of *actual* transactions,
> and calling rollback or commit on it gets passed on to all the
> transactions in the manager.
>
> If you don't do anything that needs a transaction (e.g., read-only),
> you shouldn't put your transaction in the manager.
Okay, I am mostly with you, but then you end up with a lot of
boilerplate elsewhere wherever you start a transaction and throw it
into the manager. I think we can address this in the TurboGears pylons
template somehow and automatically start a transaction and put it into
the manager on request by request basis, and provide some way to disable
it for read-only requests.
There have also been discussions of allowing you to turn it off for
specific HTTP methods, so you would never have a transaction for GET
unless you created it yourself, but POST, DELETE, etc. would usually
have transactions. This feels a touch too magical to me, since the
whole point of the automatic transaction-per-request in TurboGears was
to make things easy by default, and not difficult to understand.
I wouldn't have a problem if there were two separate pieces of
middleware though: one for rolling back active transactions on
exceptions, and another for setting up when you want those transactions
to be created automatically.
> The last thing that Ian requests here would let me do that to a
> certain extent:
>
>> from db_buffet_api import config
>> from sacontext import SAContext
>>
>> sac = SAContext(dburi=config.dburi)
>
> I'd kind of like a way of getting the "current" WSGI environment.
> Then one possible implementation of this config-getter is:
>
> def get_dburi():
> environ = get_environ()
> return environ['paste.config']['dburi']
>
> Or it could get it out of another key, of course. And in a non-web
> context you have a different get_dburi() implementation. The only
> annoying part is actually figuring out how you get that function, and
> how you provide that function.
This database layer is acting as a middleman between the configuration
system of the framework (Pylons, etc.) and the model itself, which may
not live inside the framework. Why not just have the web framework tell
the middleman how to get the dburi from the configuration, and the model
can ask for the dburi from the middleman?
> The way they do it in Zope, which is similar in ways to Paste's
> StackedObjectProxy and paste.registry, is basically something like:
>
> dburi = getUtility('get_dburi')()
>
> Except in Zope they use an interface instead of 'get_dburi'. But I
> think we should use a string.
Agreed, we should use a string.
> We might want to look at PEAK's contextual stuff, as this is basically
> addressing the same problem. I believe Phillip recently extracted
> that from PEAK. (And all this config stuff is exactly the same issue
> as getting the current transaction manager.) Personally I'm not
> terribly comfortable with paste.registry and StackedObjectProxy, as I
> feel it pretends to be more transparent than it really is; I prefer
> something more explicit like getUtility().
I'll take a look at the stuff in PEAK, but it usually breaks my brain
for at least two days before I finally grok it. But, to be fair,
StackedObjectProxy does the same thing ;)
How should we move forward with this? Is this the type of thing that
the DB-SIG cares about? Or should the discussion remain here?
Mike Orr
>
> > Personally I'd like to see something a bit different:
> >
> > * Way for a library to get "the" transaction manager.
> > * Interface for that transaction manager, maybe copied from Zope.
> > * Single convention for how to specify a database connection (ideally
> > string-based).
> > * Probably a way to get "the" configuration, so you can lazily get
> > a connection based on the configured connection.
> >
> > The frameworks would setup the transaction manager and configuration,
> > and database frameworks would basically consume this information.
>
> This would be pretty much ideal, and would satisfy my use cases very
> well. It would also allow framework authors to build in support for any
> ORM / database layer in an agnostic way, sort of like the Buffet API
> does for templating systems.
>
> Then, we could also create a simple WSGI middleware for Pylons that
> gives it TurboGears-like automatic transaction start/commit/rollback
> on a per request basis. Only, we could make it configurable so that
> it didn't happen on read-only operations. All of this would be ORM /
> database layer agnostic, which would be very nice indeed.
All this is to avoid a try-commit block in the controller method or
model function? Is it really that important? Is this what Noah means
by "controlling transactions"?
The normal Pylons strategy is to clear the SQLAlchemy session in the
base controller before calling the action method. This is effectively
a rollback, but since all the changes are in memory and haven't been
compiled to SQL yet, no rollback is necessary. Users do
sac.session.flush() to write their changes to the database, which is
essentially a commit. I'm more comfortable with that happening
explicitly in the controller method rather than vaguely in some
middleware (especially since the application is not supposed to know
or care whether a certain middleware is active).
As for direct SQL statements in SQLAlchemy, I just assume they're
autocommit. Maybe someday I'll have to get more sophistocated about
them.
> The big issue for me with SAContext right now is that it uses a
> BoundMetaData and expects you to know your dburi "up-front" as it
> were. I don't like having my model tied to my configuration system.
> I'd rather have my model fetch its configuration from some third party
> source, such as this new package we are discussing, so that I can
> create desktop applications, command line utilities, etc. that share my
> pylons application's model, without having to ship pylons or a pylons
> configuration file. The last thing that Ian requests here would let me
> do that to a certain extent:
>
> from db_buffet_api import config
> from sacontext import SAContext
>
> sac = SAContext(dburi=config.dburi)
If that's all you want, it already does this. SAContext requires an
explicit URI and options as arguments, and doesn't know about
anybody's configuration system. My hope is that it will be included
in SQLAlchemy at some point, for many diverse applications.
PylonsSAContext addresses the needs Pylons users have right now: a
standard way to configure an engine/metadata/session that's better
than pylons.database.
The overriding concerns are:
- Correctness per SQLAlchemy's recommendation (which favors
BoundMetaData, even though the SQLAlchemy manual hasn't gone that far
yet).
- Easy for users to put into their model (just a couple short lines of code).
- Automatically reads all engine options from the config file that can
be specified in a scalar format.
- Accepts other engine options from the developer which can't be
expressed in a scalar format.
If your concept requires something different than PylonsSAContext,
perhaps a different subclass, there's no reason the two can't live
side by side.
The model is "tied" to the configuration such that you can't import it
standalone, at least if you follow the SAC_Demo example. Instead you
have to use the module under "paster shell", "paster setup-app", or
manually prepare the environment for the import (which is not well
documented but websetup.py gives some hints). This is *no worse* than
existing Pylons usage, which also has the same limitation. We have
thought long about how to improve this but have not come up with a
better way. Some people put all their tables in an init function in
the model so that it can be called sometime after the actual import,
and with arguments. This is a little better in some ways but still
not a robust solution. Do you set global variables for the tables or
pass them back as the return value? Either way is messy.
Of course the engine must be initialized before any tables can be
defined or any work done. SAContext initializes the default engine in
the constructor. Is there a need to push that back to some later
stage? Which stage? Or can Ian's ideal wrapper simply delay creating
the 'sac' until it has to?
--
Mike Orr <slugg...@gmail.com>
Mike Orr
> boilerplate elsewhere wherever you start a transaction and throw it
> into the manager. I think we can address this in the TurboGears pylons
> template somehow and automatically start a transaction and put it into
> the manager on request by request basis, and provide some way to disable
> it for read-only requests.
>
> There have also been discussions of allowing you to turn it off for
> specific HTTP methods, so you would never have a transaction for GET
> unless you created it yourself, but POST, DELETE, etc. would usually
> have transactions. This feels a touch too magical to me, since the
> whole point of the automatic transaction-per-request in TurboGears was
> to make things easy by default, and not difficult to understand.
>
> I wouldn't have a problem if there were two separate pieces of
> middleware though: one for rolling back active transactions on
> exceptions, and another for setting up when you want those transactions
> to be created automatically.
It sounds like Pylons and TurboGears have very different paradigms
about how transactions are handled. Could somebody explain them side
by side, showing how the controller method interacts with the
middleware under various read-only, write-commit, write-rollback
scenarios? Especially taking into account the various reasons one
might want to roll back. As a starting point, i tried to explain the
typical Pylons usage in my previous email.
Then we can see whether SAContext or something else can meet all the needs.
--
Mike Orr <slugg...@gmail.com>
Mark Ramm
> about how transactions are handled.
I can't do a side-by-side comparison, because I am not 100% clear on
an example of the "right way" to handle transactions in Pylons.
But I can describe what TurboGears 1.0 does.
TG currently has an "automatic transaction per request" feature, which
is very widely used.
If a request fails for any reason, at any point the exception is
propigated out, and causes a transaction rollback.
We want to set up the transaction as early as possible. So that if
you have validators which hit the DB (which many do) and you want that
wrapped in a transaction that is done for you. And this is important
because you can't do that in the controller method itself since
validation happens in the decorator -- before you enter the controller
method itself. And we want the transaction closed as late as possible
so that if your internationalization fails or some other post
controller action fails, you can rollback the transaction.
Of course the problem with the current implementation of this feature
is that all kinds of pages which don't actually change anything get
transaction overhead that they don't need. So we are looking for the
best way to control transactions on a per controller basis.
--Mark Ramm
Uwe C. Schroeder
And on that note: if you're using SA with TG, SA issues a rollback on every
transaction that is not an insert or update. So if you're having a stored
procedure (which you trigger with "select * from stored_proc()" and that
stored procedure actually does updates or inserts, you're going to lose
changes - simply because SA issues a rollback on Select statements.
Quick fix for this is to modify SA to just issue a commit on every statement,
so the TG transaction can roll back or commit without being affected. IMHO
issuing a commit on a select shouldn't be more overhead than issuing a
rollback - because the db should know what to do (in this case nothing)
Maybe something to think about too, because I can't be the only one making
heavy use of stored procedures (which are far more efficient than
controller/model side code)
Uwe
--
Open Source Solutions 4U, LLC 1618 Kelly St
Phone: +1 707 568 3056 Santa Rosa, CA 95401
Cell: +1 650 302 2405 United States
Fax: +1 707 568 6416
Mike Orr
> > about how transactions are handled.
>
> I can't do a side-by-side comparison, because I am not 100% clear on
> an example of the "right way" to handle transactions in Pylons.
>
> But I can describe what TurboGears 1.0 does.
>
> TG currently has an "automatic transaction per request" feature, which
> is very widely used.
>
> If a request fails for any reason, at any point the exception is
> propigated out, and causes a transaction rollback.
The base controller's .__call__ method is your friend (but see below).
Wrap the subclass call in a try-commit, You can also use the
.__before__ and .__after__ methods unless you want to leave those
empty for users. This avoids the need for fragile middleware and
fussing with the WSGI environment.
> We want to set up the transaction as early as possible. So that if
> you have validators which hit the DB (which many do) and you want that
> wrapped in a transaction that is done for you. And this is important
> because you can't do that in the controller method itself since
> validation happens in the decorator -- before you enter the controller
> method itself. And we want the transaction closed as late as possible
> so that if your internationalization fails or some other post
> controller action fails, you can rollback the transaction.
If you find that the base .__call__ is too late, you can ask Ben
nicely for a hook method that wraps around the dispatched method call.
> Of course the problem with the current implementation of this feature
> is that all kinds of pages which don't actually change anything get
> transaction overhead that they don't need. So we are looking for the
> best way to control transactions on a per controller basis.
Would a flag variable in a Routes rule work for this? One with a
default value that means "this URL doesn't need a transaction".
How is TG handling SQLAlchemy transactions now; i.e., which SQLAlchemy
methods does it call? There's a transaction-centric pattern but I
think it only works with SQL calls, not ORM use.
--
Mike Orr <slugg...@gmail.com>
Michael Bayer
>
> I hadn't heard of DejaVu until now. One issue with SQLAlchemy is
> connect/rollback are buried in the SQL and ORM APIs, so I'm not sure
> how useful it would be to shoehorn them into this API which is "out of
> band" from the controller code.
just to clarify, not really. the entire point of the "explicit
connection" API, which 99% of the time people either arent aware of,
or find it to be "too verbose" when they see it, is that the
"transaction" going on can be completely extracted to be external to
everything SQLAlchemy. if we are looking for an ORM-agnostic API that
just wraps all internal operations in a transaction, the SAContext
need not even be involved, youd use SessionTransaction against its
current session.
Also SAContext has no dependence on "bound" metadata as it supports
alternative strategies which define the usage of engines, and that it
takes the database URI in its constructor is just a small
implementation detail thats easily changed. "Bound" metadata in its
old sense no longer exists in SQLAlchemy, as all "metadata" objects
can have a new Engine assigned to them at any time (or not at all).
Is this a full-request transaction
> the controller is expected to use? Can we assume that every
> controller has no more than one transaction, and that rollback always
> corresponds to certain HTTP status codes?
it seems we are talking about just the case where, "the entire request
is inside of a transaction", and it seems like this is mainly for the
purposes of WSGI middleware...having any fine-grained access to the
ORM in an agnostic way (i.e. anything beyond "define connection
parameters" and "transaction boundaries") would be unworkable.
Michael Bayer
On Jun 27, 8:24 pm, "Uwe C. Schroeder" <u...@oss4u.com> wrote:
>
> And on that note: if you're using SA with TG, SA issues a rollback on every
> transaction that is not an insert or update. So if you're having a stored
> procedure (which you trigger with "select * from stored_proc()" and that
> stored procedure actually does updates or inserts, you're going to lose
> changes - simply because SA issues a rollback on Select statements.
this issue can be worked around by using explicit transactions. i
also have a notion of SQL functions being marked as "transactional" to
help this issue. but this thread so far seems to be about the notion
of an entire request being marked as "transactional", which as it
turns out is a central concept of J2EE and others, which would also
eliminate the issue youre having.
> Quick fix for this is to modify SA to just issue a commit on every statement,
> so the TG transaction can roll back or commit without being affected. IMHO
> issuing a commit on a select shouldn't be more overhead than issuing a
> rollback - because the db should know what to do (in this case nothing)
this would be something that would need to be benchmarked. i do think
it would add some overhead. but beyond that, i dont like the idea of
unnecessary COMMITs for every SELECT statement at all.
>
> Maybe something to think about too, because I can't be the only one making
> heavy use of stored procedures (which are far more efficient than
> controller/model side code)
that a stored-procedure-oriented application is "far more efficient"
is *extremely* debatable and database-dependent as well. if you
really want COMMIT for every SELECT, i'd favor it being enabled via an
option passed to create_engine(). Beyond that I think the "model
implemented as stored procedure" style of development is much in the
minority these days, particularly within the "lightweight/open-source"
development community. just that the stored procedure changes the
semantics of SELECT to be a "write" operation reminds me of the
RESTful sin of using GET to post data.
Michael Bayer
On Jun 27, 6:42 pm, Ian Bicking <i...@colorstudy.com> wrote:
> The way I see this working is something like (vaguely):
>
> def transaction_middleware(app):
> def wrapper(environ, start_response):
> manager = TransactionManager()
> environ['transaction.manager'] = manager
> try:
> app_iter = app(environ, start_response)
> except:
> manager.rollback()
> raise
> else:
> manager.commit()
>
> The manager is basically a container of *actual* transactions, and
> calling rollback or commit on it gets passed on to all the transactions
> in the manager.
this is fine. we're *really* starting to imitate J2EE in some ways.
but its not a bad thing.
>
> If you don't do anything that needs a transaction (e.g., read-only), you
> shouldn't put your transaction in the manager.
just to clarify, however this works, it should be *really easy* for
individual controller methods to be marked as "transactional" or not.
I have written decorators like these already, in the style of:
class MyController(...):
def index_page(self):
.....
@transactional
def post_message(self):
.....
def display_message(self):
.....
so id want that kind of thing to be readily available, i suppose it
would communicate with the WSGI middleware on a per-request basis.
but another thing i like, is that of the response to the request being
delivered *after* the transaction commits. i have raised this issue
on the pylons list before but i dont think i impressed anyone. if you
render your response template, then your transaction fails, you have
the potential for that response template to still be delivered with
the "success!" message. anyway, might be nice for the middleware to
address this if possible.
also i think this is all outside of the scope of SAContext. SAContext
should remain as a facade to SA-specific elements; it can accept
"strategy" objects which control its internal workings, so if need be
particular strategies can be delivered in order to function with the
transaction manager, without changing the external view of SAContext.
Michael Bayer
On Jun 27, 7:44 pm, "Mike Orr" <sluggos...@gmail.com> wrote:
> The normal Pylons strategy is to clear the SQLAlchemy session in the
> base controller before calling the action method. This is effectively
> a rollback, but since all the changes are in memory and haven't been
> compiled to SQL yet, no rollback is necessary.
theres a rollback() going on ; SQLAlchemy's connection pool issues a
rollback() on all connections before theyre returned to the pool.
this is to release any locks that might be present on the connection
(and its definitely quite necessary...unless we went with the COMMIT
on every statement/connection returned idea someone has mentioned).
> Of course the engine must be initialized before any tables can be
> defined or any work done.
since ive seen people get the wrong idea about this, let me
reiterate: *you can define all your tables and mappers with no
database connection defined whatsoever*, and you can even start
creating mapped objects. SQLAlchemy does not care a whiff about your
database connection until the moment you tell it to execute a SQL
statement...and when the statement is done, it goes back to not caring
at all about your database connection; you can tear it down or modify
it to be something else all you like.
> SAContext initializes the default engine in
> the constructor. Is there a need to push that back to some later
> stage? Which stage? Or can Ian's ideal wrapper simply delay creating
> the 'sac' until it has to?
easy enough to make "uri" optional in SAContext (the way it is for
everything else in SA) and just allow binding it later via
add_engine() or whatever. SAContext is designed the way it is right
now because Pylons defines the default engine in the .ini file; theres
no need for a "delayed" connection in the usual case. im not exactly
sure why Jonathan thinks that the "url" forces one to "define my
connection up front". if youre saying, "i want to distribute the
application without an .ini file" thats fine...but when the thing is
actually *used*, there will be an .ini file or other configuration
defined when things run. if you want to define your model
independently of connection, just use an empty MetaData. Set up the
SAContext to bind the engine or individual connections to the session
instead...or give it your MetaData for it to bind when youre ready.
theres really nothing to it.
Uwe C. Schroeder
On Wednesday 27 June 2007, Michael Bayer wrote:
> On Jun 27, 8:24 pm, "Uwe C. Schroeder" <u...@oss4u.com> wrote:
> > And on that note: if you're using SA with TG, SA issues a rollback on
> > every transaction that is not an insert or update. So if you're having a
> > stored procedure (which you trigger with "select * from stored_proc()"
> > and that stored procedure actually does updates or inserts, you're going
> > to lose changes - simply because SA issues a rollback on Select
> > statements.
>
> this issue can be worked around by using explicit transactions.
actually no, it can't. Maybe I don't get it right, but the only way for me to
get a commit was actually to modify Connection._autocommit in
sqlalchemy.engine.base. Obviously SA thinks there is no transaction in TG,
so it just wraps one around it. After I got tired of searching for the
problem I just added SELECT to the above method and now get my commit.
I'm sure either I do something wrong or there's a bug in the db implementation
of TG. All I could find is that the transaction (since 1.0.2 available in
tg.sa_transaction) is just a subtransaction from the autocommit code - thus
when the outer transaction issues a rollback the inner transaction that was
committed will be rolled back too.
> i also have a notion of SQL functions being marked as "transactional" to
> help this issue. but this thread so far seems to be about the notion
> of an entire request being marked as "transactional", which as it
> turns out is a central concept of J2EE and others, which would also
> eliminate the issue youre having.
Yes - see below on the "every select statement".
>
> > Quick fix for this is to modify SA to just issue a commit on every
> > statement, so the TG transaction can roll back or commit without being
> > affected. IMHO issuing a commit on a select shouldn't be more overhead
> > than issuing a rollback - because the db should know what to do (in this
> > case nothing)
>
> this would be something that would need to be benchmarked. i do think
> it would add some overhead. but beyond that, i dont like the idea of
> unnecessary COMMITs for every SELECT statement at all.
I agree, and that is certainly DB dependent. Personally I can't imagine that
an automatically issued rollback for every select transaction is in any way
more overhead than issuing a commit. Not wrapping a select in a transaction
will definitely be the least overhead.
>
> > Maybe something to think about too, because I can't be the only one
> > making heavy use of stored procedures (which are far more efficient than
> > controller/model side code)
>
> that a stored-procedure-oriented application is "far more efficient"
> is *extremely* debatable and database-dependent as well.
I doubt it's *extremely* debatable. Just issue 100 inserts from inside a
stored procedure (or 100 updates) and do the same discretely with any kind of
db interface. In case of the interface every statement has to be parsed by
the db, whereas in a stored procedure the statement is already "compiled" of
sorts (at least Oracle and PostgreSQL do that). I had cases where moving the
application code (standard DBAPI calls, no ORM) to a stored procedure reduced
the execution time from 27 seconds to 2 seconds without changing the database
structure (ok, extreme case, but handling 1500 inserts discretely in the
application is just a lot of remote overhead, particularly when the requests
come over the network where you get an extra delay for network operations)
> if you
> really want COMMIT for every SELECT, i'd favor it being enabled via an
> option passed to create_engine().
Not every select, every transaction that didn't roll back.
I just think the default of rollback on every transaction is wrong - a
rollback should occur when there is a problem, not when the transaction was
fine. But that may just be me.
> Beyond that I think the "model
> implemented as stored procedure" style of development is much in the
> minority these days, particularly within the "lightweight/open-source"
> development community. just that the stored procedure changes the
> semantics of SELECT to be a "write" operation reminds me of the
> RESTful sin of using GET to post data.
Probably because a lot of people can't figure out how to use stored procedures
and triggers, since the "lightweight/open-source" programming is often done
on a database that has very limited support for both :-)
Personally I'm not a big fan of handling database integrity outside the
database. Take a simple portal as example, where every new user who signs up
gets a couple of mailboxes by default (inbox, outbox,drafts,trash). I could
handle that in the db structure itself, but that would make the db model
overly complicated for such a simple thing. The proper way is to handle that
in a trigger, or I can handle it in a stored procedure that just makes sure
the mailboxes are created. In my case I chose a stored procedure because it's
actually a select on the user's profile, but the stored procedure also does
certain sanity checks and inserts/updates stuff according to its findings.
Handling the same in the application code is IMHO the least desirable
solution, simply because I can test the trigger/stored procedure once and I
know everything works until I change the database structure. Application code
is meddled around with much more, so the chance of introducing a bug is
higher on that end.
You could argue that using SA with stored procedures defeats the "portability"
approach. I agree that you can't easily move a Oracle or PostgreSQL or DB2
based application with stored procedures to a different database, but how
many people do that? If you build a generic application the argument is
valid, but if you build something for a specific purpose, the database will
never be changed for the life of the application (because generally "special
purpose" applications are funded by someone and those people usually don't
invest another couple grand just to make it work with some other database).
I agree on the semantics, but I don't see a lot the folks at postgresql can do
about it: a) it's the SQL standard, b) one would have to have two different
ways of executing a stored procedure (which there are, EXECUTE and SELECT)
depending on whether the procedure has a return value or not and whether the
procedure changes data or not (in my case it has a return value and changes
data, so the SELECT is actually the correct way, but still I need a commit at
the end). In the end I think it's a SA issue - there should be a parameter
allowing to tell SA that this "select" is actually something transactional
and needs to commit if no error is raised.
Mike Orr
> > Of course the engine must be initialized before any tables can be
> > defined or any work done.
>
> since ive seen people get the wrong idea about this, let me
> reiterate: *you can define all your tables and mappers with no
> database connection defined whatsoever*,
Not if you're using autoloading or want to be able to switch
autoloading on or off for any table without having to modify the rest
of the application. To me that's important, especially when you have
to start changing the schema of an existing table (adding a field,
changing its type, etc), or if the schema is modified externally to
the application.
> > SAContext initializes the default engine in
> > the constructor. Is there a need to push that back to some later
> > stage? Which stage? Or can Ian's ideal wrapper simply delay creating
> > the 'sac' until it has to?
>
> easy enough to make "uri" optional in SAContext (the way it is for
> everything else in SA) and just allow binding it later via
> add_engine() or whatever. SAContext is designed the way it is right
> now because Pylons defines the default engine in the .ini file; theres
> no need for a "delayed" connection in the usual case.
I would probably want this optional URI in a subclass rather than in
SAContext itself. The reason 'uri' is a required argument is to
guarantee that the default engine is initialized at all times. When
we designed SAContext we (Mike & I) agreed that the bound metadata
strategy was the most straightforward and adequate for most user apps:
the 80/20 rule. Hiding the engine (connectable) as much as possible
in the metadata, while still making it accessible when you really need
it. Now we're adding one unbound metadata strategy after another.
That's fine as log as it doesn't detract from SAContex's primary
commitment to its primary userbase. I'd say a mandatory URI and bound
metadata is part of its primary commitment for an easy-to-use front
end: a normal Python TypeError for "You forgot the 'uri' argument" is
about as straightforward as one can get. Or is that changing too?
Are you having second thoughts about using bound metadata by default?
--
Mike Orr <slugg...@gmail.com>
Mike Orr
> > if you
> > really want COMMIT for every SELECT, i'd favor it being enabled via an
> > option passed to create_engine().
>
> Not every select, every transaction that didn't roll back.
> I just think the default of rollback on every transaction is wrong - a
> rollback should occur when there is a problem, not when the transaction was
> fine. But that may just be me.
But the framework doesn't know whether there's a problem, or whether
some uncommitted changes exist (maybe you called a function that
changed the database but didn't commit). It certainly doesn't know
whether it "should" commit or roll back these changes if they do
exist. But it DOES know that stray uncommitted changes should not
leak into the next web request or all hell will break loose. So it
has to commit or roll back. Rolling back is the prudent thing because
if the function really wanted to save those changes permanently, well,
it should've committed them.
I'd argue the opposite, that people don't use stored procedures and
triggers because that ties them to a certain database, and that's the
opposite of why they chose SQLAlchemy in the first place. Not only
that, you have to learn a second language that's limited to that
database, with its own quirks and limitations, and the language is
much less convenient/featureful than Python (it looks like a fossil
from 20 years ago), which is why we're using Python in the first
place. I would be more likely to change my database than change my
application, or at least just as likely. It's not that big a deal to
copy everything into a new database and adjust the program for it --
provided you don't have huge databases or stored procedures. The
"huge database" part is outside your control, but the stored procedure
part is. Tying myself to one database product gives me bad memories
of the "vendor lock-in" problems that used to be more prevelant in the
past, but are thankfully becoming rarer now that interoperability has
gotten better.
I actually have a practical example of this. I may have to switch one
application from MySQL to PostgreSQL in order to access its PostGIS
geographical functions, which I hear are more advanced than MySQL's.
So if we decide we need this functionality we'll have to switch.
Without stored procedures I can just change my DBURI string and voila,
just write the SQL expressions. With stored procedures I'd have to
port all those to the new database.
Having said all this, I have had to break down and start using views
because it made performance acceptable in one case (a mapped object on
a SELECT was just too slow). Someday I may have to write stored
procedures for that reason. But so far I haven't needed to.
--
Mike Orr <slugg...@gmail.com>
Uwe C. Schroeder
On Wednesday 27 June 2007, Mike Orr wrote:
> On 6/27/07, Uwe C. Schroeder <u...@oss4u.com> wrote:
> > > if you
> > > really want COMMIT for every SELECT, i'd favor it being enabled via an
> > > option passed to create_engine().
> >
> > Not every select, every transaction that didn't roll back.
> > I just think the default of rollback on every transaction is wrong - a
> > rollback should occur when there is a problem, not when the transaction
> > was fine. But that may just be me.
>
> But the framework doesn't know whether there's a problem, or whether
> some uncommitted changes exist (maybe you called a function that
> changed the database but didn't commit). It certainly doesn't know
> whether it "should" commit or roll back these changes if they do
> exist. But it DOES know that stray uncommitted changes should not
> leak into the next web request or all hell will break loose. So it
> has to commit or roll back. Rolling back is the prudent thing because
> if the function really wanted to save those changes permanently, well,
> it should've committed them.
If it could, yes. Right now it looks to me like SA automatically rolls every
select back without a way to tell it not to (and no, explicit transaction
didn't do the trick, but then I'm not an expert on SA, so there may be a
better way)
Interoperability has gotten better? A bit probably, but not all that much :-(
On the other hand, if you're as likely to change your DB model as you are to
modify your application, your app and db model are either very simple, or the
db model wasn't designed all that well. My current application has 513 tables
and been in production for about 5 years. The application running on top of
it was changed numerous times, the DB model had only 3 changes (all minor,
like adding a field to a table) over the years and the stored
procedures/triggers/views actually never changed. In my case there are third
party applications accessing the database (i.e. someone made fancy reports
with Crystal Reports), so changing the DB structure is a major pain with a
lot of paperwork where changing the application is not.
I'd also like to argue that everyone in the open-source field would certainly
love to see companies adopt open source over proprietary software. One of the
factors for that decision is whether the requirements for large scale
deployments are fulfilled. I'd love to see Bank of America run their
homebanking on something open-source, but I doubt they do - one of the
reasons being that their requirements aren't met by a lot of frameworks out
there.
Actually I'd also argue the "convenient" point. How is a ORM convenient?
Having a query with say 20 joins leads to a "pythonic" but pretty much
unreadable, much less comprehensible query definition. Everyone dealing with
SQL databases should be able to read SQL, but a query object definition that
doesn't even fit on 2 pages anymore is far from readable. So yes, there are
reasons why one would use ORM's and there are some against ORM's - a split SA
has actually mastered fairly well since you don't have to use the mapper
part.
> I actually have a practical example of this. I may have to switch one
> application from MySQL to PostgreSQL in order to access its PostGIS
> geographical functions, which I hear are more advanced than MySQL's.
> So if we decide we need this functionality we'll have to switch.
> Without stored procedures I can just change my DBURI string and voila,
> just write the SQL expressions. With stored procedures I'd have to
> port all those to the new database.
You'd still have to consider all the incompatibilities between databases.
PostgreSQL has far more data types, some of which are not handled by SA (i.e.
arrays, which again are not SQL standard but very useful). Also be aware that
SA doesn't handle postgresql user types correctly (particularly with
autodetect), so be prepared to patch SA when you use the geographical
functions. In the end, the moment you NEED to switch databases because of
features missing in the current one, you're already on the way to lock
yourself into one database (i.e. what if you want to move your application to
DB2? Does DB2 have geographical functions?).
> Having said all this, I have had to break down and start using views
> because it made performance acceptable in one case (a mapped object on
> a SELECT was just too slow). Someday I may have to write stored
> procedures for that reason. But so far I haven't needed to.
In the end, whether you use features like views, procedures, triggers etc. or
not, is just a question of whether it makes sense for the application. Since
Pylons and TG are frameworks for pretty much any application, they just
should support such features without having to patch core components.
Wichert Akkerman
> On Jun 27, 6:42 pm, Ian Bicking <i...@colorstudy.com> wrote:
> > The way I see this working is something like (vaguely):
> >
> > def transaction_middleware(app):
> > def wrapper(environ, start_response):
> > manager = TransactionManager()
> > environ['transaction.manager'] = manager
> > try:
> > app_iter = app(environ, start_response)
> > except:
> > manager.rollback()
> > raise
> > else:
> > manager.commit()
> >
> > The manager is basically a container of *actual* transactions, and
> > calling rollback or commit on it gets passed on to all the transactions
> > in the manager.
>
> this is fine. we're *really* starting to imitate J2EE in some ways.
> but its not a bad thing.
For what it's worth: Zope has a similar structure. The Zope publisher
starts a transaction when it receives a request and commits it when the
request processing has finished, unless an exception has been thrown or
someone did an explicit rollback. Rollback tends to be very rare so
it seems that it is not very useful to optimize for that case. The Zope
machinery allows you to add your own commit/rollback hooks, which is
used to tie SA into Zope.
Wichert.
--
Wichert Akkerman <wic...@wiggy.net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
Mike Orr
> > Tying myself to one database product gives me bad memories
> > of the "vendor lock-in" problems that used to be more prevelant in the
> > past, but are thankfully becoming rarer now that interoperability has
> > gotten better.
>
> Interoperability has gotten better? A bit probably, but not all that much :-(
Sorry, I meant that interoperability in the computer industry as a
whole has gotten better, not SQL databases in particular. Though SQL
databases have made some minor strides.
> Actually I'd also argue the "convenient" point. How is a ORM convenient?
I meant the Python language itself is more convenient than the
database procedure languages I've seen.
> Having a query with say 20 joins leads to a "pythonic" but pretty much
> unreadable, much less comprehensible query definition.
Yes, I still have my doubts about building complex joins using
SQLAlchemy's various "join" objects, whether they're really any better
than a literal SQL fragment which I can write much more quickly. Had
to make a left join recently and couldn't find the function, then
realized it's called outerjoin. That one works slick but I see what
people do with multiple mappers or joins inside mappers and I think,
is this for real?
> Everyone dealing with
> SQL databases should be able to read SQL, but a query object definition that
> doesn't even fit on 2 pages anymore is far from readable.
And this. If you have a 50-column table, by gosh SQLAlchemy is going
to select all 50 fields in full "`TableName`.columnName AS columnName
notation", which makes the query hard to read in the log.
--
Mike Orr <slugg...@gmail.com>
Uwe C. Schroeder
On Thursday 28 June 2007, Mike Orr wrote:
> And this. If you have a 50-column table, by gosh SQLAlchemy is going
> to select all 50 fields in full "`TableName`.columnName AS columnName
> notation", which makes the query hard to read in the log.
Mhhhh - a 50 column table? My old CS teacher will wiggle in his grave like a
nightcrawler. What happened to normalizing tables? :-)
But then - I've done pretty awful things myself :-)
Jonathan LaCour
> im not exactly sure why Jonathan thinks that the "url" forces one
> to "define my connection up front". if youre saying, "i want to
> distribute the application without an .ini file" thats fine...but when
> the thing is actually *used*, there will be an .ini file or other
> configuration defined when things run. if you want to define your
> model independently of connection, just use an empty MetaData. Set
> up the SAContext to bind the engine or individual connections to the
> session instead...or give it your MetaData for it to bind when youre
> ready. theres really nothing to it.
I just didn't like having my model get its URI directly from the Pylons
config, for the reasons I stated previously. However, you just solved
my issue :) I had no idea that I was able to do this with SAContext, and
effectively use it just as I want to.
Thanks for the explanation Mike!
Jonathan LaCour
> just to clarify, however this works, it should be *really easy* for
> individual controller methods to be marked as "transactional" or not.
> I have written decorators like these already, in the style of:
>
> class MyController(...):
> def index_page(self):
> .....
>
> @transactional
> def post_message(self):
> .....
>
> def display_message(self):
> .....
>
> so id want that kind of thing to be readily available, i suppose it
> would communicate with the WSGI middleware on a per-request basis.
The only issue I can see with this is that having the transaction begin
in the controller method itself might be too late if you have other
things going on in middleware or elsewhere and something goes wrong
before you even get to the controller method -- say in object dispatch,
the new TG controller's "lookup" method, or even some middleware that
stores sessions in a database for example.
Transactions should start at the earliest possible point, IMO, so that
you can handle all of these cases.
> but another thing i like, is that of the response to the request being
> delivered *after* the transaction commits. i have raised this issue
> on the pylons list before but i dont think i impressed anyone. if you
> render your response template, then your transaction fails, you have
> the potential for that response template to still be delivered with
> the "success!" message. anyway, might be nice for the middleware to
> address this if possible.
I am totally with you here, as this is the other side of the coin that
I brought up above. Open transactions shouldn't be committed until the
latest possible point.
I spent some dark days working on J2EE applications, and while there
isn't a whole lot that is worth emulating in J2EE, the transaction
handling is something that I actually liked.
> also i think this is all outside of the scope of SAContext. SAContext
> should remain as a facade to SA-specific elements; it can accept
> "strategy" objects which control its internal workings, so if need be
> particular strategies can be delivered in order to function with the
> transaction manager, without changing the external view of SAContext.
Yes, I think agree here too, as long as there is a way to hook SAContext
into whatever ends up getting decided in this (really long) thread. I
played a bit more with SAContext after these discussions, and it is
indeed very nice. Good work Mike and Mike.
Michael Bayer
On Jun 28, 1:40 am, "Uwe C. Schroeder" <u...@oss4u.com> wrote:
> On Wednesday 27 June 2007, Michael Bayer wrote:
>
> > this issue can be worked around by using explicit transactions.
>
> actually no, it can't. Maybe I don't get it right, but the only way for me to
> get a commit was actually to modify Connection._autocommit in
> sqlalchemy.engine.base.
either TG is getting in the way, or youre not getting it right. if
anyone ever needs to modify the internals of SA to get something to
work, i would *highly* prefer if they could email the ML or post a
trac ticket with their issue so that it may be handled properly.
> Obviously SA thinks there is no transaction in TG,
> so it just wraps one around it.
if TG actually has "a transaction" going on, theyd certainly have to
configure SA to be aware of it (most likely via SessionTransaction).
if not, then yes things arent going to work at all (though still, an
explicit SA transaction should work all by itself).
> I agree, and that is certainly DB dependent. Personally I can't imagine that
> an automatically issued rollback for every select transaction is in any way
> more overhead than issuing a commit. Not wrapping a select in a transaction
> will definitely be the least overhead.
we dont issue a rollback for every select transaction. we issue a
rollback when a connection is returned to the pool. you can check out
a connection explicitly and perform any number of selects on it
without any rollbacks or commits.
because the "rollback" is at the connection-pool checkin level, it
should be more apparent how inappropriate it would be to issue a
*commit* every time a connection is returned to the pool, an operation
that knows nothing about what just happened with that connection. the
rollback is to release database locks.
im thinking that it might be time to allow an option in SA that just
turns the DBAPI's "autocommit" flag on. that way you can just blame
the DBAPI for whatever issues arise.
its not always possible to "not wrap a select in a transaction".
oracle for example *always* has a transaction going on, so everything
is in a transaction in all cases.
> > that a stored-procedure-oriented application is "far more efficient"
> > is *extremely* debatable and database-dependent as well.
>
> I doubt it's *extremely* debatable.
its extremely debatable:
http://www.google.com/search?q=stored+procedures+vs
> Just issue 100 inserts from inside a
> stored procedure (or 100 updates) and do the same discretely with any kind of
> db interface. In case of the interface every statement has to be parsed by
> the db, whereas in a stored procedure the statement is already "compiled" of
> sorts (at least Oracle and PostgreSQL do that).
the debate over SP's is about a larger issue than "is an SP faster
than 5 separate INSERT statements". SP's are of course much better
for micro-benchmarks like that. its their impact on application
development and architecture where the debate comes in (read some of
the googled articles).
I am certainly not anti-SP, ive done pure SP applications before (on
projects where the DBAs controlled the SPs)...I just dont want to
start hardwiring SQLAlchemy to expect that sort of application. I
think 80/20 as applied to SELECT is that 80% of SELECTs are for read
operations and a COMMIT is inappropriate.
> > if you
> > really want COMMIT for every SELECT, i'd favor it being enabled via an
> > option passed to create_engine().
>
> Not every select, every transaction that didn't roll back.
> I just think the default of rollback on every transaction is wrong - a
> rollback should occur when there is a problem, not when the transaction was
> fine. But that may just be me.
this is the use case:
c1 = pool.connect()
row = c1.cursor().execute("select * from sometable").fetchone()
pool.return_connection(c1)
c2 = pool.connect() # (returns a connection that is not c1)
c2.cursor().execute("drop sometable") # --> deadlock
if DBAPI supported a "release_locks()" method, we'd be using that.
> Probably because a lot of people can't figure out how to use stored procedures
> and triggers, since the "lightweight/open-source" programming is often done
> on a database that has very limited support for both :-)
keep in mind youre including the vast Hibernate community, including
its creators, etc. im not sure if the "im to dum to use stored
procedures" argument can fully explain why the SP-architecture remains
a minority use case. i think the overall inconvenience of it, the
clunky old languages you have to use on DBs like Oracle and SQL
Server, as well as the harsh resistance it puts up to so-called agile
development methods are better reasons.
> Personally I'm not a big fan of handling database integrity outside the
> database.
> ....continue SP arguments.....
thats great, you can have your preferences..the google link above
should reveal that quite a few people have established their
preferences in this matter. If you are truly writing an SP-only
application which prevents direct SQL access (that was the kind of SP
app I worked on), a tool like SQLAlchemy is mostly superfluous.
for my purposes as the maintainer of SQLAlchemy, i need to support the
majority of use cases which is that of a non-SP oriented application.
not just because SQL-oriented apps are more common, but also because
SP-oriented apps aren't going to have use for higher level SQL
toolsets anyway since all the SQL is behind stored procedures.
> the end). In the end I think it's a SA issue - there should be a parameter
> allowing to tell SA that this "select" is actually something transactional
> and needs to commit if no error is raised.
this is this:
> michael bayer wrote:
> i
> also have a notion of SQL functions being marked as "transactional" to
> help this issue.
but really, i think you should get your explicit transactions
working...i would imagine you have the need to execute multiple SPs in
one transaction (otherwise that must be some enormous SP youre
running).
Michael Bayer
On Jun 28, 2:09 am, "Mike Orr" <sluggos...@gmail.com> wrote:
>
> I would probably want this optional URI in a subclass rather than in
> SAContext itself. The reason 'uri' is a required argument is to
> guarantee that the default engine is initialized at all times. When
> we designed SAContext we (Mike & I) agreed that the bound metadata
> strategy was the most straightforward and adequate for most user apps:
> the 80/20 rule. Hiding the engine (connectable) as much as possible
> in the metadata, while still making it accessible when you really need
> it. Now we're adding one unbound metadata strategy after another.
> That's fine as log as it doesn't detract from SAContex's primary
> commitment to its primary userbase. I'd say a mandatory URI and bound
> metadata is part of its primary commitment for an easy-to-use front
> end: a normal Python TypeError for "You forgot the 'uri' argument" is
> about as straightforward as one can get. Or is that changing too?
> Are you having second thoughts about using bound metadata by default?
>
OK just FTR heres why SQLAlchemy is "not a framework" and why in order
to do things without the advent of a framework you work with all these
little highly granular components, and also why im still antsy
including SAContext in SA. just trying to establish the simplest
"heres your connection" object spawns all this debate over how it
should look/work/act/etc. I would encourage those here to look at
SAContext, see that its a dead-simple piece of code, and just work up
various preferred versions of it to taste. My goal with it was more
to illustrate *how* you can build these things.
Michael Bayer
On Jun 27, 5:56 pm, Ian Bicking <i...@colorstudy.com> wrote:
> * Way for a library to get "the" transaction manager.
> * Interface for that transaction manager, maybe copied from Zope.
> * Single convention for how to specify a database connection (ideally
> string-based).
> * Probably a way to get "the" configuration, so you can lazily get a
> connection based on the configured connection.
just as a point of reference, here is zalchemy's integration of Zope
datamanager:
http://svn.zope.org/z3c.zalchemy/trunk/src/z3c/zalchemy/datamanager.py?rev=77165&view=auto
I would favor that the interface supports two-phase semantics like
zope's. sqlalchemy's "flush()" model already fits in pretty well with
the two-phase model, and will eventually be building explicit "two-
phase" hooks into SQLAlchemy's engine and session, with real
implementations for postgres to start with.
Michael Bayer
On Jun 28, 2007, at 9:58 AM, Jonathan LaCour wrote:
>
> Michael Bayer wrote:
>
>> just to clarify, however this works, it should be *really easy* for
>> individual controller methods to be marked as "transactional" or not.
>> I have written decorators like these already, in the style of:
>>
>> class MyController(...):
>> def index_page(self):
>> .....
>>
>> @transactional
>> def post_message(self):
>> .....
>>
>> def display_message(self):
>> .....
>>
>> so id want that kind of thing to be readily available, i suppose it
>> would communicate with the WSGI middleware on a per-request basis.
>
> The only issue I can see with this is that having the transaction
> begin
> in the controller method itself might be too late if you have other
> things going on in middleware or elsewhere and something goes wrong
> before you even get to the controller method -- say in object
> dispatch,
> the new TG controller's "lookup" method, or even some middleware that
> stores sessions in a database for example.
the way this idea works is that if some wrapping component knows it
needs a transaction, it opens one. then the controller method
participates in that transaction, regardless of whether or not its
marked as @transactional.
>> but another thing i like, is that of the response to the request
>> being
>> delivered *after* the transaction commits. i have raised this issue
>> on the pylons list before but i dont think i impressed anyone. if you
>> render your response template, then your transaction fails, you have
>> the potential for that response template to still be delivered with
>> the "success!" message. anyway, might be nice for the middleware to
>> address this if possible.
>
> I am totally with you here, as this is the other side of the coin that
> I brought up above. Open transactions shouldn't be committed until
> the
> latest possible point.
yes. im talking specifcally about the way Pylons passes off from
controller to view...right now you call render_response(), which
typically renders the entire template in a buffer before the
controller method has finished. id favor changing the behavior of
render_response() to creating a "deferred" render object which Pylons
explcitly places outside the scope of the controller call (however,
probably still within the scope of enclosing middleware).
>
> I spent some dark days working on J2EE applications, and while there
> isn't a whole lot that is worth emulating in J2EE, the transaction
> handling is something that I actually liked.
thats where im getting this from. theres subtleties to it that you
cant emulate casually.
Jonathan LaCour
> the way this idea works is that if some wrapping component knows
> it needs a transaction, it opens one. then the controller method
> participates in that transaction, regardless of whether or not its
> marked as @transactional.
Of course, that makes complete sense. As long at the wrapping component
knows about the transaction manager (which, clearly, I think it should).
>> I am totally with you here, as this is the other side of the coin
>> that I brought up above. Open transactions shouldn't be committed
>> until the latest possible point.
>
> yes. im talking specifcally about the way Pylons passes off from
> controller to view...right now you call render_response(), which
> typically renders the entire template in a buffer before the
> controller method has finished. id favor changing the behavior of
> render_response() to creating a "deferred" render object which Pylons
> explcitly places outside the scope of the controller call (however,
> probably still within the scope of enclosing middleware).
Makes sense to me.
>> I spent some dark days working on J2EE applications, and while there
>> isn't a whole lot that is worth emulating in J2EE, the transaction
>> handling is something that I actually liked.
>
> thats where im getting this from. theres subtleties to it that you
> cant emulate casually.
Yes, the further down the rabbit hole I go, the more clear this is
becoming. That being said, I think I am sold on the concept that we
have hashed out here.
I need to take some time to look at Zope's transaction manager at some
point, and see how hard it would be to extract it from Zope, and get rid
of any dependencies, so it can easily be packaged up as its own, simple
egg. Has anyone done this before? Is anyone interested in taking this
up as a project?
Ian Bicking
> On Jun 27, 6:42 pm, Ian Bicking <i...@colorstudy.com> wrote:
>> The way I see this working is something like (vaguely):
>>
>> def transaction_middleware(app):
>> def wrapper(environ, start_response):
>> manager = TransactionManager()
>> environ['transaction.manager'] = manager
>> try:
>> app_iter = app(environ, start_response)
>> except:
>> manager.rollback()
>> raise
>> else:
>> manager.commit()
>>
>> The manager is basically a container of *actual* transactions, and
>> calling rollback or commit on it gets passed on to all the transactions
>> in the manager.
>
> this is fine. we're *really* starting to imitate J2EE in some ways.
> but its not a bad thing.
At least someone else thought it was a good idea too ;). It's also all
very similar to Zope.
>> If you don't do anything that needs a transaction (e.g., read-only), you
>> shouldn't put your transaction in the manager.
>
> just to clarify, however this works, it should be *really easy* for
> individual controller methods to be marked as "transactional" or not.
> I have written decorators like these already, in the style of:
>
> class MyController(...):
> def index_page(self):
> .....
>
> @transactional
> def post_message(self):
> .....
>
> def display_message(self):
> .....
>
> so id want that kind of thing to be readily available, i suppose it
> would communicate with the WSGI middleware on a per-request basis.
I think that would be easy enough. All that is setup early is the
transaction container; nothing is automatically added to it. So
@transactional could just add a transaction to the container (presumably
everything would otherwise be autocommit, or perhaps automatically
rolled back?)
Presumably what you really want to do is
add-this-transaction-if-it-isn't-there-already all over the place. For
instance, if you've already started a transaction in middleware, I don't
think you'll need to start another transaction. Unless you have
subtransactions, which I suppose is possible.
> but another thing i like, is that of the response to the request being
> delivered *after* the transaction commits. i have raised this issue
> on the pylons list before but i dont think i impressed anyone. if you
> render your response template, then your transaction fails, you have
> the potential for that response template to still be delivered with
> the "success!" message. anyway, might be nice for the middleware to
> address this if possible.
Hmm... I believe paste.exceptions should do this already, even if an
exception is raised after start_response is called, as long as it is
raised before the app_iter is returned. If it's after the app_iter,
then you've somehow escaped the stack, which doesn't much concern me ;)
So as middleware I think it should be fine. If you want to catch a
problem with the commit and handle it gracefully, I think you can just
do your own commit in your code (and the middleware commit will just
commit what little happens after that).
And of course, what you put in as a transaction is up to you. You could
put in some object which uses whatever flags you want to actually pass
through a commit or else just ignore it.
> also i think this is all outside of the scope of SAContext. SAContext
> should remain as a facade to SA-specific elements; it can accept
> "strategy" objects which control its internal workings, so if need be
> particular strategies can be delivered in order to function with the
> transaction manager, without changing the external view of SAContext.
I dunno... I think if we can get this in place, then a whole lot of
things are going to become simpler. But there also still has to be
wrappers like SAContext from this fairly low-level stuff to a particular
framework.
--
Ian Bicking | ia...@colorstudy.com | http://blog.ianbicking.org
| Write code, do good | http://topp.openplans.org/careers
Michael Bayer
On Jun 28, 11:46 am, Ian Bicking <i...@colorstudy.com> wrote:
>
> So as middleware I think it should be fine. If you want to catch a
> problem with the commit and handle it gracefully, I think you can just
> do your own commit in your code (and the middleware commit will just
> commit what little happens after that).
>
> And of course, what you put in as a transaction is up to you. You could
> put in some object which uses whatever flags you want to actually pass
> through a commit or else just ignore it.
usually, applications want a single big "transaction" for the whole
request. if an app wants something that will commit independently of
the "enclosing" transaction, that can be an option to the decorator:
@transactional(independent=True)
def do_something(self):
the term "independent" is made up and maybe not so clear...but again,
this is straight out of EJB. they let you annotate session beans in
an XML file using these terms: "required", "requiresNew", "Supports",
"Mandatory", "NotSupported", "Never". I think we should consider each
of these cases and determine if we think we need to support that case,
and if so how. (also does Zope express this many variants?)
Cliff Wells
> I meant the Python language itself is more convenient than the
> database procedure languages I've seen.
Even when they're Python?
http://www.postgresql.org/docs/8.1/interactive/plpython.html
Cliff
Alberto Valverde
On Jun 28, 2007, at 3:56 AM, Mike Orr wrote:
>
> On 6/27/07, Mark Ramm <mark.mch...@gmail.com> wrote:
>>
>>> It sounds like Pylons and TurboGears have very different paradigms
>>> about how transactions are handled.
>>
>> I can't do a side-by-side comparison, because I am not 100% clear on
>> an example of the "right way" to handle transactions in Pylons.
>>
>> But I can describe what TurboGears 1.0 does.
>>
>> TG currently has an "automatic transaction per request" feature,
>> which
>> is very widely used.
>>
>> If a request fails for any reason, at any point the exception is
>> propigated out, and causes a transaction rollback.
>
> The base controller's .__call__ method is your friend (but see below).
> Wrap the subclass call in a try-commit, You can also use the
> .__before__ and .__after__ methods unless you want to leave those
> empty for users. This avoids the need for fragile middleware and
> fussing with the WSGI environment.
I'd like to add here that I've got use-cases for the transaction
boundary and session initialization being closer to the server than
what the __before__ and __after__ hooks provide.
For example, the authentication function I feed to AuthKit which uses
a DB connection to retrieve the user and it's associated roles and
permissions.
I can think of other cases where other layers of middleware (more
precisely, Middleware FrameWork Components) might want to access the
database. Having all these layers plus the controller method (or
other WSGI which sit below PylonsWSGIApp) wrapped in the same
transaction sounds like a good idea to me.
Alberto
Alberto Valverde
On Jun 28, 2007, at 12:42 AM, Ian Bicking wrote:
>
> Jonathan LaCour wrote:
>> Ian Bicking wrote:
>>
>>> Personally I'd like to see something a bit different:
>>>
>>> * Way for a library to get "the" transaction manager.
>>> * Interface for that transaction manager, maybe copied from Zope.
>>> * Single convention for how to specify a database connection
>>> (ideally
>>> string-based).
>>> * Probably a way to get "the" configuration, so you can lazily get
>>> a connection based on the configured connection.
>>>
>>> The frameworks would setup the transaction manager and
>>> configuration,
>>> and database frameworks would basically consume this information.
>>
>> This would be pretty much ideal, and would satisfy my use cases very
>> well. It would also allow framework authors to build in support
>> for any
>> ORM / database layer in an agnostic way, sort of like the Buffet API
>> does for templating systems.
>>
>> Then, we could also create a simple WSGI middleware for Pylons that
>> gives it TurboGears-like automatic transaction start/commit/rollback
>> on a per request basis. Only, we could make it configurable so that
>> it didn't happen on read-only operations. All of this would be ORM /
>> database layer agnostic, which would be very nice indeed.
>
> The way I see this working is something like (vaguely):
>
> def transaction_middleware(app):
> def wrapper(environ, start_response):
> manager = TransactionManager()
> environ['transaction.manager'] = manager
> try:
> app_iter = app(environ, start_response)
> except:
> manager.rollback()
> raise
> else:
> manager.commit()
I see you've mentioned "vaguely" and that paste.transaction
implements it properly... but I just had to say this. An almost exact
implementation of that pseudocode caused me a very embarrassing bug
in which transactions were committed after the authorization
middleware trapped security exceptions and turned them into 403s.
Luckily it was only an internal app and the bug was noticed in-
house... :)
Bottom line is that the transaction middleware should rollback any
transaction if the status code is >= 400 to handle case like this
(and that using existing, well tested code is usually better than
rolling your own no matter how fun it is :)
Alberto
Jonathan LaCour
> I'd like to add here that I've got use-cases for the transaction
> boundary and session initialization being closer to the server than
> what the __before__ and __after__ hooks provide.
>
> For example, the authentication function I feed to AuthKit which uses
> a DB connection to retrieve the user and it's associated roles and
> permissions.
>
> I can think of other cases where other layers of middleware (more
> precisely, Middleware FrameWork Components) might want to access the
> database. Having all these layers plus the controller method (or other
> WSGI which sit below PylonsWSGIApp) wrapped in the same transaction
> sounds like a good idea to me.
Fair enough to all of the above, however all the feedback from Mike and
Ian has made it clear to me that we need a few different things:
1. A standard transaction manager that can be plugged into any
application, WSGI or not. It should be similar in scope and API
to Zope's transaction manager, but free of Zope dependancies.
2. A piece of middleware that uses the transaction manager to
rollback
any active transactions in the case of an exception, or commit
them
in the other case.
3. Either some middleware or just a few lines inside the
BaseController
of the TurboGears template that starts a transaction on every
request and puts it into the transaction manager.
The first and second item are generally useful, and will allow you to
manage transactions manually if you so desire. Any general-purpose
middleware out there that needs to use a transaction can simply use the
transaction manager, and all will be well. TurboGears will have the
same behavior as it has had for a while, and since it happens in the
__call__ of the BaseController, you can tweak it to your heart's content
based upon the URI, HTTP method, or whatever you want.
And, in my opinion, items #1 and #2 should be in Pylons by default, and
the third item should be in the TurboGears template by default.
There is, of course, some extra "stuff" we could add for nested
transactions, and all of the cases he enumerated in his last email
(required, requiresNew, supports, mandatory, etc), but I think the three
things above will give a good set of building blocks for making those
things easy to deal with.
Ian Bicking
paste.transaction itself is something I should have really removed, as
it's so incomplete. But yes, it's the basic pattern, but without a
clear concept of a TransactionManager.
> Bottom line is that the transaction middleware should rollback any
> transaction if the status code is >= 400 to handle case like this
> (and that using existing, well tested code is usually better than
> rolling your own no matter how fun it is :)
Er... I'm not sure about this. I'm not sure >=400 is really a failure.
Maybe it is; I just don't feel comfortable making that claim generally.
If you have a security exception that is caught, you could also look for
the manager and roll it back exactly then, instead of expecting the
middleware to do it. The middleware is just the furthest-out boundary
for transactions, it's not the only place where you can control the
transactions.
Alberto Valverde
(I'm assuming this middleware above the PylonsWSGIApp...)
>
> 3. Either some middleware or just a few lines inside the
> BaseController
> of the TurboGears template that starts a transaction on every
> request and puts it into the transaction manager.
Hmmm, wouldn't having two transaction boundaries at 2 and 3 be
redundant? Or maybe both boundaries will use the same manager in a
similar way paste.registry has only one Registry for the whole stack?
(albeit different context boundaries every time a RegistryManager is
stacked)
One problem I see is that this might cause trouble in cases like this:
Request
|
V
db transaction middleware -> creates session A
|
V
authentication middleware -> authenticates user and keeps ref to User
instance (attached to session A)
|
V
other middleware and PylonsApp
|
V
TG controller -> creates session B inside __call__
|
V
authorization library tries to check permissions for mapped Foo
instance (attached to session B) which has a owner attribute:
foo.owner (sess B) == logged_in_user (sess A)
I haven't actually tried it so I'm not sure how to objects attached
to different sessions compare as or interact, so this might be a non-
issue (maybe Mike can shed some light on this?)
Anyway, this might be a non-issue on the whole if the transaction
middleware does not handle session creation at all.
OTOH, I do see the difficulty of having just one boundary in a single
TG/Pylons app in a layer of middleware above PylonsApp since I don't
see a clear way of making parameters passed via a decorator reach the
middleware *before* the method is actually called (which will be
needed to signal that a BEGIN should not be issued)
Alberto
Alberto Valverde
I can understand this... at least we're sure we shouldn't implicitly
*commit* the transaction
>
> If you have a security exception that is caught, you could also
> look for
> the manager and roll it back exactly then, instead of expecting the
> middleware to do it. The middleware is just the furthest-out boundary
> for transactions, it's not the only place where you can control the
> transactions.
However, this would couple the security-exception-trapping middleware
with the transaction manager. Right now, the mentioned middleware's
responsability is very simple: Trap security exceptions and turn them
into 403s or 401s (depending if there's a user logged in). it doesn't
even handle authentication (which is handled above to provide a
REMOTE_USER and/or wsgi_org.user_info and trapping 401s). I can also
think of other layers which could change status codes, expecting all
this layers to be aware that a transaction manager is in the stack
and explicitly rollback any transaction doesn't favor loose-coupling
from my POV.
I believe that in this scenario the most straightforward way to
signal that an error (or non OK) condition has ocurred is to through
the HTTP status code hence the db transaction middleware should take
it into account. The question of what status codes should be treated
as errors is still open though...
Alberto
Uwe C. Schroeder
On Thursday 28 June 2007, Michael Bayer wrote:
> On Jun 28, 1:40 am, "Uwe C. Schroeder" <u...@oss4u.com> wrote:
> > On Wednesday 27 June 2007, Michael Bayer wrote:
> > > this issue can be worked around by using explicit transactions.
> >
> > actually no, it can't. Maybe I don't get it right, but the only way for
> > me to get a commit was actually to modify Connection._autocommit in
> > sqlalchemy.engine.base.
>
> either TG is getting in the way, or youre not getting it right. if
> anyone ever needs to modify the internals of SA to get something to
> work, i would *highly* prefer if they could email the ML or post a
> trac ticket with their issue so that it may be handled properly.
I agree. I didn't put this on the ML or trac because I just don't know why it
behaves like that and frankly I just gave up on it after a couple of hours
and took the shortcut.
Obviously TG wraps every request into a transaction and commits/rollback's
that when the controller method returns the response.
There is a valid transaction object in the tg.sa_transaction variable which
should allow to end the outer transaction and start a new one on your own.
Although I can do that, the result was always the same - SA doesn't see the
transaction and runs into the base._autocommit method (which I think it
shouldn't if the transaction was handled properly before that). Tracking what
the DB gets, it never sees a rollback/commit when I tell the original
transaction object in tg.sa_transaction to rollback. The command is just
ignored.
When it comes to submitting a patch: I do have code changes for SA to handle
arrays and user defined types for postgresql in the autodetect code. So I
might roll that up and send it over when I have a minute.
>
> > Obviously SA thinks there is no transaction in TG,
> > so it just wraps one around it.
>
> if TG actually has "a transaction" going on, theyd certainly have to
> configure SA to be aware of it (most likely via SessionTransaction).
> if not, then yes things arent going to work at all (though still, an
> explicit SA transaction should work all by itself).
The tg.sa_transaction variable contains a SessionTransaction object at that
time. So they obviously do something to get that.
> > I agree, and that is certainly DB dependent. Personally I can't imagine
> > that an automatically issued rollback for every select transaction is in
> > any way more overhead than issuing a commit. Not wrapping a select in a
> > transaction will definitely be the least overhead.
>
> we dont issue a rollback for every select transaction. we issue a
> rollback when a connection is returned to the pool. you can check out
> a connection explicitly and perform any number of selects on it
> without any rollbacks or commits.
>
> because the "rollback" is at the connection-pool checkin level, it
> should be more apparent how inappropriate it would be to issue a
> *commit* every time a connection is returned to the pool, an operation
> that knows nothing about what just happened with that connection. the
> rollback is to release database locks.
>
> im thinking that it might be time to allow an option in SA that just
> turns the DBAPI's "autocommit" flag on. that way you can just blame
> the DBAPI for whatever issues arise.
:-) Yes. I bet the result would be the same in my case.
>
> its not always possible to "not wrap a select in a transaction".
> oracle for example *always* has a transaction going on, so everything
> is in a transaction in all cases.
>
> > > that a stored-procedure-oriented application is "far more efficient"
> > > is *extremely* debatable and database-dependent as well.
> >
> > I doubt it's *extremely* debatable.
>
> its extremely debatable:
>
> http://www.google.com/search?q=stored+procedures+vs
>
> > Just issue 100 inserts from inside a
> > stored procedure (or 100 updates) and do the same discretely with any
> > kind of db interface. In case of the interface every statement has to be
> > parsed by the db, whereas in a stored procedure the statement is already
> > "compiled" of sorts (at least Oracle and PostgreSQL do that).
>
> the debate over SP's is about a larger issue than "is an SP faster
> than 5 separate INSERT statements". SP's are of course much better
> for micro-benchmarks like that. its their impact on application
> development and architecture where the debate comes in (read some of
> the googled articles).
>
> I am certainly not anti-SP, ive done pure SP applications before (on
> projects where the DBAs controlled the SPs)...I just dont want to
> start hardwiring SQLAlchemy to expect that sort of application. I
> think 80/20 as applied to SELECT is that 80% of SELECTs are for read
> operations and a COMMIT is inappropriate.
I agree with your goals and I also agree that a lot of people don't use stored
procedures and similar db-dependent things. All I'd need is a way to tell SA
that this one select is something worth committing.
It's not SP only. I have maybe 20 SP's. The majority is views and very little
access goes to the tables directly (only for changes). The majority of data
comes from 2 different client applications, neither of which uses SA. The
third is a web interface which is mainly readonly, so SA simplifies a lot of
things on the web-end.
> for my purposes as the maintainer of SQLAlchemy, i need to support the
> majority of use cases which is that of a non-SP oriented application.
> not just because SQL-oriented apps are more common, but also because
> SP-oriented apps aren't going to have use for higher level SQL
> toolsets anyway since all the SQL is behind stored procedures.
Again, not SP only. I only use SP's for pieces where I just don't want to
recode business logic for the different client applications. Coding this
stuff in each client is a nightmare to maintain (particularly because one of
them is an old legacy system - AS400 ring a bell? Yeah, the financial
industry is still a long way behind....)
> > the end). In the end I think it's a SA issue - there should be a
> > parameter allowing to tell SA that this "select" is actually something
> > transactional and needs to commit if no error is raised.
>
> this is this:
> > michael bayer wrote:
> > i
> > also have a notion of SQL functions being marked as "transactional" to
> > help this issue.
>
> but really, i think you should get your explicit transactions
> working...i would imagine you have the need to execute multiple SPs in
> one transaction (otherwise that must be some enormous SP youre
> running).
If I could figure out how, I probably would. The code from the docs doesn't
work, because TG does something around that. I guess one of the major
drawbacks of TG is it's documentation (or lack thereof) when it comes to
using SA together with it. I guess that will change once TG uses SA by
default.
Mike Orr
> 3. Either some middleware or just a few lines inside the
> BaseController
> of the TurboGears template that starts a transaction on every
> request and puts it into the transaction manager.
>
> the third item should be in the TurboGears template by default.
... and available to non-TG Pylons applications. The @transactional
decorator sounds like a good idea, possibly with a False flag if the
default is transactional.
This is a good point to ask, how will the TG template differ from the
standard Pylons template? Will there be a base controller, or at what
point will they diverge? Where will CherryPy-style dispatching be
implemented? I imagine you will override PylonsApp and/or
PylonsBaseWSGIApp in pylons.wsgiapp, no?
--
Mike Orr <slugg...@gmail.com>
Alberto Valverde
The dispatching is currently implemented in a "route" method in a
PylonsWSGIController subclass. There's a route defined like:
map.connect('*url', controller='root', action='route')
That handles it url to further dispatch it, cherrypy style. Here's
all the meat:
http://trac.turbogears.org/browser/trunk/tg/controllers.py#L58
Alberto
Noah Gift
Alberto
Jonathan LaCour
> ... and available to non-TG Pylons applications. The @transactional
> decorator sounds like a good idea, possibly with a False flag if the
> default is transactional.
Agreed.
> This is a good point to ask, how will the TG template differ from
> the standard Pylons template? Will there be a base controller,
> or at what point will they diverge? Where will CherryPy-style
> dispatching be implemented? I imagine you will override PylonsApp
> and/or PylonsBaseWSGIApp in pylons.wsgiapp, no?
The TurboGears template, as of right now, is basically the same as
a Pylons template, only it includes a default route to a special
WSGIController subclass called TurboGearsController that implements
all sorts of nice things, like object-dispatch, content-negotation,
validation, and TG style @expose decoration. This controller is called
the "Root" controller and is preconfigured in the routes configuration:
map.connect('*url', controller='root', action='route')
You can find the controller here:
http://trac.turbogears.org/browser/trunk/tg/controllers.py
The route method of the controller probably needs to be split up
into some smaller, overridable chunks. The nice thing about the
approach that we have taken thus far is that you can mix TG-style
object-dispatching controllers and regular Pylons controllers if you
like.
Mike Orr
> a Pylons template, only it includes a default route to a special
> WSGIController subclass called TurboGearsController that implements
> all sorts of nice things, like object-dispatch, content-negotation,
> validation, and TG style @expose decoration. This controller is called
> the "Root" controller and is preconfigured in the routes configuration:
>
> map.connect('*url', controller='root', action='route')
>
> You can find the controller here:
>
> http://trac.turbogears.org/browser/trunk/tg/controllers.py
>
> The route method of the controller probably needs to be split up
> into some smaller, overridable chunks. The nice thing about the
> approach that we have taken thus far is that you can mix TG-style
> object-dispatching controllers and regular Pylons controllers if you
> like.
So if i want to use this for all URLs under a certain prefix, I would
direct the route to my subclass of TurboGearsController, using action
"route" (always?).
Line 66 (object_dispatch) returns my TG action method in all its
decorated glory, line 85 calls it and receives a dict, and line 88
calls a special render_response that invokes the template.
render_response is copying things out of the 'c' variable, so
templates would use '${foo}' rather than '${c.foo}' for data values.
Is this right?
I have long thought about making a TG-style dispatcher for Pylons.
It's great that somebody with more expertise has actually done it.
Noah wrote:
> The controller is also slightly different as it is based a bit on Rick Copeland's > q-lookup Traversable system to a degree:
> http://code.feestjeproject.com/feestje/browser/trunk/pyatl/lib/generic.py
This reminds me of Quixote, especially the '._q_lookup' method. Did
they borrow it from him or did he borrow it from them?
--
Mike Orr <slugg...@gmail.com>
Jonathan LaCour
> So if i want to use this for all URLs under a certain prefix, I would
> direct the route to my subclass of TurboGearsController, using action
> "route" (always?).
Well, thats the idea. It needs lots of love and testing, though.
> Line 66 (object_dispatch) returns my TG action method in all its
> decorated glory, line 85 calls it and receives a dict, and line
> 88 calls a special render_response that invokes the template.
> render_response is copying things out of the 'c' variable, so
> templates would use '${foo}' rather than '${c.foo}' for data values.
> Is this right?
Yes. Most of us TurboGears folks are not big fans of single
character globals that represent stacked object proxies :) The
TurboGearsController will work like TurboGears 1.0 controllers in that
methods return dicts and those dicts are passed into the namespace of
the template.
> I have long thought about making a TG-style dispatcher for Pylons.
> It's great that somebody with more expertise has actually done it.
Well, most of the credit goes to Rick Copeland, since he hacked most of
the code for the new controller.
> This reminds me of Quixote, especially the '._q_lookup' method. Did
> they borrow it from him or did he borrow it from them?
You'll have to ask Rick where he came up with the idea, all I know is
that I find it to be extremely useful!
Noah Gift
--
Jonathan LaCour
http://cleverdevil.org
--~--~---------~--~----~------ ------~-------~--~----~
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To post to this group, send email to pylons-...@googlegroups.com
To unsubscribe from this group, send email to pylons-discuss-unsubscribe @googlegroups.com
For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en
-~----------~----~----~----~-- ----~----~------~--~---
--
http://www.blog.noahgift.com
Michael Bayer
On Jun 28, 2:32 pm, "Uwe C. Schroeder" <u...@oss4u.com> wrote:
> If I could figure out how, I probably would. The code from the docs doesn't
> work, because TG does something around that. I guess one of the major
> drawbacks of TG is it's documentation (or lack thereof) when it comes to
> using SA together with it. I guess that will change once TG uses SA by
> default.
my random guess is that they create a SessionTransaction, then they
forget about it. if you then go and do some straight SQL like
select(foo).execute(), that goes against the engine, but that engine
needs to be set up explicitly within the SessionTransaction:
http://www.sqlalchemy.org/docs/unitofwork.html#unitofwork_transaction_sql
alternatively, you can pull out the connection straight from the
session and use that, via session.connection() - this also registers
the engine with the current transaction.
when using just the ORM for Query/flush(), the Session does this for
you when it first uses an engine while the transaction is present...so
it would seem like it "just works", when really it only "just works"
for ORM logic.
Michael Bayer
On Jun 28, 2:01 pm, Alberto Valverde <albe...@toscat.net> wrote:
> One problem I see is that this might cause trouble in cases like this:
>
> Request
> |
> V
> db transaction middleware -> creates session A
> |
> V
> authentication middleware -> authenticates user and keeps ref to User
> instance (attached to session A)
> |
> V
> other middleware and PylonsApp
> |
> V
> TG controller -> creates session B inside __call__
> |
> V
> authorization library tries to check permissions for mapped Foo
> instance (attached to session B) which has a owner attribute:
>
> foo.owner (sess B) == logged_in_user (sess A)
you generally should not share objects between sessions, things will
break. for example, the right way to use the user from session a into
session b would be :
foo.owner = sessionb.merge(user_from_sessiona)
Noah Gift
drawbacks of TG is it's documentation (or lack thereof) when it comes to
> using SA together with it.
I guess that will change once TG uses SA by
> default.
http://www.blog.noahgift.com
rikl...@gmail.com
that make thinks easier and quicker
- wrappers for various backends ("tg-user" for authkit and others)
- set of helpers (toscawidgets, webhelpers)
Uwe C. Schroeder
Funny you mention Twisted. May app is actually Twisted based - and yes, I
agree. I had some good (and some bad) experiences with Glyph emailing back
and forth to get things resolved :-)
--
UC
Mike Orr
> I am pretty sure Turbogears riding on Pylons will be on a vengeance to
> document things correctly and not just the API, but usage scenarios for
> everything.
Joining forces will make a big difference. TG and Pylons combined
probably account for half of the non-Zope Python web applications
written in the past two years, give or take ten percent, and roughly
similar for the amount of developer activity. Having twice as many
people working on the same codebase and docs rather than two separate
ones should accellerate developement. There may still be two wiki
demos but they'll be able to borrow half their code from each other
rather than just the look & feel. Our skills are also complementary.
TG developers do a better job of documentation and newbie handholding,
while Pylons has attracted developers who are obsessed with modularity
and reuse to a greater degree. (TG developers may doubt this, but
many people came to Pylons because they found TG more monolithic than
they liked.) Nine months till the next PyCon: it'll be exciting to
see where we're at then, and what sprints we'll be having.
--
Mike Orr <slugg...@gmail.com>
Noah Gift
--
Mike Orr <slugg...@gmail.com>
http://www.blog.noahgift.com
Mike Orr
> Yes, this is really exciting stuff. I vote we capitalize on the energy will
> it is hot and start making subprojects and delegating things. One excellent
> strategy is to use community user groups like the one I help
> organize....http://www.pyatl.org. Give us some group
> projects and then do that same thing for groups throughout the world.
The Seattle Python group would certainly be interested, any excuse for
pair programming.
I started a wiki page for this.
http://docs.pythonweb.org/display/pylonscommunity/Community+Development+Wishlist
--
Mike Orr <slugg...@gmail.com>