Adding transactions to @expose

[Sean Cazzell]

There has been a lot of interest on the list of having a way to
automatically wrap a request in a transaction. The most obvious way to
do this is to add it into expose. Michele has done a great job putting
together the relevant information (with suggestions from Ian Bicking) on
ticket #80 here:

http://trac.turbogears.org/turbogears/ticket/80

Since I am already doing some hacking on expose (moving the validator
and inputform functionality out to separate decorators) I thought I'd
pick this one up.

My question is - do we want this on by default, or do we want to
explicitly enable it with a parameter to expose?

Enabled by default (must explicitly disable):
@expose(transaction=False)

Disabled by default (must explicitly enable):
@expose(transaction=True)

Obviously the latter is backwards compatible, but if everyone is going
to want a transaction for all of their controller methods, maybe it
should be enabled by default?


Sean

[Michele Cella]

Sean CazzellCazzell wrote:
> There has been a lot of interest on the list of having a way to
> automatically wrap a request in a transaction. The most obvious way to
> do this is to add it into expose. Michele has done a great job putting

> together the relevant information (with suggestions frBickingBicking) on

> ticket #80 here:
>
> http://trac.turbogears.org/turbogears/ticket/80
>

Thanks Sean for taking into this ticket.

> Since I am already doing some hacking on expose (movivalidatorvalidatorinputform
inputforminputformcorators) I thought I'd

> pick this one up.
>
> My question is - do we want this on by default, or do we want to
> explicitly enable it with a parameter to expose?
>
> Enabled by default (must explicitly disable):
> @expose(transaction=False)
>
> Disabled by default (must explicitly enable):
> @expose(transaction=True)
>
> Obviously the latter is backwards compatible, but if everyone is going
> to want a transaction for all of their controller methods, maybe it
> should be enabled by default?
>

I vote for making it disabled by default, since you will only use this
when you are saving to the db having:

@expose(transaction=True)

is more explicit, and (as you noted) backwards compatible.

>
> Sean

Ciao
Michele

[Jonathan LaCour]

> Since I am already doing some hacking on expose (moving the validator
> and inputform functionality out to separate decorators) I thought I'd
> pick this one up.

Fantastic! This is probably my biggest gripe with TurboGears right
now, and I really was hoping someone would step up and work on this!

> My question is - do we want this on by default, or do we want to
> explicitly enable it with a parameter to expose?

> <snip snip>

> Obviously the latter is backwards compatible, but if everyone is going
> to want a transaction for all of their controller methods, maybe it
> should be enabled by default?

I for one *expected* it to be on by default, if not only because it
is desired behavior for most cases that I can think of. In my
current project, it would reduce the size of my controllers by at
least 50%.

I do have a question for you: how are you going to handle Exceptions
and the ability to display nice errors with `flash`? Typically, I
end up with a try:except:finally block where I `flash('some nice
error message')` and do a `hub.rollback()`. I may have a different
error message for different types of Exceptions...

Maybe this could be handled sort of like validators?

@turbogears.expose(transaction=True,
err_map={Exception : 'generic msg',
SQLObjectNotFound : 'specific msg'}
def my_method(self):
frob = frobbler.frobble()
return dict(frob=frob)

On an exception being raised, the transaction would automatically be
rolled back, and the appropriate error message would be flashed on
the page. Just thinking out loud, and this obviously isn't perfect.
But, its an important wrinkle to figure out.

-- Jon

[Ian Bicking]

Jonathan LaCour wrote:
> I do have a question for you: how are you going to handle Exceptions
> and the ability to display nice errors with `flash`? Typically, I end
> up with a try:except:finally block where I `flash('some nice error
> message')` and do a `hub.rollback()`. I may have a different error
> message for different types of Exceptions...

I think "expected" exceptions (including redirects) should not roll back
the transaction. Unexcepted exceptions (everything else should).
However, you should be able to explicitly roll back or commit the
transaction if the decorator doesn't do what you want.

So in your example you'd have to explicitly roll back the transaction if
you didn't want your otherwise normal-looking response to cause a
rollback.

--
Ian Bicking / ia...@colorstudy.com / http://blog.ianbicking.org

[Sean Cazzell]

On Tue, 2005-11-15 at 17:31 -0500, Jonathan LaCour wrote:
> Maybe this could be handled sort of like validators?
>
> @turbogears.expose(transaction=True,
> err_map={Exception : 'generic msg',
> SQLObjectNotFound : 'specific msg'}
> def my_method(self):
> frob = frobbler.frobble()
> return dict(frob=frob)
>
> On an exception being raised, the transaction would automatically be
> rolled back, and the appropriate error message would be flashed on
> the page. Just thinking out loud, and this obviously isn't perfect.
> But, its an important wrinkle to figure out.

I would recommend you put this in a separate decorator like this:

@turbogears.expose(...)
@exception_flash({Exception: 'generic msg'})
def mymethod(self):
... etc ...

It may be cool to have an exception_flash decorator built in to
turbogears. Though I really wish cherrypy had an event system or better
filtering system so we didn't have to push all of this functionality
into decorators.


Sean

[Jonathan LaCour]

On Nov 15, 2005, at 5:34 PM, Ian Bicking wrote:
>> I do have a question for you: how are you going to handle
>> Exceptions and the ability to display nice errors with `flash`?
>> Typically, I end up with a try:except:finally block where I `flash
>> ('some nice error message')` and do a `hub.rollback()`. I may
>> have a different error message for different types of Exceptions...
>
> I think "expected" exceptions (including redirects) should not roll
> back the transaction. Unexcepted exceptions (everything else
> should). However, you should be able to explicitly roll back or
> commit the transaction if the decorator doesn't do what you want.

Agreed, especially when it comes to redirects. This is actually one
reason that I don't like redirect being an Exception... I would
prefer it to be a `return` instead... it causes all sorts of confusion.

But, I think you missed my point on how to set the error message to
be `flashed` in the case of an exceptional Exception ... I don't want
to have to continue to use a try:except: clause to handle error
messages in the context of an automatic request-per-transaction. I
would like this to be automated as well, since it probably needs to
happen inside of the transaction `try:except:` in the decorator.

From the wiki, I think your suggestion was:

hub.begin()
try:
do stuff
except:
hub.rollback()
raise
else:
hub.commit()

This still leaves me to catch the re-raised exception and display an
error message. Why not go ahead and handle the setting of the error
message?

hub.begin()
try:
do stuff
except Exception, e:
hub.rollback()
message = err_map.get(e.__class__, 'Generic message')
flash(message)
else:
hub.commit()

This would reduce a lot of boilerplate error handling. Most of my
controller methods look like this:

@turbogears.expose()
def update_frobble(self, frob_id, new_name):
hub.begin()
try:
frobbler = Frobbler.get(frob_id)
frobbler.set(name=new_name)
except SQLObjectNotFound:
flash("Couldn't find frobbler!')
hub.rollback()
else:
hub.commit()
raise cherrypy.Redirect('/some/page')

It would be great if all I had to write was:

@turbogears.expose(err_map={SQLObjectNotFound :
"Couldn't find frobbler!"})
def update_frobble(self, frob_id, new_name):
frobbler = Frobbler.get(frob_id)
frobbler.set(name=new_name)
raise cherrypy.Redirect('/some/page')

> So in your example you'd have to explicitly roll back the
> transaction if you didn't want your otherwise normal-looking
> response to cause a rollback.

I almost always want Exceptions to trigger a rollback, but I also
frequently want to display an error message using `flash` that
describes to the user what happened.

Does this make sense, or am I alone here?

-- Jon

[Leandro Lucarella]

Sean Cazzell, el martes 15 de noviembre a las 17:09 me escribiste:

What about a different decorator? I don't think is a good idea to keep
adding complexity to expose. And the transaction is not orthogonal to
exposure, you can have no-exposed event wrapped in a transaction an vice
versa.

So you can have:

@expose(template='something')
@transaction
def list():
# ...

or

@transaction
def save(something):
# ...

Better name accepted =)

--
Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/
.------------------------------------------------------------------------,
\ GPG: 5F5A8D05 // F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05 /
'--------------------------------------------------------------------'
Hey you, standing in the road
always doing what you're told,
Can you help me?

[Sean Cazzell]

On Tue, 2005-11-15 at 16:34 -0600, Ian Bicking wrote:
> I think "expected" exceptions (including redirects) should not roll back
> the transaction. Unexcepted exceptions (everything else should).
> However, you should be able to explicitly roll back or commit the
> transaction if the decorator doesn't do what you want.

See, I knew I was forgetting something. I wonder what a reasonable list
of "expected exceptions" that we don't rollback on is? It should
probably be short, I don't want to not rollback if someone is expecting
it. I agree with Ian on:

cherrypy.HTTPRedirect

What about:

cherrypy.NotFound
cherrypy.HTTPError
cherrypy.InternalRedirect

I think we should include InternalRedirect and not the other two. The
list of cherrypy errors can be found in _cperror.py.


Sean Cazzell

[Brian Dorsey]

On 15/11/05, Sean Cazzell <seanc...@gmail.com> wrote:
> Disabled by default (must explicitly enable):
> @expose(transaction=True)
>
> Obviously the latter is backwards compatible, but if everyone is going
> to want a transaction for all of their controller methods, maybe it
> should be enabled by default?

I'm a TG and SQLObject newbie, so I may be missing something. But, as
a database guy, I'd prefer to see it off by default.

I'm making the assumption that some if not most pages on a site will
be reading, but not modifying data, and that the database transactions
are based on locks of some sort. In that case, it makes more sense
from the database side to only use a transaction when actually
modifying data, especially in a multi-user application. Wrapping it up
in expose is a nice clean way of doing that, but I'm (perhaps
unreasonably?) worried that wrapping every single request in a
transaction by default is unneccesary overhead.

One other thought: My current TurboGears application doesn't even use
SQLObject, I'm using direct database queries and an XML-RPC server.
I'd love to see this kind of scenario keep working.

Thanks for doing the work on the transactions!

Take care,
-Brian

[Sean Cazzell]

On Tue, 2005-11-15 at 20:54 -0300, Leandro Lucarella wrote:
> What about a different decorator? I don't think is a good idea to keep
> adding complexity to expose. And the transaction is not orthogonal to
> exposure, you can have no-exposed event wrapped in a transaction an vice
> versa.

I am in the process of ripping most of the functionality out of expose
so it will be much simpler than the current expose. You will also have
the option of using the in_transaction decorator directly if you want to
go that route (see below).


Based on feedback, I am planning to add a configuration directive to let
you enable transactions by default for exposed methods if you want.
This way those that want transactions on by default and those that want
them off can both have their way.

If you have transactions off, then you can enable them for a particular
method by passing transaction=True to expose, or you can use the
in_transaction (name suggestions are welcome) decorator directly.

Also, the HTTPRedirect and InternalRedirect exceptions will not cause a
rollback when in a transaction - all other exceptions will. If anyone
wants any other exceptions to not cause a rollback, let me know.

I think this covers all of the issues everyone has raised so far? Thank
you everyone for your feedback.


Sean

[Roman Roelofsen]

> If you have transactions off, then you can enable them for a particular
> method by passing transaction=True to expose, or you can use the
> in_transaction (name suggestions are welcome) decorator directly.

What if I want to factor out my DB logic (save a user, etc) into a "service
layer" (Fowler, PoEAA) because I want to expose a certain amount of my
functionality to xmlrpc or a web service? Then, there may be a need for
chaining the methods in this service layer. Using a decorator for my
transaction management, there should be a way to say if I need a new
transaction or e.g. an active transaction.

IMHO, the name "in_transaction" is then a bit imprecise.

Sean, what do you think about this naming styles:
new_transaction()
in_transaction()
-- OR --
transaction(new=True)
transaction(new=False) or transaction(participate=True)

If a database systems allows the use of nested transaction, these naming
styles would allow to use a decorator like

nested_transaction()
-- OR --
transaction(nested=True)

The second naming style is open for improvements and I think it is not less
readable:

@in_transaction()
def meth(self):

@transaction()
def meth(self):

By the way: TurboGears is a great piece of software! Thank all of you for
making this possible and sharing it.

Best regards,

Roman

[Kevin Dangoor]

On 11/15/05, Sean Cazzell <seanc...@gmail.com> wrote:
> It may be cool to have an exception_flash decorator built in to
> turbogears. Though I really wish cherrypy had an event system or better
> filtering system so we didn't have to push all of this functionality
> into decorators.

How would you, in an ideal world, see the API as being different with
events or "better filtering"? Always best to start with the ideal and
then make tradeoffs as needed.

I actually think there are certain advantages to the decorator model,
which is why I had started building up functionality in expose. (In
fact, before TurboGears 0.5 there was a point where the
dictionary->Kid/JSON functionality was in a filter). What's nice about
it is that the method defines the functionality that you're exposing
to the outside world. The decorator is basically the metadata about
how you want to expose it. And it's all neatly kept together.

Kevin

[Jonathan LaCour]

On Nov 16, 2005, at 9:23 AM, Kevin Dangoor wrote:
> How would you, in an ideal world, see the API as being different with
> events or "better filtering"? Always best to start with the ideal and
> then make tradeoffs as needed.

At least in the case of transactions, it would be nice if it wasn't a
transaction per controller method, but a transaction per *request*,
which isn't really handled by the decorator case. I think it _could_
be handled better by a filter-like approach.

What if my controller method calls another controller method that is
also decorated to run inside a transaction? Will the decorator be
smart enough to use the current transaction? In the case of a filter-
based approach, you could instead start a transaction
`onRequestStart`, rollback and display an error message
`onRequestError`, and commit `onRequestEnd`. This way, irrespective
of the code path, you are guaranteed one transaction-per-request in a
simple and straightforward manner. In this way, I think of
TurboGears as a container for my WSGI application. I manage the
logic, it manages the boilerplate for transactions, logging,
exceptions, etc.

In my darker days of ASP.NET development (*shudder*), I frequently
used Global.asax, which provides filter capabilities very much like
the above, for doing transactions and error handling. Its one of the
only parts of ASP.NET that isn't entirely nauseating :)

-- Jonathan

[Kevin Dangoor]

On 11/16/05, Jonathan LaCour <jonatha...@cleverdevil.org> wrote:
>
> On Nov 16, 2005, at 9:23 AM, Kevin Dangoor wrote:
> > How would you, in an ideal world, see the API as being different with
> > events or "better filtering"? Always best to start with the ideal and
> > then make tradeoffs as needed.
>
> At least in the case of transactions, it would be nice if it wasn't a
> transaction per controller method, but a transaction per *request*,
> which isn't really handled by the decorator case. I think it _could_
> be handled better by a filter-like approach.

For the potentially many methods that are just reading data, you don't
really want a transaction, and decorators are a nice way to configure
where you want the transaction. Now that i think about it,
transactions are already maintained on the thread level so it should
be fairly straightforward to keep reusing the same transaction. As
Roman points out, there may be times where you want to use nested
transactions (just make sure you're not using sqlite). So the trick is
finding the best defaults and an easy way to diverge from the default.

Kevin

[vinjvinj]

>>One other thought: My current TurboGears application doesn't even use
>>SQLObject, I'm using direct database queries and an XML-RPC server.
>>I'd love to see this kind of scenario keep working.

I second that as well. My application also does not use the SQLObject
stuff yet.
Even when I finally use it, there will some portions of the application
which will
use direct queries to DB. I would like it to be turned off by default.

[Jonathan LaCour]

> For the potentially many methods that are just reading data, you don't
> really want a transaction, and decorators are a nice way to configure
> where you want the transaction.

I fail to see the harm in starting a transaction for read-only
methods? What's the big deal? It seems like transactional integrity
is pretty important, and automating it would be really nice :)

Assuming that most methods are "read only" is assuming things about
the application. That isn't safe to do, because there are many types
of applications. Some will be largely reading, others may be largely
update/create/delete, others may be an even split. You don't know
which is the primary case. What you *do* know is that read-only
methods can safely run in a transaction and that other methods *have
to* run in a transaction. If you run in a transaction in both cases,
everything is good. If you don't, then something can go wrong.
Seems like we have found our default, no?

In addition, maybe methods are read-only, but something behind the
scenes is doing some database work, possibly in a filter, like
logging certain views to the database, or updating a "last_access"
field in a session database backend. In these scenarios, you might
want a transaction. Its just too hard to tell.

Seems like "transaction-per-request always" is the absolute safest
bet, at least to me, and I don't see the harm in making it the
default. Of course, I am only going on my experience, and I am sure
that there are people who will disagree with me there.

> Now that i think about it,
> transactions are already maintained on the thread level so it should
> be fairly straightforward to keep reusing the same transaction. As
> Roman points out, there may be times where you want to use nested
> transactions (just make sure you're not using sqlite). So the trick is
> finding the best defaults and an easy way to diverge from the default.

Fair enough, but you are *assuming* threads, which is another unsafe
assumption. We need to make sure to conform to the WSGI spec
regarding multi-process vs. multi-thread. People like me just aren't
going to be deploying their apps inside a multi-threaded server. It
should be easier in that case anyway, because then its one connection
per process :) So, you just use the processes' current transaction
instead of the thread's current transaction.

As for defaults, I still vote for request-per-transaction by
default. It is the absolute safest thing to do, in my opinion,
because any other decision is making assumptions about the
applications that people are writing with TurboGears, how they are
deploying it, etc.

The only other alternative that I can think of, is to automatically
realize when you need to run inside a transaction by having SQLObject
start one automatically when a create, update, or delete occurs and
then have TurboGears commit or rollback any open transactions at the
conclusion of a request depending on whether or not an "exceptional"
Exception occurred.

-- Jon

[Jonathan LaCour]

Fair enough, and I just sent an email railing on making assumptions
about people's applications, but it seems to me like your case is not
the case to optimize for. SQLObject is a core piece of TurboGears.
Sure, you aren't using it, but its part of the core, its part of all
the screencasts, and I think most people will be using it. Again,
this is an assumption, and I am open to being proved wrong here :)

Set the default for the common and safe case, and let people who
don't use SQLObject flip a switch.

-- Jonathan

[Kevin Dangoor]

On 11/16/05, Jonathan LaCour <jonatha...@cleverdevil.org> wrote:
>

> > For the potentially many methods that are just reading data, you don't
> > really want a transaction, and decorators are a nice way to configure
> > where you want the transaction.
>
> I fail to see the harm in starting a transaction for read-only
> methods? What's the big deal? It seems like transactional integrity
> is pretty important, and automating it would be really nice :)

I absolutely agree that transaction integrity is very important. I
always work in transactions when I'm updating data.

The issue is that SQLObject uses a separate cache for each
transaction, but one cache when you're not working in a transaction.
If you have lots of reads, you'll get lots of cache hits if you don't
put those read operations in a transaction.

> Assuming that most methods are "read only" is assuming things about
> the application. That isn't safe to do, because there are many types
> of applications. Some will be largely reading, others may be largely
> update/create/delete, others may be an even split. You don't know
> which is the primary case. What you *do* know is that read-only
> methods can safely run in a transaction and that other methods *have
> to* run in a transaction. If you run in a transaction in both cases,
> everything is good. If you don't, then something can go wrong.
> Seems like we have found our default, no?

When coming up with defaults, we do have to make some assumption about
what people are doing.

I agree with you that transactions are the safer route, and there are
many people who don't think about using transactions. (I wonder how
many read/write databases still use MyISAM tables?) By turning them on
by default, we automatically help people get a bit more consistency to
their data. That sounds reasonable to me. As long as it's an easy bit
to flip, I have no qualms about choosing what is ultimately a safer
default.

> > Now that i think about it,
> > transactions are already maintained on the thread level so it should
> > be fairly straightforward to keep reusing the same transaction. As
> > Roman points out, there may be times where you want to use nested
> > transactions (just make sure you're not using sqlite). So the trick is
> > finding the best defaults and an easy way to diverge from the default.
>
> Fair enough, but you are *assuming* threads, which is another unsafe
> assumption. We need to make sure to conform to the WSGI spec
> regarding multi-process vs. multi-thread. People like me just aren't
> going to be deploying their apps inside a multi-threaded server. It
> should be easier in that case anyway, because then its one connection
> per process :) So, you just use the processes' current transaction
> instead of the thread's current transaction.

Yes, I said thread because that's the default mode of operation for
CherryPy, but working with a process-wide connection should work
equally well. (Assuming we fix the caching problems you were seeing.)

Kevin

[Ian Bicking]

Jonathan LaCour wrote:
>
>> For the potentially many methods that are just reading data, you don't
>> really want a transaction, and decorators are a nice way to configure
>> where you want the transaction.
>
>
> I fail to see the harm in starting a transaction for read-only
> methods? What's the big deal? It seems like transactional integrity
> is pretty important, and automating it would be really nice :)

Well, just as a data point, in Zope (where you always get a transaction
whether you like it or not, and aren't given any control over how it
works) this has kicked me in the ass before. It's also usually nice.
But the ass-kicking really sucks. One of Zope's features is that if you
get a conflict error, it'll re-run the request for you, essentially
creating a situation where when contention gets high you don't require
the user to hit submit over and over, you hit submit over and over *for*
the user. Lovely.

However, conflict errors are inevitable when you start running
everything in transactions, and that's something that should probably be
taken into account.

That said, there certainly is a genuine appeal to putting everything in
transactions.

> The only other alternative that I can think of, is to automatically
> realize when you need to run inside a transaction by having SQLObject
> start one automatically when a create, update, or delete occurs and
> then have TurboGears commit or rollback any open transactions at the
> conclusion of a request depending on whether or not an "exceptional"
> Exception occurred.

SQLObject can't current do this, but I don't think this would be too
difficult. Basically you'd create another Transaction subclass, and it
would watch for updating methods, and would only start using its own
transactional connection once one had gone by. I think this could
reduce the likelihood of contention quite a bit, so long as you aren't
too concerned about absolutely isolated transactions. I.e., your first
select may not be consistent with your second select, but if you run
that in a transaction (with the right isolation) it would be.

There are some open caching issues with transactions as well, that
require some thinking about what has to be expired when. And, really,
transactions could probably make more use of caching than they currently do.

[Jeff Watkins]

Jonathan LaCour wrote:
> What if my controller method calls another controller method that is
> also decorated to run inside a transaction? Will the decorator be smart
> enough to use the current transaction? In the case of a filter-based
> approach, you could instead start a transaction `onRequestStart`,
> rollback and display an error message `onRequestError`, and commit
> `onRequestEnd`.

Unfortunately, CherryPy filters will only allow us to rollback or
commit. We can't actually display a meaningful page. I filed a ticket
for this #392 with the CherryPy team, but they seem to think it is
working correctly. I'll have to write a short example to show exactly
what's going wrong.

Basically, whenever an exception was raised, in my case an Identity
related exception, I wanted to catch the exception and display an error
page. I tried everything I could think of and certainly everything that
was documented, but I could never manage to convince CherryPy to do
anything other than report a 500 error with a traceback.

Maybe this has been fixed since I tried (which was pre-2.1 release), but
I'm guessing I didn't word my ticket specifically enough.

My other ticket (#391) was summarily closed too: I think it's reasonable
to raise an InternalRedirect in a filter rather than muck about with the
nearly undocumented cherrypy.request.objectPath attribute.


--
Jeff Watkins
http://newburyportion.com/

[Jonathan LaCour]

>> What if my controller method calls another controller method that
>> is also decorated to run inside a transaction? Will the decorator
>> be smart enough to use the current transaction? In the case of a
>> filter-based approach, you could instead start a transaction
>> `onRequestStart`, rollback and display an error message
>> `onRequestError`, and commit `onRequestEnd`.
>
> Unfortunately, CherryPy filters will only allow us to rollback or
> commit. We can't actually display a meaningful page. I filed a
> ticket for this #392 with the CherryPy team, but they seem to think
> it is working correctly. I'll have to write a short example to show
> exactly what's going wrong.

This got me thinking that maybe filters are the wrong approach, as
are decorators. Why not make both the transaction-per-request and
automated-error-handling WSGI middleware? I am not an expert on the
WSGI specification, but it seems at a cursory glance to be possible,
and then we wouldn't have to convince CherryPy to change anything.

-- Jonathan

[Jonathan LaCour]

> Well, just as a data point, in Zope (where you always get a
> transaction whether you like it or not, and aren't given any
> control over how it works) this has kicked me in the ass before.
> It's also usually nice. But the ass-kicking really sucks. One of
> Zope's features is that if you get a conflict error, it'll re-run
> the request for you, essentially creating a situation where when
> contention gets high you don't require the user to hit submit over
> and over, you hit submit over and over *for* the user. Lovely.

In my experience, Zope is a valuable data point for how *not* to do
things. Like you, my ass still has bruises from Zope 2. I hope this
doesn't negate my initial argument :)

> That said, there certainly is a genuine appeal to putting
> everything in transactions.

Agreed, and it appears that Kevin agrees.

>> The only other alternative that I can think of, is to
>> automatically realize when you need to run inside a transaction
>> by having SQLObject start one automatically when a create,
>> update, or delete occurs and then have TurboGears commit or
>> rollback any open transactions at the conclusion of a request
>> depending on whether or not an "exceptional" Exception occurred.
>
> SQLObject can't current do this, but I don't think this would be
> too difficult. Basically you'd create another Transaction
> subclass, and it would watch for updating methods, and would only
> start using its own transactional connection once one had gone by.
> I think this could reduce the likelihood of contention quite a bit,
> so long as you aren't too concerned about absolutely isolated
> transactions. I.e., your first select may not be consistent with
> your second select, but if you run that in a transaction (with the
> right isolation) it would be.

Given that this is also possible, it seems like there are several
different ways to do this off the top of my head:

1. WSGI middleware (I think)
2. CherryPy filters
3. SQLObject modifications
4. Decorators
5. Some mixture of the above

What is the best way? I have no idea, but I am intrigued by option
#3, because I thought it might be hard, and it would take into
account the actual use of the model objects automatically... of
course it doesn't take into account anything that happens directly
against the DB not through SQLObject...

That being said, I really don't know what the best option is. Kevin,
have you decided on a direction to take here? How can I help?

> There are some open caching issues with transactions as well, that
> require some thinking about what has to be expired when. And,
> really, transactions could probably make more use of caching than
> they currently do.

I am very interested in the direction that you are planning on
taking, and how I can help.

-- Jonathan

[Ian Bicking]

Jonathan LaCour wrote:
> This got me thinking that maybe filters are the wrong approach, as are
> decorators. Why not make both the transaction-per-request and
> automated-error-handling WSGI middleware? I am not an expert on the
> WSGI specification, but it seems at a cursory glance to be possible,
> and then we wouldn't have to convince CherryPy to change anything.

I'm not sure what the status is of WSGI middleware and CherryPy, and how
easy they are to put together. Mostly in terms of configuring a stack
of middleware. But anyway, I posted a kind of scarecrow implementation
of this a while back:
http://blog.ianbicking.org/more-perfect-app-server-wsgi-transactions.html

Also, exceptions have to be raised all the way to this middleware for
the system to work.

[Kevin Dangoor]

Yes, I have a feeling that this would not be the easiest path to
implementation, for both of these reasons.

I think the decorator path is likely the easiest, *and* it potentially
allows someone to specify that no transaction is required for a given
method.

Kevin

[Sylvain Hellegouarch]

> Given that this is also possible, it seems like there are several
> different ways to do this off the top of my head:
>
> 1. WSGI middleware (I think)
> 2. CherryPy filters
> 3. SQLObject modifications
> 4. Decorators
> 5. Some mixture of the above
>
> What is the best way? I have no idea, but I am intrigued by option
> #3, because I thought it might be hard, and it would take into
> account the actual use of the model objects automatically... of
> course it doesn't take into account anything that happens directly
> against the DB not through SQLObject...
>
> That being said, I really don't know what the best option is. Kevin,
> have you decided on a direction to take here? How can I help?
>
> > There are some open caching issues with transactions as well, that
> > require some thinking about what has to be expired when. And,
> > really, transactions could probably make more use of caching than
> > they currently do.
>
> I am very interested in the direction that you are planning on
> taking, and how I can help.

I would not go with CherryPy filters. They are not the right place to do so IMO.
CherryPy filters are point cuts around the HTTP request/response processing.

Therefore, those filters have no knowledge of the stack built on top of it.
Transactions mean nothing to CP. Thus CP can't help you the right way to use
them.

Besides, future versions of CP (the 3 branch) might as well change all of that
and thus TG would be stuck with CP 2.x

I'd say that you should always keep in mind that CP is only dealing with HTTP
requests/responses so don't try to overload CP with higher level features :)

The decorator seem to be the best place per se because it doesn't involve a
complex machinery as would to use a WSGI middleware solution (which would
by-pass CP I guess).

As for the SQLObject one, I don't know enough of it to say if it could be the
right place.

Hope this helped a bit.
- Sylvain




----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

[Ian Bicking]

Kevin Dangoor wrote:
>>I'm not sure what the status is of WSGI middleware and CherryPy, and how
>>easy they are to put together. Mostly in terms of configuring a stack
>>of middleware. But anyway, I posted a kind of scarecrow implementation
>>of this a while back:
>>http://blog.ianbicking.org/more-perfect-app-server-wsgi-transactions.html
>>
>>Also, exceptions have to be raised all the way to this middleware for
>>the system to work.
>
>
> Yes, I have a feeling that this would not be the easiest path to
> implementation, for both of these reasons.
>
> I think the decorator path is likely the easiest, *and* it potentially
> allows someone to specify that no transaction is required for a given
> method.

The method I gave does allow significant flexibility at runtime, in that
it merely calls certain methods at certain times; the logic is really in
the Manager class, and could support any number of kinds of
transactions. Well, not any number -- the Zope transaction manager
(mentioned in the comments on that post) handles two-phase commit and
some other fancy stuff. But anyway, the middleware really just ensures
certain boundaries for a transaction (bounded by the request itself), if
you add the transaction to the middleware. I.e., a transaction added to
the middleware must be rolled back or committed by the end of the
transaction, with certain defaults in the case of a successful or
unsuccessful request.

Irregardless of the way transaction boundaries are determined --
middleware or decorator (both of which are very similar in both function
and implementation, just applied at different stages) -- the possibility
of a generic manager should be considered, that can manage a variety of
transactions (both in SQLObject or elsewhere).

[Sean Cazzell]

On Wed, 2005-11-16 at 13:45 -0500, Jonathan LaCour wrote:
> Given that this is also possible, it seems like there are several
> different ways to do this off the top of my head:
>
> 1. WSGI middleware (I think)
> 2. CherryPy filters
> 3. SQLObject modifications
> 4. Decorators
> 5. Some mixture of the above

1. Unfortunately, CherryPy really only has very basic WSGI support - I
don't know of any reasonable way to hook WSGI middleware up to Cherrypy
at this point (maybe in a future release).

2. Filters in CherryPy won't work either, they don't allow you to "wrap"
the request and I don't think there is any way to guarantee that they
are actually run - especially if an exception occurs.

3. SQLObject modifications would be great, but until they are done we'll
need to have an interim solution.

4. Which leaves decorators. My plan now is to implement a transaction
decorator which will look something like:

def in_transaction(func):
hub = AutoConnectHub()
def newfunc(*args, **kw):
hub.begin()
try:
func(*args, **kw)
except cherrypy.HTTPRedirect:
hub.commit()
raise

except:
hub.rollback()
raise
else:
hub.commit()

newfunc.func_name = func.func_name
newfunc.func_doc = func.func_doc
return newfunc

The expose decorator will check the config / it's parameters and if a
transaction is desired it will wrap itself in the in_transaction
decorator.

This doesn't give us strict per-request transactions if you have
database code in your cherrypy filters. But I think any database code
in filters would probably mostly be doing reads, so this doesn't seem to
be a huge issue. It is probably possible to monkeypatch CherryPy to
give us true per-request transactions if we really need them.

The AutoConnectHub.begin method checks to see if we're already in a
transaction, so calling one transaction-wrapped method from another will
result in them both running in one transaction. I think this is the
desired default behavior.

If you need more transaction stuff beyond this, you can always write
your own decorators to do things like nested transactions. And if there
is enough demand we can add support for such things to TG in the future.

Of course, if you want you can disable all of this automatic transaction
stuff in the config and handle transactions yourself or not use
transactions / database at all.


Sean Cazzell

[Kevin Dangoor]

On 11/17/05, Sean Cazzell <seanc...@gmail.com> wrote:
> def in_transaction(func):
> hub = AutoConnectHub()
> def newfunc(*args, **kw):
> hub.begin()
> try:
> func(*args, **kw)
> except cherrypy.HTTPRedirect:
> hub.commit()
> raise
> except:
> hub.rollback()
> raise
> else:
> hub.commit()
> newfunc.func_name = func.func_name
> newfunc.func_doc = func.func_doc
> return newfunc
>
> The expose decorator will check the config / it's parameters and if a
> transaction is desired it will wrap itself in the in_transaction
> decorator.

Unfortunately, it really needs to use the hub that's in use by the
model classes... from a random model.py:

__connection__ = AutoConnectHub()
# (or PackageHub)

class Foo(SQLObject):
...

In a multi-application world, it'll be possible to have different
parts of a site going to different databases. that won't be the common
case, though.

Sticking to the common case, we just need a nice way to get ahold of
the hub in question. There are probably a small handful of ways we
could find the model module and get the hub from it. Unfortunately,
I'm out of time for exploring that now, and I'll be offline for a
while. I hope I'll be able to get back online at least briefly later
today...

Kevin

[Sean Cazzell]

On Thu, 2005-11-17 at 12:28 -0500, Kevin Dangoor wrote:
> Unfortunately, it really needs to use the hub that's in use by the
> model classes... from a random model.py:
>
> __connection__ = AutoConnectHub()
> # (or PackageHub)
>
> class Foo(SQLObject):
> ...
>
> In a multi-application world, it'll be possible to have different
> parts of a site going to different databases. that won't be the common
> case, though.
>
> Sticking to the common case, we just need a nice way to get ahold of
> the hub in question. There are probably a small handful of ways we
> could find the model module and get the hub from it. Unfortunately,
> I'm out of time for exploring that now, and I'll be offline for a
> while. I hope I'll be able to get back online at least briefly later
> today...

Kevin, I was worried that I might have to get the hub from the model. I
would definitely like to support the multiple-apps case if at all
possible.

I have to take off as well, I'll look into this more Sunday when I get
home.


Sean