sqlalchemy and run_with_transaction
Dennis Muhlestein
I find having the framework manage the database session for you OK 95%
of the time. What do you do in the other 5% of the time?
Example 1: I have certain credit card processing methods that require
commits to the database before and after the card is processed.
Example 2: I have some methods that require no database interaction.
(So why create a session? .. also.. how much overhead is there in
creating the session?)
Anyone have any best practice ideas for situations like this?
Here is what I did on previous websites:
Background:
1- When I 1st started Using SQLa, TG had no support for it but it
worked great by manually managing it.
2- SQLalchemy support didn't work with TG.run_with_transaction for a
time when SQLA moved to version 2.x
Solutions:
1-Manually manage SQLAlchemy transations
2-Hack TG in my start script to avoid run_with_transaction altogether:
==snip==
# hack for sa 0.2.x
def nothing(func,*args,**kw):
return func(*args,**kw)
turbogears.database.run_with_transaction=nothing
==end snip==
Anyway, I'm doing some new work and it would be the perfect time to get
rid of that hack but I don't have a real good solution for the 2
scenarios I mentioned above.
Thoughts?
-Dennis
Alberto Valverde
run_with_transaction is a generic function you can specialize to fit
your needs, for example:
from turbogears.database import run_with_transaction, _use_sa
@run_with_transaction.when("getattr(func, '_no_trans', False) and
_use_sa()")
def _no_trans_for_this_method(func, *args, **kw):
return func(*args, **kw)
Now in any controller method you can set a _no_trans attribute to
True so this specialized version which doesn't start a session or
begins any transaction runs when run_with_transaction is called:
@expose()
def fooo(self):
#do something, calling this method will not run under a transaction.
fooo._no_trans = True
Much more flexible than monkey-patching (http://tinyurl.com/uz3s9),
isn't it? :)
Alberto
Dennis Muhlestein
> from turbogears.database import run_with_transaction, _use_sa
>
> @run_with_transaction.when("getattr(func, '_no_trans', False) and
> _use_sa()")
> def _no_trans_for_this_method(func, *args, **kw):
> return func(*args, **kw)
>
Wonderful! I hadn't seen any reference to decorators for the
run_with_transaction method in the documentation. Perhaps I missed it
if it is there.
Thanks
Dennis Muhlestein
> > -Dennis run_with_transaction is a generic function you can specialize to fit
> your needs, for example:
>
> from turbogears.database import run_with_transaction, _use_sa
>
> @run_with_transaction.when("getattr(func, '_no_trans', False) and
> _use_sa()")
> def _no_trans_for_this_method(func, *args, **kw):
> return func(*args, **kw)
>
> Now in any controller method you can set a _no_trans attribute to
> True so this specialized version which doesn't start a session or
> begins any transaction runs when run_with_transaction is called:
>
This is a nice approach to this problem. I learned how generic
functions work and checked out how run_with_transaction is implemented
using that concept.
There is one issue though, when using the approach mentioned, my
_no_trans_ method still isn't called. The _use_sa() function returns
another the sa_rwt method because it evaluates to true before this rule
is evaluated. I see an implementation for Ordered generic functions in
the genericfunctions.py file but that method is not being used by
run_with_transaction.
If I could call:
@run_with_transaction("getattr(func,"_no_trans",False) and
_use_sa()",order=0)
Then this would work I believe.
Should TG be patched to handle this or am I missing something?
Thanks
-Dennis
Alberto Valverde
That should have worked (unless I'm missing something) because the
rule that I wrote in the example that checks for _use_sa() *and*
getattr(...) is more specific than _use_sa() by itself, however, for
some strange reason (at least strange to me) it seems it doesn't
work. Have you tried swapping the predicates?
"""_use_sa() and getattr(func,"_no_trans",False)"""
If that still doesn't make it, fortunately, as you've noticed, TG has
the infrastruture to handle the "order" argument to the
GenericFunction decorators. You'll need to patch
database.run_with_transaction so it looks like this:
from turbogears.genericfunctions import MultiorderGenericFunction
@generic(MultiorderGenericFunction)
def run_with_transaction(....)
You'll need to set the order to -1 so it runs before the default
which is 0.
If this solution works and all tests pass you could submit a ticket
with the patch so we can commit it.
However, TG 1.0 is in a feature-freeze standstill, so the dilema is:
is this a new feature or a bug fix? I'd consider it a "bug" because
the fact that "run_with_transaction" is a generic function suggests
that the kind of thing you're trying to do should be possible... I
have no problem in comitting it if it doesn't break anything. Opinions?
Alberto
Elvelind Grandin
--
cheers
elvelind grandin
Dennis Muhlestein
appear to send.
> work. Have you tried swapping the predicates?
>
> """_use_sa() and getattr(func,"_no_trans",False)"""
That didn't seem to affect anything. I also put a log statement in the
condition and turned the logging level to debug. I noticed that
conditions seem to be evaluated at load time. I thought then that
perhaps the _no_trans attribute is not being set until after the
condition is evaluated:
example:
@expose()
def something(self):
pass
something._no_trans=True
is the same as
def something(self):
pass
expose()(something)
something._no_trans=True
So I changed my code to be this:
def something(self):
pass
something._no_trans=True
expose()(something)
Unfortunately, the effect was the same and the sa_rwt method is still
called.
>
> If that still doesn't make it, fortunately, as you've noticed, TG has
> the infrastruture to handle the "order" argument to the
> GenericFunction decorators. You'll need to patch
> database.run_with_transaction so it looks like this:
>
> from turbogears.genericfunctions import MultiorderGenericFunction
>
> @generic(MultiorderGenericFunction)
> def run_with_transaction(....)
>
> You'll need to set the order to -1 so it runs before the default
> which is 0.
>
I tried patching TG for the MultiorderGenericFunction in combination
with the change I mentioned above and the sa_rwt method is still
called. I'm a little stumpted at this point. The condition is indeed
being evaluated (I checked that with the logging) but I haven't been
able to figure out what I need to change to get run_with_transaction to
call my no_trans method instead of sa_rwt.
Suggestions?
Thanks
Dennis
Alberto Valverde
> That didn't seem to affect anything. I also put a log statement in the
> condition and turned the logging level to debug. I noticed that
> conditions seem to be evaluated at load time.
Not exactly. As I understand RD's implementation, Conditions are
"compiled" at load time but are evaluated on each call to the generic
function (else there'll be no fun ;)
> I thought then that
> perhaps the _no_trans attribute is not being set until after the
> condition is evaluated:
>
> example:
> @expose()
> def something(self):
> pass
> something._no_trans=True
>
> is the same as
> def something(self):
> pass
> expose()(something)
> something._no_trans=True
Aha! I think I now know where the problem is :) Try this:
something = expose()(something) # don't forget resetting the attribute
something._no_trans = True
If this works tell me as the solution could be optimized....
> I tried patching TG for the MultiorderGenericFunction in combination
> with the change I mentioned above and the sa_rwt method is still
> called. I'm a little stumpted at this point. The condition is indeed
> being evaluated (I checked that with the logging) but I haven't been
> able to figure out what I need to change to get run_with_transaction to
> call my no_trans method instead of sa_rwt.
I believe the problem is that the "func" the conditions are checking is
the un-decorated function which hasn't got the _no_trans attribute you're
adding as the @ syntax decorates the function before you have a chance to
set the attr.
The best way yo implement this IMO would be to write a decorator that sets
this attribute in the *original* function and check for this attribute on
the *original* function in the condition. Something like:
form turbogears.decorator import func_original
def no_transaction(func):
func_original(func)._no_trans = True
return func
@run_with_transaction.when("_use_sa() and getattr(func_original(func),
'_no_trans', False)")
def _no_trans_for_me(func, *args, **kw):
return func(*args, **kw)
now you should (I hope) be able to decorate any method which you don't
want a transaction to begin/end when called with "no_transaction"
@no_transaction
@expose()
def foo(self, bar):
....
You *should* be able to stack "no_transaction" in any order....
Tell me how it goes...
Alberto
Dennis Muhlestein
On Dec 12, 11:46 am, "Alberto Valverde" <albe...@toscat.net> wrote:
> > That didn't seem to affect anything. I also put a log statement in the
> > condition and turned the logging level to debug. I noticed that
> > conditions seem to be evaluated at load time.Not exactly. As I understand RD's implementation, Conditions are
> "compiled" at load time but are evaluated on each call to the generic
> function (else there'll be no fun ;)
>
> > I thought then that
> > perhaps the _no_trans attribute is not being set until after the
> > condition is evaluated:
>
> > example:
> > @expose()
> > def something(self):
> > pass
> > something._no_trans=True
>
> > is the same as
> > def something(self):
> > pass
> > expose()(something)
> > something._no_trans=True
Aha! I think I now know where the problem is :) Try this:
> something = expose()(something) # don't forget resetting the attribute
> something._no_trans = True
I thought that seemed like a red herring too, but it also didn't fix
the problem.
>
> If this works tell me as the solution could be optimized....
>
> > I tried patching TG for the MultiorderGenericFunction in combination
> > with the change I mentioned above and the sa_rwt method is still
> > called. I'm a little stumpted at this point. The condition is indeed
> > being evaluated (I checked that with the logging) but I haven't been
> > able to figure out what I need to change to get run_with_transaction to
> > call my no_trans method instead of sa_rwt.I believe the problem is that the "func" the conditions are checking is
> the un-decorated function which hasn't got the _no_trans attribute you're
> adding as the @ syntax decorates the function before you have a chance to
> set the attr.
>
> The best way yo implement this IMO would be to write a decorator that sets
> this attribute in the *original* function and check for this attribute on
> the *original* function in the condition. Something like:
>
> form turbogears.decorator import func_original
>
> def no_transaction(func):
> func_original(func)._no_trans = True
> return func
>
> @run_with_transaction.when("_use_sa() and getattr(func_original(func),
> '_no_trans', False)")
> def _no_trans_for_me(func, *args, **kw):
> return func(*args, **kw)
>
> now you should (I hope) be able to decorate any method which you don't
> want a transaction to begin/end when called with "no_transaction"
>
> @no_transaction
> @expose()
> def foo(self, bar):
> ....
>
> You *should* be able to stack "no_transaction" in any order....
>
> Tell me how it goes...
I modified my code as you suggested. It's still calling the sa_rwt :(
Worthy to note that I still have the MultiorderGenericFunction being
used as well.
I think this problem has to do with the evaluation of the
run_with_transaction.when rule. When I logged something in that
condition, I only got output during load time when the methods were
being exposed. I never got logging when I actually accessed the
method. You mentioned that it was supposed to be evaluated at call
time though.
-Dennis
Alberto Valverde
> I modified my code as you suggested. It's still calling the sa_rwt :(
>
> Worthy to note that I still have the MultiorderGenericFunction being
> used as well.
>
:(
> I think this problem has to do with the evaluation of the
> run_with_transaction.when rule. When I logged something in that
> condition, I only got output during load time when the methods were
> being exposed. I never got logging when I actually accessed the
> method. You mentioned that it was supposed to be evaluated at call
> time though.
I'm pretty sure that rules that cannot be determined at "compile" time are
evaluated at call time:
In [1]: from dispatch import generic, strategy
In [2]: @generic()
...: def foo(b):
...: pass
...:
In [3]: def b_gt_2(b):
...: print "checking b"
...: return b>2
...:
In [4]: foo.when("b_gt_2(b)")(lambda b: b+2)
Out[4]: <function <lambda> at 0xb77c72cc>
In [5]: foo(1)
checking b
---------------------------------------------------------------------------
dispatch.interfaces.NoApplicableMethods
In [6]: foo(3)
checking b
Out[6]: 5
As you can see, "checking b" is printed in every call... What gets
evaluated at "compile" time are "static" rules because if the solution can
be determined at "compile" time then RD can optimize those rules in order
to minimize checks (please forgive my simplistic explanation, I'm no RD
guru... ;)
Hmmm, this makes me wonder... have you tried:
def _check_no_trans(func):
return getattr(func_original(func), '_no_trans', False) and _use_sa()
@run_with_transaction("_check_no_trans(func)")
def sadsadfsdf(....):
.....
Alberto
Dennis Muhlestein
>
> Hmmm, this makes me wonder... have you tried:
>
> def _check_no_trans(func):
> return getattr(func_original(func), '_no_trans', False) and _use_sa()
>
> @run_with_transaction("_check_no_trans(func)")
> def sadsadfsdf(....):
> .....
Well, I tried this:
@run_with_transaction("_use_sa() and True")
...
and it works.
So the problem is definitely in the checking for the _no_trans
attribute and not in the dispatch code.
Next, I'm going to play around with different versions of the decorator
for setting the _no_trans. See, I'm using multiple decorators in my
code:
@expose()
@error_handler(...)
@validate(...)
def ...
So I think the problem is that func_original probably may not be
returning the _real_ original function. I haven't looked at that a lot
yet but I suspect that is the problem.
Thanks
Dennis
Dennis Muhlestein
> So I think the problem is that func_original probably may not be
> returning the _real_ original function. I haven't looked at that a lot
> yet but I suspect that is the problem.
Just to confirm, I placed logging in the condition again. The function
being returned by func_original is _expose, which does not have the
_no_trans attribute.
So perhaps the bug in all this is that func_original is not returning
my submit function.
Kevin Dangoor
This is precisely why I'd like to replace the decorators with some
other kind of marker for TG 2.0.
Note that another approach would be to put your _no_trans on
cherrypy.request.
Kevin
Dennis Muhlestein
>
> Note that another approach would be to put your _no_trans on
> cherrypy.request.
That solves part of the problem. (func_original returning not the
original function)
This is a two part problem though.
The 2nd part of the problem is that the condition is not being
evaluated at call time. In my debugging, the condition for rwt.when is
being evaluated when it is loaded.
example: @run_with_transaction.when('True')
will call my no_trans method.
but
@run_with_transaction.when('getattr(cherrypy.request,"_no_trans",False)')
will not call the no_trans method even thought the _no_trans attr is
set by the decorator at run time.
I haven't come up with a solution for that part of the problem yet.
BTW, I agree that perhaps the decorators need an overhaul or re-think.
Thanks
Dennis
Dennis Muhlestein
appears Alberto was right about the rules being optimized if they could
be determined statically. I didn't detect that earlier though because
the func_original function wasn't returning the correct function.
Anyhow.. here is how this works.
1) Patch TG to user MultiorderGenericFunction
I got an ambiguous function def error when I tried it with stock TG.
Index: turbogears/database.py
===================================================================
--- turbogears/database.py (revision 2078)
+++ turbogears/database.py (working copy)
@@ -13,6 +13,7 @@
import turbogears
from turbogears import config, errorhandling
+from turbogears.genericfunctions import MultiorderGenericFunction
log = logging.getLogger("turbogears.database")
@@ -231,7 +232,7 @@
for hub in hub_registry:
hub.end()
-[dispatch.generic()]
+[dispatch.generic(MultiorderGenericFunction)]
def run_with_transaction(func, *args, **kw):
pass
2) Create a no_transaction decorator
from turbogears.decorator import weak_signature_decorator
def no_transaction():
def wrap(func):
def no_trans(func,*args,**kw):
cherrypy.request._no_trans=True
return func(*args,**kw)
return no_trans
return weak_signature_decorator(wrap)
3) create the test method for no_trans
This has to be in a method for the reasons stated at the top of the
post.
def is_no_trans():
return getattr(cherrypy.request,'_no_trans',False)
4) create the no trans method. Order=-1 is important so you go before
the default sa_rwt, and so_rwt conditions.
@run_with_transaction.when("is_no_trans()",order=-1)
def _no_trans_for_this_method(func, *args, **kw):
log.debug ( "No Trans Method" )
return func(*args, **kw)
5) In your controller code you can now decorate like this:
@no_transaction()
@expose()
def something(self): pass
Note that order is indeed important for no_transaction because the
request attribute must be set before it is tested for in the expose
method.
Well, this has been a long thread. This solution works for me but I'm
not sure it's very suited to suggest to others in the same boat.
Anyone have a better solution?
-Dennis
Alberto Valverde
On Dec 13, 2006, at 12:07 AM, Dennis Muhlestein wrote:
>
> OK, I Have a working solution but I don' t think it's very pretty. It
> appears Alberto was right about the rules being optimized if they
> could
> be determined statically. I didn't detect that earlier though because
> the func_original function wasn't returning the correct function.
> Anyhow.. here is how this works.
Actually func_original was doing what it should... The problem was
that the "func" passed to rwt is not the function that we thought it
was but a gf created on the fly by "expose" to support multiple
output formats by expose (take a look at turbogears.tests.test_expose
if interesed).
>
> 1) Patch TG to user MultiorderGenericFunction
> I got an ambiguous function def error when I tried it with stock TG.
> .....
>
> 2) Create a no_transaction decorator
> from turbogears.decorator import weak_signature_decorator
>
> ....
>
> 4) create the no trans method. Order=-1 is important so you go before
> the default sa_rwt, and so_rwt conditions.
> @run_with_transaction.when("is_no_trans()",order=-1)
> def _no_trans_for_this_method(func, *args, **kw):
> log.debug ( "No Trans Method" )
> return func(*args, **kw)
>
>
> 5) In your controller code you can now decorate like this:
> @no_transaction()
> @expose()
> def something(self): pass
>
> Note that order is indeed important for no_transaction because the
> request attribute must be set before it is tested for in the expose
> method.
>
>
> Well, this has been a long thread. This solution works for me but I'm
> not sure it's very suited to suggest to others in the same boat.
> Anyone have a better solution?
>
Phew! We should do something, really, in TG 2.0 regarding
decorators... this is really going through rims of fire, snow, ice
and splinters under one's toe nails for something that should have
been *much* easier... :D Glad you finally got something that works :)
I can't really thing of a better solution given the current state of
affairs... ;)
Alberto
P.S Should I comitt the patch? If so please file a ticket with it at
the Trac so we don't forget it at the CHANGELOG for 1.0b3
P.P.S. Something I've learnt from this thread and write in a big
sticker in front of my desk: don't ever, ever again think about
abusing decorators :)
Dennis Muhlestein
> P.S Should I comitt the patch? If so please file a ticket with it at
> the Trac so we don't forget it at the CHANGELOG for 1.0b3
I added ticket. I think it should definitely make it in as a bug fix
because a) I don't believe it breaks anything and b) I don't know
another way to accomplish this fix yet.
http://trac.turbogears.org/turbogears/ticket/1201
Thanks for your help!
-Dennis
Bob Ippolito
>
> This is more of a best practice question:
>
> I find having the framework manage the database session for you OK 95%
> of the time. What do you do in the other 5% of the time?
>
> Example 1: I have certain credit card processing methods that require
> commits to the database before and after the card is processed.
> Example 2: I have some methods that require no database interaction.
> (So why create a session? .. also.. how much overhead is there in
> creating the session?)
>
> Anyone have any best practice ideas for situations like this?
>
I just wrote different controller classes that stuck the right (or no)
db connection in the context for each request. I have one for
read-only, one for read/write, and one that doesn't hit the db at all.
To do this I had to write my own transaction management stuff of
course.
I eventually switched from TG to Pylons for that app, but I'm using
the same strategy with that framework.
-bob
Dennis Muhlestein
> did you mean 'transaction' when you said 'session' below?
> Also, did you ever get a concise clear set of mods to apply to turn
> off auto-transactions?
The solution was that run_with_transaction is a generic function. You
can add a rule and a function to make it do certain things depending
on criteria.
here is what I did.
1) Create a wrapper function that sets an attribute on a function
telling it NOT to use run_with_transaction
from turbogears.decorator import weak_signature_decorator
def no_transaction():
def wrap(func):
def no_trans(func,*args,**kw):
cherrypy.request._no_trans=True
return func(*args,**kw)
return no_trans
return weak_signature_decorator(wrap)
def is_no_trans():
return getattr(cherrypy.request,'_no_trans',False)
2) create an alternate function to run instead of run_with_transaction:
@run_with_transaction.when("is_no_trans()",order=-1)
def _no_trans_for_this_method(func, *args, **kw):
log.debug ( "No Trans Method" )
return func(*args, **kw)
3) In your code, you just need to use your new decorator:
@no_transation()
@expose()
def etc...etc...
A note: The order of the decorator is important. You have to put it
before expose. That is because I used the cherrypy.request attribute.
There may be a way to do it that doesn't worry about the order but
this method is working for me.
-Dennis
> I'm trying to do the same (sqlalchemy user) and have found this thread
> quite difficult to follow completely to a conclusion of any kind.
> thanks,
> Joel
>
> On Dec 6 2006, 11:40 am, "Dennis Muhlestein" <djmuhlest...@gmail.com>
--
You can see what's happening at http://muhlesteins.com