Transactions in the admin
Jonathan Daugherty
I'm interested to know about the status of transaction support in
general, with particular regard to the admin interface and inline
editing. I see that there was once a short thread started by Kapil,
the submitter of a patch to implement transaction support in ticket
#9.
(Google groups thread link)
http://tinyurl.com/97gge
Are there any plans of further discussion or integration of the patch?
Adrian tells me that this is pending further discussion and a good
implementation. I'm gearing up to use django for a big project at
work and I'm interested to know where this stands.
Thanks for django!
--
Jonathan Daugherty
http://www.parsed.org
Adrian Holovaty
> I'm interested to know about the status of transaction support in
> general, with particular regard to the admin interface and inline
> editing. I see that there was once a short thread started by Kapil,
> the submitter of a patch to implement transaction support in ticket
> #9.
come up with a definitive game plan.
I haven't tested the patch to #9, but it requires (at least part of)
Zope, which is a big reason not to use it. Also, I don't know whether
coupling transactions to Web requests is a good idea.
Anybody have ideas on implementations? Let's get this going.
Adrian
--
Adrian Holovaty
holovaty.com | djangoproject.com | chicagocrime.org
Joshua D. Drake
>I haven't tested the patch to #9, but it requires (at least part of)
>Zope, which is a big reason not to use it. Also, I don't know whether
>coupling transactions to Web requests is a good idea.
>
>
form that is associated
to multiple tables, I expect that if all the data is required within the
form , all the data
will make it into the database, or none will. Otherwise I end up with
inconsistent data which
is a big no no.
Sincerely,
Joshua D. Drake
>Anybody have ideas on implementations? Let's get this going.
>
>Adrian
>
>--
>Adrian Holovaty
>holovaty.com | djangoproject.com | chicagocrime.org
>
>
Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240
PostgreSQL Replication, Consulting, Custom Programming, 24x7 support
Managed Services, Shared and Dedicated Hosting
Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/
Adrian Holovaty
> >I haven't tested the patch to #9, but it requires (at least part of)
> >Zope, which is a big reason not to use it. Also, I don't know whether
> >coupling transactions to Web requests is a good idea.
> >
> >
> Think of it from the perspective of an inline form. If I have a single
> form that is associated
> to multiple tables, I expect that if all the data is required within the
> form , all the data
> will make it into the database, or none will. Otherwise I end up with
> inconsistent data which
> is a big no no.
of data in multiple tables, it should certainly fall within a single
transaction.
But my point was that transactions shouldn't *have* to be tied to Web
requests. We should be able to use transactions outside the scope of a
Web request -- in, say, a Python script or the interactive
interpreter. Transactions shouldn't be coupled with any other layers
of the stack.
Jonathan Daugherty
# interpreter. Transactions shouldn't be coupled with any other layers
# of the stack.
I agree. It seems to me that "transaction support" doesn't need to be
anything more complicated than three calls (one to start a transaction
on the django core database connection, one to commit, and one to roll
back). Is that oversimplification? Admin code that knows inline
editing is taking place can call BEGIN, COMMIT, and ROLLBACK when
appropriate, but ordinary code runs in "autocommit-style" mode unless
it wants to run something transactionally. Maybe others have
grandiose plans for transaction support, but again, ISTM that's all we
really need. The developer gets to choose, and -- paradigmatically
speaking -- it's no different or less powerful than transaction usage
in ordinary web programming. Does django's persistent connection come
to bear at all on this approach?
On a related but somewhat tangential note, I've noticed that django
creates new database connections periodically (when I use
'runserver'); under what circumstances are new connections opened and
closed?
Joshua D. Drake
># We should be able to use transactions outside the scope of a Web
># request -- in, say, a Python script or the interactive
># interpreter. Transactions shouldn't be coupled with any other layers
># of the stack.
>
>I agree. It seems to me that "transaction support" doesn't need to be
>anything more complicated than three calls (one to start a transaction
>on the django core database connection, one to commit, and one to roll
>back). Is that oversimplification? Admin code that knows inline
>editing is taking place can call BEGIN, COMMIT, and ROLLBACK when
>appropriate, but ordinary code runs in "autocommit-style" mode unless
>it wants to run something transactionally.
>
to be able to
catch an exception and process it via a savepoint but there is no reason
why that has
to be a 1.0 rev type thing.
Joshua D. Drake
> Maybe others have
>grandiose plans for transaction support, but again, ISTM that's all we
>really need. The developer gets to choose, and -- paradigmatically
>speaking -- it's no different or less powerful than transaction usage
>in ordinary web programming. Does django's persistent connection come
>to bear at all on this approach?
>
>On a related but somewhat tangential note, I've noticed that django
>creates new database connections periodically (when I use
>'runserver'); under what circumstances are new connections opened and
>closed?
>
>
>
--
Eugene Lazutkin
to developers. Right now database backends expose a connection, which can be
used to obtain cursors, which can be used to execute SQL. Model and view
code doesn't work with connections and cursors directly. It's in a different
abstraction plane now.
Additionally admin code should be modified to use it.
Thanks,
Eugene
"Jonathan Daugherty" <cyg...@cprogrammer.org> wrote
in message news:20051012030...@vulcan.cprogrammer.org...
hugo
>requests. We should be able to use transactions outside the scope of a
>Web request -- in, say, a Python script or the interactive
>interpreter. Transactions shouldn't be coupled with any other layers
>of the stack.
Actually with web apps we might get into deep trouble if transactions
are _not_ tied to one specific place, but users are allowed to do
transactions themselves. The problem is in the fact that
commit/rollback are not bound to the cursor, but to the connection
(only psycopg works differently there - it allows commit on the cursor,
but that's not DBAPI2).
So we can't just pass cursors around and let users decide what cursor
to commit - the commit would have to be on the connection. And it would
commit all open database changes, even from cursors the user didn't
really know about.
Think about a web project where you use a foreign app that uses
explicit transaction control and your own apps use a commit-on-response
transaction control. As soon as the foreign app would trigger a commit,
the changes you did in your apps are commited, too - but you don't
expect that to happen, as your transaction isn't commited, yet - you
think.
I think the best way to do commits is to tie them to the request by
using a middleware that does the commit-rollback-handling. Only needed
change to django would be a process_exception middleware hook that is
only triggered if the viewfunc throws an exception - to do the
automatic rollback. Additionally the middleware would set a global flag
"don't commit on .save()" that would be adhered to by the model code -
that way the user can either have the current commit-on-.save()
functionality or load the transaction control middleware to get
commit-on-response functionality. Since middleware runs for _all_ apps
in a project, all apps are handled the same way.
For batch scripts, since there is no transaction middleware active,
this would end in using the current .save() transaction system. But it
could be easily changed by giving utility functions that
disable_commit_on_save() in that global flag and give a manual_commit()
and manual_rollback() function to do manual transaction control. Since
commits and rollbacks are defined on the database connection, there is
no need to pass along cursors or such stuff.
Of course this doesn't take advantage of more finegrained transaction
stuff in databases like named cursors, savepoints and stuff like that -
but that could be left for 1.1, I think for 1.0 the above stuff would
be just fine.
bye, Georg
Jonathan Daugherty
# using a middleware that does the commit-rollback-handling. Only
# needed change to django would be a process_exception middleware hook
# that is only triggered if the viewfunc throws an exception - to do
# the automatic rollback. Additionally the middleware would set a
# global flag "don't commit on .save()" that would be adhered to by
# the model code - that way the user can either have the current
# commit-on-.save() functionality or load the transaction control
# middleware to get commit-on-response functionality. Since middleware
# runs for _all_ apps in a project, all apps are handled the same way.
Doesn't this deviate from the standard use of middlewares? The docs
say, "It's a light, low-level "plugin" system for globally altering
Django's input and/or output." This sounds like a rather invasive
exception, and it sounds like something that makes this middleware
more than a plugin.
Jonathan Daugherty
# cursor to commit - the commit would have to be on the
# connection. And it would commit all open database changes, even from
# cursors the user didn't really know about.
But *are* cursors being passed around? I guess I'm assuming that
users will run all of their database interactions through the django
core where things like this can be controlled. I suppose that users
can go poking around if they want to get cursors or connections, but
IMHO that's a case of too much rope.
hugo
>say, "It's a light, low-level "plugin" system for globally altering
>Django's input and/or output." This sounds like a rather invasive
>exception, and it sounds like something that makes this middleware
>more than a plugin.
I don't think so - it's just a rather lightweight modification to the
request - it only does the commit/rollback and the global flag setting.
The middleware would only be a rather small thing - some handfull of
lines at max.
And middleware already has side-effects like changing elements of the
request object, for example. Or storing stuff in the cache - the
caching middleware actually would be a much "bigger" middleware than
the transaction middleware :-)
The nice thing about using a middleware for binding transactions to the
request would be that the transaction stuff won't be coupled with the
request machinery at all - it's just a middleware the user can decide
to run or not to run.
bye, Georg
Jonathan Daugherty
# the request object, for example. Or storing stuff in the cache - the
# caching middleware actually would be a much "bigger" middleware than
# the transaction middleware :-)
Ok, good point. (But caching still has more to do with the content
itself than does the transactionality of the request. That's what
inspired the comment.)
# The nice thing about using a middleware for binding transactions to
# the request would be that the transaction stuff won't be coupled
# with the request machinery at all - it's just a middleware the user
# can decide to run or not to run.
But in many cases the transactional nature of the request *is* part of
the request logic, and a simple all-or-nothing wrapper won't do. I
might want the form-saving aspects of my request to be transactional,
whereas other things, like updating the user's session, etc., would
commit instantly. You might call those operations "out-of-band" from
the primary task of the request (like inserting an order into the
database), so I think that should be considered in the
implementation. With your proposal, can you illustrate how one would
accomplish this?
hugo
>the primary task of the request (like inserting an order into the
>database), so I think that should be considered in the
>implementation. With your proposal, can you illustrate how one >would
>accomplish this?
Sure: in those cases you just won't use the "easy-way-out" middleware,
but just use the "disable-commit-on-save" machinery and do explicit
commits and rollbacks yourself.
This discussion is titled "transactions in the admin" - there you
usually don't have those out-of-band activity, you only have the
all-or-nothing-approach. The middleware would make stuff like
transactional admin or simple transactional sites quite easy.
For more complex sites, just use the decorator_from_middleware to
create a decorator for transactionality and use that for those view
functions that should use the all-or-nothing approach and use manual
transactions in all other cases.
My idea is just to provide a really easy way to do the all-or-nothing
approache without taking away the possibility of explicit control for
those who know exactly what to do. And the combined
middleware/decorator thingy is working very nicely with the caching
system, so I just thought it would be the ideal choice for the
transactionality system, too :-)
Maybe it would be easier to discuss this with some code at hand. If
#616 would be applied, django would have the needed "process_exception"
mechanism (that would be usefull regardless wether we use it for
transactions or not, I think). Then I could hack up the middleware and
decorator as I envision it and we could play with it to see wether it's
worth to do more investigation. It's not much work, so I won't mind to
do it even if it's later thrown away for a better approach.
BTW: the "passing around cursors" was a reference to the idea adrian
posted to ticket #9. There he was passing around cursors to signal
explicit transactions.
bye, Georg
Jonathan Daugherty
# middleware, but just use the "disable-commit-on-save" machinery and
# do explicit commits and rollbacks yourself.
Yeah, that's what I was thinking, but it wasn't yet clear from your
proposal whether it would be easy to toggle between the two. (As you
say below, code would make all of this easier. :) )
# This discussion is titled "transactions in the admin" - there you
# usually don't have those out-of-band activity, you only have the
# all-or-nothing-approach. The middleware would make stuff like
# transactional admin or simple transactional sites quite easy.
Yeah, true. I was thinking about mostly non-admin things.
# For more complex sites, just use the decorator_from_middleware to
# create a decorator for transactionality and use that for those view
# functions that should use the all-or-nothing approach and use manual
# transactions in all other cases.
Great; is decorator_from_middleware implemented or still an idea? I
can't find it mentioned in the docs.
# My idea is just to provide a really easy way to do the
# all-or-nothing approache without taking away the possibility of
# explicit control for those who know exactly what to do. And the
# combined middleware/decorator thingy is working very nicely with the
# caching system, so I just thought it would be the ideal choice for
# the transactionality system, too :-)
Yeah, it sounds like decorators would work very well.
hugo
>can't find it mentioned in the docs.
from django.utils.decorators import decorator_from_middleware
from django.middleware.middlewaremodule import MiddlewareClass
decoratorfunc = decorator_from_middleware(MiddlewareClass)
:-)
Went into the source with the recent cache reworkings. I didn't like
the code duplication of cache management between the middleware and the
decorator.
bye, Georg
kap...@gmail.com
transaction support, its the biggest thing stopping me from using
django for production work.
regarding the patch, i'll address the transaction coupling in another
respone in this thread, but the zope usage, i wanted to clarify. first
the dependency is on parts of zope3 not zope2, and as jason alluded to
in the initial thread, zope3 is designed from the ground up to be
reusable as a set of python libraries, and is internally packaged that
way ( using the zpkg tool, and eggs in the future ). as libraries they,
unlike zope2, don't do any magic initialization or alter the runtime,
meaning that they can play well with frameworks like django that do ;-)
and their designed to be pythonic. the portions of zope3 that are
needed for this patch can be trimmed down to 2 small packages.
i'm not clear if the reason that zope3 is 'big reason not to include'
is related to the above or perhaps nih, but imho zope3 has a lot of
quality engineering work done on it, backed up by thousands of unit
tests, and i think its good to benefit from the experience of others
who have been solving complex problems with python for years. the patch
itself is small, in part due to the functionality of the libraries z3
offers, and the nice design of django, but more importantly in addition
to making django transactional it provides a solid foundation as well
for future work with transactions in django, ie integration of multiple
databases (simulatenous), distributed cache transactions, sending
emails out (after transaction commit, or to disk for separate
processing), etc.
-kapil
kap...@gmail.com
web layer, with it you can use a python script or interactive
interpreter, via the api provided by the transaction manager. ie.
import transaction
# do some work
# and
transaction.commit()
# or
transaction.abort()
additionally there are more options and apis available to those who
want to use them, like transaction observers, or doing transactional
file system interactions.
integration was done on the web layer there because coming from a zope
world, where thats the norm, i'm loathe to go back to manual
transaction management, or none at all, and imho the cases where you
don't want a web request to map to a transaction are extremly rare,
only one i can think of atm is bulk loading. web requests that mutate
data logically map onto semantic actions, and those actions should
either succeed or not. pushing the burden onto the programmer is the
wrong thing for a framework todo, it gets complex exponentially as the
apps do, and the best thing a framework can do is keep it simple, by
freeing the programmer from having to think about it. unless of course
they need to, in which case the patch still allows for manual
transaction management even in a web request, ie. you can still
directly commit and abort the transaction(s) within a web request.
-kapil
kap...@gmail.com
transaction library defines a transaction manager, with apis for
retrieving transaction objects and manipulating them and some convience
items for commiting, aborting directly from the manager. data managers
are objects that register with the transaction and recieve transaction
messages, in a django context, a database connection wrapper would be a
data manager. because the dbapi specifies transactions on a connection
level, at implementation level, you'd need multiple database
connections to achieve your goal of manipulation of db resources non
transactionally while performing other ops transactionally, which would
need additional support in django for multiple database connections,
the other database connection wrapper for sessions, can simply not
choose not to register with the transaction. in general though, as i
mentioned before i think this sort of thing becomes extremely fragile
outside of a specific app context, and the non transactional app
behavior has to be carried over to any other app components using
sessions. ie. if others are working with the same deployment they have
to be aware of the details of the session usage as regards
transactions. although i could definitely see storing sessions in as
pickles in something like memcached but thats outside the txn framework
entirely anyways, unless integration work is done.
-kapil
kap...@gmail.com
managers that support it. the postgres impl currently doesn't but it
would be straightforward to make it so.
cheers,
-kapil
hugo
>Maybe it would be easier to discuss this with some code at hand. If
>#616 would be applied, django would have the needed "process_exception"
>mechanism (that would be usefull regardless wether we use it for
>transactions or not, I think). Then I could hack up the middleware and
>decorator as I envision it and we could play with it to see wether it's
>worth to do more investigation. It's not much work, so I won't mind to
>do it even if it's later thrown away for a better approach.
I just did that and added it to ticket #9 - so you could have a look on
how I think transaction management could be done and what ways to
control transactions this would give.
bye, Georg
Jason Huggins
kap...@gmail.com wrote:
> ... transaction support, its the biggest thing stopping me from using
> django for production work.
Me, too. I'd like to use Django for a time and expense app, but it's a
no go without some kind of answer for bootstrapping transactions into
my code.
-jason
Kapil Thangavelu
patch allows the freedom to choose transaction management modes as a
developer finds appropriate, and keeps in the django philosophy of loose
coupling. one question about the the implementation design, i guess i'm
not clear why a stack is needed at all? unless there is some sort of
mixing between different modes is expected on the same request, which i
think would best to avoid due to the subleties and complexities of the
stack transaction blocks interactions, ie. any automatic blocks in the
stack prior to a managed will have their lifecycle dictated by it, as
opposed to the semantic notion in the docs of tying to the
request/response cycle, basically requiring global knowledge of the
management txn management of all the code being executed, to make
behavior guarantees.
regarding the actual implementation, it would be nice to see the core
transaction manager api pluggable by encompassing the commit/rollback
function implementations into a private global instance (or at least a
class namespace) and having the existing functions delegate to that
manager, so those needing custom transaction behavior (managing multiple
txn aware connections for example) can plug the manager directly in a
single place, with the existing implementation serving as the default
manager impl. additionally the error handling on the patch looks to need
a little work, since the database connections can toss errors on commit
or abort, of wrapping calls to db.db.commit/db.db.rollback in
try/finally block guarding manipulation of the bookeeping info.
cheers,
kapil
hugo
>coupling. one question about the the implementation design, i guess i'm
>not clear why a stack is needed at all? unless there is some sort of
>mixing between different modes is expected on the same request, which i
>think would best to avoid due to the subleties and complexities of the
>stack transaction blocks interactions, ie. any automatic blocks in the
>stack prior to a managed will have their lifecycle dictated by it, as
>opposed to the semantic notion in the docs of tying to the
>request/response cycle, basically requiring global knowledge of the
Not everybody will use the django code in webapps - if you write batch
programs, you need a way to do transaction handling, too. And the
decorators already "stack" - you might have the transaction middlware
active, so your transactions are bound to request/response, but you
want to use the commit_manually decorator on some functions where you
want to controll transactions manually.
>regarding the actual implementation, it would be nice to see the core
>transaction manager api pluggable by encompassing the commit/rollback
>function implementations into a private global instance (or at least a
Please provide some sample code how you think this would look like.
It's easier to talk about code than about ideas, as ideas can easily be
misunderstood (or not understood), while code is just that: code. :-)
bye, Georg