Transaction management and atomic

[Cristiano Coelho]

Hello there,

I have recently updated to django 1.7, from 1.5 and one of the main changes I had to do was to replace all deprecated transaction.commit_manually to transaction.atomic, so far so good.

After this, I have found out that if an IntegrityError or DatabaseError exception happen inside a code that is decorated with @transaction.atomic (or inside a with transaction.atomic) and the exception is handled (not throwing it out from the atomic block) the whole transaction gets invalid, and any other database access will fail, just as described in the docs (silly me I didn't read them with enough depth).

As mentioned in the docs, the solution is to also put the "exception throwing code" inside another atomic block, and catch the exception outside of it. I can not describe how anoying this is, compared to the old behaviour where I could easly decide when to commit or rollback, now I have to review my whole code to detect the places where a database save is performed and the exception is handled, and add another atomic block to it.

I believe this was an issue heavily discussed with the develpers and they came to this as the best option, but there needs to be another easier way to handle this kind of issue.

What are the complications of leaving the transaction in a correct state even if an operation raises a database error and the exception is handled silently (not thrown outside of the atomic block)? This was totally possible with the deprecated transaction functions, where you could do all your logic and just at the end handle the transaction commit or rollback, and didn't matter what happened inside of it. But now this is imposible, you need to keep a sharp eye on every database save you perform and surround it with another atomic block just in case it raises a database error exception.

To make the issue clear, here's a sample code:

First one shows what would my current code look like, where add_children() would raise an exception because generate_relationships() was not inside another atomic block.

With the old transactions api, I could easily surround the whole code in another try/except, and at the very end commit or rollback, and everything would be fine even if generate_relationships() throws an exception, it would be siltently ignored.

from django.db import IntegrityError, transaction

@transaction.atomic
def viewfunc(request):
    create_parent()

    try:        
        generate_relationships()
    except IntegrityError:
        handle_exception()

    add_children()

Now this is how the code should actually be with current django 1.7 in order to prevent an error and get the excepted behaviour.

from django.db import IntegrityError, transaction

@transaction.atomic
def viewfunc(request):
    create_parent()

    try:
        with transaction.atomic():
            generate_relationships()
    except IntegrityError:
        handle_exception()

    add_children()

It would be great, if the transaction api could work as the first code, with the results of the second one. Meaning, even if generate_relationships() raises an exception, and it is handled correctly, the transaction would still be valid to be used.

Thanks!

[Curtis Maloney]

This is primarily a function of the DBMS.  When your operations cause a DB-level error, the DB needs to roll back the transaction to clean up.

If you don't, you're operating on an unknown state.

Django can not control this beyond wrapping the "risky" operation in a transaction.

Consider -- exactly what should Django do to clean up the database thrown integrity error without a transaction point to roll back to?

--

Curtis

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e347fe6d-78c8-4c15-848d-3a82415c3550%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Anssi Kääriäinen]

On Wed, 2014-10-01 at 14:31 +1000, Curtis Maloney wrote:
> This is primarily a function of the DBMS. When your operations cause
> a DB-level error, the DB needs to roll back the transaction to clean
> up.

This depends on the database. On PostgreSQL, after any error the
transaction (or savepoint) must be rolled back. But on MySQL and Oracle
it is OK to continue the transaction. The PostgreSQL behavior is much
safer, and that is why Django defaults to that behavior.

There are query patterns where IntegrityError is expected - for example
try to insert, if IntegrityError, then update.

Django does allow to continue the transaction if you explicitly choose
to do so. After handling the error, you must use set_rollback[1]. This
is useful for those rare cases where you want to continue after error in
transaction.

- Anssi

1. https://docs.djangoproject.com/en/dev/topics/db/transactions/#django.db.transaction.set_rollback

[Florian Apolloner]

On Wednesday, October 1, 2014 6:58:28 AM UTC+2, Anssi Kääriäinen wrote:

Django does allow to continue the transaction if you explicitly choose
to do so. After handling the error, you must use set_rollback[1]. This
is useful for those rare cases where you want to continue after error in
transaction.

Imo, this is mainly useful to rollback if no error occurred! The other use of this feature might work but is quite dangerous imo.

Cheers,
Florian

[Florian Apolloner]

On Wednesday, October 1, 2014 6:58:28 AM UTC+2, Anssi Kääriäinen wrote:

Django does allow to continue the transaction if you explicitly choose
to do so. After handling the error, you must use set_rollback[1]. This
is useful for those rare cases where you want to continue after error in
transaction.

[Aymeric Augustin]

2014-10-01 9:21 GMT+02:00 Florian Apolloner <f.apo...@gmail.com>:

Imo, this is mainly useful to rollback if no error occurred! The other use of this feature might work but is quite dangerous imo.

If you really know what you are doing, it might be possible to use it safely while taking advantage of a database's statement-level transactional integrity guarantees. I don't know MySQL's guarantees sufficiently well to tell whether this is a viable strategy.

--
Aymeric.