Database inserts with autocommit set to off.

[Maciej Gol]

Hey!

I've been recently working (porting to Django 1.8) on a project where we need to use a few transactions in a single request, and these transactions cannot be correctly handled by the `atomic` decorator due to functions calls nesting. Basically, we are sending celery tasks at the end of some processing, and it requires the results to be visible in the database. I'm doing this for each element of a list, thus the celery task sending is done right after saving the data to the database. The commit should happen between the save and posting the task, and since the processing logic is complex, I can't use a decorator here.

The issue is, when autocommit is set to off, whenever I try to `.update()` a `QuerySet` or `.save()` a `Model`, it results in `TransactionManagementError: The outermost 'atomic' block cannot use savepoint = False when autocommit is off.` error, which is kind of sad because I could handle the eventual rollback myself gracefully. Instead, django throws me this error in the face.

Have you got any solution to this? My logic looks more or less like this:

def process_events(events):
    # ...some generic logic here...
    transaction.set_autocommit(False)
    for event in events:
        react_to_event(event)

    # ...finalization...

def react_to_event(event):
    if event.a:
        do_a(event)
    elif event.b:
        do_b(event)
    elif event.c:
        do_c(event)
    else:
        do_err(event)

def do_[abc](event):
    # ...process changes...
    event.save()
    # Make the changes visible to celery tasks and others.
    transaction.commit()

    # ...run signals...
    # ...send celery tasks...    

Cheers

[Carl Meyer]

Hi Maciej,

On 07/27/2015 07:03 AM, Maciej Gol wrote:
> I've been recently working (porting to Django 1.8) on a project where we
> need to use a few transactions in a single request, and these
> transactions cannot be correctly handled by the `atomic` decorator due
> to functions calls nesting. Basically, we are sending celery tasks at
> the end of some processing, and it requires the results to be visible in
> the database. I'm doing this for each element of a list, thus the celery
> task sending is done right after saving the data to the database. The
> commit should happen between the save and posting the task, and since
> the processing logic is complex, I can't use a decorator here.

I think a better solution to this situation is to use
transaction.atomic, and then use django-transaction-hooks [1] to delay
creation of the Celery task(s) until the transaction is successfully
committed. (Django 1.9 will have transaction-hooks integrated into core.)

[1] http://django-transaction-hooks.readthedocs.org/en/latest/

> The issue is, when autocommit is set to off, whenever I try to
> `.update()` a `QuerySet` or `.save()` a `Model`, it results in
> `TransactionManagementError: The outermost 'atomic' block cannot use
> savepoint = False when autocommit is off.` error, which is kind of sad
> because I could handle the eventual rollback myself gracefully. Instead,
> django throws me this error in the face.

Yes, this is a known issue in 1.8:
https://code.djangoproject.com/ticket/24921

The ticket describes the needed solution in some detail, it just remains
for someone to code up the patch with a test. Since the issue is a
regression in 1.8, I think such a patch would be backported to the 1.8
branch and appear in the next 1.8.x release.

Carl

[Maciej Gol]

Thanks a lot! I've ended up using `django-transaction-hooks`.