Non-atomic migrations in Django?

[Ludwig Hähne]

Hi all,

I'd like to get your feedback on supporting non-atomic migrations in Django.

Database transactions are always wrapped in transactions (on DBs that support transactional DDL like Postgres). Generally this is a good idea.

However, one can't do batched updates in data migrations which is essential for performing changes on big tables on a live DB [1]. It's also not possible to create indexes concurrently on Postgres from inside a transaction.

Therefore, I'd like to have support for non-atomic migrations in Django because it's pretty messy to work around not having proper support for that [2].

Here's a proof-of-concept implementation [3] of exempting specific migrations from being wrapped in a transaction by setting `atomic = False` on the migration:

https://github.com/django/django/compare/master...Ableton:non-atomic-migrations

Do you agree that non-atomic migrations should be supported by Django?

Is setting an `atomic` property on a migration a good API for that?

If there's a chance this will be merged, I'd add documentation, incorporate your feedback, and open a ticket or PR.

Thanks,
Ludwig

[1] http://pankrat.github.io/2015/django-migrations-without-downtimes/
[2] http://stackoverflow.com/questions/31247810/commit-manually-in-django-data-migration
[3] https://github.com/django/django/compare/master...Ableton:non-atomic-migrations

[Anssi Kääriäinen]

I don't see any problem with optional non-transactional migrations. So, +1 for the idea and API. I haven't looked at the implementation, so no comments about that.

 - Anssi

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/c062736d-2ebc-42cd-83a5-fe4d064cb24e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Aymeric Augustin]

I like the API as well. Surprisingly, I couldn't find a Trac ticket about this.

-- 

Aymeric.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CALMtK1GDj0_wheW5dHEdp1ZLYcXa-6OpxfHgcnHqWtMcT9YG0g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Aymeric.

[Marc Tamlyn]

+1 to idea and API. I've wished I had this recently - even if it's just so I can check up on the progress of a slow running data generation migration so it flushes the data every few 100 records.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANE-7mUiytuj26np%3DzP-kX4fHmPv3i0u7Hn-a8GTGuHk4sF3qw%40mail.gmail.com.

[Andrew Godwin]

I also agree this looks sensible - I think South even had an attribute like this on the migration class, it just never got ported over. +1

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMwjO1Goz60WtuFrwpX0h-SUcf9%2BVn1neTnOAnZeBwoKn9yAkQ%40mail.gmail.com.

[Markus Holtermann]

Hi Ludwig,

the API looks clean and sensible to me. +1 for getting that into 1.10

/Markus

[Ludwig Hähne]

Thanks for your feedback and support. I've opened a ticket and pull request:

https://code.djangoproject.com/ticket/25833