Automatic reverse migrations

127 views
Skip to first unread message

pa...@qwil.co

unread,
Apr 14, 2017, 2:04:42 PM4/14/17
to Django users
In a modern Continuous Delivery environment, it's expected that there is an automated process for deploying code, and therefore performing database migrations. This is all straightforward.

I haven't been able to find any good solutions for automatic rollback though. The main problem that I see is that there is (AFAIK) no easy way to definitively know which migrations to unapply to roll back to the previous verison. If you try to rollback from the new N+1 version, you have the migrations, but no recording of the previous version's state. If you rollback from the previous N version, you don't have the new migration files to do the DB rollback.

What I'd really like is a way of recording a 'db migration checkpoint' which could be generated per-release (or whenever else you care to checkpoint your migration state), and would say something like `v1: {app1:0002,0003, app2: 0004}, v2: {app1:0004, app2: 0005, 0006}`, thus letting me roll back all of the migrations in the v2 deploy with a single command.

Does anyone have suggestions or references here? I may try rolling the above solution if there is no prior art, but I want to avoid reinventing the wheel here, as it seems that this issue must have been hit by many other users before me.

Andrew Godwin

unread,
Apr 14, 2017, 2:26:51 PM4/14/17
to django...@googlegroups.com
Hi Paul,

I have tried this in the past, but it's basically an unsolvable problem in the generic case. Databases do not take well to snapshotting or changing schema, and some operations are naturally irreversible. If you find a way that works well, I suggest you write it up so others can learn from it!

Andrew

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Camilo Torres

unread,
Apr 15, 2017, 9:01:03 PM4/15/17
to Django users
Hi,
May both of you try to implement "blue-green deployments", and switch only if all tests are OK.
You may also be interested in running a test deployment in a testing environment prior to production.
Don't know of a solution integrated into django.


On Friday, April 14, 2017 at 2:26:51 PM UTC-4, Andrew Godwin wrote:
Hi Paul,

I have tried this in the past, but it's basically an unsolvable problem in the generic case. Databases do not take well to snapshotting or changing schema, and some operations are naturally irreversible. If you find a way that works well, I suggest you write it up so others can learn from it!

Andrew
On Fri, Apr 14, 2017 at 10:33 AM, <pa...@qwil.co> wrote:
In a modern Continuous Delivery environment, it's expected that there is an automated process for deploying code, and therefore performing database migrations. This is all straightforward.

I haven't been able to find any good solutions for automatic rollback though. The main problem that I see is that there is (AFAIK) no easy way to definitively know which migrations to unapply to roll back to the previous verison. If you try to rollback from the new N+1 version, you have the migrations, but no recording of the previous version's state. If you rollback from the previous N version, you don't have the new migration files to do the DB rollback.

What I'd really like is a way of recording a 'db migration checkpoint' which could be generated per-release (or whenever else you care to checkpoint your migration state), and would say something like `v1: {app1:0002,0003, app2: 0004}, v2: {app1:0004, app2: 0005, 0006}`, thus letting me roll back all of the migrations in the v2 deploy with a single command.

Does anyone have suggestions or references here? I may try rolling the above solution if there is no prior art, but I want to avoid reinventing the wheel here, as it seems that this issue must have been hit by many other users before me.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

pa...@qwil.co

unread,
Apr 17, 2017, 7:58:31 PM4/17/17
to Django users
Thanks Andrew, will see what I can rustle up.

Camilo -- blue/green deploys don't really help with this problem. As each application deployment still talks to the same database, once you have run the migration you can't just abandon your deployment, you need to unwind the db migration first.

Mike Dewhirst

unread,
Apr 17, 2017, 8:21:57 PM4/17/17
to django...@googlegroups.com
On 18/04/2017 9:58 AM, pa...@qwil.co wrote:
> Thanks Andrew, will see what I can rustle up.
>
> Camilo -- blue/green deploys don't really help with this problem. As
> each application deployment still talks to the same database, once you
> have run the migration you can't just abandon your deployment, you
> need to unwind the db migration first.

I agree it is an unsolvable problem to automate it. Or rather, not worth
the development cost of making an automatic "unwinder".

For the number of times it is likely to be needed in general it would be
less costly to do another new migration towards the previous state.
> <https://groups.google.com/group/django-users>.
> <https://groups.google.com/d/msgid/django-users/6cafecca-2c71-4a6f-b93b-fe3e64847bc2%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com
> <mailto:django-users...@googlegroups.com>.
> To post to this group, send email to django...@googlegroups.com
> <mailto:django...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/8d8f46a7-8819-4d18-908d-9681d8b6a39e%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/8d8f46a7-8819-4d18-908d-9681d8b6a39e%40googlegroups.com?utm_medium=email&utm_source=footer>.

Paul Tiplady

unread,
Apr 18, 2017, 1:50:09 PM4/18/17
to django...@googlegroups.com
I think it's possible to do better than the current status quo with a bit more metadata, and without changing the current flow for users that don't want/need more -- essentially all I'm looking for is a way to record which migrations were applied in a given run of `./manage.py migrate`, so that I can revert the same set in a downgrade. This would be equally useful for reversing manually applied migrations as for doing an automated rollback.

The API I'm thinking of is something like this (zero thought invested in naming):

```
# ./manage.py migratewithtag 1.2.3
> running migration app1.0002
> running migration app2.0004
> writing tag before and after state to django_migrationtags table {app1:{before: 0001, after:0002},...}
# ./manage.py rollbacktag 1.2.3
> running migration app1.0001
> running migration app2.0003
```

This is essentially just a layer of metadata above the current migrations table.

I've yet to start playing with an implementation, but of course suggestions welcome for potential problems with this approach.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com <mailto:django-users+unsubscrib...@googlegroups.com>.
To post to this group, send email to django...@googlegroups.com <mailto:django-users@googlegroups.com>.

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/39Un2Jm--aU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages