Migration challenges

[Shai Berger]

Hi all,

I gave a talk to a local user-group about the migrations in 1.7, and some
people in the audience raised things they would like to be able to do, and are
not supported by the current framework; thought it would be nice to bring them
here.

Two issues were about handling migrations at the project level rather than app
level:

1) Cross-app migrations. The act of moving a model from one app to another
should be a common-enough refactoring; especially for beginners, breaking a
piece of an app out to a separate app is something a user is very likely to
want. Current migrations, being app-oriented, make this possible, but a little
awkward.

2) Roll back project to point in history. This is requested by people who want
to work on feature branches (or any separate branches, each with its own
migrations). It is a bit of a hard problem I've run int myself, and I suspect
it requires some integration with source-control systems to be done right. The
solution I recommended (and used) was to keep separate databases for separate
branches, but that is quite cumbersome in a large project.

The third issue is an intriguing idea. It was presented as "keep more than one
state of migration history in the database", but I think that a general
multiple-migration-history mechanism is neither required nor sufficient for the
user's goal. What he wanted was:

3) Separate "destructive" migrations from "non-destructive" -- if I got it
right, "destructive" in the sense that "the old code can no longer work with
the database after the migration". So, adding a nullable column, or a new
table, would generally be non-destructive. If you have several servers running
on the same database, with this separation you can do a rolling upgrade --
first make non-destructive changes, then upgrade the servers one by one, and
only when they all have the new code, do the destructive migration.

Your comments are welcome,

Shai.

[Josh Smeaton]

I have some comments regarding 3.

The idea, if I've understood your recount of the problem, is a good one. You want to be able to run a backwards-compatible migration (database works with app-state X and X+1), upgrade your application servers, and then run another migration that works with app-state X+1. The best example is adding a nullable column, incrementally upgrading your applications, then changing the nullable column into a non-nullable column.

I don't think that django migrations should try to understand the difference between compatible/destructive migration operations though. This is best handled (currently) by doing two individual deployments, with separate migrations. The first commit would have the addition of the nullable column -> deploy/migrate. The second commit would remove the null constraint -> deploy/migrate.

I was thinking about support for applying a subset of migrations:

./manage.py migrate app -n 1

But then your model and the state of the migrations are out of sync. (Probably) much better to separate out the deployment into two discrete operations and code.

Josh

[Josh Smeaton]

I have some comments regarding 3.

The idea, if I've understood your recount of the problem, is a good one. You want to be able to run a backwards-compatible migration (database works with app-state X and X+1), upgrade your application servers, and then run another migration that works with app-state X+1. The best example is adding a nullable column, incrementally upgrading your applications, then changing the nullable column into a non-nullable column.

I don't think that django migrations should try to understand the difference between compatible/destructive migration operations though. This is best handled (currently) by doing two individual deployments, with separate migrations. The first commit would have the addition of the nullable column -> deploy/migrate. The second commit would remove the null constraint -> deploy/migrate.

I was thinking about support for applying a subset of migrations:

./manage.py migrate app -n 1

But then your model and the state of the migrations are out of sync. (Probably) much better to separate out the deployment into two discrete operations and code.

Josh

On Wednesday, 24 September 2014 08:54:28 UTC+10, Shai Berger wrote:

[schinckel]

On Wednesday, September 24, 2014 8:24:28 AM UTC+9:30, Shai Berger wrote:

Hi all,

I gave a talk to a local user-group about the migrations in 1.7, and some
people in the audience raised things they would like to be able to do, and are
not supported by the current framework; thought it would be nice to bring them
here.

Two issues were about handling migrations at the project level rather than app
level:

1) Cross-app migrations. The act of moving a model from one app to another
should be a common-enough refactoring; especially for beginners, breaking a
piece of an app out to a separate app is something a user is very likely to
want. Current migrations, being app-oriented, make this possible, but a little
awkward.

I had a need to do something similar to this: basically adding an extra attribute

to a contrib app (and the database column too). It is possible to write a custom

migration operation that allows you to alter a model that is not part of the same

app, but, yes, it's a pain in the proverbial.

I guess the 'changing app of a model' is something that could appear.

However, it's going to require having a db_table entry in Meta at the least,

or migrations to rename the table. I guess you could replace the creation

migration with one that renames the other one.

 

2) Roll back project to point in history. This is requested by people who want
to work on feature branches (or any separate branches, each with its own
migrations). It is a bit of a hard problem I've run int myself, and I suspect
it requires some integration with source-control systems to be done right. The
solution I recommended (and used) was to keep separate databases for separate
branches, but that is quite cumbersome in a large project.

It's no different to what exists with south though, or any other migration system.

The backwards migrations need to be done while the checked out version

contains them. You could have a pre-update hook that examines if the changeset

you are checking out is missing any migrations, and warn that reverse migrations

may need to be run first.

Matt.

[Andrew Godwin]

I'm still on holiday, but just wanted to quickly feed back on some of these:

On Wed, Sep 24, 2014 at 8:54 AM, Shai Berger <sh...@platonix.com> wrote:

Hi all,

I gave a talk to a local user-group about the migrations in 1.7, and some
people in the audience raised things they would like to be able to do, and are
not supported by the current framework; thought it would be nice to bring them
here.

Two issues were about handling migrations at the project level rather than app
level:

1) Cross-app migrations. The act of moving a model from one app to another
should be a common-enough refactoring; especially for beginners, breaking a
piece of an app out to a separate app is something a user is very likely to
want. Current migrations, being app-oriented, make this possible, but a little
awkward.

This was always a complaint with South, too. There's nothing stopping you doing it with Django migrations, provided you use the right operation and dependencies, but it could be made easier and possibly autodetected.

 

2) Roll back project to point in history. This is requested by people who want
to work on feature branches (or any separate branches, each with its own
migrations). It is a bit of a hard problem I've run int myself, and I suspect
it requires some integration with source-control systems to be done right. The
solution I recommended (and used) was to keep separate databases for separate
branches, but that is quite cumbersome in a large project.

This idea comes up every year or so at least. The only true solution is, as you suggest, separate databases (or schemas in PostgreSQL) for each branch; the destructive nature of some migration operations means that nothing else is going to work as people suspect.

 

The third issue is an intriguing idea. It was presented as "keep more than one
state of migration history in the database", but I think that a general
multiple-migration-history mechanism is neither required nor sufficient for the
user's goal. What he wanted was:

3) Separate "destructive" migrations from "non-destructive" -- if I got it
right, "destructive" in the sense that "the old code can no longer work with
the database after the migration". So, adding a nullable column, or a new
table, would generally be non-destructive. If you have several servers running
on the same database, with this separation you can do a rolling upgrade --
first make non-destructive changes, then upgrade the servers one by one, and
only when they all have the new code, do the destructive migration.

What people define as "destructive" varies - is it being incompatible with currently-running code? Changing sequences so that you can't roll back? Deleting cached data? Actually causing data loss?

At one point I was looking into a warning system that classified operations not only by destructiveness but by time it would take to apply - for example, it would know that adding a NULL column on PostgreSQL was super quick, but the opposite on MySQL was slow.

[Erik Ashepa]

Hi,
I've presented the third feature in the user group.

I'll try to clarify what I've meant.

A destructive operation is any operation which would break backward compatibility with a previous version of the application.
I proposed implementing the ability to have multiple south migration sequences mapped to multiple south tables in the database.
And having the ability to indicate to which migration sequence I'm adding the migration via the cli
for example two migration sequences :
1. Additive: new tables, columns
2. Destructive: delete tables, columns , migrate data to new scheme.

This feature could also be of value when migrating a model between diango apps.

Erik.

[Marc Tamlyn]

Adding a not null column can be a breaking operation

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1af61e31-7a77-45e4-9a99-fd7e990fa28f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.