South's --update equivalent for Django 1.7 migrations

[Petri Lehtinen]

Hi!

The database migration system in Django 1.7 is really great and
seems to work very well. However, since I'm a long-time South
user, there's one feature that I'm missing: the possibility to
update an existing migration (the --update flag to South's
schemamigration command).

I think it's an important feature because it allows you to make
massive changes to your models when developing a new feature, because
it rolls back the migration and rewrites it, effectively wiping
everything and starting from scratch for that migration. I can rename
tables, change foreign keys to normal fields and vice versa,
add/remove/rename M2M fields freely, etc., things that the migrations
don't normally handle very well automatically. When developing,
there's no need to retain data, so it's OK to roll back and lose all
the data in newly created tables, for example. It's also not very
efficient to leave all those intermediate steps in the migration
history to wait until squashmigrations is used some time in the
future.

As a proof-of-concept, I created a shell script that does the
migration updating for a given app. It's not an universal
solution because it's tied to a certain directory hierarchy,
though. Here it is:

#!/bin/sh

project=PROJECTNAME
app=$1

cur=$(./manage.py migrate --list $app | tail -n 1 | grep '\[X\]' | sed 's/ \[X\] //')
prev=$(./manage.py migrate --list $app | tail -n 2 | head -n 1 | grep '\[X\]' | sed 's/ \[X\] //')
if [ -z "$cur" -o -z "$prev" ]; then
echo "Unable to find out current or previous migration. Have you run all migrations?"
exit 1
fi
echo "Rolling back to:" $prev
echo "Deleting and updating:" $cur
echo "Are you sure? [yN]"
read x
[ "$x" != "y" ] && exit 0
./manage.py migrate "$app" "$prev"
rm $project/$app/migrations/$cur.py
./manage.py makemigrations "$app"

What do you think? Could functionality like this be added to Django?
Are there problems I've missed with this approach?

Petri

[Curtis Maloney]

Since I didn't know about the update feature in South... can you explain how it would be different from migration squashing in 1.7?

--

C

Petri

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20141008063316.GE32167%40p29.
For more options, visit https://groups.google.com/d/optout.

[Petri Lehtinen]

On Thursday, October 9, 2014 12:01:45 AM UTC+3, Curtis Maloney wrote:

Since I didn't know about the update feature in South... can you explain how it would be different from migration squashing in 1.7?

South's schemamigration --update has very little to do with migration squashing.

Squashing can be used to simplify the migration history by squashing many migrations into one. This has two functions:

- Having less files lying around

- Optimizing the initialization of an empty database

For a project that has already been deployed to a production server, the only advantage is having less migration files. The squashed migration will never be applied on the production server.

For clarity, let's go through migration squashing first. Suppose you have the following migrations defined for myapp:

myapp

 [X] 0001_initial

 [X] 0002_foo

 [X] 0003_bar

 [X] 0004_baz

If you run ./manage.py squashmigrations myapp 0004, you get the following migration history:

myapp

 [X] 0001_squashed_0004_baz (4 squashed migrations)

Now, if someone starts *with an empty database*, it's enough that he runs this single optimized migration instead of the 4 original migrations.

What the "makemigrations --update" command I'm proposing would do is this. Suppose you have these migrations defined for myapp:

myapp

 [X] 0001_initial

 [X] 0002_foo

 [X] 0003_bar

 [X] 0004_baz

Now you run "./manage.py makemigrations --update myapp". First, it rolls back the last migration:

myapp

 [X] 0001_initial

 [X] 0002_foo

 [X] 0003_bar

 [ ] 0004_baz

Then it removes the last migration altogether:

myapp

 [X] 0001_initial

 [X] 0002_foo

 [X] 0003_bar

Then it creates a new migration based on what has changed since 0003_bar:

myapp

 [X] 0001_initial

 [X] 0002_foo

 [X] 0003_bar

 [ ] 0004_auto_20141009_0826

And that's it! So, instead of creating a new migration on top of 0004_baz, it wiped 0004_baz and created the new migration that replaces it.

This is useful when developing a new feature. A single developer can test his changes by creating an unfinished migration and applying it, then writing some code that uses the changed database schema. If he finds that the new schema isn't good, he can change what is needed and update the migration. When the feature is ready, he'll commit the migration to version control and deploy it to production. In the production server, a single migration is run that contains *the final version* of his changes.

If he hadn't used --update (which he currently can't do), he ends up creating many intermediate migrations. With South, updating a migration 5 times is not uncommon for me, so I with Django's built-in migrations I would end up with 5 migrations, when only one would have been enough.

Futhermore, by unapplying and re-creating a migration I can make changes that would normally be harder to do. For example, changing a new model's name or changing a foreign key field to point to another table are not a problem. And changes like these really happen when developing a new feature.

Petri

[Andrew Godwin]

--update was a feature contributed to South that I didn't write, but pulled in and everyone found useful; I didn't get it in for the 1.7 release as there were time constraints, but I'd definitely be open to seeing it re-added.

The logic is not too complex, but the feature would need to be dependency-aware and delete any migrations that depended on the updated migration before it re-ran makemigrations; in addition, if no migration name is provided (as to how makemigrations runs by default), the strategy might change a bit.

I've not got the time or my energy back to work on this yet, but if someone wants to take it on I'm happy to provide guidance and code review.

Andrew

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5fd91632-a4fa-46a0-8c30-3a9a81fd5918%40googlegroups.com.

[Petri Lehtinen]

Andrew Godwin wrote:
> --update was a feature contributed to South that I didn't write, but pulled in
> and everyone found useful; I didn't get it in for the 1.7 release as there were
> time constraints, but I'd definitely be open to seeing it re-added.
>
> The logic is not too complex, but the feature would need to be dependency-aware
> and delete any migrations that depended on the updated migration before it
> re-ran makemigrations; in addition, if no migration name is provided (as to how
> makemigrations runs by default), the strategy might change a bit.
>
> I've not got the time or my energy back to work on this yet, but if someone
> wants to take it on I'm happy to provide guidance and code review.

I think I'll try it myself. Should I email you directly guidance
and/or review, or post here?

Petri

[Andrew Godwin]

Either is fine, depends on the dynamic you want to take. Direct to me will probably get more attention; I don't read -developers nearly as often as I should.

Andrew

Petri

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20141009070343.GA31004%40p29.

[Shai Berger]

On Thursday 09 October 2014 09:53:44 Andrew Godwin wrote:
>
> The logic is not too complex, but the feature would need to be
> dependency-aware and delete any migrations that depended on the updated
> migration before it re-ran makemigrations; in addition, if no migration
> name is provided (as to how makemigrations runs by default), the strategy
> might change a bit.
>

There are a few other minor complications; for example, the script suggested
might fail if the last migration depended on more than one migration in the
local app (which naturally happens if you needed "makemigrations --merge").

Also, I'd be quite apprehensive about updating migrations which have
dependents. It seems like a high potential for breakage.

Shai.

[Petri Lehtinen]

Yeah. I already figured out that it's probably the best to refuse to
update if a migration has dependents.

My next question would have been whether it's possible that a
migration has multiple local dependencies, but you already answered
that. Updating a migration is quite a limited use case, so it's
probably best to also refuse to update a merge migration.

Thanks,

Petri