dmigrations

[Trey Piepmeier]

Simon Willison (one of the creators of Django) just released a new
database migration tool for Django called dmigrations:
http://simonwillison.net/2008/Sep/3/dmigrations/

It's still not as nice as what's built into Rails, in my opinion. I'd
prefer to have the migration files be written in Python (as opposed to
raw SQL), like Rails migration files are written in Ruby. This looks
like it's probably better than any other option available for Django
right now.

Have any of you used any migration tool in Django?

South: http://south.aeracode.org/
django-evolution: http://code.google.com/p/django-evolution/

[Patrick Altman]

I ended up rolling my own and it has been working fairly well:
http://paltman.com/2008/07/03/managing-database-changes-in-django/

It was less code/effort than researching all the available ones out
there as I knew what I wanted to accomplish.

Since doing this, haven't had enough "pain level" to look at some of
the other solutions.

Unlike you, Trey, I prefer to write the migrations in SQL instead of
Python for a couple of reasons. The primary of these reasons is that
not all migrations are simple DDL changes. Sometimes there is the need
to fix or otherwise manipulate data well, and it just feels as if you
get a lot more control when writing in a language that the database
understands.

[Trey Piepmeier]

> to fix or otherwise manipulate data well, and it just feels as if you  
> get a lot more control when writing in a language that the database  
> understands.

I can dig that, but I like the idea of having everything be database
agnostic. I might want to develop a site using SQLite and then move
to MySQL in production.

[Freddie Palmer]

Just my 2 centavos :)

If you ever think you are going to move to MySql just use it from the
get-go. You could be setting yourself up for some pain later on.

[Simon Willison]

Trey Piepmeier wrote:
> It's still not as nice as what's built into Rails, in my opinion. I'd
> prefer to have the migration files be written in Python (as opposed to
> raw SQL), like Rails migration files are written in Ruby. This looks
> like it's probably better than any other option available for Django
> right now.

Hello,

The migrations are written in Python, to a certain extent. Here's a
migration that adds an index to a table:

from dmigrations.mysql import migrations as m
import datetime
migration = m.AddIndex('sessions', 'session', 'expire_date')

AddIndex is a class that "knows" the SQL for adding an index.
AddColumn, DropColumn and DropIndex are others. I'm hoping that people
will write more migration classes for other common operations.

Note that table creation still uses a raw SQL migration rather than a
AddTable class. This was just the simplest thing to do when the
migration system was first written; I certainly wouldn't oppose
creating a more Pythonic abstraction, and doing so would not require
any changes to dmigrations as it exists now. The only contract it
expects is that a migration is an object with an up() and a down()
method.

Hope that clears things up a bit, and sorry for the group invasion!

Cheers,

Simon

[Trey Piepmeier]

Simon,

Thanks for the input. That makes sense.

So basically, there are parts that are already abstracted to Python,
but not all of it yet. And as more abstractions are added, they will
be usable as-is with the API.

Thanks for your work on this and Django in general. I hope this
thread didn't come off as too negative, I just like to wax
philosophical on the differences between the popular frameworks! :)

-Trey

[Simon Willison]

On Sep 3, 9:38 pm, Trey Piepmeier <tpiepme...@gmail.com> wrote:
> Thanks for your work on this and Django in general.  I hope this
> thread didn't come off as too negative, I just like to wax
> philosophical on the differences between the popular frameworks! :)

Not at all. Rails migrations are definitely much more mature - they've
been solving the problem really well for years now. dmigrations is
baby steps at the moment, but it's proved itself in production with a
reasonably large team. And like you said, I'm keen on getting basic
framework right in a way that's extensible. I have high hopes for the
discussion about this stuff at DjangoCon in a few days time.

[George Titsworth]

So what is everyone's opinion on dmigrations vs. django-evolutions vs. whatever the hell rails does.  I personally don't like having to individually declare each database change in dmigrations.  I like that django-evolutions will auto detect what is needed.  However, I like the sequential migration process that dmigrations sets up and how it allows you to unapply your migrations.  I am not sure if django-evolutions allows this or not.

I'm sure a wrapper could be implemented around dmigrations to do the autodetection of changes and go ahead and create the migrations, at least on request.  Something like:

./manage.py dmigrate autogenerate

That spits out a few migrations, maybe one for each table that is changing in some way.

Admittedly I haven't used either.  The only django project I have worked on had already been designed and I just blindly copied the design and haven't changes my models since.  But we are in the process of redesigning and I expect I will definately run into a case in which I will need to use one of these tools.  So I am eager to hear about real life success/failure stories with each of these. 

Side question: Simon said this was going to be a panel topic at Django-Con... were they planning on live casting anything or just doing the normal slideshare approach?

George

[Simon Willison]

On Sep 3, 10:30 pm, "George Titsworth" <titswo...@gmail.com> wrote:
> So what is everyone's opinion on dmigrations vs. django-evolutions vs.
> whatever the hell rails does.  I personally don't like having to
> individually declare each database change in dmigrations.  I like that
> django-evolutions will auto detect what is needed.  However, I like the
> sequential migration process that dmigrations sets up and how it allows you
> to unapply your migrations.  I am not sure if django-evolutions allows this
> or not.
>
> I'm sure a wrapper could be implemented around dmigrations to do the
> autodetection of changes and go ahead and create the migrations, at least on
> request.  Something like:
>
> ./manage.py dmigrate autogenerate

I would absolutely love this to be the case. I'm looking forward to
talking this over with the django-evolution people at DjangoCon - my
requirements for a migration system are that all changes are
explicitly defined in a file somewhere and I can go both up() and
down(), but if something can mostly automate the process of creating
those migration scripts I'm all for it. django-evolution gets most of
the way there; if it could support manual migration scripts ala
dmigrations there would be no reason for dmigrations to exist at all.

> Side question: Simon said this was going to be a panel topic at
> Django-Con... were they planning on live casting anything or just doing the
> normal slideshare approach?

As far as I know the plan is to video every single session, and share
the video online as soon as possible after the event.

[Andreas]

> I would absolutely love this to be the case. I'm looking forward to
> talking this over with the django-evolution people at DjangoCon - my
> requirements for a migration system are that all changes are
> explicitly defined in a file somewhere and I can go both up() and
> down(), but if something can mostly automate the process of creating
> those migration scripts I'm all for it. django-evolution gets most of
> the way there; if it could support manual migration scripts ala
> dmigrations there would be no reason for dmigrations to exist at all.

Would be interesting to see how many times down() actually is used.
Feels like its only providing sense of safety because with the rapid
pace of todays web development you almost always go forward, and never
back.

[Trey Piepmeier]

> Would be interesting to see how many times down() actually is used.
> Feels like its only providing sense of safety because with the rapid
> pace of todays web development you almost always go forward, and never
> back.

Probably as often as people revert code in a source code repository
rather than just plugging away to fix a mistake. It good to have the
functionality in case something really bad happens, though.

[Simon Willison]

On Sep 3, 11:35 pm, Andreas <andrii...@gmail.com> wrote:
> Would be interesting to see how many times down() actually is used.
> Feels like its only providing sense of safety because with the rapid
> pace of todays web development you almost always go forward, and never
> back.

That's exactly why we felt it was so important to automate the
creation of the down() method for all of the common cases. Developers
aren't going to bother most of the time if they have to write the
down() method from scratch. It turns out it's actually most useful in
development - you can create a migration for a new feature, up() it,
mess around a bit, decide that it wasn't quite right, down() it,
delete the migration and try something else - all the while never
actually checking your migration in to source control.

[Andreas]

>
> That's exactly why we felt it was so important to automate the
> creation of the down() method for all of the common cases. Developers
> aren't going to bother most of the time if they have to write the
> down() method from scratch. It turns out it's actually most useful in
> development - you can create a migration for a new feature, up() it,
> mess around a bit, decide that it wasn't quite right, down() it,
> delete the migration and try something else - all the while never
> actually checking your migration in to source control.

Yeah this was exactly what i was thinking. I dont have any clue on how
evolution works but dmigs seems really nice!