Thoughts on diff-based migrations with django?

[djrobstep]

Hi all,

Some thoughts on schema migrations that may interest you. By way of background, I'm the author of migra (https://github.com/djrobstep/migra), a schema comparison/diff tool for Postgres.

I initially wrote this tool because I wanted to be able to generate migration scripts automatically, without needing historical migration files.

It means that you can do things like sync your database to your models mostly automatically, and also explicitly test for a matching schema. Instead of a chain of migration files you only ever need to keep track of one, containing any pending changes).

I've used this approach with success on a number of projects now, and it seems to work pretty well. I also talked about this approach to migrations at the most recent PostgresOpen (https://www.youtube.com/watch?v=xr498W8oMRo).

The downside of this approach is that it's slightly more operationally complex (you need have a temporary copy of your production schema available for comparison purposes, typically a --schema-only dump file) and also that it's postgres-only at this point. It's also conceptually quite different to the typical Django/Rails style of migrations which represents a barrier to entry.

In spite of this, I think doing migrations this way has some compelling advantages, that are relevant to users of Django.

I'm interested in your opinions on this topic as Django developers. What do you think? Is it something worth offering as an option for managing Django migrations? If your feelings are positive, I'm happy to take a look at extending the code accordingly.

[Curtis Maloney]

Hi,

I must say your idea intrigues me... and I'd certainly like to discuss
it further.

The first thought that comes to mind is how would such a system handle
data migrations?

Being able to leap from one schema state to another is great, buy
typically in the life-cycle of a reusable app it's required to also
migrate old data into new schema.

This also happens within project-specific apps.

--
Curtis

> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com
> <mailto:django-develop...@googlegroups.com>.
> To post to this group, send email to django-d...@googlegroups.com
> <mailto:django-d...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[djrobstep]

Yes, this approach certainly doesn't magically solve data migration issues, however the data migrations question has been a common one.

The nature of data migrations is that they are very one-off and context dependent and so still something you'd need to script up manually. You might use the diff tool to generate the migration script required but then edit it to add any queries needed for carrying out the data migration.

And doing things this way, there's nothing stopping you continuing to chain multiple migration scripts together to support multiple versions in the context of a reusable app.

> <mailto:django-developers+unsub...@googlegroups.com>.

[Andrew Godwin]

Hi,

This was an approach we deliberately avoided during development of the current migration system - it works poorly with diverging feature branches and requires that your developers have access to production schema at all times (and additionally that you don't have divergent production/staging/QA deploys, like many environments do).

I think it would be a fine addition as an optional third-party tool, but it's not something that really makes sense to include in Django at this point I think.

Andrew

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com.

[djrobstep]

Yes, I don't think it would make sense to include in django directly. Because the diffing approach works at the database directly, below the layer of the models, the two parts don't really need to be integrated. All that is really required for django to support this as an option is to provide a documented way to emit the from-scratch creation SQL for the currently defined models.

As far as I can tell there used to be an admin command available for this in version 1.x but it was removed? Is there already another supported way to do this?

Incidentally, I've used this approach on some larger projects with diverging branches and it worked pretty well. You do have to set some things up differently tho - keep meaning to write a blog post or something about this. The need to have access to a production schema is an additional hurdle, but well worth it for the additional testability and robustness, in my experience.

On Wednesday, 21 February 2018 18:44:30 UTC+11, Andrew Godwin wrote:

Hi,

This was an approach we deliberately avoided during development of the current migration system - it works poorly with diverging feature branches and requires that your developers have access to production schema at all times (and additionally that you don't have divergent production/staging/QA deploys, like many environments do).

I think it would be a fine addition as an optional third-party tool, but it's not something that really makes sense to include in Django at this point I think.

Andrew

On Mon, Feb 19, 2018 at 10:05 PM, djrobstep <robert...@gmail.com> wrote:

Hi all,

Some thoughts on schema migrations that may interest you. By way of background, I'm the author of migra (https://github.com/djrobstep/migra), a schema comparison/diff tool for Postgres.

I initially wrote this tool because I wanted to be able to generate migration scripts automatically, without needing historical migration files.

It means that you can do things like sync your database to your models mostly automatically, and also explicitly test for a matching schema. Instead of a chain of migration files you only ever need to keep track of one, containing any pending changes).

I've used this approach with success on a number of projects now, and it seems to work pretty well. I also talked about this approach to migrations at the most recent PostgresOpen (https://www.youtube.com/watch?v=xr498W8oMRo).

The downside of this approach is that it's slightly more operationally complex (you need have a temporary copy of your production schema available for comparison purposes, typically a --schema-only dump file) and also that it's postgres-only at this point. It's also conceptually quite different to the typical Django/Rails style of migrations which represents a barrier to entry.

In spite of this, I think doing migrations this way has some compelling advantages, that are relevant to users of Django.

I'm interested in your opinions on this topic as Django developers. What do you think? Is it something worth offering as an option for managing Django migrations? If your feelings are positive, I'm happy to take a look at extending the code accordingly.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

[djrobstep]

Bump! Is there a documented way to generate the from-scratch creation SQL for the currently defined models? If not, is it possible to add something that spits this out? Happy to take a look at implementing this.

[Florian Apolloner]

Just for one migration; but there is the sqlmigrate management command which you can use as building base.

Cheers,
Florian