A new approach for migrations?

[Uri P]

I have to admit I don't know the ins and outs of the new migration system in django1.7+, and I don't quite understand why it fails to handle circular dependencies which the old fashioned syncdb does. I would love to hear an explanation, but my main question is:

If the initial syncdb is actually capable of creating the database for the models despite of the circular dependencies, why not base the migrating mechanism simply on some kind of DIFF between the existing database and the one that would have been created by syncdb for the newly changed models?

[Joe Tennies]

First this is really a question for the django-users group. This is more for actually developing Django. I think the root of your issue is actually the app loading feature and not migrations. Your models should be lazy loading each other (using the string version not directly importing).

Secondly, the diff approach is difficult to get right, especially when you consider data migration as well as schema migration. You can have a tested path from a to d via b and c. It's tough to know how to get from a to d without b and c.  And the permutations grow exponentially. The Python Community and especially the Django people are heavily into testing being an important (read required) part of development and change integration testing.

On Oct 16, 2014 5:38 AM, "Uri P" <uri...@gmail.com> wrote:

I have to admit I don't know the ins and outs of the new migration system in django1.7+, and I don't quite understand why it fails to handle circular dependencies which the old fashioned syncdb does. I would love to hear an explanation, but my main question is:

If the initial syncdb is actually capable of creating the database for the models despite of the circular dependencies, why not base the migrating mechanism simply on some kind of DIFF between the existing database and the one that would have been created by syncdb for the newly changed models?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/cef5b6e4-9f2e-4c81-aaee-0eede10d55b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Godwin]

Hi Uri,

That was the approach taken by an older problem called django-evolution; the main problem with that approach is that it means the migrations that happen are different on every system depending on the database, down to the point where a system with frequent schema updates might detect a rename whereas one that gets the schema lumped on it in one go might diff incorrectly (as it's not seen the intermediary stages) and start deleting and making models when it shouldn't.

In addition, there's no place to put things like data migrations.

The second option (which was taken by the old dmigrations project) was project-wide migrations rather than per-app, allowing circular dependencies to all be done in a single migration, much like syncdb did them all in a single pass. The problem here, of course, is that this makes it impossible to ship migrations with third-party apps, so they don't have any way of doing more complex changes than adding/removing a field because of the "dumb diff" issues - it's impossible to always pick a field/model rename versus a model addition and deletion.

So, we have the app-based migration current system, which allows people to run the diff (that's what the Autodetector and makemigrations does), and then review and refine its output first before preserving the final, polished migrations on disk in a distributable format that follows Django's app-based structure.

I hope that helps.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CACiOJ6s4APj4fWcugVA3Bn2_Z1NvtE64dWg65Xoch5vb8xSQ8w%40mail.gmail.com.

[Uri P]

Your models should be lazy loading each other (using the string version not directly importing).

Lazy loading is actually what got me into this mess - that's exactly what enabled my predecessors in my current project, to create those circular dependencies. They left me for example with two models in two apps pointing at each other. Python would have thrown them an ImportError if it weren't for lazy loading.

The migrations feature, however, doesn't like that too much.

[Uri P]

Thanks Andrew for enlightening me. That was helpful.

Anyway, for me personally, data migration is the easiest part, I can do it in a heartbeat by a simple script.

In contrary, solving circular dependencies seems to be very difficult in my case, especially in light of the currently still limited amount of documentation on the issue...

But at least now I understand the different approaches. Thanks.

[Shai Berger]

Hi Uri,

On Friday 17 October 2014 01:30:10 Uri P wrote:
> Thanks Andrew for enlightening me. That was helpful.
>
> Anyway, for me personally, data migration is the easiest part, I can do it
> in a heartbeat by a simple script.
> In contrary, solving circular dependencies seems to be very difficult in my
> case, especially in light of the currently still limited amount of
> documentation on the issue...
>

You've said that the circular dependencies are "enabled" by lazy loading,
which implies that the references are more-or-less regular foreign keys (not,
for example, the implicit one-to-one key generated for multi-table
inheritance). In that case, breaking the circular dependency is not hard:
comment out one of the offending FKs (so there is no longer a circular
dependency), run makemigrations, un-comment it, and run makemigrations again.
You will get, for the app where you did it, two migrations -- one creating the
model without the FK, and one adding it, with all the dependencies
automatically set up.

It may be useful to add something like this to the migrations documentation,
although I suspect there wll always be cases left uncovered, and solving such
problems in general will ultimately require one to "know the ins and outs of
the new migration system".

HTH,

Shai.

[Andrew Godwin]

It's worth pointing out that the migration system does try to solve circular references itself using this same strategy - we have a pretty extensive set of tests that make sure the autodetector splits apart circular reference cycles in a variety of situations. The only case where it's known to fail is swappable models (so AUTH_USER_MODEL), as we can't predict the value of it.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/201410170315.11295.shai%40platonix.com.

[Uri P]

Thanks Shai! That trick helped me a lot. I eventually managed to add migrations successfully using this trick and some manual adjustments to the migration files.

I'm optimistic that things will flow more easily from now on. Migrations is such an awesome addition to django, but I have to admit the circular dependencies were a serious bummer to handle. A guide on the subject by someone who knows its "ins and outs" would probably be extremely helpful.

Thanks again, Andrew.