Keeping apps without migrations?

[Claude Paroz]

Tim recently did a fabulous job of removing deprecated code for the
future 1.9 on master. Thanks for that.

However, one thing he removed was support for apps without migrations.
https://github.com/django/django/commit/7e8cf74dc74539f40f4cea53c1e8bba82791fcb6

Considering that we have to keep internal support for Django's own test
suite anyway, I wonder if we should remove that support at all for
"normal" projects. I think one of Andrew's motivation was not to have to
keep two schema editing code bases. But now that the old syncdb also
uses the new schema editor, I think that this argument doesn't stand.

So what about keeping support for apps without migrations in the longer
term. Of course, the fact that those apps cannot depend on migrated apps
limits its use, but I think that for quick prototyping and initial
developement, migrations could be more of a hindrance. Would you see
major drawbacks with keeping this support?

Opinions welcome.

Claude
--
www.2xlibre.net

[Andrew Godwin]

My main argument for removing them entirely was the dependency issues between apps with and without migrations. Having syncdb use SchemaEditor is a big step and one I'm happy we've got to, but the only advantage of keeping syncdb is for the test suite and I'd rather we approach that more as "migrations made and run at runtime as a special case" rather than "syncdb".

If nothing else, I'd like to see the end-developer-facing parts, like the syncdb command itself, gone.

Andrew

--
www.2xlibre.net

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1421617394.5741.4.camel%40doulos.
For more options, visit https://groups.google.com/d/optout.

[Markus Holtermann]

Creating in-memory migrations for all apps that don't have migration
files seems to be an option to solve the dependency problem. This would
even allow apps without migrations to depend on those with migrations.

We have to consider though, that there are tens of apps and hundreds of
models in our own test suite, and generating all migrations during start
seems to be quite an expensive task. And I'm not even talking about the
migration optimizer which probably needs to get a lot smarter if we take
this road.

syncdb, which leaves a developer with a database scheme that cannot be
altered automatically, is something we should get rid of as soon as
possible, especially since Django has a out-of-the-box migration system.

/Markus

[Tim Graham]

With the alpha for 1.9 coming up in 6 weeks, we need to decide whether or not to continue support for apps without migrations (currently in master no tables are created for such apps (as the deprecation timeline says), but it might be appropriate to add a warning or error message for this case if things don't change).

Is anyone sufficiently interested in continuing support for apps without migrations to commit to completing it by the alpha? If not, do you consider it a release blocker that I should spend time on?

Related tickets:

https://code.djangoproject.com/ticket/25144 - No way to create tables for apps without migrations
https://code.djangoproject.com/ticket/24919 - Add an option not to run migrations when running tests
https://code.djangoproject.com/ticket/24481 - Improve sqlmigrate to be more flexible and allow bypassing migrations on disk
https://code.djangoproject.com/ticket/24901 - makemigrations should create empty migrations dir for any installed app without it
https://code.djangoproject.com/ticket/24588 - Improve handling apps without migrations while running migrate command.

[Andrew Godwin]

Do we know exactly what "support for apps without migrations" would consist of at this point? I have that half-done code for replicating syncdb with the autodetector stuck onto the migration executor, but it's not especially speedy and would need some work to make it a sensible speed before it goes forward.

Alternately, we could modify the autodetector to have a "create only" mode that's more efficient or just write a new generator that is quick because it knows it's only making nonmigrated apps' migrations.

Andrew

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/89839628-6b11-4c61-86ed-4d9a076e9e3d%40googlegroups.com.

[Markus Holtermann]

Instead of rushing to a decision if and of how we continue to support
"apps w/o migrations" I would argue that, since Django's test suite does
infact create tables for apps w/o migrations, we could add a commandline
flag (e.g. --run-syncdb or whatever) to the migrate command and warn the
user if there are apps w/o migrations an the user doesn't supplied:
https://github.com/django/django/blob/master/django/core/management/commands/migrate.py#L136

/Markus

>> <https://groups.google.com/d/msgid/django-developers/89839628-6b11-4c61-86ed-4d9a076e9e3d%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .

>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>

>--
>You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
>To post to this group, send email to django-d...@googlegroups.com.
>Visit this group at http://groups.google.com/group/django-developers.

>To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1urojSqXkaDnjVdrPHrs5TKpQC-baz9TZLtvOZ_3UJ%2BM2Q%40mail.gmail.com.

>For more options, visit https://groups.google.com/d/optout.

--

[Tim Graham]

It's a one line change in the code to keep the status quo from 1.8. The main drawback I see is continued support tickets about dependencies between apps with and without migrations. I already removed the documentation describing the problem and "won't fixed" tickets that suggested doing more to detect and throw errors about these cases.

I guess I'd like here why migrations could be a hindrance "for quick prototyping and initial development." I'd argue that starting with migrations from the start (manage.py makemigrations && manage.py migrate) will save the need to learn about --fake-initial if you decide to add migrations at some point later (and of course avoid dependency issues). Is "manage.py makemigrations && manage.py migrate" more difficult that the proposed ""manage.py migrate --allow-apps-without-migrations"? Am I missing a motive behind the request?

diff --git a/django/core/management/commands/migrate.py b/django/core/management
index 2f04824..dec86e1 100644
--- a/django/core/management/commands/migrate.py
+++ b/django/core/management/commands/migrate.py
@@ -133,7 +133,7 @@ class Command(BaseCommand):
             targets = executor.loader.graph.leaf_nodes()
 
         plan = executor.migration_plan(targets)
-        run_syncdb = options.get('run_syncdb') and executor.loader.unmigrated_a
+        run_syncdb = bool(executor.loader.unmigrated_apps)
 
         # Print some useful info
         if self.verbosity >= 1:

[Tim Graham]

It would be great if we could decide this by next Friday (September 4) or so, so that we have some time to do the necessary implementations and testing before alpha. (For example, running manage.py test without migrations currently gives "no such table ...". It would be nice to give a friendlier message if we require migrations for running tests (https://github.com/django/django/pull/5100 does this to some extent, but could be improved I think).

I know at least Markus said he would like to present an argument for keeping support for apps without migrations.

[Markus Holtermann]

The general benefit of *not* having migrations is the incredible better performance to setup the tables. Especially for larger projects this is something we should at least keep in mind (see https://code.djangoproject.com/ticket/24743). I spent a fair bit of time on the issue and have not yet found a satisfying solution.

Supporting apps without migrations would essentially continue to support the old `syncdb` behavior, though not as a standalone command, but as a flag for `manage.py migrate`. It takes the models from those apps and calls `schema_editor.create_model()`, taking the deferred SQL and running it after all models of apps w/o migrations have been created (current behavior [in tests] I think). This would continue to raise issues if models in apps w/o migrations refer to models in apps w/ migrations (missing constraints), but that is a limitation we had since 1.7 (silent failure; 1.8 fails loud).

While migrations should provide *the* way to maintain your database I see the points that have been made in other recent discussions. Although Django doesn't force you to use migrations there are some things (e.g. not create the `django_migrations` table) we probably should not do unless you actually use migrations.

We might want to consider looking at the `MIGRATION_MODULES` settings variable and allowing to set values to `None` in order to disable migrations for that app, but that's a different story.

/Markus

[Andrew Godwin]

I think I agree with Markus that we do not yet have a good enough performance story to make migrations mandatory. If you want to ditch them and do your own database management, we should let you - which we could do with some kind of MIGRATIONS_MODULES opt-out that basically sets everything to pseudo-unmanaged, but I see no reason not to have the syncdb-like solution keep working as it at least runs off of SchemaEditor now.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b079f398-92fb-43e4-8ab5-6c1c8dbc9b09%40googlegroups.com.

[Anssi Kääriäinen]

Could we allow applications to have fake migration files? I'm thinking
of something like operations.AllModelsUpdated(). After this operation
is ran, Django will use the present models for that app. You could
place this as your last migration, and run direct SQL in previous
migrations (or use some completely different mechanism) . Direct SQL
without model state changes will be fast. The last migration would
supposedly be fast, as it only needs to construct up to date images of
the current models.

The problem is that I'm not at all sure how this will mix with
migrations from other applications, so this proposal is perhaps
technically impossible to do. But if this is feasible, then it should
be possible to mix migrations and manual database changes pretty much
seamlessly. This one would work much better if we had project level
migrations - implementing a migration operation that tells Django that
all of the project's models are up to date is almost trivial.

I believe a fake migration would work pretty well for the use case
where you want to use manual database control for your project, but
use 3rd party applications with migrations.

Another idea I've been thinking is that maybe we could have snapshots
for migrations. The reason I believe this would work is that building
the model state is expensive, but in many cases it is completely
redundant to build the model state from scratch.

The snapshots would essentially do the following operation: after
migrations X, Y and Z are applied the model state will be as presented
in this migration. The snapshots would only contain model state, not
any operations. When you run manage.py migrate, the app state builder
will be able to skip to latest snapshot that has all dependencies
already applied, and then continue building model state from there.
When performance begins to degrade, you can create a new snapshot. The
snapshots would probably have to be project level to work properly.
This would seriously speed up migrations, except for the case where
you run the migrations against fresh database.

- Anssi

> https://groups.google.com/d/msgid/django-developers/CAFwN1ur_yiBmRuj%2BY26bFNJZe7meZpVsQnTuZLRWB9dj6GBqNg%40mail.gmail.com.

[Andrew Godwin]

On Thu, Aug 27, 2015 at 10:25 PM, Anssi Kääriäinen <akaa...@gmail.com> wrote:

Could we allow applications to have fake migration files? I'm thinking
of something like operations.AllModelsUpdated(). After this operation
is ran, Django will use the present models for that app. You could
place this as your last migration, and run direct SQL in previous
migrations (or use some completely different mechanism) . Direct SQL
without model state changes will be fast. The last migration would
supposedly be fast, as it only needs to construct up to date images of
the current models.

The problem is that I'm not at all sure how this will mix with
migrations from other applications, so this proposal is perhaps
technically impossible to do. But if this is feasible, then it should
be possible to mix migrations and manual database changes pretty much
seamlessly. This one would work much better if we had project level
migrations - implementing a migration operation that tells Django that
all of the project's models are up to date is almost trivial.

The problem is mainly swappable models. We can only allow one direction of "migrated depending on unmigrated" or "unmigrated depending on migrated", so I chose "migrated depending on unmigrated". The problem is all about when to run the "syncdb" phase; if you run it before migrations you get one solution, and after the other.

Allowing apps to depend on these fake migrations would only work if "executing" them actually performed the syncdb phase of the cycle, which unfortunately is written in an entire-project-at-once way, so it doesn't have to handle FK dependencies. Solving that problem would seem to be very difficult to me, and probably impossible (you'd need to be able to syncdb individual apps at once, and correctly detect dependencies to migrated apps and other unmigrated apps, and so on, until you just have given them migrations in-memory).

 

I believe a fake migration would work pretty well for the use case
where you want to use manual database control for your project, but
use 3rd party applications with migrations.

Another idea I've been thinking is that maybe we could have snapshots
for migrations. The reason I believe this would work is that building
the model state is expensive, but in many cases it is completely
redundant to build the model state from scratch.

The snapshots would essentially do the following operation: after
migrations X, Y and Z are applied the model state will be as presented
in this migration. The snapshots would only contain model state, not
any operations. When you run manage.py migrate, the app state builder
will be able to skip to latest snapshot that has all dependencies
already applied, and then continue building model state from there.
When performance begins to degrade, you can create a new snapshot. The
snapshots would probably have to be project level to work properly.
This would seriously speed up migrations, except for the case where
you run the migrations against fresh database.

I think Markus was toying with this idea, but I'm not sure if it was similar or not. It definitely sounds faster - this is how South did it, of course, and given we now have all the serialisation/deserialisation code for models it would seem to be not too hard to do.

 

Andrew

[Markus Holtermann]

Hey Anssi,

thanks for the ideas. A few comments inline

On Friday, August 28, 2015 at 3:25:34 PM UTC+10, Anssi Kääriäinen wrote:

Could we allow applications to have fake migration files? I'm thinking
of something like operations.AllModelsUpdated(). After this operation
is ran, Django will use the present models for that app. You could
place this as your last migration, and run direct SQL in previous
migrations (or use some completely different mechanism) . Direct SQL
without model state changes will be fast. The last migration would
supposedly be fast, as it only needs to construct up to date images of
the current models.

That is something you can already do by using `SeparateDatabaseAndState` https://docs.djangoproject.com/en/1.8/ref/migration-operations/#django.db.migrations.operations.SeparateDatabaseAndState. Use RunSQL for the Database part and CreateModel for the state part.

The problem is that I'm not at all sure how this will mix with
migrations from other applications, so this proposal is perhaps
technically impossible to do. But if this is feasible, then it should
be possible to mix migrations and manual database changes pretty much
seamlessly. This one would work much better if we had project level
migrations - implementing a migration operation that tells Django that
all of the project's models are up to date is almost trivial.

I don't see how project level migrations would make that easier. Unless you have only changes to the data in the database Django won't change any SQL and will happily reuse the latest models up to that state.

I believe a fake migration would work pretty well for the use case
where you want to use manual database control for your project, but
use 3rd party applications with migrations.

Another idea I've been thinking is that maybe we could have snapshots
for migrations. The reason I believe this would work is that building
the model state is expensive, but in many cases it is completely
redundant to build the model state from scratch.

The snapshots would essentially do the following operation: after
migrations X, Y and Z are applied the model state will be as presented
in this migration. The snapshots would only contain model state, not
any operations. When you run manage.py migrate, the app state builder
will be able to skip to latest snapshot that has all dependencies
already applied, and then continue building model state from there.
When performance begins to degrade, you can create a new snapshot. The
snapshots would probably have to be project level to work properly.
This would seriously speed up migrations, except for the case where
you run the migrations against fresh database.

You are only half right: the performance problem is the *rendering* of the models from the model states. However I don't see how having checkpoints would help, reusing the state. It would help to not need to create the state up until that point, but the performance issue there got resolved I think. The main problem is models having 'object references' to other models (`models.ForeignKey(User)` vs `models.ForeignKey('auth.User')`). In order to have consistent references between two model classes (i.e. `ModelA._meta.get_field('fk_to_modelb').related_model is ModelB`) all related models have to be rerendered once one changes: https://github.com/django/django/blob/master/django/db/migrations/state.py#L32

 

 - Anssi

/Markus

[Tim Graham]

In the absence of a more sophisticated implementation, here's a possible simple solution: https://github.com/django/django/pull/5212

Maybe we can come up with a better name for the migrate option (currently --run-syncdb).

[Tim Graham]

As far as how well apps without migrations would work compared to 1.8, a couple things come to mind:

1. There's no management commands to get the SQL for these apps.
2. There's no more support for initial data/custom SQL.