Django migrations taking all the memory on Django 1.9

[andrea crotti]

We have a very big Django project with > 100 apps and > 300 models.

We had some massive issues with Django 1.8 and migrations, which would take forever and just take all the computer memory after a while.

Now with Django 1.9 things improved, however we are again back with some extremely bad performances and massive memory usages (up to 8GB of RAM for example), sometimes just to run ONE single migration.

It's not even entirely deterministic though sometimes killing the process and doing it again just works.

I'm attaching the profile graph (done wtith gprof2dot) from running "./manage migrate" and one thing that clearly looks bad is that *render* for example is called 44355 times, which is definitively not normal.

Any idea about about what we can do about this and what could be the problem?

I have the impression that it's related with the amount of models and how they are interconnected, and mabye some caching would avoid all this extra computation.

Noone else has similar issues?

Thanks

[Markus Holtermann]

Thanks Andrea,

this is a known issue. Let me give you a bit of a history on this:

In 1.7 Django didn't do any caching. That was awfully slow for huge
projects like yours.

In 1.8 Django introduced heavy caching. Things improved speed wise but
failed horrifically when you were altering relations or related models

In 1.8.1 (I believe it was the .1 patch release, not sure right now)
Django reduced the caching and ensured that related models are
"reloaded" [1, 2]. At least that was the plan.

In 1.9 Django slightly changed the way it detected the migration plan
and dropped an undocumented and unimplemented feature [3, 4]. This
improved the forward migrations but remains slow for backwards
migrations.

My plan is to get rid of the "render()" method during schema
alterations. Only when using RunPython the requested models would need
to be rendered. I started implementing that in [5]. If you're looking
into contributing there, I'm happy to talk to you in more detail,
preferably on IRC in #django-dev.

Cheers,

/Markus

[1] https://github.com/django/django/commit/a1ba4627931591b80afa46e38e261f354151d91a
[2] https://code.djangoproject.com/ticket/24225
[3] https://github.com/django/django/commit/5aa55038ca9ac44b440b56d1fc4e79c876e51393
[4] https://code.djangoproject.com/ticket/24745
[5] https://github.com/MarkusH/django/commits/schemaeditor-modelstate

>--
>You received this message because you are subscribed to the Google Groups "Django users" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
>To post to this group, send email to django...@googlegroups.com.
>Visit this group at https://groups.google.com/group/django-users.
>To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5b67eccf-d48f-4926-86ec-4bd2a8427dcb%40googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.

[andrea crotti]

Ok great thanks for the answer Markus.

And yes I can try to help, it's quite a big issue for us..

Anything else we can do to improve the situation in the meanwhile?

In general I was wondering why do we need to handle all the "baggage" of supporting backward migrations and doing things 100% properly while running migrations just for tests.

I think that in general if there was just a way to render all the SQL that needs to be run statically for tests it would be great.

[Tim Graham]

As long as your tests don't rely on any data migrations, you can set MIGRATIONS_MODULES['app'] = None for all apps in a test settings file as described in https://docs.djangoproject.com/en/stable/ref/settings/#std:setting-MIGRATION_MODULES to disable migrations while testing. There's a ticket to make that a bit easier: https://code.djangoproject.com/ticket/25388

[andrea crotti]

Since we also use --keepdb I thought we could simply avoid running all the migrations, and maybe dynamically even skip all the apps that were not changed at all. 

So I just tried something silly like this:

NO_MIGRATIONS = os.environ.get('NO_MIGRATIONS', 'false').lower() in ('true', '1')

if NO_MIGRATIONS:

    for app in INSTALLED_APPS:

        if app not in MIGRATION_MODULES:

            MIGRATION_MODULES[app] = None

But I get this error:

django.db.migrations.exceptions.NodeNotFoundError: Migration something.0001_initial dependencies reference nonexistent parent node ('something_else', u'0001_initial')

So I guess it's not as simple (and maybe that ticket is what that's about, but sadly it's not even merged in master now from my understanding).

PS. so to clarify our tests do rely on migrations (and there are some data migrations too) but since we normally use --keepdb anyway we definitively not need to run only a few migrations at a time max (and even then sometimes it takes ages and uses all the RAM).

[Tim Graham]

I think that error suggests that you disabled migrations for an app (something_else) that's a dependency of another app that doesn't have migrations disabled (something). That isn't permitted. I guess there's no good solution for your requirements as of now.

[andrea crotti]

Thanks Marcus and others, this issue is in general getting worse and worse.

We did some more investigation and we should probably just help out the work you are doing.

Do you think it would be possible to apply these changes also on Django 1.9.9 or it has to be from master/1.10 only?

I also tried another approach:

- use "./manage.py sqlmigrate <app> <name>" to generate the SQL

- add in the generated SQL the INSERT to django_migrations

- run the SQL manually in the DB

This actually works and could be automated, but I'm quite sure it's not a good idea..

But what do you think could go wrong with this approach?

I would say that as long as the dependencies are met and things are done in the right order it might be safe enough, but of course we get into muddy waters..

[Tim Graham]

Per our support versions policy, Django 1.9.x is only receiving fixes for security and data loss issues. Migrations optimizations are unlikely to qualify for a backport to Django 1.10.x either. Hopefully that doesn't demotivate you from contributing an improvement.

https://docs.djangoproject.com/en/dev/internals/security/#supported-versions

[andrea crotti]

Ah ok thanks, well generally it's not a blocker, except that we first have to upgrade to 1.10.

We have done a lot of migrations in the last few months and given the size of the project it always takes a long time from one version to the other, so just not sure when I'll be able to test these changes on our project..