[Django] #34841: Reverse migrations model state rendering slow with moderate to large migrations
Django
migrations
------------------------------------------------+------------------------
Reporter: David Sanders | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Migrations | Version: 4.2
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------------+------------------------
Relevant discussion: https://forum.djangoproject.com/t/migrating-
backwards-takes-orders-of-magnitude-longer-than-migrating-forwards/20873/2
Investigations revealed 2 things.
1. this block of code in `db/migrations/excecutor.py` was responsible for
a large portion of processing time:
https://github.com/django/django/blob/b8b2f7451201f3ff60891b6ce55f177400700d7a/django/db/migrations/executor.py#L222-L232
In my small project with only a moderate number of migrations, a simple
reverse migrate of the last migration on the main app took ~20s. The block
below was responsible for ~16s.
{{{
# Generate the post migration state by starting from the state
before
# the last migration is unapplied and mutating it to include all
the
# remaining applied migrations.
last_unapplied_migration = plan[-1][0]
state = states[last_unapplied_migration]
for index, (migration, _) in enumerate(full_plan):
if migration == last_unapplied_migration:
for migration, _ in full_plan[index:]:
if migration in applied_migrations:
migration.mutate_state(state, preserve=False)
break
}}}
2. these 2 lines appeared to be responsible for the slowdown, commenting
them out made the block above run almost instantly:
https://github.com/django/django/blob/b8b2f7451201f3ff60891b6ce55f177400700d7a/django/db/migrations/executor.py#L203-L204
{{{
if "apps" not in state.__dict__:
state.apps # Render all -- performance critical
}}}
I have a small patch to move record the unapplied state before these 2
lines and clone it, removing the `state.apps` attribute.
--
Ticket URL: <https://code.djangoproject.com/ticket/34841>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
migrations
Reporter: David Sanders | Owner: David
Type: | Sanders
Cleanup/optimization | Status: assigned
Component: Migrations | Version: 4.2
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by David Sanders):
* owner: nobody => David Sanders
* status: new => assigned
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/34841#comment:1>
Django
migrations
-------------------------------------+-------------------------------------
Reporter: David Sanders | Owner: David
Type: | Sanders
Cleanup/optimization | Status: assigned
Component: Migrations | Version: 4.2
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Simon Charette (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/34841#comment:2>
Django
migrations
-------------------------------------+-------------------------------------
Reporter: David Sanders | Owner: David
Type: | Sanders
Cleanup/optimization | Status: assigned
Component: Migrations | Version: 4.2
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak):
* cc: Markus Holtermann, Shai Berger, Josh Smeaton (added)
* stage: Unreviewed => Accepted
Comment:
Tentatively accepted for the investigation.
--
Ticket URL: <https://code.djangoproject.com/ticket/34841#comment:3>
Django
migrations
-------------------------------------+-------------------------------------
Reporter: David Sanders | Owner: David
Type: | Sanders
Cleanup/optimization | Status: assigned
Component: Migrations | Version: 4.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/34841#comment:4>
Django
migrations
-------------------------------------+-------------------------------------
Reporter: David Sanders | Owner: David
Type: | Sanders
Cleanup/optimization | Status: assigned
Component: Migrations | Version: 4.2
Severity: Normal | Resolution:
| checkin
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/34841#comment:5>
Django
migrations
-------------------------------------+-------------------------------------
Reporter: David Sanders | Owner: David
Type: | Sanders
Component: Migrations | Version: 4.2
Severity: Normal | Resolution: fixed
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* resolution: => fixed
* status: assigned => closed
Comment:
In [changeset:"b6e2b83901d841cf4d2179f650d1e7dca7e90c16" b6e2b83]:
{{{#!CommitTicketReference repository=""
revision="b6e2b83901d841cf4d2179f650d1e7dca7e90c16"
Fixed #34841 -- Avoided rendering apps on state still requiring mutation.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/34841#comment:6>