[Django] #36992: Migrations that belong to apps that are always routed to different databases can be shortcut to be "faked" earlier

[Django]

#36992: Migrations that belong to apps that are always routed to different
databases can be shortcut to be "faked" earlier
-------------------------------------+-------------------------------------
Reporter: Suzannah Cooper | Type:
| Cleanup/optimization
Status: new | Component:
| Migrations
Version: 6.0 | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
One usage of routers is to be able to say which database tables within a
Django app should be routed to. Within the application I am working on, we
have an implementation of `allow_migrate` which takes in the `app_label`
and database and always returns `False` if it is for an `app_label` that
should go to a different database.

I've been working within a large Django codebase, which has many
migrations (that should have been squashed, but for whatever reason have
not been squashed yet). We have been setting up another database. Django
requires that all the migrations that are routed to the default database
also be applied on this new database. Given that those old migrations
should never be routed, we expected these to only be no-ops and performing
an insert to show that the migration has been marked as completed.

When profiling locally with a sample size of ~3000 migrations that would
always be routed elsewhere, we found that the
`executor.py::apply_migration` function was taking up a large chunk of the
time. For example, this took somewhere in the realm of half an hour to run
all migrations. The `migration.apply(state, schema_editor)` is where the
`allow_migrate` function was being checked within, another few function
calls deep, within the migration `database_forwards` logic itself.

When we added a check for `allow_migrate` within the `apply_migration`
function, skipping the `migration.apply` if `allow_migrate` returned
False, we had a speedup so significant that it changed from the order of
an hour or so to a minute.

Attaching some profiled logs screenshots for the individual functions. Can
provide more upon request
--
Ticket URL: <https://code.djangoproject.com/ticket/36992>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36992: Migrations that belong to apps that are always routed to different
databases can be shortcut to be "faked" earlier
-------------------------------------+-------------------------------------

Reporter: Suzannah Cooper | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Migrations | Version: 6.0
Severity: Normal | Resolution:

Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Suzannah Cooper):

* Attachment "image-20260318-164031.png" added.

[Django]

#36992: Migrations that belong to apps that are always routed to different
databases can be shortcut to be "faked" earlier
-------------------------------------+-------------------------------------
Reporter: Suzannah Cooper | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Migrations | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Suzannah Cooper):

* Attachment "image-20260318-164138.png" added.

Profiling using cProfile before - can see 3408 instances of calling
`apply_migration` with cumulative time of 1000

[Django]

#36992: Migrations that belong to apps that are always routed to different
databases can be shortcut to be "faked" earlier
-------------------------------------+-------------------------------------
Reporter: Suzannah Cooper | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Migrations | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Suzannah Cooper):

* Attachment "image-20260318-164031.png" removed.

[Django]

#36992: Migrations that belong to apps that are always routed to different
databases can be shortcut to be "faked" earlier
-------------------------------------+-------------------------------------
Reporter: Suzannah Cooper | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Migrations | Version: 6.0
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Suzannah Cooper):

* Attachment "image-20260318-164231.png" added.

Profiling using cProfile after - can see 3408 instances of calling
`apply_migration` with cumulative time of ~0.66

[Django]

#36992: Migrations that belong to apps that are always routed to different
databases can be shortcut to be "faked" earlier
-------------------------------------+-------------------------------------
Reporter: Suzannah Cooper | Owner: (none)

Type: | Status: closed

Cleanup/optimization |
Component: Migrations | Version: 6.0

Severity: Normal | Resolution: duplicate

Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Simon Charette):

* resolution: => duplicate
* status: new => closed

Comment:

The database operations rely on `Operation.allow_migrate_model` (which
calls `Options.can_migrate`) and `Router.allow_migrate_model` which
eventually delegate to `Router.allow_migrate` to determine if a model
should be migrated.

The `allow_migrate_model` functions expect a fully built model class to be
provided as it relies on `required_db_features` and `required_db_vendor`
which are both options we don't track in migrations.

In my opinion this is a duplicate of #29898 which tracks making the
migration framework operate from `ModelState` instead of dynamically
rendering model classes (I suspect a lot of time is spent in
`ModelBase.__new__`, `ModelState.render`, and `ModelState.render_multiple`
here).
--
Ticket URL: <https://code.djangoproject.com/ticket/36992#comment:1>