[Django] #37061: Add migration_recorder_class and migration_executor_class hooks to BaseDatabaseWrapper so third-party backends can customise migration infrastructure without monkey-patching Django internals.

[Django]

#37061: Add migration_recorder_class and migration_executor_class hooks to
BaseDatabaseWrapper so third-party backends can customise migration
infrastructure without monkey-patching Django internals.
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Type:
| Cleanup/optimization
Status: new | Component:
| Migrations
Version: 6.0 | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
== Context

I'd like to use clickhouse inside my django project. The main project to
do that is [https://github.com/jayvynl/django-clickhouse-backend django-
clickhouse-backend],
but it currently patches too many things related to migrations, that's an
issue in my project.

Third-party database backends that need to customise Django's migration
behaviour currently
have no way to extend relevant classes. The only available mechanism is to
monkey-patch
`MigrationRecorder` (to customise migration tracking) and
`Migration.apply` (to inject
per-operation logic before execution), affecting the entire Django process
globally.

With `migration_executor_class`, a backend can instead subclass
`MigrationExecutor` and
override `apply_migration` — the method that calls `migration.apply()`.
Per-operation
logic moves up to the executor level, and `Migration.apply` itself never
needs to be
touched.

== Proposed change

Add two `None` defaulting class attributes to `BaseDatabaseWrapper`,
following the exact same pattern already used for `schema_editor_class`,
`creation_class`, `introspection_class`, `ops_class`, and
`validation_class`:

{{{
# django/db/backends/base/base.py
class BaseDatabaseWrapper:
...
migration_recorder_class = None # defaults to MigrationRecorder
migration_executor_class = None # defaults to MigrationExecutor
}}}

And update the four files that instantiate these classes to respect
the hook:

- `django/db/migrations/executor.py` — `MigrationExecutor.__init__`
- `django/db/migrations/loader.py` — `MigrationLoader.build_graph`
- `django/core/management/commands/migrate.py` — `Command.handle`
- `django/core/management/commands/showmigrations.py` — `show_list`

With these two hooks, `django-clickhouse-backend` (and any future backend
with similar needs) can:

1. Set `migration_recorder_class = ClickHouseMigrationRecorder` on its
`DatabaseWrapper` — a proper subclass scoped to ClickHouse connections.
2. Set `migration_executor_class = ClickHouseMigrationExecutor` on its
`DatabaseWrapper` — a proper subclass that adds cluster logic without
touching `Migration.apply`.


== Why both hooks?

**`migration_recorder_class`** allows a backend to replace the
`django_migrations` tracking table with a backend-appropriate equivalent
(e.g. a ClickHouse `MergeTree` table instead of a standard Django model,
with backend-specific semantics for `record_applied`, `flush`, etc.).

**`migration_executor_class`** allows a backend to inject cluster-aware
logic *above* `Migration.apply` — for example, skipping a migration
operation that was already executed on a remote replica — without needing
to touch `Migration.apply` itself. This is exactly the scenario that
currently forces `django-clickhouse-backend` to patch `Migration.apply`
globally.

== Why `None` as default instead of pointing at the built-in classes?

Using `None` (resolved at call sites via `or MigrationRecorder`) avoids
a circular import: `base.py` is imported very early and importing
`MigrationRecorder` or `MigrationExecutor` there would pull in the
migrations module graph before it is needed. The `None` sentinel makes the
intent explicit and keeps the default behaviour identical to today.
--
Ticket URL: <https://code.djangoproject.com/ticket/37061>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#37061: Add migration_recorder_class and migration_executor_class hooks to
BaseDatabaseWrapper so third-party backends can customise migration
infrastructure without monkey-patching Django internals.

--------------------------------------+------------------------------------
Reporter: Laurent Tramoy | Owner: (none)
Type: Cleanup/optimization | Status: new
Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

--------------------------------------+------------------------------------
Changes (by Tim Graham):

* cc: Tim Graham (added)
* stage: Unreviewed => Accepted
* version: 6.0 => dev

Comment:

Without evaluating the proposal's specifics, Django is certainly open to
patches that make it more extensible for third-party database backends.
Along with your proposed patch for Django, you should provide a PR for
django-clickhouse-backend to demonstrate that the hooks in Django will be
useful.
--
Ticket URL: <https://code.djangoproject.com/ticket/37061#comment:1>

[Django]

#37061: Add migration_recorder_class and migration_executor_class hooks to
BaseDatabaseWrapper so third-party backends can customise migration
infrastructure without monkey-patching Django internals.

-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: Héctor
Type: | Castillo
Cleanup/optimization | Status: assigned

Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Héctor Castillo):

* owner: (none) => Héctor Castillo
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/37061#comment:2>

[Django]

#37061: Add migration_recorder_class and migration_executor_class hooks to
BaseDatabaseWrapper so third-party backends can customise migration
infrastructure without monkey-patching Django internals.
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: Héctor
Type: | Castillo
Cleanup/optimization | Status: assigned
Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Laurent Tramoy):

Replying to [comment:1 Tim Graham]:

> Without evaluating the proposal's specifics, Django is certainly open to
patches that make it more extensible for third-party database backends.
Along with your proposed patch for Django, you should provide a PR for
django-clickhouse-backend to demonstrate that the hooks in Django will be
useful.

Sorry I forgot to answer here. `clickhouse_connect`, the official
Clickhouse - python lib, now support migrations using `alembic` (+
SQLAlchemy tables), so we decided to go with that lib, to ensure a better
support.

I have another use-case that requires more extensible migration-related
classes, directly related to https://forum.djangoproject.com/t/very-slow-
migrations-with-large-numbers-of-tables/29038/7 : I think a customizable
`ProjectState` class would allow to test this kind of patch more easily,
and be able to override it in downstream projects, without relying on
monkeypatches. I'll try to work on a PR
--
Ticket URL: <https://code.djangoproject.com/ticket/37061#comment:3>