[Django] #35707: Squash migrations in django.contrib.auth for better performance and nicer output

[Django]

#35707: Squash migrations in django.contrib.auth for better performance and nicer
output
-------------------------------------+-------------------------------------
Reporter: Tim Abbott | Type:
| Cleanup/optimization
Status: new | Component:
| contrib.auth
Version: 5.0 | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Provisioning a fresh database for any project using Django's contrib.auth
system ends up spending about 0.3 seconds, and probably more importantly a
nontrivial amount of output on the screen running the 12 migrations in
django.contrib.auth in sequence. This can be optimized significantly by
squashing the migrations using the `manage.py squashmigrations` algorithm.

Sample output using `ts "%b %d %H:%M:%.S"` for timing output:

Aug 23 17:07:23.116481 Applying
auth.0002_alter_permission_name_max_length... OK
Aug 23 17:07:23.131759 Applying auth.0003_alter_user_email_max_length...
OK
Aug 23 17:07:23.149192 Applying auth.0004_alter_user_username_opts... OK
Aug 23 17:07:23.166695 Applying auth.0005_alter_user_last_login_null...
OK
Aug 23 17:07:23.169690 Applying auth.0006_require_contenttypes_0002...
OK
Aug 23 17:07:23.191648 Applying
auth.0007_alter_validators_add_error_messages... OK
Aug 23 17:07:23.207522 Applying
auth.0008_alter_user_username_max_length... OK
Aug 23 17:07:23.234117 Applying
auth.0009_alter_user_last_name_max_length... OK
Aug 23 17:07:23.332708 Applying auth.0010_alter_group_name_max_length...
OK
Aug 23 17:07:23.417119 Applying auth.0011_update_proxy_permissions... OK
Aug 23 17:07:23.433407 Applying
auth.0012_alter_user_first_name_max_length... OK

The django.contrib.auth.migrations tree hasn't see any new migrations
since migration `0012_alter_user_first_name_max_length.py` was added in
2020, so it seems fairly likely that this won't need to be done on a
recurring basis.
--
Ticket URL: <https://code.djangoproject.com/ticket/35707>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35707: Squash migrations in django.contrib.auth for better performance and nicer
output
-------------------------------------+-------------------------------------

Reporter: Tim Abbott | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: contrib.auth | Version: 5.0
Severity: Normal | Resolution:

Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jacob Walls):

Saving 0.3s from test runs is great, but it's worth noting this will
impose a breaking change on users who have explicit dependencies on these
migrations in their own migration files.
--
Ticket URL: <https://code.djangoproject.com/ticket/35707#comment:1>

[Django]

#35707: Squash migrations in django.contrib.auth for better performance and nicer
output
-------------------------------------+-------------------------------------
Reporter: Tim Abbott | Owner: (none)

Type: | Status: closed

Cleanup/optimization |
Component: contrib.auth | Version: 5.0

Severity: Normal | Resolution: wontfix

Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* resolution: => wontfix
* status: new => closed

Comment:

Here is a bench from my development machine.

Python 3.12.1
Apple M2 Pro
Django 5.2.a-0@cdcd604ef8f650533eff6bd63a517ebb4ffddf96
Postgres 15.4

**Current: 0.137s**
{{{
Running migrations:
Applying auth.0001_initial... OK (0.071s)
Applying auth.0002_alter_permission_name_max_length... OK (0.005s)
Applying auth.0003_alter_user_email_max_length... OK (0.003s)
Applying auth.0004_alter_user_username_opts... OK (0.003s)
Applying auth.0005_alter_user_last_login_null... OK (0.004s)
Applying auth.0006_require_contenttypes_0002... OK (0.001s)
Applying auth.0007_alter_validators_add_error_messages... OK (0.003s)
Applying auth.0008_alter_user_username_max_length... OK (0.010s)
Applying auth.0009_alter_user_last_name_max_length... OK (0.003s)
Applying auth.0010_alter_group_name_max_length... OK (0.004s)
Applying auth.0011_update_proxy_permissions... OK (0.026s)
Applying auth.0012_alter_user_first_name_max_length... OK (0.004s)
}}}

**Squashed: 0.093s**
{{{
Running migrations:
Applying
auth.0001_initial_squashed_0012_alter_user_first_name_max_length... OK
(0.093s)
}}}

----

That's a savings of about 1/3 of the migration time, or 0.044s on an
admittedly fast machine (so double it for cloud CI). Given that this would
impose a breaking change on users (and the verbosity can already be
silenced with `-v0`), I don't see the value outweighing the cost. We also
have guidance in the migrations docs about not worrying about the number
of migrations until you have
[https://docs.djangoproject.com/en/5.1/topics/migrations/#squashing-
migrations hundreds].
--
Ticket URL: <https://code.djangoproject.com/ticket/35707#comment:2>

[Django]

#35707: Squash migrations in django.contrib.auth for better performance and nicer
output
-------------------------------------+-------------------------------------
Reporter: Tim Abbott | Owner: (none)
Type: | Status: closed
Cleanup/optimization |
Component: contrib.auth | Version: 5.0
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Natalia Bidart):

Hello Tim, thank you for your ticket! I agree with the resolution from
this ticket and with the commentary from Jacob.

Squashing `auth` migrations without a proper deprecation path, and without
considering all the possible scenarios of usage by Django projects, will
likely be a breaking change and the gain does not feel worth producing
that breakage.

Because of the above, this report should be treated similarly to a "new
feature request" for Django. For cases like this, the recommended path
forward is to first propose and discuss the idea/request with the
community and gain consensus. To do that, please consider starting a new
conversation on the [https://forum.djangoproject.com/c/internals/5 Django
Forum], where you'll reach a wider audience and likely get extra feedback.

If the community agrees with the proposal, please return to this ticket
and reference the forum discussion so we can re-open it. For more
information, please refer to
[https://docs.djangoproject.com/en/stable/internals/contributing/bugs-and-
features/#requesting-features the documented guidelines for requesting
features].
--
Ticket URL: <https://code.djangoproject.com/ticket/35707#comment:3>

[Django]

#35707: Squash migrations in django.contrib.auth for better performance and nicer
output
-------------------------------------+-------------------------------------
Reporter: Tim Abbott | Owner: (none)
Type: | Status: closed
Cleanup/optimization |
Component: contrib.auth | Version: 5.0
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Abbott):

Thanks for investigating!

> this will impose a breaking change on users who have explicit
dependencies on these migrations in their own migration files

Hmm, I wasn't aware of any breaking change concern with squashing
migrations, as long as one doesn't delete the original migrations when
doing the squashing. I'll see if I can find time to start a proper Django
forum discussion.
--
Ticket URL: <https://code.djangoproject.com/ticket/35707#comment:4>

[Django]

#35707: Squash migrations in django.contrib.auth for better performance and nicer
output
-------------------------------------+-------------------------------------
Reporter: Tim Abbott | Owner: (none)
Type: | Status: closed
Cleanup/optimization |
Component: contrib.auth | Version: 5.0
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jacob Walls):

> as long as one doesn't delete the original migrations when doing the
squashing.

Sorry, I took that to be part of the point of the part of the proposal,
but it looks like that was an assumption on my part.
--
Ticket URL: <https://code.djangoproject.com/ticket/35707#comment:5>

[Django]

#35707: Squash migrations in django.contrib.auth for better performance and nicer
output
-------------------------------------+-------------------------------------
Reporter: Tim Abbott | Owner: (none)
Type: | Status: closed
Cleanup/optimization |
Component: contrib.auth | Version: 5.0
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Natalia Bidart):

Replying to [comment:5 Jacob Walls]:

> > as long as one doesn't delete the original migrations when doing the
squashing.
>

> Sorry, I took that to be part of the point of the proposal, but it looks

like that was an assumption on my part.

I did assume that the original migrations would not be deleted. But, my
thinking when mentioning breakages was more related to: when new `auth`
migrations need to be added, which migration would these new files depend
on? the latest non-squashed? the squashed one? As far as I understand
either answer breaks "the other case" (our docs are not that explicit
about a case similar to this).

Because of the impact that this can (potentially) have, I think that, if
we want to consider re-opening, we need a forum discussion with some
general agreement that this is both:
* desired, and
* does not break current or future Django projects.
--
Ticket URL: <https://code.djangoproject.com/ticket/35707#comment:6>

[Django]

#35707: Squash migrations in django.contrib.auth for better performance and nicer
output
-------------------------------------+-------------------------------------
Reporter: Tim Abbott | Owner: (none)
Type: | Status: closed
Cleanup/optimization |
Component: contrib.auth | Version: 5.0
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Klaas van Schelven):

My 2c:

I agree with the usefulness of this (speed, cleanup).

In principle, I would say this is _not_ (or should not be) a breaking
change _at all_. Given how squashmigrations work, the replacing migration
"should just take over when it can". Notes on some questions:

* Indeed, the original migrations should not be deleted. In the general
case Django recommends "Once you’ve squashed [..] distribute this change
to all running instances of your application, making sure that they run
migrate to store the change in their database." but this is infeasible and
undesirable in the general case.
* When new auth migrations after the squashed migration are created, they
should depend on the new migration, because in the general case it has the
longest life-expectancy. In my experience Django does this correctly, but
I didn't see this documented. Other than the life-expectancy, it shouldn't
matter in principle: one migration is declared to be a valid replacement
of a set of others, so they should be interchangeable.

In general, I'd like to point out that any doubts raised in the above are
really doubts about Django's squashmigration machinery, which (though
there are surely bugs) in principle works well, and should equally work
well for Django's own migrations.
--
Ticket URL: <https://code.djangoproject.com/ticket/35707#comment:7>

[Django]

#35707: Squash migrations in django.contrib.auth for better performance and nicer
output
-------------------------------------+-------------------------------------
Reporter: Tim Abbott | Owner: (none)
Type: | Status: closed
Cleanup/optimization |
Component: contrib.auth | Version: 5.0
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Klaas van Schelven):

Oh and I've actually tested the above assertion regarding "what happens
with migration 0013?" i.e. when experimenting with this myself I found
that Django depends on the replacing migration, and that databases through
either path (straight through the replacing migration, or the longer
original path) will migrate to the new endpoint as promised.
--
Ticket URL: <https://code.djangoproject.com/ticket/35707#comment:8>

[Django]

#35707: Squash migrations in django.contrib.auth for better performance and nicer
output
-------------------------------------+-------------------------------------
Reporter: Tim Abbott | Owner: (none)
Type: | Status: closed
Cleanup/optimization |
Component: contrib.auth | Version: 5.0
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Klaas van Schelven):

[https://github.com/bugsink/squashed-users Available as a package] now
--
Ticket URL: <https://code.djangoproject.com/ticket/35707#comment:9>