[Django] #35508: Add an --ignore-deps flag to squashmigrations

[Django]

#35508: Add an --ignore-deps flag to squashmigrations
-------------------------------------+-------------------------------------
Reporter: Shai | Owner: nobody
Berger |
Type: New | Status: new
feature |
Component: | Version: dev
Migrations | Keywords: squash
Severity: Normal | squashmigrations
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 1 | Needs tests: 1
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
In large, long-going projects -- the ones whose need for migration
squashing is the greatest -- apps have a tendency to develop dependencies
upon each other. This entails problems with migrations -- dependencies
could be circular, or an app could be removed from the project but the
migrations that hold dependencies on it still remain, and
{{{squashmigrations}}} faces constraints that makes it very hard to do
correctly. Current implementation often generates migrations that make no
sense, with dependencies on several migrations from the same app.

As a partial solution, let's add a {{{--ignore-deps}}} flag to
{{{squashmigrations}}}. When this flag is applied, the generated squashed
migration will

- Only depend on the dependencies of the first migration in the range
squashed
- Not include the operations that create or modify relation fields, if
these fields refer to models from other apps (deletion can be included)

The idea is that such a migration will be correct, but not complete.
Assuming the range of migrations includes the last migrations of the app,
it will then be possible to add the missing pieces by a regular
{{{makemigrations}}}; but the squashing will still include all the non-
ellidable {{{RunSQL}}} and {{{RunPython}}} operations -- the
{{{squashmigrations --ignore-deps}}} + {{{makemigrations}}} dance is
proposed as a safer alternative to "migration bankruptcy", the practice of
just removing all the migrations and stating afresh

On a side note: It may be safer to add an operation which enforces that it
is indeed called with the last migration of the app in range, and maybe
even automatically invokes the following {{{makemigrations}}}, but I think
we should trust developers with the lower-level building bricks, which may
be useful in other situations as well.

This follows from a [https://forum.djangoproject.com/t/idea-make-squash-
migrations-no-deps/23986 discussion on the forum].
--
Ticket URL: <https://code.djangoproject.com/ticket/35508>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35508: Add an --ignore-deps flag to squashmigrations
-------------------------------------+-------------------------------------

Reporter: Shai Berger | Owner: Clinton
| Christian
Type: New feature | Status: assigned
Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: squash | Triage Stage:
squashmigrations | Unreviewed
Has patch: 0 | Needs documentation: 1

Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Clinton Christian):

* owner: nobody => Clinton Christian
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/35508#comment:1>

[Django]

#35508: Add an --ignore-deps flag to squashmigrations
-------------------------------------+-------------------------------------
Reporter: Shai Berger | Owner: Clinton
| Christian
Type: New feature | Status: assigned
Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: squash | Triage Stage:
squashmigrations | Unreviewed
Has patch: 0 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Clinton Christian):

I'm working on this and will finalize a PR shortly.
--
Ticket URL: <https://code.djangoproject.com/ticket/35508#comment:2>

[Django]

#35508: Add an --ignore-deps flag to squashmigrations
-------------------------------------+-------------------------------------
Reporter: Shai Berger | Owner: Clinton
| Christian
Type: New feature | Status: assigned
Component: Migrations | Version: dev
Severity: Normal | Resolution:

Keywords: squash | Triage Stage: Accepted
squashmigrations |

Has patch: 0 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* cc: Mariusz Felisiak, David Wobrock, David Sanders, Andrew Godwin
(added)
* stage: Unreviewed => Accepted

Comment:

Hi Shai, thank you for raising this ticket
cc-ing a few people here in case they have further insights or opinions on
this
Tentatively accepting as the discussion seems positive and I can see the
value of the feature 👍
--
Ticket URL: <https://code.djangoproject.com/ticket/35508#comment:3>

[Django]

#35508: Add an --ignore-deps flag to squashmigrations
-------------------------------------+-------------------------------------
Reporter: Shai Berger | Owner: Clinton
| Christian
Type: New feature | Status: assigned
Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: squash | Triage Stage: Accepted
squashmigrations |

Has patch: 1 | Needs documentation: 1

Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Clinton Christian):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/35508#comment:4>

[Django]

#35508: Add an --ignore-deps flag to squashmigrations
-------------------------------------+-------------------------------------
Reporter: Shai Berger | Owner: Clinton
| Christian
Type: New feature | Status: assigned
Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: squash | Triage Stage: Accepted
squashmigrations |
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Clinton Christian):

Filtering out multi-app `FieldOperation`s is pretty straight forward, but
should we mutate a `ModelOperation` to exclude multi-app foreign key
references, or exclude the `ModelOperation` entirely?
--
Ticket URL: <https://code.djangoproject.com/ticket/35508#comment:5>

[Django]

#35508: Add an --ignore-deps flag to squashmigrations
-------------------------------------+-------------------------------------
Reporter: Shai Berger | Owner: Clinton
| Christian
Type: New feature | Status: assigned
Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: squash | Triage Stage: Accepted
squashmigrations |
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Shai Berger):

Replying to [comment:5 Clinton Christian]:

> Filtering out multi-app `FieldOperation`s is pretty straight forward,
but should we mutate a `ModelOperation` to exclude multi-app foreign key
references, or exclude the `ModelOperation` entirely?

Coming back to this after two years, with the response I put on Clinton's
[https://github.com/django/django/pull/18276 PR]:

Many {{{ModelOperation}}}s are, in fact, irrelevant, because they do not
operate on fields; so it is better to focus at the more-specific operation
level. and even then, it changes by case. In the case of
{{{CreateModel}}}, it is better to modify the operation, because it is
likely that migrations down the line will rely on the model existing --
even if they don't rely on the relation field. In the case of Index or
Constraint operations (some of which, by the way, do not inherit
{{{ModelOperation}}}), it is better to remove the whole operation, because
the constraint or index will be entirely incorrect if it is missing a
field.
--
Ticket URL: <https://code.djangoproject.com/ticket/35508#comment:6>

[Django]

#35508: Add an --ignore-deps flag to squashmigrations
-------------------------------------+-------------------------------------
Reporter: Shai Berger | Owner: Clinton
| Christian
Type: New feature | Status: assigned
Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: squash | Triage Stage: Accepted
squashmigrations |
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Shai Berger):

Thinking more about that deferral-of-relations issue:

I think the operations and parameters that are filtered out should be
recorded, as comments, in the migration file. I suspect in many cases,
later operations -- e.g. {{{RunPython}} -- will rely on the results of the
deferred operations, and if there's no record of their removal in the
migration file, the errors will be quite difficult to debug.

This requires some support in the migration-writing machinery.
--
Ticket URL: <https://code.djangoproject.com/ticket/35508#comment:7>

[Django]

#35508: Add an --ignore-deps flag to squashmigrations
-------------------------------------+-------------------------------------
Reporter: Shai Berger | Owner: Clinton
| Christian
Type: New feature | Status: assigned
Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: squash | Triage Stage: Accepted
squashmigrations |
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Shai Berger):

And one more related, but separate idea:

It may be better to optimize the squashed migration before filtering it
for external dependencies -- if a dependency is added and then removed, it
is better to have it "organically" deleted as an optimization, rather than
have just the addition removed as a "no-deps" effect.
--
Ticket URL: <https://code.djangoproject.com/ticket/35508#comment:8>