[Django] #36643: Migrate should not check for consistent history when faking migrations

[Django]

#36643: Migrate should not check for consistent history when faking migrations
-----------------------------------+--------------------------------------
Reporter: Alexandru Chirila | Type: Bug
Status: new | Component: Migrations
Version: 5.2 | Severity: Normal
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-----------------------------------+--------------------------------------
If for some reason or another you end up needing to fake apply a
migration, it will still run the `check_consistent_history`.

So if you ended up by accident (or on purpose somehow?) with a invalid
migration history, there is no way of fixing it except manually doing
things in the `django_migrations` table.

It seems like a scenario like this ought to be fixable using the "fake"
apply of migrations, but as far as I can tell there is no way of doing
that as the check will always fell and throw an error, so the fake
aplication can never be done.

{{{
.venv/lib/python3.12/site-packages/django/db/migrations/loader.py", line
327, in check_consistent_history
raise InconsistentMigrationHistory(
django.db.migrations.exceptions.InconsistentMigrationHistory: Migration
shortner.0001_initial is applied before its dependency charts.0001_initial
on database 'default'.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36643>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36643: Migrate should not check for consistent history when faking migrations
-----------------------------------+--------------------------------------

Reporter: Alexandru Chirila | Owner: (none)

Type: Bug | Status: new
Component: Migrations | Version: 5.2

Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by Andrew Williamson):

Hi, I'm struggling to think of the scenario where you'd fake a migration
over an inconsistent history, and it'd leave a consistent state that you'd
not need to fake the next time. I.e. it'd always be inconsistent, every
time. Can you provide an example of where this would help?
--
Ticket URL: <https://code.djangoproject.com/ticket/36643#comment:1>

[Django]

#36643: Migrate should not check for consistent history when faking migrations
-----------------------------------+--------------------------------------
Reporter: Alexandru Chirila | Owner: (none)
Type: Bug | Status: new
Component: Migrations | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by Alexandru Chirila):

Replying to [comment:1 Andrew Williamson]:

> Hi, I'm struggling to think of the scenario where you'd fake a migration
over an inconsistent history, and it'd leave a consistent state that you'd
not need to fake the next time. I.e. it'd always be inconsistent, every
time. Can you provide an example of where this would help?

Sure, here's what happend to me:

- I had some squashed migrations for some of the django apps in my
project.
- I had **incorrectly** verified that all instances have indeed been
updated to the squashed migrations
- Considering I thought all instances were updated, I removed the old
migrations from the code.
- When I updated the one instance that did NOT received the squashed
migrations before, the `manage.py migrate` command was run automatically
- One of the apps did not have any squashed migrations, so it remained
marked as applied and appeared updated (i.e. `shortner` in this case)
- The same app `shortner` also was depending on a another app `charts`
that has had squashed migration, and as far as Django is concerned it
looked like none of the migrations of applied for `charts` but all
migrations were applied to `shortner`. Because the when the migrate script
ran it marked all migrations as being unapplied for `charts`.
- The consistency check was seeing that migrations for `shortner` were
applied before `charts`. Even though both apps were up to date will all
the schema changes.

----

Beeing in this scenario, I thought that: "Hey no problem, the DB itself is
up to date. Django has this 'fake' apply migration utility just to bail
you out when you mess up massively as I did."

Well, apparently I could not be bailed out from this scenario by the "fake
apply" migration because of this "consistency check". So I ended up
messing around directly into django internal tables (which I would rather
always avoid) to fix my issue because the "migrate" command refused to do
anything at all.

----

Definetly this was a big user error on my part, combined with an
unfortunate set of circumstances. That said, I don't think that there is
any good reason to perform a consistency check when running the "fake
apply", since presumably you are in weird state anyway and you're using
the "fake apply" to fix something that was caused by some manual error.

If there is a good reason to run the check in this scenario as well, a
simple flag that allows you to disable it would be useful I think. I did
try "--skip-checks" but that doesn't work as far as I can tell.
--
Ticket URL: <https://code.djangoproject.com/ticket/36643#comment:2>

[Django]

#36643: Migrate should not check for consistent history when faking migrations
-----------------------------------+--------------------------------------
Reporter: Alexandru Chirila | Owner: (none)
Type: Bug | Status: new
Component: Migrations | Version: 5.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by Augusto):

Hi, good evening, i tried to replicate what you did, and i believe that i
came up with a simple solution, that can ignore the
{{{check_consistent_history}}}, and basically heres what i did: i searched
into the{{{django/core/management/commands/migrate.py}}} and
{{{django/core/management/commands/makemigrations.py}}} path, and i add
this command line here at the {{{add_arguments}}}(on both paths) method:


{{{
parser.add_argument(
'--bypass-consistency-check',
action='store_true',
dest='bypass_consistency_check',
help='Bypass the migration history consistency check. Use with
caution.',
)
}}}

and on the {{{handle}}} method at the
{{{django/core/management/commands/migrate.py}}} i changed this part:

(line: 124)
{{{
# Raise an error if any migrations are applied before their
# dependencies.
if not options['bypass_consistency_check']:
executor.loader.check_consistent_history(connection)
}}}


at the {{{django/core/management/commands/makemigrations.py}}} i edited
this line here, before the verification:

(line: 155)
{{{
if not options['bypass_consistency_check']:
for alias in sorted(aliases_to_check):
....
}}}

to test this, you can type on the terminal:
{{{
python makemigrations.py your_app --bypass-consistency-check
}}}

{{{
python manage.py migrate your_app your_migration_file --fake --bypass-
consistency-check
}}}

see if this works
--
Ticket URL: <https://code.djangoproject.com/ticket/36643#comment:3>

[Django]

#36643: Migrate should not check for consistent history when faking migrations
-----------------------------------+--------------------------------------
Reporter: Alexandru Chirila | Owner: (none)

Type: New feature | Status: closed
Component: Migrations | Version: 5.2
Severity: Normal | Resolution: wontfix

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------
Changes (by Natalia Bidart):

* easy: 1 => 0
* resolution: => wontfix
* status: new => closed
* type: Bug => New feature

Comment:

Hello! Thank you for taking the time to create this ticket. To me, this is
more of a feature request than a bug. The current behavior is intentional:
Django always checks for a consistent migration history to protect
projects from entering an invalid state, even when using `--fake`.

The need to bypass this check seems very specific to a niche recovery
scenario rather than something that applies broadly. Django aims to
provide robust, predictable behavior for common cases, so changing this
would fall outside that scope.

Given the above, I'll close the ticket accordingly. If you'd like to
pursue this as a potential feature, please review the
[https://docs.djangoproject.com/en/stable/internals/contributing/bugs-and-
features/#requesting-features feature request guidelines] for more
details.
--
Ticket URL: <https://code.djangoproject.com/ticket/36643#comment:4>

[Django]

#36643: Migrate should not check for consistent history when faking migrations
-----------------------------------+--------------------------------------
Reporter: Alexandru Chirila | Owner: (none)
Type: New feature | Status: closed
Component: Migrations | Version: 5.2
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by Natalia Bidart):

After some further research, it's worth mentioning that this seems
somewhat related to #28250, which focused on handling a specific edge case
(--fake-initial) where ignoring the consistency check may be considered
safe.
--
Ticket URL: <https://code.djangoproject.com/ticket/36643#comment:5>

[Django]

#36643: Migrate should not check for consistent history when faking migrations
-----------------------------------+--------------------------------------
Reporter: Alexandru Chirila | Owner: (none)
Type: New feature | Status: closed
Component: Migrations | Version: 5.2
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by Alexandru Chirila):

Replying to [comment:4 Natalia Bidart]:

> The current behavior is intentional: Django always checks for a
consistent migration history to protect projects from entering an invalid
state, even when using `--fake`.

I'm not sure I understand, if the consistency check is in place to prevent
**entering** an invalid state, why is it being run **before** the
migration runs? Wouldn't it make sense to run it **after** the migration
is run (but before commit)? Or alternatively it should check consistency
in a "dry run" mode, to compute the state of the dependency graph after
the the migrate runs. Either of these would cover my case as well, and
make the consistency check actually useful.

Because as it stands the consistency checks, does not in fact prevent
entering an invalid state. The only thing it achieves is blocking using
the migrate script if you are already in an invalid state.
--
Ticket URL: <https://code.djangoproject.com/ticket/36643#comment:6>

[Django]

#36643: Migrate should not check for consistent history when faking migrations
-----------------------------------+--------------------------------------
Reporter: Alexandru Chirila | Owner: (none)
Type: New feature | Status: closed
Component: Migrations | Version: 5.2
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by Simon Charette):

> Because as it stands the consistency checks, does not in fact prevent
entering an invalid state. The only thing it achieves is blocking using
the migrate script if you are already in an invalid state.

I believe things are this way because the `migrate` command, with and
without `--fake`, makes graph manipulation expecting the history graph to
be in a consistent state prior to making these changes and that the
operations it performs could venture into undefined behaviour territory
without these per-requisite.

To make a ''typing'' analogy, if a piece of code expect a graph to follow
specific schema (e.g. single head per app, acyclic) and you force it
through the pipe in an unexpected form then it can result in damageable
ways.

Given undefined behaviour is not something we want anywhere close to
database schema alterations I believe the current approach is reasonable.
--
Ticket URL: <https://code.djangoproject.com/ticket/36643#comment:7>

[Django]

#36643: Migrate should not check for consistent history when faking migrations
-----------------------------------+--------------------------------------
Reporter: Alexandru Chirila | Owner: (none)
Type: New feature | Status: closed
Component: Migrations | Version: 5.2
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by Alexandru Chirila):

While I don't agree with the conclusion here, I do understand where it's
coming from, so I'll leave it be.

-----

I'll leave here the hacky solution for this problem, for any unfortunate
soul that finds themselves in such a terrible mess that I've created for
myself.

* Check the status of migrations using `./manage.py showmigrations`, looks
for the ones that are applied and are confusing the Django consistency
check.
* Remove from the DB directly the applied migrations for the app(s) in
question:

{{{
delete from django_migrations where app='APPNAME';
}}}

* Fake apply migrations for the one just removed:

{{{
./manage.py migrate APPNAME --fake
}}}

* Fake apply any other migrations for the other apps that have been
squashed
--
Ticket URL: <https://code.djangoproject.com/ticket/36643#comment:8>

[Django]

#36643: Migrate should not check for consistent history when faking migrations
-----------------------------------+--------------------------------------
Reporter: Alexandru Chirila | Owner: (none)
Type: New feature | Status: closed
Component: Migrations | Version: 5.2
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------

Comment (by David Sanders):

I just want to chime in here that I think we ought to consider this
option.

Squashing migrations either through the command or manually using the
replacements process is notoriously unreliable (a patch was submitted
recently to address false circular errors and there's another issue I've
encountered that blocks this).

This leaves developers to brute force squash by simply deleting &
regenerating migrations. I was hoping that `--fake` would work but am
being blocked by `InconsistentMigrationHistory`.

The only option left was to wipe out the migrations table then fake. This
was easy for me to do because I have access but I just wonder whether this
would be a step too far for some folks.
--
Ticket URL: <https://code.djangoproject.com/ticket/36643#comment:9>