[Django] #24628: Cannot apply migration that depends on squashed migration
Django
----------------------------+--------------------
Reporter: dwt | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------+--------------------
After some time in the chat with MarkusH, we decided that this should be a
bug report.
Some context: I have quite some migrations (33) of which the first already
was a squash of several migrations. It did get it's 'replaced' field
removed though, as the original migrations it replaced are long gone
already. The goal now was to get that long list of migrations down to one
again, for peace of mind.
This led to the following symptoms: When I created the squashed migration
everything seemed fine. ./manage.py migrate did say that nothing was to be
done (but then again, all the migrations to be squashed where already
applied).
But adding another migration after that ended in tears - ./manage migrate
didn't want to apply it and told me so in no uncertain terms.
{{{
(pycess)dwt@atlan ~/Code/Projekte/pycess/pycess (git)-[master] %
./manage.py migrate
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File
"/Users/dwt/Code/Projekte/pycess/django/django/core/management/__init__.py",
line 330, in execute_from_command_line
utility.execute()
File
"/Users/dwt/Code/Projekte/pycess/django/django/core/management/__init__.py",
line 322, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File
"/Users/dwt/Code/Projekte/pycess/django/django/core/management/base.py",
line 347, in run_from_argv
self.execute(*args, **cmd_options)
File
"/Users/dwt/Code/Projekte/pycess/django/django/core/management/base.py",
line 398, in execute
output = self.handle(*args, **options)
File
"/Users/dwt/Code/Projekte/pycess/django/django/core/management/commands/migrate.py",
line 86, in handle
executor = MigrationExecutor(connection,
self.migration_progress_callback)
File
"/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/executor.py",
line 19, in __init__
self.loader = MigrationLoader(self.connection)
File
"/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py",
line 47, in __init__
self.build_graph()
File
"/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py",
line 281, in build_graph
_reraise_missing_dependency(migration, parent, e)
File
"/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py",
line 264, in _reraise_missing_dependency
raise exc
File
"/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py",
line 274, in build_graph
self.graph.add_dependency(migration, key, parent)
File
"/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/graph.py",
line 124, in add_dependency
parent
django.db.migrations.graph.NodeNotFoundError: Migration
process.0002_auto_20150411_1005 dependencies reference nonexistent parent
node ('process', '0001_squashed_initial_2')
}}}
Here we had quite some discussion in #django-dev with MarkusH, of which
the result to me was that a) django doesn't seem to ever record that a
squashed migration is applied, at least as long as it is recognizable by
django as a squashed migration (i.e. it has a .replaced property). b)
there seems to be no way to tell django that the replacing squashed
migration really is already applied for this database.
The last one turned out to be achievable if you remove the old migrations
and the .replaces property from the squashed migration and then apply it
with --fake
MarkusH might want to say more here that he can describe better.
For ease of reproduction I'm attaching the project where this occurred for
me.
--
Ticket URL: <https://code.djangoproject.com/ticket/24628>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: dwt | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: master
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by dwt):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
Since the upload limit was too small, I've put it on dropbox:
https://dl.dropboxusercontent.com/u/765996/pycess.small.zip
--
Ticket URL: <https://code.djangoproject.com/ticket/24628#comment:1>
Django
Reporter: dwt | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by MarkusH):
* version: master => 1.8
* stage: Unreviewed => Accepted
Comment:
The behavior from the second part of that ticket (once the replaced
migrations are removed and the `replaces` section is gone) sounds to me
like a regression in 1.8: in 1.7 the squashed migration would
automatically be fake applied due to 1.7's automatic behavior. This
changed in 1.8 and at least requires a documentation update!
--
Ticket URL: <https://code.djangoproject.com/ticket/24628#comment:2>
Django
are
----------------------------+------------------------------------
Reporter: dwt | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Old description:
New description:
(I am attempting a shorter and clearer description of this bug, but
leaving the original description intact below - carljm).
In Django 1.8, consider an app `A` with migrations `1` and `2` and a
squashed migration `1_squashed_2` that replaces both `1` and `2`. Consider
a deployment of this app which has only `1` applied. When it receives the
update including `2` and `1_squashed_2` and is migrated, `2` is marked as
applied in the database but `1_squashed_2` is not.
This does not immediately appear to be a problem, because as long as
migrations `1` and `2` exist and `1_squashed_2` is marked as replacing
them, Django automatically considers `1_squashed_2` to be applied (and
shows it as such in `showmigrations`). But Django never actually records
`1_squashed_2` itself as applied in the database.
At some point, once all deployments have migrated through `2`, the idea is
that `1` and `2` can be removed, and the `replaces` tag removed from
`1_squashed_2`. At this point we have a problem, because now Django
considers `1_squashed_2` to not be applied, and tries to apply it again,
causing errors because tables already exist, etc. The only solution is to
manually `--fake` squashed migrations on all deployments when their
`replaces` tag is removed, which is a pain and eliminates much of the
convenience of the migration system; `--fake` should not be necessary in
the normal course of things.
The solution appears simple: anytime a migration is marked as having been
applied, Django should check if it is part of a replaced set, and if that
replaced set is now fully applied, the replacing migration should also be
marked as applied.
It may be that this check should be performed by `migrate` even when it
hasn't applied any migrations in the current run, as that would help to
correct the corruption already caused by this bug in many existing
databases, but not yet noticed because the replaced migrations haven't yet
been removed. (This correction should probably be mentioned to the user so
it doesn't happen silently.)
Original report follows:
--
Comment (by carljm):
I just ran into this as well. I think it is definitely a bug and should be
fixed in code, not just in documentation.
--
Ticket URL: <https://code.djangoproject.com/ticket/24628#comment:3>
Django
are
----------------------------+------------------------------------
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by carljm):
* has_patch: 0 => 1
Comment:
PR: https://github.com/django/django/pull/4738
--
Ticket URL: <https://code.djangoproject.com/ticket/24628#comment:4>
Django
are
Reporter: dwt | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/24628#comment:5>
Django
are
----------------------------+---------------------------------------------
Type: Bug | Status: closed
Component: Migrations | Version: 1.8
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carl Meyer <carl@…>):
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"492537ac18df56c36f8a2335d773aa0fa01a61a3" 492537a]:
{{{
#!CommitTicketReference repository=""
revision="492537ac18df56c36f8a2335d773aa0fa01a61a3"
Fixed #24628 -- Fixed applied status for squashed migrations.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24628#comment:6>
Django
are
----------------------------+---------------------------------------------
Type: Bug | Status: closed
Component: Migrations | Version: 1.8
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carl Meyer <carl@…>):
In [changeset:"efdcd13c340cee944531747c24be890af1e46710" efdcd13]:
{{{
#!CommitTicketReference repository=""
revision="efdcd13c340cee944531747c24be890af1e46710"
[1.8.x] Fixed #24628 -- Fixed applied status for squashed migrations.
Backport of 492537ac18df56c36f8a2335d773aa0fa01a61a3 from master.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24628#comment:7>
Django
are
----------------------------+---------------------------------------------
Type: Bug | Status: closed
Component: Migrations | Version: 1.8
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carl Meyer <carl@…>):
In [changeset:"5c085ea7b3f3ff10389aeed327e018581791876a" 5c085ea7]:
{{{
#!CommitTicketReference repository=""
revision="5c085ea7b3f3ff10389aeed327e018581791876a"
Refs #24628 -- Added a second test and a docstring comment to avoid
regression.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24628#comment:8>
Django
are
----------------------------+---------------------------------------------
Type: Bug | Status: closed
Component: Migrations | Version: 1.8
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carl Meyer <carl@…>):
In [changeset:"feed5ad2a07223d14f8304920fa761c972559b07" feed5ad]:
{{{
#!CommitTicketReference repository=""
revision="feed5ad2a07223d14f8304920fa761c972559b07"
[1.8.x] Refs #24628 -- Added a second test and a docstring comment to
avoid regression.
Backport of 5c085ea7b3f3ff10389aeed327e018581791876a from master.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24628#comment:9>