[Django] #35595: Generated migration for removing related models can't migrate backwards

45 views
Skip to first unread message

Django

unread,
Jul 11, 2024, 5:44:22 PM7/11/24
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-------------------------------------+-------------------------------------
Reporter: Timothy Schilling | Type:
| Uncategorized
Status: new | Component:
| Migrations
Version: 5.0 | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
The migration that's created when removing two related models, results in
a migration that can't be migrated backwards. However, this is only the
case when the model with the foreign key has a `Meta.indexes` specified
that references the foreign key field.

Given the following models:

{{{

class Animal(models.Model):
name = models.CharField(max_length=100)


class Dog(models.Model):
name = models.CharField(max_length=100)
animal = models.ForeignKey(Animal, on_delete=models.CASCADE)

class Meta:
indexes = [models.Index(fields=("animal", "name"))]
}}}


Doing the following results in the error at the end.

1. `python manage.py makemigrations [app]` -> 0001_initial.py (in my test
it was the second migration, in case you can't reproduce it)
2. Remove both models
3. `python manage.py makemigrations [app]` -> 0002_remove.py
4. `python manage.py sqlmigrate [app] 0002 --backwards`

Error:


{{{
❯ ./manage.py sqlmigrate endpoint 0003 --backwards
Traceback (most recent call last):
File "site-packages/django/db/models/options.py", line 681, in get_field
return self.fields_map[field_name]
~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'animal'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/schillingt/Projects/aspiredu/./manage.py", line 11, in
<module>
execute_from_command_line(sys.argv)
File "site-packages/django/core/management/__init__.py", line 442, in
execute_from_command_line
utility.execute()
File "site-packages/django/core/management/__init__.py", line 436, in
execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "site-packages/django/core/management/base.py", line 413, in
run_from_argv
self.execute(*args, **cmd_options)
File "site-packages/django/core/management/commands/sqlmigrate.py", line
38, in execute
return super().execute(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "site-packages/django/core/management/base.py", line 459, in
execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "site-packages/django/core/management/commands/sqlmigrate.py", line
80, in handle
sql_statements = loader.collect_sql(plan)
^^^^^^^^^^^^^^^^^^^^^^^^
File "site-packages/django/db/migrations/loader.py", line 383, in
collect_sql
state = migration.unapply(state, schema_editor, collect_sql=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "site-packages/django/db/migrations/migration.py", line 193, in
unapply
operation.database_backwards(
File "site-packages/django/db/migrations/operations/models.py", line
394, in database_backwards
schema_editor.create_model(model)
File "site-packages/django/db/backends/base/schema.py", line 513, in
create_model
self.deferred_sql.extend(self._model_indexes_sql(model))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "site-packages/django/db/backends/base/schema.py", line 1624, in
_model_indexes_sql
output.append(index.create_sql(model, self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "site-packages/django/db/models/indexes.py", line 112, in
create_sql
model._meta.get_field(field_name)
File "site-packages/django/db/models/options.py", line 683, in get_field
raise FieldDoesNotExist(
django.core.exceptions.FieldDoesNotExist: Dog has no field named 'animal'

}}}

The generated migrations for me are:

0001:

{{{
operations = [
migrations.CreateModel(
name="Animal",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
("name", models.CharField(max_length=100)),
],
),
migrations.CreateModel(
name="Dog",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
("name", models.CharField(max_length=100)),
(
"animal",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
to="[app].animal",
),
),
],
options={
"indexes": [
models.Index(
fields=["animal", "name"],
name="[app]_do_animal__b82a2f_idx"
)
],
},
),
]
}}}

0002:

{{{
operations = [
migrations.RemoveField(
model_name="dog",
name="animal",
),
migrations.DeleteModel(
name="Animal",
),
migrations.DeleteModel(
name="Dog",
),
]
}}}



My environment is:
- Django 5.0.6
- Postgres 15
- python 3.12.4
- M1 mac

Removing the `Meta.indexes` results in a generated migration that can be
migrated backwards.

This appears by solveable in at least two ways:

1. Adding the following operation to the top of the removal migration
solves the problem.


{{{
migrations.RemoveIndex(
model_name="dog",
name="[app]_do_animal__b82a2f_idx",
),
}}}

2. Removing the `RemoveField` operation from the removal migration and
rearranging the models.


{{{
operations = [
migrations.DeleteModel(
name="Dog",
),
migrations.DeleteModel(
name="Animal",
),
]
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35595>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,
Jul 12, 2024, 1:50:42 AM7/12/24
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-----------------------------------+------------------------------------
Reporter: Timothy Schilling | Owner: (none)
Type: Bug | Status: new
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------
Changes (by Sarah Boyce):

* stage: Unreviewed => Accepted
* type: Uncategorized => Bug

Comment:

Thank you for the report! Replicated on main 👍
--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:1>

Django

unread,
Jul 14, 2024, 11:19:59 AM7/14/24
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-------------------------------------+-------------------------------------
Reporter: Timothy Schilling | Owner: Akash
| Kumar Sen
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Akash Kumar Sen):

* owner: (none) => Akash Kumar Sen
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:2>

Django

unread,
Mar 16, 2025, 4:35:54 AM3/16/25
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-----------------------------------+------------------------------------
Reporter: Timothy Schilling | Owner: wookkl
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------
Changes (by wookkl):

* owner: Akash Kumar Sen => wookkl

--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:3>

Django

unread,
Apr 8, 2025, 12:08:14 PM4/8/25
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-----------------------------------+------------------------------------
Reporter: Timothy Schilling | Owner: wookkl
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------
Changes (by wookkl):

* has_patch: 0 => 1

Comment:

PR [https://github.com/django/django/pull/19361]
--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:4>

Django

unread,
Apr 8, 2025, 1:18:19 PM4/8/25
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-----------------------------------+------------------------------------
Reporter: Timothy Schilling | Owner: wookkl
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------
Changes (by Simon Charette):

* needs_better_patch: 0 => 1

Comment:

I think the actual issue is less about the migration `sqlmigrate` crashing
but more about the auto-detector generating an invalid migration.

The resulting migration operations of removing all models should include a
`RemoveIndex` operation that precedes the `RemoveField`

{{{#!python
operations = [
migrations.RemoveIndex(
model_name="dog",
name="animal",
),
migrations.RemoveField(
model_name="dog",
name="animal",
),
migrations.DeleteModel(
name="Animal",
),
migrations.DeleteModel(
name="Dog",
),
]
}}}

This is a test that should pass

{{{#!diff
diff --git a/tests/migrations/test_autodetector.py
b/tests/migrations/test_autodetector.py
index ac725d317e..30085e5c9b 100644
--- a/tests/migrations/test_autodetector.py
+++ b/tests/migrations/test_autodetector.py
@@ -2798,6 +2798,36 @@ def test_remove_indexes(self):
changes, "otherapp", 0, 0, model_name="book",
name="book_title_author_idx"
)

+ def test_remove_field_and_index(self):
+ before_state = [
+ ModelState("testapp", "Animal", []),
+ ModelState(
+ "testapp",
+ "Dog",
+ fields=[
+ ("name", models.CharField(max_length=100)),
+ (
+ "animal",
+ models.ForeignKey("testapp.Animal",
on_delete=models.CASCADE),
+ ),
+ ],
+ options={
+ "indexes": [
+ models.Index(fields=("animal", "name"),
name="animal_name_idx")
+ ]
+ },
+ ),
+ ]
+ changes = self.get_changes(before_state, [])
+ # Right number/type of migrations?
+ self.assertNumberMigrations(changes, "testapp", 1)
+ self.assertOperationTypes(
+ changes,
+ "testapp",
+ 0,
+ ["RemoveIndex", "RemoveField", "DeleteModel", "DeleteModel"],
+ )
+
def test_rename_indexes(self):
book_renamed_indexes = ModelState(
"otherapp",
}}}

and the likely culprit lies in
`MigrationAutodetector.generate_deleted_models` which should add a
`RemoveIndex` operations for each index like we do for `unique_together`
being turned into `None`. We'll likely need to add a new
`OperationDependency.Type.REMOVE_INDEX_OR_CONSTRAINT` so `RemoveField`
have their `dependencies` are always sorted after the newly introduced
`RemoveIndex`.

Small demonstration of how
[https://github.com/django/django/compare/main...charettes:django:ticket-35595
this can be done].

FWIW the same issue will happens for `constraints` so we should add a test
for that as well.
--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:5>

Django

unread,
Apr 8, 2025, 1:47:23 PM4/8/25
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-----------------------------------+------------------------------------
Reporter: Timothy Schilling | Owner: wookkl
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------
Comment (by Simon Charette):

This is the twin ticket of #35962 for `indexes` instead of `constraints`.
--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:6>

Django

unread,
Apr 8, 2025, 9:49:30 PM4/8/25
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-----------------------------------+------------------------------------
Reporter: Timothy Schilling | Owner: wookkl
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------
Comment (by wookkl):

Thank you for the review. I'll try to resolve it based on the small
demonstration!
--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:7>

Django

unread,
Apr 15, 2025, 12:23:46 PM4/15/25
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-----------------------------------+------------------------------------
Reporter: Timothy Schilling | Owner: wookkl
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------
Changes (by wookkl):

* needs_better_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:8>

Django

unread,
Apr 21, 2025, 1:26:05 PM4/21/25
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-----------------------------------+------------------------------------
Reporter: Timothy Schilling | Owner: wookkl
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------
Comment (by Simon Charette):

Patch LGTM and solves #35962 at the same time.
--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:9>

Django

unread,
May 9, 2025, 10:03:24 AM5/9/25
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-----------------------------------+------------------------------------
Reporter: Timothy Schilling | Owner: wookkl
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------
Changes (by Sarah Boyce):

* needs_tests: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:10>

Django

unread,
Jun 17, 2025, 1:09:37 PM6/17/25
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-----------------------------------+------------------------------------
Reporter: Timothy Schilling | Owner: wookkl
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------
Changes (by wookkl):

* needs_tests: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:11>

Django

unread,
Jun 18, 2025, 10:02:01 AM6/18/25
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-----------------------------------+------------------------------------
Reporter: Timothy Schilling | Owner: wookkl
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+------------------------------------
Changes (by Sarah Boyce):

* needs_tests: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:12>

Django

unread,
Jun 27, 2025, 6:07:30 AM6/27/25
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-------------------------------------+-------------------------------------
Reporter: Timothy Schilling | Owner: wookkl
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Sarah Boyce):

* needs_tests: 1 => 0
* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:13>

Django

unread,
Jun 27, 2025, 11:17:57 AM6/27/25
to django-...@googlegroups.com
#35595: Generated migration for removing related models can't migrate backwards
-------------------------------------+-------------------------------------
Reporter: Timothy Schilling | Owner: wookkl
Type: Bug | Status: closed
Component: Migrations | Version: 5.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Sarah Boyce <42296566+sarahboyce@…>):

* resolution: => fixed
* status: assigned => closed

Comment:

In [changeset:"29f5e1e97d660855d34c1ee5edbd5cfe094b6ca7" 29f5e1e9]:
{{{#!CommitTicketReference repository=""
revision="29f5e1e97d660855d34c1ee5edbd5cfe094b6ca7"
Fixed #35595, #35962 -- Removed indexes and constraints before fields in
migrations.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35595#comment:14>
Reply all
Reply to author
Forward
0 new messages