[Django] #35424: Migration autodetector fails when order_with_respect_to is removed, but an _order field remains

[Django]

#35424: Migration autodetector fails when order_with_respect_to is removed, but an
_order field remains
--------------------------------------+------------------------
Reporter: stuarta0 | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 4.2
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 1
UI/UX: 0 |
--------------------------------------+------------------------
We have a project with a large number of migrations that began failing
during migrate/makemigrations due to order_with_respect_to having been
removed, and an explicit _order field added. The historic migration
contains this operation:

{{{
migrations.AlterOrderWithRespectTo(
name='choice',
order_with_respect_to=None,
),
}}}

This causes an issue with the code introduced in commit
https://github.com/django/django/commit/aa4acc164d1247c0de515c959f7b09648b57dc42
which was subsequently "fixed" in
https://code.djangoproject.com/ticket/33449 but remains failing in the
case described above.

A minimal reproducible case has been created here:
https://github.com/stuarta0/django-order-with-respect-to based on the
django docs polls app.

A possible solution may be to fall back to field_name if
order_with_respect_to is either missing, or present with None (which is
what occurs above):

{{{
def get_field(self, field_name):
if field_name == "_order":
field_name = self.options.get("order_with_respect_to") or
field_name
return self.fields[field_name]
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35424>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35424: Migration autodetector fails when order_with_respect_to is removed, but an
_order field remains

------------------------------------+------------------------------------
Reporter: Stuart Attenborrow | Owner: nobody

Type: Bug | Status: new
Component: Migrations | Version: 4.2

Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0

------------------------------------+------------------------------------
Changes (by Sarah Boyce):

* stage: Unreviewed => Accepted

Comment:

Thank you for the report, managed to replicate on main
Given an existing migrated model
{{{#!python
class Foo(models.Model):
order = models.IntegerField(default=999999)

class Meta:
order_with_respect_to = "order"
}}}
when you change that model to
{{{#!python
class Foo(models.Model):
_order = models.IntegerField(default=999999)

class Meta:
ordering = ("_order",)
}}}
that generates a migration like
{{{#!python
# Generated by Django 5.1 on 2024-05-03 12:02

from django.db import migrations


class Migration(migrations.Migration):

dependencies = [
('app3', '0004_foo'),
]

operations = [
migrations.AlterModelOptions(
name='foo',
options={'ordering': ('_order',)},
),
migrations.RenameField(
model_name='foo',
old_name='order',
new_name='_order',
),
migrations.AlterOrderWithRespectTo(
name='foo',
order_with_respect_to=None,
),
]
}}}
Which fails with `django.core.exceptions.FieldDoesNotExist: Foo has no
field named 'order'`
If you move `AlterOrderWithRespectTo` to be the first operation, this
migration can be applied.
--
Ticket URL: <https://code.djangoproject.com/ticket/35424#comment:1>

[Django]

#35424: Migration autodetector fails when order_with_respect_to is removed, but an
_order field remains

------------------------------------+--------------------------------------

Reporter: Stuart Attenborrow | Owner: nobody

Type: Bug | Status: closed
Component: Migrations | Version: 4.2
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0

------------------------------------+--------------------------------------
Changes (by Sarah Boyce):

* resolution: => needsinfo
* stage: Accepted => Unreviewed
* status: new => closed

--
Ticket URL: <https://code.djangoproject.com/ticket/35424#comment:2>

[Django]

#35424: Migration autodetector fails when order_with_respect_to is removed, but an
_order field remains
------------------------------------+--------------------------------------
Reporter: Stuart Attenborrow | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 4.2
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+--------------------------------------

Comment (by Stuart Attenborrow):

Thanks for looking into it. Due to the error you're seeing, the migrations
had to be staggered to achieve the changes. e.g.:

- [https://github.com/stuarta0/django-order-with-respect-
to/blob/main/mysite/polls/migrations/0002_alter_choice_order_with_respect_to.py
0002_alter_choice_order_with_respect_to]: arranging the test.
- [https://github.com/stuarta0/django-order-with-respect-
to/blob/main/mysite/polls/migrations/0003_choice_order.py
0003_choice_order]: we wanted to move to our own ordering, but retain the
data from the existing hidden field. Removing order_with_respect_to
would've lost the data, so we added a placeholder field to store the
result, and copied the values across with a simple python migration.
- [https://github.com/stuarta0/django-order-with-respect-
to/blob/main/mysite/polls/migrations/0004_alter_choice_options_and_more.py
0004_alter_choice_options_and_more]: the order_with_respect_to field can
be removed, which drops the hidden column.
- [https://github.com/stuarta0/django-order-with-respect-
to/blob/main/mysite/polls/migrations/0005_alter_choice_options_rename_order_choice__order.py
0005_alter_choice_options_rename_order_choice__order]: finally, we can
rename our placeholder field.

Since the code is 8 years old and written by another author, I can't give
you an explicit reason why the "_order" name was reused other than my
assumption that "order" is a reserved word and "_order" was good enough
for django internals at the time. However, it hasn't been an issue because
django never had code that was structured in the way it is now.

The answer for new projects is probably to remake or squash migrations to
avoid it, but it's quite difficult on a legacy project with this scale of
migrations. My suggested change in the original report fixes the issue for
this case, but I haven't run django tests to see if it fails in other
cases.
--
Ticket URL: <https://code.djangoproject.com/ticket/35424#comment:3>

[Django]

#35424: Migration autodetector fails when order_with_respect_to is removed, but an
_order field remains
------------------------------------+--------------------------------------
Reporter: Stuart Attenborrow | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 4.2

Severity: Normal | Resolution: worksforme

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+--------------------------------------

Changes (by Sarah Boyce):

* resolution: needsinfo => worksforme

Comment:

Hi Stuart,
I have cloned your test repo installed Django 4.2, ran migrations and they
all apply without an error for me 🤔
I think you also need to share the error that you are seeing, I can't
determine that Django is at fault here.
--
Ticket URL: <https://code.djangoproject.com/ticket/35424#comment:4>

[Django]

#35424: Migration autodetector fails when order_with_respect_to is removed, but an
_order field remains
------------------------------------+--------------------------------------
Reporter: Stuart Attenborrow | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 4.2
Severity: Normal | Resolution: worksforme
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+--------------------------------------

Changes (by Stuart Attenborrow):

* Attachment "django-order-with-respect-to.zip" added.

--
Ticket URL: <https://code.djangoproject.com/ticket/35424>

[Django]

#35424: Migration autodetector fails when order_with_respect_to is removed, but an
_order field remains
------------------------------------+--------------------------------------
Reporter: Stuart Attenborrow | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 4.2
Severity: Normal | Resolution: worksforme
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+--------------------------------------

Comment (by Stuart Attenborrow):

Error shown below. I've added a zip that contains the same project, but
with docker compose, and a poetry lock file to ensure we're looking at the
same thing. Note I can reproduce this both in a docker container, and on a
production EC2 instance using system python and virtual envs.

{{{
/code/mysite# python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, contenttypes, polls, sessions
Running migrations:
No migrations to apply.
Traceback (most recent call last):
File "/code/mysite/manage.py", line 22, in <module>
main()
File "/code/mysite/manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/opt/venv/lib/python3.11/site-
packages/django/core/management/__init__.py", line 442, in
execute_from_command_line
utility.execute()
File "/opt/venv/lib/python3.11/site-
packages/django/core/management/__init__.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/venv/lib/python3.11/site-
packages/django/core/management/base.py", line 412, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/venv/lib/python3.11/site-
packages/django/core/management/base.py", line 458, in execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-
packages/django/core/management/base.py", line 106, in wrapper
res = handle_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-
packages/django/core/management/commands/migrate.py", line 335, in handle
changes = autodetector.changes(graph=executor.loader.graph)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-
packages/django/db/migrations/autodetector.py", line 46, in changes
changes = self._detect_changes(convert_apps, graph)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-
packages/django/db/migrations/autodetector.py", line 197, in
_detect_changes
self.generate_altered_fields()
File "/opt/venv/lib/python3.11/site-
packages/django/db/migrations/autodetector.py", line 1117, in
generate_altered_fields
old_field = self.from_state.models[app_label,
old_model_name].get_field(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-
packages/django/db/migrations/state.py", line 765, in get_field
return self.fields[field_name]
~~~~~~~~~~~^^^^^^^^^^^^
KeyError: None
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35424#comment:5>

[Django]

#35424: Migration autodetector fails when order_with_respect_to is removed, but an
_order field remains
------------------------------------+--------------------------------------
Reporter: Stuart Attenborrow | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 4.2
Severity: Normal | Resolution: worksforme

Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
------------------------------------+--------------------------------------
Changes (by Sarah Boyce):

* easy: 1 => 0

* stage: Unreviewed => Accepted

Comment:

Hi thank you Stuart for the extra context, I'm able to replicate your
error too.
Note for others, it can "migrate" without error on the first run but then
error on future calls to migrate.
--
Ticket URL: <https://code.djangoproject.com/ticket/35424#comment:6>

[Django]

#35424: Migration autodetector fails when order_with_respect_to is removed, but an
_order field remains
------------------------------------+------------------------------------

Reporter: Stuart Attenborrow | Owner: nobody

Type: Bug | Status: new

Component: Migrations | Version: 4.2
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------------+------------------------------------

Changes (by Sarah Boyce):

* resolution: worksforme =>
* status: closed => new

--
Ticket URL: <https://code.djangoproject.com/ticket/35424#comment:7>