[Django] #35359: Got `django.core.exceptions.FieldError` when adding GeneratedField

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField
--------------------------------------+--------------------------------
Reporter: wd0517 | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 5.0
Severity: Normal | Keywords: GeneratedField
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
--------------------------------------+--------------------------------
Here is an exists model

{{{
class Foo1(models.Model):
age = models.IntegerField()
}}}

Then adding GeneratedField


{{{
class Foo1(models.Model):
age = models.IntegerField()
name = models.CharField(max_length=10)
lower_name = models.GeneratedField(
expression=Lower("name"),
output_field=models.CharField(max_length=11),
db_persist=True,
)
}}}

Run makemigrations get below file


{{{
class Migration(migrations.Migration):

dependencies = [
('temp_django_test', '0006_foo1'),
]

operations = [
migrations.AddField(
model_name='foo1',
name='lower_name',
field=models.GeneratedField(db_persist=True,
expression=django.db.models.functions.text.Lower('name'),
output_field=models.CharField(max_length=11)),
),
migrations.AddField(
model_name='foo1',
name='name',
field=models.CharField(default='a', max_length=10),
preserve_default=False,
),
]
}}}


Then run migrate, will got error


{{{
temp-django-5.0 ❯ python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, contenttypes, sessions,
temp_django_test
Running migrations:
Applying temp_django_test.0007_foo1_lower_name_foo1_name...Traceback
(most recent call last):
File "/Users/wangdi/Desktop/test_scripts.nosync/temp-
django-5.0/manage.py", line 22, in <module>
main()
File "/Users/wangdi/Desktop/test_scripts.nosync/temp-
django-5.0/manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/Users/wangdi/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/core/management/__init__.py", line 442, in
execute_from_command_line
utility.execute()
File "/Users/wangdi/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/core/management/__init__.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/wangdi/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/core/management/base.py", line 413, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/core/management/base.py", line 459, in execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/core/management/base.py", line 107, in wrapper
res = handle_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/core/management/commands/migrate.py", line 356, in handle
post_migrate_state = executor.migrate(
^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/migrations/executor.py", line 135, in migrate
state = self._migrate_all_forwards(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/migrations/executor.py", line 167, in
_migrate_all_forwards
state = self.apply_migration(
^^^^^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/migrations/executor.py", line 252, in apply_migration
state = migration.apply(state, schema_editor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/migrations/migration.py", line 132, in apply
operation.database_forwards(
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/migrations/operations/fields.py", line 108, in
database_forwards
schema_editor.add_field(
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/backends/mysql/schema.py", line 103, in add_field
super().add_field(model, field)
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/backends/base/schema.py", line 709, in add_field
definition, params = self.column_sql(model, field,
include_default=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/backends/base/schema.py", line 369, in column_sql
" ".join(
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/backends/base/schema.py", line 336, in _iter_column_sql
generated_sql, generated_params = self._column_generated_sql(field)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/backends/base/schema.py", line 434, in
_column_generated_sql
expression_sql, params = field.generated_sql(self.connection)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/models/fields/generated.py", line 58, in generated_sql
resolved_expression = self.expression.resolve_expression(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/models/expressions.py", line 975, in resolve_expression
c.source_expressions[pos] = arg.resolve_expression(
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/models/expressions.py", line 854, in resolve_expression
return query.resolve_ref(self.name, allow_joins, reuse, summarize)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/models/sql/query.py", line 2014, in resolve_ref
join_info = self.setup_joins(
^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/models/sql/query.py", line 1867, in setup_joins
path, final_field, targets, rest = self.names_to_path(
^^^^^^^^^^^^^^^^^^^
File "/Users/wd0517/Desktop/test_scripts.nosync/temp-
django-5.0/.venv/lib/python3.11/site-
packages/django/db/models/sql/query.py", line 1772, in names_to_path
raise FieldError(
django.core.exceptions.FieldError: Cannot resolve keyword 'name' into
field. Choices are: age, id, lower_name
}}}

Environment:
- Python 3.11.8
- Django 5.0.4
- MySQL 8.0.34

I think the bug is caused by the order of operations in Migration, after
adjusting it, then can be migrate successfully
--
Ticket URL: <https://code.djangoproject.com/ticket/35359>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField

---------------------------------+------------------------------------

Reporter: wd0517 | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 5.0

Severity: Release blocker | Resolution:
Keywords: GeneratedField | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

---------------------------------+------------------------------------
Changes (by Simon Charette):

* severity: Normal => Release blocker
* stage: Unreviewed => Accepted

Comment:

The auto-detector has to be adapted to make the `AddField` of generated
field depend on any other `AddField` or `AlterField` of field it
references through `.expression`. Since `GeneratedField` cannot depend on
other `GeneratedField` the easier way of doing to avoid introspecting
`.expression` is likely to make them depend on all other
non-`GeneratedField` fields being added per model so they are always last.
--
Ticket URL: <https://code.djangoproject.com/ticket/35359#comment:1>

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField
---------------------------------+------------------------------------
Reporter: wd0517 | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 5.0
Severity: Release blocker | Resolution:
Keywords: GeneratedField | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Comment (by David Sanders):

For anyone picking up this ticket: Simon & I were discussing ways to get
fields from expressions in this PR:
https://github.com/django/django/pull/16054
--
Ticket URL: <https://code.djangoproject.com/ticket/35359#comment:2>

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField
---------------------------------+------------------------------------
Reporter: wd0517 | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 5.0
Severity: Release blocker | Resolution:
Keywords: GeneratedField | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Changes (by David Sanders):

* cc: David Sanders (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/35359#comment:3>

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField
---------------------------------+------------------------------------
Reporter: wd0517 | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 5.0
Severity: Release blocker | Resolution:
Keywords: GeneratedField | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Changes (by Simon Charette):

* cc: Simon Charette (added)

Comment:

Thanks David, I knew this was implemented somewhere but I lost track of
where exactly!

I think that relying on this property once it lands is definitely the way
to go so it's reused by multiple code paths but in the case of stable/5.x
we might be better off by always adding them last. Thoughts?
--
Ticket URL: <https://code.djangoproject.com/ticket/35359#comment:4>

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField
---------------------------------+------------------------------------
Reporter: wd0517 | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 5.0
Severity: Release blocker | Resolution:
Keywords: GeneratedField | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Comment (by David Sanders):

> Move "assigned tenant users" into "assigned client users" - need to
figure out because acu requires an agreement

Hm can't fks depend on these though because the constraints for those are
also grouped together last. 🤔
--
Ticket URL: <https://code.djangoproject.com/ticket/35359#comment:5>

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField
---------------------------------+------------------------------------
Reporter: wd0517 | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 5.0
Severity: Release blocker | Resolution:
Keywords: GeneratedField | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Comment (by Simon Charette):

> Hm can't fks depend on these though because the constraints for those
are also grouped together last. 🤔

Right. Since `GeneratedField` cannot be foreign keys though it should
still allow for the following grouping of fields?

`non_fk_non_generated_field | generated_fields | fk_fields`
--
Ticket URL: <https://code.djangoproject.com/ticket/35359#comment:6>

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField
---------------------------------+------------------------------------

Reporter: wd0517 | Owner: Bhuvnesh
Type: Bug | Status: assigned

Component: Migrations | Version: 5.0
Severity: Release blocker | Resolution:
Keywords: GeneratedField | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Changes (by Bhuvnesh):

* has_patch: 0 => 1
* owner: nobody => Bhuvnesh
* status: new => assigned

Comment:

[https://github.com/django/django/pull/18068 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/35359#comment:7>

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField
---------------------------------+------------------------------------
Reporter: wd0517 | Owner: Bhuvnesh
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Release blocker | Resolution:
Keywords: GeneratedField | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"97d48cd3c6f409584b5cc19fbddfca917bae78fd" 97d48cd]:
{{{#!CommitTicketReference repository=""
revision="97d48cd3c6f409584b5cc19fbddfca917bae78fd"
Refs #34007, Refs #35359 -- Added Q.referenced_based_fields property.

Thank you to Mariusz Felisiak and Natalia Bidart for the reviews.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35359#comment:8>

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField
---------------------------------+------------------------------------
Reporter: wd0517 | Owner: Bhuvnesh

Type: Bug | Status: closed
Component: Migrations | Version: 5.0
Severity: Release blocker | Resolution: fixed

Keywords: GeneratedField | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Changes (by nessita <124304+nessita@…>):

* resolution: => fixed
* status: assigned => closed

Comment:

In [changeset:"9aeb38c296c69532c7e64b5e3d706a5eb17b3f12" 9aeb38c2]:
{{{#!CommitTicketReference repository=""
revision="9aeb38c296c69532c7e64b5e3d706a5eb17b3f12"
Fixed #35359 -- Fixed migration operations ordering when adding fields
referenced by GeneratedField.expression.

Thank you to Simon Charette for the review.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35359#comment:9>

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField

-------------------------------------+-------------------------------------

Reporter: wd0517 | Owner: Bhuvnesh
Type: Bug | Status: closed
Component: Migrations | Version: 5.0
Severity: Release blocker | Resolution: fixed

Keywords: GeneratedField | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Natalia Bidart):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/35359#comment:10>

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField
-------------------------------------+-------------------------------------
Reporter: wd0517 | Owner: Bhuvnesh
Type: Bug | Status: closed
Component: Migrations | Version: 5.0
Severity: Release blocker | Resolution: fixed
Keywords: GeneratedField | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Natalia <124304+nessita@…>):

In [changeset:"fa202d5cb1f16b9bbfd9da72eb03125fabc34bb8" fa202d5c]:
{{{#!CommitTicketReference repository=""
revision="fa202d5cb1f16b9bbfd9da72eb03125fabc34bb8"
[5.0.x] Refs #34007, Refs #35359 -- Added Q.referenced_based_fields

property.

Thank you to Mariusz Felisiak and Natalia Bidart for the reviews.

Backport of 97d48cd3c6f409584b5cc19fbddfca917bae78fd from main
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35359#comment:11>

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField
-------------------------------------+-------------------------------------
Reporter: wd0517 | Owner: Bhuvnesh
Type: Bug | Status: closed
Component: Migrations | Version: 5.0
Severity: Release blocker | Resolution: fixed
Keywords: GeneratedField | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Natalia <124304+nessita@…>):

In [changeset:"24f54c3b09ab4f19bfc11c5d7ba80985ed870103" 24f54c3]:
{{{#!CommitTicketReference repository=""
revision="24f54c3b09ab4f19bfc11c5d7ba80985ed870103"
[5.0.x] Fixed #35359 -- Fixed migration operations ordering when adding

fields referenced by GeneratedField.expression.

Thank you to Simon Charette for the review.

Backport of 9aeb38c296c69532c7e64b5e3d706a5eb17b3f12 from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35359#comment:12>

[Django]

#35359: Got `django.core.exceptions.FieldError` when adding GeneratedField
-------------------------------------+-------------------------------------
Reporter: wd0517 | Owner: Bhuvnesh
Type: Bug | Status: closed
Component: Migrations | Version: 5.0
Severity: Release blocker | Resolution: fixed
Keywords: GeneratedField | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by nessita <124304+nessita@…>):

In [changeset:"ac9e18f1c4f17d956d203779df1b22faeffa670f" ac9e18f]:
{{{#!CommitTicketReference repository=""
revision="ac9e18f1c4f17d956d203779df1b22faeffa670f"
[5.0.x] Refs #35359 -- Fixed OperationTests.test_add_generate_field() test
on PostgreSQL.

Concat() in Django 5.0 is not immutable on PostgreSQL and cannot be used
in GeneratedField, see 6364b6ee1071381eb3a23ba6b821fc0d6f0fce75.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35359#comment:13>