Re: [Django] #36754: Broken migration created when GeneratedField's expression references a foreign key that has been deferred to another initial migration file (was: Bug in GeneratedField when it references a related field (e.g. ForeignKey) with 2 conditions (django happens to create multiple 000x_inital.py for the app && the ForeignKey is first initialized in the later file 000x_inital.py))

[Django]

#36754: Broken migration created when GeneratedField's expression references a
foreign key that has been deferred to another initial migration file
-------------------------------------+-------------------------------------
Reporter: Ou7law007 | Owner: Ou7law007
Type: Bug | Status: assigned
Component: Migrations | Version: 5.0
Severity: Normal | Resolution:
Keywords: autodetector | Triage Stage: Accepted
GeneratedField |
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Jacob Walls):

* component: Database layer (models, ORM) => Migrations
* keywords: migrations autodetector => autodetector GeneratedField
* needs_tests: 0 => 1
* owner: (none) => Ou7law007
* stage: Unreviewed => Accepted
* status: new => assigned
* summary:
Bug in GeneratedField when it references a related field (e.g.
ForeignKey) with 2 conditions (django happens to create multiple
000x_inital.py for the app && the ForeignKey is first initialized in
the later file 000x_inital.py)
=>
Broken migration created when GeneratedField's expression references a
foreign key that has been deferred to another initial migration file
* version: 5.2 => 5.0

Comment:

Thanks for the report, reproduced at
ce36c35e76f82f76cdfa5777456e794d481e5afc and at 5.0.9.

It's all good, but next time, please include some portion of the
stacktrace, as it can help clarify that the error manifests at migration
time, not makemigrations time.

Reproduced with these models, riffing on
`AutodetectorTests.test_arrange_for_graph_with_multiple_initial`. It's
likely that this can be reduced even further.

{{{#!py
# testapp/models.py
from django.db import models

class Author(models.Model):
name = models.CharField(max_length=200)
book = models.ForeignKey("otherapp.Book", models.CASCADE,
related_name="+")
}}}

{{{
./manage.py makemigrations
./manage.py migrate
}}}

{{{#!py
# otherapp/models.py
from django.db import models
from django.db.models.functions import Concat

class Book(models.Model):
author = models.ForeignKey("testapp.Author", models.CASCADE,
related_name="+")
title = models.CharField(max_length=200)


class Attribution(models.Model):
author = models.ForeignKey("testapp.Author", models.CASCADE)
book = models.ForeignKey("otherapp.Book", models.CASCADE)
author_book_pairs = models.GeneratedField(
expression=Concat(models.F("author_id"), models.Value("-"),
models.F("book_id")),
output_field=models.CharField(max_length=50),
db_persist=True,
unique=True,
)
}}}

{{{#!py
File "/Users/jwalls/django/django/core/management/commands/migrate.py",
line 354, in handle
post_migrate_state = executor.migrate(
targets,
...<3 lines>...
fake_initial=fake_initial,
)
File "/Users/jwalls/django/django/db/migrations/executor.py", line 137,
in migrate
state = self._migrate_all_forwards(
state, plan, full_plan, fake=fake, fake_initial=fake_initial
)
File "/Users/jwalls/django/django/db/migrations/executor.py", line 169,
in _migrate_all_forwards
state = self.apply_migration(
state, migration, fake=fake, fake_initial=fake_initial
)
File "/Users/jwalls/django/django/db/migrations/executor.py", line 257,
in apply_migration
state = migration.apply(state, schema_editor)
File "/Users/jwalls/django/django/db/migrations/migration.py", line 132,
in apply
operation.database_forwards(
~~~~~~~~~~~~~~~~~~~~~~~~~~~^
self.app_label, schema_editor, old_state, project_state
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/Users/jwalls/django/django/db/migrations/operations/models.py",
line 100, in database_forwards
schema_editor.create_model(model)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
File "/Users/jwalls/django/django/db/backends/base/schema.py", line 511,
in create_model
sql, params = self.table_sql(model)
~~~~~~~~~~~~~~^^^^^^^
File "/Users/jwalls/django/django/db/backends/base/schema.py", line 222,
in table_sql
definition, extra_params = self.column_sql(model, field)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/Users/jwalls/django/django/db/backends/base/schema.py", line 392,
in column_sql
" ".join(
~~~~~~~~^
# This appends to the params being returned.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<7 lines>...
)
^
),
^
File "/Users/jwalls/django/django/db/backends/base/schema.py", line 357,
in _iter_column_sql
generated_sql, generated_params = self._column_generated_sql(field)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
File "/Users/jwalls/django/django/db/backends/base/schema.py", line 457,
in _column_generated_sql
expression_sql, params = field.generated_sql(self.connection)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/Users/jwalls/django/django/db/models/fields/generated.py", line
58, in generated_sql
resolved_expression = self.expression.resolve_expression(
self._query, allow_joins=False
)
File "/Users/jwalls/django/django/db/models/expressions.py", line 301,
in resolve_expression
expr.resolve_expression(query, allow_joins, reuse, summarize,
for_save)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwalls/django/django/db/models/expressions.py", line 301,
in resolve_expression
expr.resolve_expression(query, allow_joins, reuse, summarize,
for_save)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwalls/django/django/db/models/expressions.py", line 904,
in resolve_expression
return query.resolve_ref(self.name, allow_joins, reuse, summarize)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwalls/django/django/db/models/sql/query.py", line 2070, in
resolve_ref
join_info = self.setup_joins(
field_list, self.get_meta(), self.get_initial_alias(),
can_reuse=reuse
)
File "/Users/jwalls/django/django/db/models/sql/query.py", line 1920, in
setup_joins
path, final_field, targets, rest = self.names_to_path(
~~~~~~~~~~~~~~~~~~^
names[:pivot],
^^^^^^^^^^^^^^
...<2 lines>...
fail_on_missing=True,
^^^^^^^^^^^^^^^^^^^^^
)
^
File "/Users/jwalls/django/django/db/models/sql/query.py", line 1825, in
names_to_path
raise FieldError(
...<2 lines>...
)
django.core.exceptions.FieldError: Cannot resolve keyword 'author_id' into
field. Choices are: author_book_pairs, id
}}}
----
> Feel free to test it.

Would you like to submit your solution as a pull request? You could likely
riff on `AutodetectorTests.test_arrange_for_graph_with_multiple_initial`
to create a test for this.
--
Ticket URL: <https://code.djangoproject.com/ticket/36754#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.