[Django] #32775: LiveServerTestCase fails when query expressions are used for model ordering
Django
---------------------------------------------+------------------------
Reporter: Gergely Kalmár | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.2
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
= Reproducing the bug =
Create a `models.py` file as follows:
{{{
#!python
from django.db import models
from django.db.models.functions import Lower
class Order(models.Model):
id = models.CharField(max_length=10, primary_key=True)
class Meta:
ordering = [Lower('id')]
class Offer(models.Model):
order = models.OneToOneField(Order, primary_key=True,
on_delete=models.PROTECT)
}}}
Create a test case:
{{{
#!python
from django.contrib.staticfiles.testing import LiveServerTestCase
class Test(LiveServerTestCase):
def test_case(self):
pass
}}}
Run the tests with `./manage.py test`:
{{{
> ./manage.py test
Creating test database for alias 'default'...
Traceback (most recent call last):
File "./manage.py", line 17, in <module>
main()
File "./manage.py", line 13, in main
execute_from_command_line(sys.argv)
File ".../lib/python3.8/site-
packages/django/core/management/__init__.py", line 419, in
execute_from_command_line
utility.execute()
File ".../lib/python3.8/site-
packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File ".../lib/python3.8/site-
packages/django/core/management/commands/test.py", line 23, in
run_from_argv
super().run_from_argv(argv)
File ".../lib/python3.8/site-packages/django/core/management/base.py",
line 354, in run_from_argv
self.execute(*args, **cmd_options)
File ".../lib/python3.8/site-packages/django/core/management/base.py",
line 398, in execute
output = self.handle(*args, **options)
File ".../lib/python3.8/site-
packages/django/core/management/commands/test.py", line 55, in handle
failures = test_runner.run_tests(test_labels)
File ".../lib/python3.8/site-packages/django/test/runner.py", line 725,
in run_tests
old_config = self.setup_databases(aliases=databases)
File ".../lib/python3.8/site-packages/django/test/runner.py", line 643,
in setup_databases
return _setup_databases(
File ".../lib/python3.8/site-packages/django/test/utils.py", line 179,
in setup_databases
connection.creation.create_test_db(
File ".../lib/python3.8/site-
packages/django/db/backends/base/creation.py", line 90, in create_test_db
self.connection._test_serialized_contents =
self.serialize_db_to_string()
File ".../lib/python3.8/site-
packages/django/db/backends/base/creation.py", line 136, in
serialize_db_to_string
serializers.serialize("json", get_objects(), indent=None, stream=out)
File ".../lib/python3.8/site-
packages/django/core/serializers/__init__.py", line 129, in serialize
s.serialize(queryset, **options)
File ".../lib/python3.8/site-packages/django/core/serializers/base.py",
line 90, in serialize
for count, obj in enumerate(queryset, start=1):
File ".../lib/python3.8/site-
packages/django/db/backends/base/creation.py", line 133, in get_objects
yield from queryset.iterator()
File ".../lib/python3.8/site-packages/django/db/models/query.py", line
353, in _iterator
yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch,
chunk_size=chunk_size)
File ".../lib/python3.8/site-packages/django/db/models/query.py", line
51, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch,
chunk_size=self.chunk_size)
File ".../lib/python3.8/site-packages/django/db/models/sql/compiler.py",
line 1162, in execute_sql
sql, params = self.as_sql()
File ".../lib/python3.8/site-packages/django/db/models/sql/compiler.py",
line 513, in as_sql
extra_select, order_by, group_by = self.pre_sql_setup()
File ".../lib/python3.8/site-packages/django/db/models/sql/compiler.py",
line 56, in pre_sql_setup
order_by = self.get_order_by()
File ".../lib/python3.8/site-packages/django/db/models/sql/compiler.py",
line 372, in get_order_by
resolved = expr.resolve_expression(self.query, allow_joins=True,
reuse=None)
File ".../lib/python3.8/site-packages/django/db/models/expressions.py",
line 247, in resolve_expression
c.set_source_expressions([
File ".../lib/python3.8/site-packages/django/db/models/expressions.py",
line 248, in <listcomp>
expr.resolve_expression(query, allow_joins, reuse, summarize)
File ".../lib/python3.8/site-packages/django/db/models/expressions.py",
line 678, in resolve_expression
c.source_expressions[pos] = arg.resolve_expression(query, allow_joins,
reuse, summarize, for_save)
File ".../lib/python3.8/site-packages/django/db/models/expressions.py",
line 578, in resolve_expression
return query.resolve_ref(self.name, allow_joins, reuse, summarize)
File ".../lib/python3.8/site-packages/django/db/models/sql/query.py",
line 1754, in resolve_ref
join_info = self.setup_joins(field_list, self.get_meta(),
self.get_initial_alias(), can_reuse=reuse)
File ".../lib/python3.8/site-packages/django/db/models/sql/query.py",
line 1623, in setup_joins
path, final_field, targets, rest = self.names_to_path(
File ".../lib/python3.8/site-packages/django/db/models/sql/query.py",
line 1537, in names_to_path
raise FieldError("Cannot resolve keyword '%s' into field. "
django.core.exceptions.FieldError: Cannot resolve keyword 'id' into field.
Choices are: order, order_id
}}}
If there is any workaround that I can use in the meantime please let me
know. Thank you!
--
Ticket URL: <https://code.djangoproject.com/ticket/32775>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Gergely Kalmár | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.2
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
We'll have to figure out exactly what ORM query is being constructed here.
The error doesn't happen on Django's main branch after
3089018e951dcca568574c0e0ebf9d8aab112389 but I'd guess the underlying
problem still exists.
--
Ticket URL: <https://code.djangoproject.com/ticket/32775#comment:1>
Django
Reporter: Gergely Kalmár | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.2
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Simon Charette):
* stage: Unreviewed => Accepted
Comment:
You'll want to make sure to generate migrations for the app containing
these models otherwise you won't be able to reproduce.
This bug is two fold. First we likely just want to have the querysets
generated during serialization order by the primary key without involving
possible related object semantic
{{{#!python
diff --git a/django/db/backends/base/creation.py
b/django/db/backends/base/creation.py
index 81cb34bd9f..f49c31edaf 100644
--- a/django/db/backends/base/creation.py
+++ b/django/db/backends/base/creation.py
@@ -129,7 +129,7 @@ def get_objects():
):
queryset = model._base_manager.using(
self.connection.alias,
- ).order_by(model._meta.pk.name)
+ ).order_by('pk')
yield from queryset.iterator()
# Serialize to a string
out = StringIO()
}}}
The above happens to address the reported crash but it doesn't deal with
the fundamental issue which is that
`Referent.objects.order_by('reference')` crashes if
`Referred.Meta.ordering` contains expressions because they won't be
transposed to the origin of the queryset (e.g. `Lower('id') ->
Lower('order__id')` in the reported case). It can be reproduced with the
same set of models and the following test
{{{#!python
from django.test import TestCase
from .models import Offer
class Test(TestCase):
def test_case(self):
list(Offer.objects.order_by('order'))
}}}
The issue lies in `SQLCompiler.find_ordering_name` but
[https://github.com/django/django/blob/5e04e84d67da8163f365e9f5fcd169e2630e2873/django/db/models/sql/compiler.py#L773-L774
is not trivial to address] as we'd need a way to prefix all field
references in an `Expression` so I'd suggest we raise a
`NotImplementedError` instead for now. [https://djangoci.com/job/django-
coverage/HTML_20Coverage_20Report/_home_jenkins_workspace_django-
coverage_django_db_models_sql_compiler_py.html#t774 Not sure when this
path is covered by the suite] but I would assume it would only work when
MTI inheritance is involved and referenced field references don't need to
be prefixed as they are inherited?
--
Ticket URL: <https://code.djangoproject.com/ticket/32775#comment:2>
Django
Reporter: Gergely Kalmár | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.2
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Gergely Kalmár):
Thank you for the help! It seems that I can work around this by setting
{{{
#!python
DATABASES = {
'default': {
...
'TEST': {'SERIALIZE': False}
}
}
}}}
which is fine for now, I think. It would be great if the recommended patch
for at least fixing the immediate issue could be added though in case we'd
need tests to be serialized for some reason in the future.
--
Ticket URL: <https://code.djangoproject.com/ticket/32775#comment:3>
Django
Reporter: Gergely Kalmár | Owner: nobody
Component: Testing framework | Version: 3.2
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* status: new => closed
* resolution: => fixed
Comment:
Fixed in 2798c937deb6625a4e6a36e70d4d60ce5faac954 (#29538).
--
Ticket URL: <https://code.djangoproject.com/ticket/32775#comment:4>