[Django] #29159: ModelChoiceIterator triggers duplicate queries when choices are cast to list

[Django]

#29159: ModelChoiceIterator triggers duplicate queries when choices are cast to
list
-------------------------------------+-------------------------------------
Reporter: François | Owner: François Freitag
Freitag |
Type: | Status: assigned
Uncategorized |
Component: Forms | Version: 1.8
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
When there are no `prefetch_related()` on `ModelChoiceIterator`'s
queryset, it attempts to use the `iterator()` method to avoid loading all
choices in memory.
([https://docs.djangoproject.com/en/2.0/ref/models/querysets/#prefetch-
related because iterator() and prefetch_related() don't make sense
together.])

That results in a duplicate query if the choices have been evaluated
before, because calling `iterator()` will clone the `QuerySet`, resetting
its result cache in the process.

{{{
class TestModelChoiceField(TestCase):
def test_queryset_result_cache_is_reused(self):
choice = ChoiceOptionModel.objects.create(name="choice 1")
f = ModelChoiceField(ChoiceOptionModel.objects.all())
with self.assertNumQueries(1):
self.assertEqual(
# list calls both __len__ and __iter__
list(f.choices),
[('', '---------'), (choice.pk, str(choice))],
)
}}}

Fails with:
{{{
FAIL: test_queryset_result_cache_is_reused
(forms_tests.test_tmp.TestModelChoiceField)
----------------------------------------------------------------------
Traceback (most recent call last):
File "django/tests/forms_tests/test_tmp.py", line 19, in
test_queryset_result_cache_is_reused
[('', '---------'), (choice.pk, str(choice))],
File "django/django/test/testcases.py", line 80, in __exit__
'%d. %s' % (i, query['sql']) for i, query in
enumerate(self.captured_queries, start=1)
AssertionError: 2 != 1 : 2 queries executed, 1 expected
Captured queries were:
1. SELECT "forms_tests_choiceoptionmodel"."id",
"forms_tests_choiceoptionmodel"."name" FROM
"forms_tests_choiceoptionmodel" ORDER BY
"forms_tests_choiceoptionmodel"."name" ASC
2. SELECT "forms_tests_choiceoptionmodel"."id",
"forms_tests_choiceoptionmodel"."name" FROM
"forms_tests_choiceoptionmodel" ORDER BY
"forms_tests_choiceoption$odel"."name" ASC
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/29159>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#29159: ModelChoiceIterator triggers duplicate queries when choices are cast to
list
-------------------------------------+-------------------------------------

Reporter: François Freitag | Owner: François
| Freitag
Type: Uncategorized | Status: assigned
Component: Forms | Version: 1.8
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by François Freitag):

* has_patch: 0 => 1

Comment:

[https://github.com/django/django/pull/9726 PR]

--
Ticket URL: <https://code.djangoproject.com/ticket/29159#comment:1>

[Django]

#29159: ModelChoiceIterator triggers duplicate queries when choices are cast to
list
-------------------------------------+-------------------------------------

Reporter: François Freitag | Owner: François
Type: | Freitag
Cleanup/optimization | Status: assigned
Component: Forms | Version: 1.8
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tim Graham):

* type: Uncategorized => Cleanup/optimization
* stage: Unreviewed => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/29159#comment:2>

[Django]

#29159: ModelChoiceIterator triggers duplicate queries when choices are cast to
list
-------------------------------------+-------------------------------------

Reporter: François Freitag | Owner: François
Type: | Freitag

Cleanup/optimization | Status: closed
Component: Forms | Version: 1.8
Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tim Graham <timograham@…>):

* status: assigned => closed
* resolution: => fixed

Comment:

In [changeset:"a2e97abd8149e78071806a52282a24c27fe8236b" a2e97abd]:
{{{
#!CommitTicketReference repository=""
revision="a2e97abd8149e78071806a52282a24c27fe8236b"
Fixed #29159 -- Made ModelChoiceIterator reuse QuerySet result cache.

When __len__() is called (e.g. when casting to list or tuple), the
QuerySet is evaluated and the result cache populated. iterator()
shouldn't be called on the QuerySet after that, as it would reset the
result cache and trigger a second query.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/29159#comment:3>