[Django] #29984: Support prefetch_related with Queryset.iterator
Django
-------------------------------------+-------------------------------------
Reporter: Taylor H | Owner: nobody
Type: New | Status: new
feature |
Component: Database | Version: 2.1
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
I was surprise when I found out that `prefetch_related` calls are ignored
when using `Queryset.iterator`. I noticed in the docs here
https://docs.djangoproject.com/en/2.1/ref/models/querysets/#iterator that
it is because the "these two optimizations do not make sense together."
That may have been true in the past, but it is definitely not the case
now. The iterator allows chunking (by default 2000) and it would be very
helpful to prefetch related for each chunk.
--
Ticket URL: <https://code.djangoproject.com/ticket/29984>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: Taylor H | Owner: nobody
Component: Database layer | Version: 2.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
Comment:
The situation isn't completely straightforward. There's a discussion on
[https://groups.google.com/d/topic/django-
developers/ADgUd6jRvdw/discussion django-developers]. Accepting the ticket
to do something.
--
Ticket URL: <https://code.djangoproject.com/ticket/29984#comment:1>
Django
Reporter: Taylor H | Owner: nobody
Component: Database layer | Version: 2.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Raphael Kimmig):
I've created a proof of concept implementation
https://github.com/django/django/pull/10707/
I think prefetching is probably closer to what a user would expect from
the api.
--
Ticket URL: <https://code.djangoproject.com/ticket/29984#comment:2>
Django
Reporter: Taylor H | Owner: nobody
Component: Database layer | Version: 2.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_docs: 0 => 1
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/29984#comment:3>
Django
Reporter: Taylor H | Owner: nobody
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* version: 2.1 => master
Old description:
> I was surprise when I found out that `prefetch_related` calls are ignored
> when using `Queryset.iterator`. I noticed in the docs here
> https://docs.djangoproject.com/en/2.1/ref/models/querysets/#iterator that
> it is because the "these two optimizations do not make sense together."
> That may have been true in the past, but it is definitely not the case
> now. The iterator allows chunking (by default 2000) and it would be very
> helpful to prefetch related for each chunk.
New description:
I was surprised when I found out that `prefetch_related` calls are ignored
when using `Queryset.iterator`. I noticed in the docs here
https://docs.djangoproject.com/en/dev/ref/models/querysets/#iterator that
it is because the "these two optimizations do not make sense together."
That may have been true in the past, but it is definitely not the case
now. The iterator allows chunking (by default 2000) and it would be very
helpful to prefetch related for each chunk.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/29984#comment:4>
Django
Reporter: Taylor H | Owner: nobody
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
Comment:
Patch needs both documentation and a deprecation warning when
`self._prefetch_related_lookups and chunk_size is None`. The idea is to
eventually turn this deprecation warning into an error and never performs
prefetching unless a chunk size is specified.
--
Ticket URL: <https://code.djangoproject.com/ticket/29984#comment:5>
Django
Reporter: Taylor H | Owner: Jacob
| Walls
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => Jacob Walls
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/29984#comment:6>
Django
Reporter: Taylor H | Owner: Jacob
| Walls
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
* needs_docs: 1 => 0
Comment:
[https://github.com/django/django/pull/15334 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/29984#comment:7>
Django
Reporter: Taylor H | Owner: Jacob
| Walls
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/29984#comment:8>
Django
Reporter: Taylor H | Owner: Jacob
| Walls
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/29984#comment:9>
Django
Reporter: Taylor H | Owner: Jacob
| Walls
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/29984#comment:10>
Django
Reporter: Taylor H | Owner: Jacob
| Walls
Component: Database layer | Version: dev
(models, ORM) |
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"edbf930287cb72e9afab1f7208c24b1146b0c4ec" edbf9302]:
{{{
#!CommitTicketReference repository=""
revision="edbf930287cb72e9afab1f7208c24b1146b0c4ec"
Fixed #29984 -- Added QuerySet.iterator() support for prefetching related
objects.
Co-authored-by: Raphael Kimmig <raphael...@ampad.de>
Co-authored-by: Simon Charette <chare...@gmail.com>
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29984#comment:11>
Django
Reporter: Taylor H | Owner: Jacob
| Walls
Type: New feature | Status: closed
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"139135627650ed6aaaf4c755b82c3bd43f2b8f51" 1391356]:
{{{
#!CommitTicketReference repository=""
revision="139135627650ed6aaaf4c755b82c3bd43f2b8f51"
Refs #29984 -- Made QuerySet.iterator() without chunk_size raise
ValueError after prefetch_related().
Per deprecation timeline.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29984#comment:12>