[Django] #26690: Add the option in the prefetch_related to have the prefetched results returned as a list or as a queryset

[Django]

#26690: Add the option in the prefetch_related to have the prefetched results
returned as a list or as a queryset
-------------------------------------+-------------------------------------
Reporter: Ursidours | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) | Keywords: Prefetch
Severity: Normal | prefetch_related queryset
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
In his answer in ticket #26676, Simon Charette suggested me to open this
ticket.

Currently, the result stored by prefetch_related is by default a queryset,
unless Prefetch with to_attr is used, in which case it is a list.
According to the
[https://docs.djangoproject.com/en/1.9/ref/models/querysets/#django.db.models.Prefetch
documentation about Prefetch], this trades the flexibility offered by the
queryset for a considerable performance improvement.

I can see the rationale to have this behavior by default, but it seems to
me it would be nicer to have an option to explicitly declare whether the
results should be returned as a queryset or as a list:
1. In many cases, there is no need to chain further queries to the default
prefetch_related. By explicitly asking for a list it would then become
possible to benefit from the speed improvement.
2. Conversely, some developers may be happy to take the performance hit in
order to further refine a Prefetch with to_attr.

This could be done via an optional boolean in prefetch_related,
is_chainable or force_queryset or something like this (if left unset the
current default behavior would apply). Does that sound like a good idea?

--
Ticket URL: <https://code.djangoproject.com/ticket/26690>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#26690: Add the option in the prefetch_related to have the prefetched results
returned as a list or as a queryset
-------------------------------------+-------------------------------------
Reporter: Ursidours | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) |

Severity: Normal | Resolution:
Keywords: Prefetch | Triage Stage:
prefetch_related queryset | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by timgraham):

* cc: charettes (added)
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0

Comment:

Simon, care to add your thoughts?

--
Ticket URL: <https://code.djangoproject.com/ticket/26690#comment:1>

[Django]

#26690: Add the option in the prefetch_related to have the prefetched results
returned as a list or as a queryset
-------------------------------------+-------------------------------------
Reporter: Ursidours | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) |

Severity: Normal | Resolution:
Keywords: Prefetch | Triage Stage:
prefetch_related queryset | Unreviewed
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by charettes):

Personally I'm -0 on adding such a feature for the following reasons:

> In many cases, there is no need to chain further queries to the default
prefetch_related. By explicitly asking for a list it would then become
possible to benefit from the speed improvement.

The default `prefetch_related()` results are stored in a `QuerySet()` in
order to make sure the returned instances' prefetched relations still
behave the same as if they were not prefetched. Storing lists in there
would make such instances/querysets unusable by many components of Django
such as forms and model level validation where it would fail in non-
obvious ways.

Using `Prefetch('relation', to_attr='relation_as_list')` already serves
this purpose and don't think introducing another way to do it such as
`Prefetch('relation', as_list=True)` is worth it. If we'd really want to
speed up default prefetches I think we should investigate assigning
objects that quack like `QuerySet` (e.g. `PrefetchedQuerySet`) but only
lazily build their underlying query when required as this is the part that
slow down performance here.

> Conversely, some developers may be happy to take the performance hit in
order to further refine a Prefetch with to_attr.

I've never personally required to refine a prefetched result, that sounds
like an anti-pattern to me.

--
Ticket URL: <https://code.djangoproject.com/ticket/26690#comment:2>

[Django]

#26690: Add the option in the prefetch_related to have the prefetched results
returned as a list or as a queryset
-------------------------------------+-------------------------------------
Reporter: Ursidours | Owner: nobody

Type: New feature | Status: closed

Component: Database layer | Version: master
(models, ORM) |

Severity: Normal | Resolution: wontfix

Keywords: Prefetch | Triage Stage:
prefetch_related queryset | Unreviewed
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by timgraham):

* status: new => closed
* resolution: => wontfix

--
Ticket URL: <https://code.djangoproject.com/ticket/26690#comment:3>

[Django]

#26690: Add the option in the prefetch_related to have the prefetched results
returned as a list or as a queryset
-------------------------------------+-------------------------------------
Reporter: Ursidours | Owner: nobody

Type: New feature | Status: closed

Component: Database layer | Version: master
(models, ORM) |

Severity: Normal | Resolution: wontfix
Keywords: Prefetch | Triage Stage:
prefetch_related queryset | Unreviewed
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by charettes):

FWIW the `PrefetchedQuerySet` idea was already suggested 3 years ago by
Anssi (#20577).

--
Ticket URL: <https://code.djangoproject.com/ticket/26690#comment:4>