[Django] #28600: Prefetch Related Support in RawQuerySet

Django

unread,

Sep 15, 2017, 8:28:09 AM9/15/17

to django-...@googlegroups.com

- There is no support to prefetch related models to increase performance
- Query results need to be cached manually

--
Ticket URL: <https://code.djangoproject.com/ticket/28600>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Sep 15, 2017, 8:28:47 AM9/15/17

to django-...@googlegroups.com

#28600: Prefetch Related Support in RawQuerySet
-------------------------------------+-------------------------------------

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Adnan Umer):

* status: new => assigned
* owner: nobody => Adnan Umer

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:1>

Django

unread,

Sep 26, 2017, 8:30:24 PM9/26/17

to django-...@googlegroups.com

#28600: Add prefetch related support to RawQuerySet

Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tim Graham):

* has_patch: 1 => 0
* stage: Unreviewed => Accepted

Comment:

Tentatively accepting, although I haven't looked into the history to
understand if there's a design reason why it's not implemented. A
[https://groups.google.com/forum/#!topic/django-users/98IZGnqtsYo post on
django-users] suggests that using `prefetch_related_objects()` (which has
since been [https://docs.djangoproject.com/en/dev/ref/models/querysets
/#prefetch-related-objects documented as a public API]) works.

I'm unchecking "Has patch" as I don't see a patch anywhere. If there's a
patch somewhere, please provide a link.

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:2>

Django

unread,

Sep 27, 2017, 4:27:24 AM9/27/17

to django-...@googlegroups.com

Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Adnan Umer):

* has_patch: 0 => 1
* needs_tests: 0 => 1

Comment:

Here is the link to PR that enabled this feature
https://github.com/django/django/pull/9151

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:3>

Django

unread,

Oct 11, 2017, 3:33:17 PM10/11/17

to django-...@googlegroups.com

Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Дилян Палаузов):

Would you mind implementing the same for bulk_create() #28692?

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:4>

Django

unread,

Oct 11, 2017, 3:35:20 PM10/11/17

to django-...@googlegroups.com

Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Дилян Палаузов):

* cc: Дилян Палаузов (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:5>

Django

unread,

Oct 11, 2017, 4:28:58 PM10/11/17

to django-...@googlegroups.com

Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Adnan Umer):

Replying to [comment:4 Дилян Палаузов]:

> Would you mind implementing the same for bulk_create() #28692?

Once someone accept the ticket, I'll post commit solution of that. BTW you
can temporarily avoid that situation this way

{{{
models = ModelA.objects.select_related('b').bulk_create([ModelA(...),
ModelA(...), ModelA(...)])
from django.db.models import prefetch_related_objects
prefetch_related_objects(models, 'b')
for m in models:
print(m.b.description) # No more extra SELECT query here
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:6>

Django

unread,

Nov 17, 2017, 12:02:05 AM11/17/17

to django-...@googlegroups.com

Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Asif Saifuddin Auvi):

* needs_better_patch: 0 => 1
* needs_docs: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:7>

Django

unread,

Nov 26, 2017, 3:43:29 PM11/26/17

to django-...@googlegroups.com

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Adnan Umer):

* needs_tests: 1 => 0

Old description:

> When using custom raw SQL queries with Django ORM,
>
> - There is no support to prefetch related models to increase performance
> - Query results need to be cached manually

New description:

When using custom raw SQL queries with Django ORM,

- There is no support to prefetch related models to increase performance
- Query results need to be cached manually

PS: __len__ & __bool__ implementation can also be added as that was not
there because of no internal caching in RawQuerySet.

--

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:8>

Django

unread,

Nov 26, 2017, 3:57:45 PM11/26/17

to django-...@googlegroups.com

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

Unless `__bool__` and `__len__` support is required for `prefetch_related`
to work they should be added/tracked in another PR/ticket.

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:9>

Django

unread,

Nov 26, 2017, 4:27:12 PM11/26/17

to django-...@googlegroups.com

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Adnan Umer):

Replying to [comment:9 Simon Charette]:

> Unless `__bool__` and `__len__` support is required for
`prefetch_related` to work they should be added/tracked in another
PR/ticket.

Got that. Moving that to separate PR.

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:10>

Django

unread,

Nov 26, 2017, 4:27:33 PM11/26/17

to django-...@googlegroups.com

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Description changed by Adnan Umer:

Old description:

> When using custom raw SQL queries with Django ORM,
>
> - There is no support to prefetch related models to increase performance
> - Query results need to be cached manually
>

> PS: __len__ & __bool__ implementation can also be added as that was not

> there because of no internal caching in RawQuerySet.

New description:

When using custom raw SQL queries with Django ORM,

- There is no support to prefetch related models to increase performance
- Query results need to be cached manually

--

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:11>

Django

unread,

Nov 26, 2017, 6:15:07 PM11/26/17

to django-...@googlegroups.com

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Adnan Umer):

* needs_better_patch: 1 => 0
* needs_docs: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:12>

Django

unread,

Nov 27, 2017, 2:02:20 AM11/27/17

to django-...@googlegroups.com

#28600: Add prefetch related support to RawQuerySet
-------------------------------------+-------------------------------------

Reporter: Adnan Umer | Owner: (none)
Type: New feature | Status: new

Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Adnan Umer):

* owner: Adnan Umer => (none)
* status: assigned => new

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:13>

Django

unread,

Dec 6, 2017, 1:50:19 AM12/6/17

to django-...@googlegroups.com

#28600: Add prefetch related support to RawQuerySet
-------------------------------------+-------------------------------------

Reporter: Adnan Umer | Owner: Adnan
| Umer
Type: New feature | Status: assigned

Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Adnan Umer):

* owner: (none) => Adnan Umer

* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:14>

Django

unread,

Mar 17, 2018, 5:11:21 PM3/17/18

to django-...@googlegroups.com

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tim Graham):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:15>

Django

unread,

Mar 18, 2018, 3:15:07 PM3/18/18

to django-...@googlegroups.com

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Adnan Umer):

* needs_better_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:16>

Django

unread,

Apr 11, 2018, 10:45:14 AM4/11/18

to django-...@googlegroups.com

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Carlton Gibson):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:17>

Django

unread,

Apr 16, 2018, 6:14:46 AM4/16/18

to django-...@googlegroups.com

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Adnan Umer):

* needs_better_patch: 1 => 0

Comment:

Did requested changes. Requesting to review the PR again.

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:18>

Django

unread,

Apr 18, 2018, 3:17:39 AM4/18/18

to django-...@googlegroups.com

Has patch: 1 | Needs documentation: 1

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Carlton Gibson):

* needs_docs: 0 => 1

Comment:

Patch is looking good. Just needs some adjustments to the documentation
changes.

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:19>

Django

unread,

Apr 18, 2018, 4:01:30 AM4/18/18

to django-...@googlegroups.com

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Carlton Gibson):

* cc: Carlton Gibson (added)

Comment:

Replying to [comment:10 Adnan Umer]:

> Replying to [comment:9 Simon Charette]:
> > Unless `__bool__` and `__len__` support is required for
`prefetch_related` to work they should be added/tracked in another
PR/ticket.
>
> Got that. Moving that to separate PR.

Did that happen? (I
[https://github.com/django/django/pulls?utf8=✓&q=is%3Apr+author%3Auadnan
can't see another PR by Adnan])

I ask because the warning in
[https://github.com/django/django/blob/7a22d9f75125e3cfbea0979a876efe4634f6fe05/docs/topics/db/sql.txt#L88-L97
docs/db/sql.txt] about this will also need updating:

`__bool__()` and `__len__()` are not defined in `RawQuerySet`, and
thus all ``RawQuerySet`` instances are considered `True`. The reason
these methods are not implemented in `RawQuerySet` is that
implementing
them without internal caching would be a performance drawback and
adding
such caching would be backward incompatible.

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:20>

Django

unread,

Apr 18, 2018, 6:41:46 AM4/18/18

to django-...@googlegroups.com

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Adnan Umer):

Replying to [comment:20 Carlton Gibson]:

> Did that happen? (I
[https://github.com/django/django/pulls?utf8=✓&q=is%3Apr+author%3Auadnan
can't see another PR by Adnan])

I havn't yet opened any new PR for that. I was just waiting on
approval/merge of my existing ticket. Once that is done I'll create a new
PR for that.

> I ask because the warning in
[https://github.com/django/django/blob/7a22d9f75125e3cfbea0979a876efe4634f6fe05/docs/topics/db/sql.txt#L88-L97
docs/db/sql.txt] about this will also need updating:

Yes, I've created a new ticket https://code.djangoproject.com/ticket/29337
and assigned that to myself. Not sure should I open PR once this one got
merged or before merge?

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:21>

Django

unread,

Apr 18, 2018, 6:43:23 AM4/18/18

to django-...@googlegroups.com

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Adnan Umer):

* needs_docs: 1 => 0

Comment:

Unchecking "Needs Documentation" to have PR reviewed again after doing
requested changes.

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:22>

Django

unread,

Apr 18, 2018, 10:11:59 AM4/18/18

to django-...@googlegroups.com

Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Carlton Gibson):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:23>

Django

unread,

Apr 19, 2018, 2:30:04 PM4/19/18

to django-...@googlegroups.com

#28600: Add prefetch related support to RawQuerySet
-------------------------------------+-------------------------------------
Reporter: Adnan Umer | Owner: Adnan
| Umer

Type: New feature | Status: closed

Component: Database layer | Version: master
(models, ORM) |

Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tim Graham <timograham@…>):

* status: assigned => closed
* resolution: => fixed

Comment:

In [changeset:"534d8d875eebac6aee278f0ffa6cc59760dac546" 534d8d87]:
{{{
#!CommitTicketReference repository=""
revision="534d8d875eebac6aee278f0ffa6cc59760dac546"
Fixed #28600 -- Added prefetch_related() support to RawQuerySet.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/28600#comment:24>

Django

unread,

Apr 20, 2018, 10:34:34 AM4/20/18

to django-...@googlegroups.com

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------