[Django] #24272: prefetch_related GenericRelation via related_query_name
Django
-------------------------------------+-------------------------------------
Reporter: asdator | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: 1.7
(models, ORM) | Keywords: prefetch_related,
Severity: Normal | GenericRelation, related_query_name
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Consider the following model structure:
{{{#!python
from django.db import models
from django.contrib.contenttypes.fields import GenericForeignKey,
GenericRelation
from django.contrib.contenttypes.models import ContentType
class TaggedItem(models.Model):
tag = models.SlugField()
content_type = models.ForeignKey(ContentType)
object_id = models.PositiveIntegerField()
content_object = GenericForeignKey('content_type', 'object_id')
def __unicode__(self):
return self.tag
class Director(models.Model):
name = models.CharField(max_length=100)
def __unicode__(self):
return self.name
class Movie(models.Model):
name = models.CharField(max_length=100)
director = models.ForeignKey(Director)
tags = GenericRelation(TaggedItem, related_query_name='movies')
def __unicode__(self):
return self.name
class Author(models.Model):
name = models.CharField(max_length=100)
def __unicode__(self):
return self.name
class Book(models.Model):
name = models.CharField(max_length=100)
author = models.ForeignKey(Author)
tags = GenericRelation(TaggedItem, related_query_name='books')
def __unicode__(self):
return self.name
}}}
And some initial data:
{{{#!python
>>> a = Author.objects.create(name='E L James')
>>> b1 = Book.objects.create(name='Fifty Shades of Grey', author=a)
>>> b2 = Book.objects.create(name='Fifty Shades Darker', author=a)
>>> b3 = Book.objects.create(name='Fifty Shades Freed', author=a)
>>> d = Director.objects.create(name='James Gunn')
>>> m1 = Movie.objects.create(name='Guardians of the Galaxy', director=d)
>>> t1 = TaggedItem.objects.create(content_object=b1, tag='roman')
>>> t2 = TaggedItem.objects.create(content_object=b2, tag='roman')
>>> t3 = TaggedItem.objects.create(content_object=b3, tag='roman')
>>> t4 = TaggedItem.objects.create(content_object=m1, tag='action movie')
}}}
Now using the `GenericForeignKey` we are able to:
1. `prefetch` only one level deep from `querysets` containing different
type of `content_object`
{{{#!python
>>> TaggedItem.objects.all().prefetch_related('content_object')
[<TaggedItem: roman>, <TaggedItem: roman>, <TaggedItem: roman>,
<TaggedItem: action movie>]
}}}
2. `prefetch` many levels but from `querysets` containing only one type of
`content_object`.
{{{#!python
>>> TaggedItem.objects.filter(books__author__name='E L
James').prefetch_related('content_object__author')
[<TaggedItem: roman>, <TaggedItem: roman>, <TaggedItem: roman>]
}}}
But we can't do 1) and 2) together (//`prefetch` many levels from
`querysets` containing different types of `content_objects`//)
{{{#!python
>>> TaggedItem.objects.all().prefetch_related('content_object__author')
Traceback (most recent call last):
...
AttributeError: 'Movie' object has no attribute 'author_id'
}}}
For such tasks this API is inconvenient and became unconvincing in more
complex examples. For example if we want all `TaggedItems` with
`prefetched` `movies` with their `directors` and `prefetched` `books` with
their `author`.
One silly attempt would look like this:
{{{#!python
>>> TaggedItem.objects.all().prefetch_related(
... 'content_object__author',
... 'content_object__director',
... )
Traceback (most recent call last):
...
AttributeError: 'Movie' object has no attribute 'author_id'
}}}
Or like this:
{{{#!python
>>> TaggedItem.objects.all().prefetch_related(
... Prefetch('content_object',
queryset=Book.objects.all().select_related('author')),
... Prefetch('content_object',
queryset=Movie.objects.all().select_related('director')),
... )
Traceback (most recent call last):
...
ValueError: Custom queryset can't be used for this lookup.
}}}
What I suggest is to use the API which we used to filter `TaggedItems` by
their `book` `author`. This is not working right now.
{{{#!python
>>> TaggedItem.objects.filter(books__author__name='E L
James').prefetch_related('books')
Traceback (most recent call last):
...
AttributeError: 'Book' object has no attribute 'object_id'
}}}
This way we would have and a nice solution for the more complex example
mentioned above:
{{{#!python
>>> TaggedItem.objects.all().prefetch_related(
... 'books__author',
... 'movies__director',
... )
Traceback (most recent call last):
...
AttributeError: 'Book' object has no attribute 'object_id'
}}}
Or like this:
{{{#!python
>>> TaggedItem.objects.all().prefetch_related(
... Prefetch('books',
queryset=Book.objects.all().select_related('author')),
... Prefetch('movies',
queryset=Movie.objects.all().select_related('director')),
... )
Traceback (most recent call last):
...
AttributeError: 'Book' object has no attribute 'object_id'
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24272>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: asdator | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: 1.7
(models, ORM) |
Keywords: prefetch_related, | Triage Stage:
GenericRelation, | Unreviewed
related_query_name |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
I don't think what you have asked for can be implemented, but I'll leave
this open for confirmation by an ORM expert. See #21422 which is to
document the limitation.
--
Ticket URL: <https://code.djangoproject.com/ticket/24272#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: asdator | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: 1.7
(models, ORM) |
Keywords: prefetch_related, | Triage Stage:
GenericRelation, | Unreviewed
related_query_name |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by timgraham):
This issue is duplicated on Stackoverflow:
http://stackoverflow.com/questions/28127135/is-django-prefetch-related-
supposed-to-work-with-genericrelation
--
Ticket URL: <https://code.djangoproject.com/ticket/24272#comment:2>
Django
-------------------------------------+-------------------------------------
Reporter: asdator | Owner: nobody
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
GenericRelation, |
related_query_name |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* version: 1.7 => master
* type: New feature => Cleanup/optimization
* stage: Unreviewed => Accepted
Comment:
If the proposal can't be implemented, I think it would be helpful to at
least throw a more helpful error message.
--
Ticket URL: <https://code.djangoproject.com/ticket/24272#comment:3>
Django
Reporter: asdator | Owner: nobody
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: prefetch_related, | Triage Stage: Accepted
GenericRelation, |
related_query_name |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by asdator):
I think the proposal is implementable. Here is an django-app which I just
found which seems like implements the main part of the problem (where
django get's confused to prefetch different type of FK's from
'content_object').
[https://bitbucket.org/andrew_pashkin/django-deep-prefetch/overview
django-deep-prefetch]
Found the app from this ticket:
[https://code.djangoproject.com/ticket/22014 #22014]
--
Ticket URL: <https://code.djangoproject.com/ticket/24272#comment:4>
Django
-------------------------------------+-------------------------------------
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: prefetch_related, | Triage Stage: Accepted
GenericRelation, |
related_query_name |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Todor Velichkov):
I did some debugging and I think I find out why `prefetch_related` on
`GenericRelation` using `related_query_name` is not working. i.e.
`TaggedItem.objects.filter(books__author__name='E L
James').prefetch_related('books')`
It starts from the
[https://github.com/django/django/blob/1.11.2/django/db/models/fields/related_descriptors.py#L596
get_prefetch_queryset] at `related_descriptors`, where `self.field` is a
`GenericRelation` field
(`<django.contrib.contenttypes.fields.GenericRelation: tags>` in this
example) the
[https://github.com/django/django/blob/1.11.2/django/contrib/contenttypes/fields.py#L276
GenericRelation] class inherits from `ForeignObject` but does not
implement
[https://github.com/django/django/blob/1.11.2/django/db/models/fields/related.py#L655
get_local_related_value] and `get_foreign_related_value` methods which
looks incompatible with the `GenericRelation` interface, because they
search for `object_id` attribute inside a `Book` model.
I would love to try to fix this, but I still can't fully understand the
code, I'm not even sure what needs to be returned here, all tags related
to this book maybe? If thats the case, then the code in
`get_prefetch_queryset` looks like its expected to be returned only a
single instance, not many, this is confusing me.
--
Ticket URL: <https://code.djangoproject.com/ticket/24272#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: Todor Velichkov | Owner: nobody
Cleanup/optimization |
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution: duplicate
Keywords: prefetch_related, | Triage Stage: Accepted
GenericRelation, |
related_query_name |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => duplicate
Comment:
I think that we can actually close this one as a duplicate of #33651 as
the latter would allow the following syntax to be used
{{{#!python
TaggedItem.objects.prefetch_related(
GenericPrefetch(
"content_object", [
Book.objects.select_related('author'),
Movie.objects.select_related('director'),
]
),
)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24272#comment:6>