[Django] #27554: Queryset evaluation fails with mix of nested and flattened prefetches (AttributeError on RelatedManager)
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
Reporter: Anthony | Owner: nobody
Leontiev |
Type: Bug | Status: new
Component: Database | Version: 1.10
layer (models, ORM) | Keywords: prefetch
Severity: Normal | AttributeError prefetch_related
Triage Stage: | nested RelatedManager
Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Under certain circumstances, querysets with nested prefetches fail to
evaluate in Django 1.10.
With these models (populated with data):
{{{
# A <-> B <-- C --> D
# \__________/
class A(models.Model):
pass
class B(models.Model):
a = models.OneToOneField('A', null=True, related_name='b')
d = models.ManyToManyField('D', through='C', related_name='b')
class C(models.Model):
b = models.ForeignKey('B', related_name='c')
d = models.ForeignKey('D', related_name='c')
class D(models.Model):
pass
}}}
... and this query/evaluation:
{{{
queryset = A.objects.all().prefetch_related(
Prefetch(
'b',
queryset=B.objects.all().prefetch_related(
Prefetch('c__d')
)
)
)
content = queryset[0].b.c.all()[0].d.pk
}}}
Django throws this exception:
{{{
File "/Users/ant/code/django-bugs/prefetchbug/.venv/lib/python2.7/site-
packages/django/db/models/query.py", line 256, in __iter__
self._fetch_all()
File "/Users/ant/code/django-bugs/prefetchbug/.venv/lib/python2.7/site-
packages/django/db/models/query.py", line 1087, in _fetch_all
self._prefetch_related_objects()
File "/Users/ant/code/django-bugs/prefetchbug/.venv/lib/python2.7/site-
packages/django/db/models/query.py", line 663, in
_prefetch_related_objects
prefetch_related_objects(self._result_cache,
*self._prefetch_related_lookups)
File "/Users/ant/code/django-bugs/prefetchbug/.venv/lib/python2.7/site-
packages/django/db/models/query.py", line 1460, in
prefetch_related_objects
(through_attr, first_obj.__class__.__name__, lookup.prefetch_through))
AttributeError: Cannot find 'd' on RelatedManager object, 'b__c__d' is an
invalid parameter to prefetch_related()
}}}
Interestingly enough, this practically identical variant of the above
query with a fully "nested" prefetch structure succeeds:
{{{
queryset = A.objects.all().prefetch_related(
Prefetch(
'b',
queryset=B.objects.all().prefetch_related(
Prefetch(
'c',
queryset=C.objects.all().prefetch_related(
'd'
)
)
)
)
)
content = queryset[0].b.c.all()[0].d.pk
}}}
To reproduce, download the attachment and run:
{{{
pip install -r requirements.txt
python manage.py migrate
python manage.py runserver
}}}
... and navigate to {localhost:8000/flat} and {localhost:8000/nested}.
This only seems to impact Django 1.10; the bug does not reproduce if the
Django version in the {requirements.txt} file is changed to 1.9 or 1.8.
Discovered in https://github.com/AltSchool/dynamic-rest/pull/147
Similar to https://code.djangoproject.com/ticket/26801
--
Ticket URL: <https://code.djangoproject.com/ticket/27554>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.10
(models, ORM) |
Severity: Normal | Resolution:
Keywords: prefetch | Triage Stage:
AttributeError prefetch_related | Unreviewed
nested RelatedManager |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* Attachment "prefetchbug.tar.gz" added.
test project to reproduce the bug
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.10
(models, ORM) |
Severity: Normal | Resolution:
Keywords: prefetch | Triage Stage:
AttributeError prefetch_related | Unreviewed
nested RelatedManager |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Anthony Leontiev):
Discovered in an API integration test that relies on nested prefetching
during an upgrade of Django 1.7 to Django 1.10.
--
Ticket URL: <https://code.djangoproject.com/ticket/27554#comment:1>
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.10
(models, ORM) |
Severity: Normal | Resolution:
AttributeError prefetch_related |
nested RelatedManager |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
Comment:
I haven't tried reproducing myself but the report is complete enough that
I trust the developer's judgment.
Anthony, can you confirm this is also an issue when running
`A.objects.prefetch_related('b__c__d'')`? What about
`B.objects.prefetch_related('c__d')`?
[https://docs.djangoproject.com/en/1.10/internals/contributing/triaging-
tickets/#bisecting-a-regression Bisecting the exact commit that introduced
the regression] would also be useful.
--
Ticket URL: <https://code.djangoproject.com/ticket/27554#comment:2>
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.10
(models, ORM) |
Severity: Normal | Resolution:
Keywords: prefetch | Triage Stage: Accepted
AttributeError prefetch_related |
nested RelatedManager |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Vimarsh Chaturvedi):
Hey guys,
My first time posting here.
I tried reproducing the bug, and was able to.
I added the following cases as per Simon Charette's suggestion and Django
didn't throw any exception.
views.py
{{{
#Code in orinigal test project by Anthony Leontiev
def setup():
a = A.objects.create()
b = B.objects.create()
b.a = a
b.save()
d = D.objects.create()
C.objects.create(b=b, d=d)
#Succeeded: As reported above.
def nested_prefetch(request, *args, **kwargs):
setup()
queryset = A.objects.all().prefetch_related(
Prefetch(
'b',
queryset=B.objects.all().prefetch_related(
Prefetch(
'c',
queryset=C.objects.all().prefetch_related(
'd'
)
)
)
)
)
content = queryset[0].b.c.all()[0].d.pk
return HttpResponse(content)
#Failed: As reported above.
def flat_prefetch(request, *args, **kwargs):
setup()
queryset = A.objects.all().prefetch_related(
Prefetch(
'b',
queryset=B.objects.all().prefetch_related(
Prefetch('c__d')
)
)
)
content = queryset[0].b.c.all()[0].d.pk
return HttpResponse(content)
#The following three cases were added by me
#Failed: As expected.
def flat_prefetch_one(request, *args, **kwargs):
setup()
queryset = A.objects.all().prefetch_related(
Prefetch(
'b',
queryset=B.objects.all().prefetch_related('c__d')
)
)
content = queryset[0].b.c.all()[0].d.pk
return HttpResponse(content)
#Succeeded: I was expecting this and the following case to fail too, but
they didn't.
def flat_prefetch_two(request, *args, **kwargs):
setup()
queryset = A.objects.all().prefetch_related(
Prefetch('b__c__d')
)
content = queryset[0].b.c.all()[0].d.pk
return HttpResponse(content)
#succeeded
def flat_prefetch_three(request, *args, **kwargs):
setup()
queryset = A.objects.all().prefetch_related('b__c__d')
content = queryset[0].b.c.all()[0].d.pk
return HttpResponse(content)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27554#comment:3>
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.10
(models, ORM) |
Severity: Normal | Resolution:
Keywords: prefetch | Triage Stage: Accepted
AttributeError prefetch_related |
nested RelatedManager |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* Attachment "flattened-prefetch.zip" added.
test cases for bug
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.10
(models, ORM) |
Severity: Normal | Resolution:
Keywords: prefetch | Triage Stage: Accepted
AttributeError prefetch_related |
nested RelatedManager |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by vimarshc):
Hey,
Just finished "bisecting the exact commit which introduced the
regression".
Posting my results here.
Git output:
{{{
bdbe50a491ca41e7d4ebace47bfe8abe50a58211 is the first bad commit
commit bdbe50a491ca41e7d4ebace47bfe8abe50a58211
Author: François Freitag <francois...@polyconseil.fr>
Date: Sun Jan 10 17:54:57 2016 +0100
Fixed #25546 -- Prevented duplicate queries with nested
prefetch_related().
:040000 040000 5ad2af54d2f98566bbfdc23e04493e4c98262568
f3488a5f680bc5d4b18249aed299b6abd5f86de9 M django
:040000 040000 59faff56062b7a347956e24315e12beeb5c7d704
c4a686067f06b7056dafffa1013bfe3c018c5e52 M tests
bisect run success
}}}
I'll start trying to locate and fix the bug. I'll request the bug be
assigned to me once I'm confident I can resolve the issue.
I'm attaching a directory with the models and tests inside it.
This directory was inside the tests directory.
I've written three test cases out of which 2 are passing and one is
failing. (the one that is failing is
"test_case_for_nested_and_flat_prefetches")
[https://github.com/vimarshc/django Here] is a link to the django repo I
forked and I've pushed the aforementioned tests here too if someone wants
to get them from here.
They're on the nested-bug-1.10 branch.
--
Ticket URL: <https://code.djangoproject.com/ticket/27554#comment:4>
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.10
(models, ORM) |
Severity: Normal | Resolution:
Keywords: prefetch | Triage Stage: Accepted
AttributeError prefetch_related |
nested RelatedManager |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: mail@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/27554#comment:5>
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.10
(models, ORM) |
Severity: Normal | Resolution:
Keywords: prefetch | Triage Stage: Accepted
AttributeError prefetch_related |
nested RelatedManager |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):
#27744 was a duplicate.
--
Ticket URL: <https://code.djangoproject.com/ticket/27554#comment:6>
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
| Freitag
Type: Bug | Status: assigned
Component: Database layer | Version: 1.10
(models, ORM) |
Severity: Normal | Resolution:
Keywords: prefetch | Triage Stage: Accepted
AttributeError prefetch_related |
nested RelatedManager |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => François Freitag
* status: new => assigned
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/8143 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/27554#comment:7>
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
| Freitag
Type: Bug | Status: assigned
Component: Database layer | Version: 1.10
(models, ORM) |
Severity: Normal | Resolution:
Keywords: prefetch | Triage Stage: Accepted
AttributeError prefetch_related |
nested RelatedManager |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Pascal Briet):
Any update on this issue ?
It prevents us from migrating to 1.10
Thanks !
--
Ticket URL: <https://code.djangoproject.com/ticket/27554#comment:8>
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
| Freitag
Type: Bug | Status: assigned
Component: Database layer | Version: 1.10
(models, ORM) |
Keywords: prefetch | Triage Stage: Ready for
AttributeError prefetch_related | checkin
nested RelatedManager |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* severity: Normal => Release blocker
* stage: Accepted => Ready for checkin
Comment:
The ticket wasn't tagged as a regression (release blocker) so
unfortunately we missed the time to incorporate the fix in 1.10 which is
now receiving only security updates. I'll backport the fix to 1.11.
--
Ticket URL: <https://code.djangoproject.com/ticket/27554#comment:9>
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
| Freitag
Component: Database layer | Version: 1.10
(models, ORM) |
Keywords: prefetch | Triage Stage: Ready for
AttributeError prefetch_related | checkin
nested RelatedManager |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"c0a2b9508aa2c6eaa22e9787010276edca887f0b" c0a2b950]:
{{{
#!CommitTicketReference repository=""
revision="c0a2b9508aa2c6eaa22e9787010276edca887f0b"
Fixed #27554 -- Fixed prefetch_related() crash when fetching relations in
nested Prefetches.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27554#comment:10>
Django
(AttributeError on RelatedManager)
-------------------------------------+-------------------------------------
| Freitag
Type: Bug | Status: closed
Component: Database layer | Version: 1.10
(models, ORM) |
Severity: Release blocker | Resolution: fixed
Keywords: prefetch | Triage Stage: Ready for
AttributeError prefetch_related | checkin
nested RelatedManager |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tim Graham <timograham@…>):
In [changeset:"c679ac04499798e366ca594f97e4f95d3da3682f" c679ac04]:
{{{
#!CommitTicketReference repository=""
revision="c679ac04499798e366ca594f97e4f95d3da3682f"
[1.11.x] Fixed #27554 -- Fixed prefetch_related() crash when fetching
relations in nested Prefetches.
Backport of c0a2b9508aa2c6eaa22e9787010276edca887f0b from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27554#comment:11>