[Django] #24873: Error while using Prefech on more than two levels

[Django]

#24873: Error while using Prefech on more than two levels
-------------------------------------+-------------------------------------
Reporter: Gagaro | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) | Keywords: Prefetch
Severity: Normal | prefetch_related
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Given the following models:

{{{
class A(models.Model):
pass

class B(models.Model):
a = models.ForeignKey(A, related_name='b')

class C(models.Model):
b = models.ForeignKey(B, related_name='c')

class D(models.Model):
c = models.ForeignKey(C, related_name='d')
}}}

And the following tests:

{{{
class PrefetchTest(TestCase):
def setUp(self):
a = A.objects.create()
b = B.objects.create(a=a)
c = C.objects.create(b=b)
D.objects.create(c=c)

def test_with_prefetch(self):
c = C.objects.prefetch_related(Prefetch('d'))
b = B.objects.prefetch_related(Prefetch('c', queryset=c))
a = A.objects.prefetch_related(Prefetch('b', queryset=b))
list(a)

def test_without_prefetch(self):
c = C.objects.prefetch_related('d')
b = B.objects.prefetch_related(Prefetch('c', queryset=c))
a = A.objects.prefetch_related(Prefetch('b', queryset=b))
list(a)
}}}

The first test will fail while the second will succeed. The backtrace with
Django 1.8 is:

{{{
Traceback (most recent call last):
File
"/home/gagaro/django/modules/test/django_test/models_test/tests.py", line
19, in test_with_prefetch
list(a)
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 162, in __iter__
self._fetch_all()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 967, in _fetch_all
self._prefetch_related_objects()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 591, in
_prefetch_related_objects
prefetch_related_objects(self._result_cache,
self._prefetch_related_lookups)
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 1516, in
prefetch_related_objects
obj_list, additional_lookups = prefetch_one_level(obj_list,
prefetcher, lookup, level)
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 1615, in prefetch_one_level
prefetcher.get_prefetch_queryset(instances,
lookup.get_current_queryset(level)))
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/fields/related.py", line 729, in
get_prefetch_queryset
for rel_obj in queryset:
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 162, in __iter__
self._fetch_all()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 967, in _fetch_all
self._prefetch_related_objects()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 591, in
_prefetch_related_objects
prefetch_related_objects(self._result_cache,
self._prefetch_related_lookups)
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 1505, in
prefetch_related_objects
(through_attr, first_obj.__class__.__name__, lookup.prefetch_through))
AttributeError: Cannot find 'c' on C object, 'c__d' is an invalid
parameter to prefetch_related()
}}}

I tested with Django 1.7, 1.8, 1.8.2 and master (commit
24718b7dccd64dfa118fe8136e7b175babec679b).

--
Ticket URL: <https://code.djangoproject.com/ticket/24873>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#24873: Error while using Prefech on more than two levels
-------------------------------------+-------------------------------------
Reporter: Gagaro | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) |

Severity: Normal | Resolution:
Keywords: Prefetch | Triage Stage: Accepted
prefetch_related |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by timgraham):

* needs_docs: => 0
* needs_better_patch: => 0
* needs_tests: => 0
* stage: Unreviewed => Accepted

Comment:

I don't have much experience with `prefetch_related()` but as far as I
understand, the two tests should work identically.

--
Ticket URL: <https://code.djangoproject.com/ticket/24873#comment:1>

[Django]

#24873: Error while using Prefech on more than two levels
-------------------------------------+-------------------------------------
Reporter: Gagaro | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) |

Severity: Normal | Resolution:
Keywords: Prefetch | Triage Stage: Accepted
prefetch_related |
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Description changed by Gagaro:

Old description:

> Given the following models:
>

> {{{
> class A(models.Model):
> pass
>

New description:

Given the following models:

{{{
# Foreign Keys

class A(models.Model):
pass

class B(models.Model):
a = models.ForeignKey(A, related_name='b')

class C(models.Model):
b = models.ForeignKey(B, related_name='c')

class D(models.Model):
c = models.ForeignKey(C, related_name='d')

#ManyToMany Fields

class E(models.Model):
pass

class F(models.Model):
e = models.ManyToManyField(E, related_name='f')

class G(models.Model):
f = models.ManyToManyField(F, related_name='g')

class H(models.Model):
g = models.ManyToManyField(G, related_name='h')
}}}

And the following tests:

{{{
class PrefetchForeignKeyTest(TestCase):

def setUp(self):
a = A.objects.create()
b = B.objects.create(a=a)
c = C.objects.create(b=b)
D.objects.create(c=c)

def test_with_prefetch(self):
c = C.objects.prefetch_related(Prefetch('d'))
b = B.objects.prefetch_related(Prefetch('c', queryset=c))
a = A.objects.prefetch_related(Prefetch('b', queryset=b))
list(a)

a.first().b.first().c.first().d.first()

def test_without_prefetch(self):
c = C.objects.prefetch_related('d')
b = B.objects.prefetch_related(Prefetch('c', queryset=c))
a = A.objects.prefetch_related(Prefetch('b', queryset=b))
list(a)

a.first().b.first().c.first().d.first()

class PrefetchManyTest(TestCase):
def setUp(self):
e = E.objects.create()
f = F.objects.create()
g = G.objects.create()
h = H.objects.create()
f.e.add(e)
g.f.add(f)
h.g.add(g)

def test_with_prefetch(self):
g = G.objects.prefetch_related(Prefetch('h'))
f = F.objects.prefetch_related(Prefetch('g', queryset=g))
e = E.objects.prefetch_related(Prefetch('f', queryset=f))
list(e)
e.first().f.first().g.first().h.first()

def test_without_prefetch(self):
g = G.objects.prefetch_related('h')
f = F.objects.prefetch_related(Prefetch('g', queryset=g))
e = E.objects.prefetch_related(Prefetch('f', queryset=f))
list(e)
e.first().f.first().g.first().h.first()
}}}

The tests will fail. The backtrace with master is:

{{{
======================================================================
ERROR: test_with_prefetch (models_test.tests.PrefetchManyTest)
----------------------------------------------------------------------

Traceback (most recent call last):
File
"/home/gagaro/django/modules/test/django_test/models_test/tests.py", line

49, in test_with_prefetch
e.first().f.first().g.first().h.first()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 543, in first
objects = list((self if self.ordered else self.order_by('pk'))[:1])
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 258, in __iter__
self._fetch_all()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 1065, in _fetch_all
self._prefetch_related_objects()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 649, in

_prefetch_related_objects
prefetch_related_objects(self._result_cache,
self._prefetch_related_lookups)
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-

packages/django/db/models/query.py", line 1426, in
prefetch_related_objects
(through_attr, first_obj.__class__.__name__, lookup.prefetch_through))
AttributeError: Cannot find 'f' on F object, 'f__f__g' is an invalid
parameter to prefetch_related()

======================================================================
ERROR: test_without_prefetch (models_test.tests.PrefetchManyTest)
----------------------------------------------------------------------

Traceback (most recent call last):
File
"/home/gagaro/django/modules/test/django_test/models_test/tests.py", line

56, in test_without_prefetch
e.first().f.first().g.first().h.first()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 543, in first
objects = list((self if self.ordered else self.order_by('pk'))[:1])
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 258, in __iter__
self._fetch_all()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 1065, in _fetch_all
self._prefetch_related_objects()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 649, in

_prefetch_related_objects
prefetch_related_objects(self._result_cache,
self._prefetch_related_lookups)
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-

packages/django/db/models/query.py", line 1426, in
prefetch_related_objects
(through_attr, first_obj.__class__.__name__, lookup.prefetch_through))
AttributeError: Cannot find 'f' on F object, 'f__f__g' is an invalid
parameter to prefetch_related()

======================================================================
ERROR: test_with_prefetch (models_test.tests.PrefetchForeignKeyTest)
----------------------------------------------------------------------

Traceback (most recent call last):
File
"/home/gagaro/django/modules/test/django_test/models_test/tests.py", line

23, in test_with_prefetch
list(a)
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 258, in __iter__
self._fetch_all()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 1065, in _fetch_all
self._prefetch_related_objects()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 649, in

_prefetch_related_objects
prefetch_related_objects(self._result_cache,
self._prefetch_related_lookups)
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-

packages/django/db/models/query.py", line 1437, in

prefetch_related_objects
obj_list, additional_lookups = prefetch_one_level(obj_list,
prefetcher, lookup, level)
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-

packages/django/db/models/query.py", line 1536, in prefetch_one_level
prefetcher.get_prefetch_queryset(instances,
lookup.get_current_queryset(level)))
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/fields/related.py", line 753, in

get_prefetch_queryset
for rel_obj in queryset:
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-

packages/django/db/models/query.py", line 258, in __iter__
self._fetch_all()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 1065, in _fetch_all
self._prefetch_related_objects()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 649, in

_prefetch_related_objects
prefetch_related_objects(self._result_cache,
self._prefetch_related_lookups)
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-

packages/django/db/models/query.py", line 1426, in

prefetch_related_objects
(through_attr, first_obj.__class__.__name__, lookup.prefetch_through))
AttributeError: Cannot find 'c' on C object, 'c__d' is an invalid
parameter to prefetch_related()

======================================================================
ERROR: test_without_prefetch (models_test.tests.PrefetchForeignKeyTest)
----------------------------------------------------------------------

Traceback (most recent call last):
File
"/home/gagaro/django/modules/test/django_test/models_test/tests.py", line

31, in test_without_prefetch
a.first().b.first().c.first().d.first()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 543, in first
objects = list((self if self.ordered else self.order_by('pk'))[:1])
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 258, in __iter__
self._fetch_all()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 1065, in _fetch_all
self._prefetch_related_objects()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 649, in

_prefetch_related_objects
prefetch_related_objects(self._result_cache,
self._prefetch_related_lookups)
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-

packages/django/db/models/query.py", line 1437, in

prefetch_related_objects
obj_list, additional_lookups = prefetch_one_level(obj_list,
prefetcher, lookup, level)
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-

packages/django/db/models/query.py", line 1536, in prefetch_one_level
prefetcher.get_prefetch_queryset(instances,
lookup.get_current_queryset(level)))
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/fields/related.py", line 753, in

get_prefetch_queryset
for rel_obj in queryset:
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-

packages/django/db/models/query.py", line 258, in __iter__
self._fetch_all()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 1065, in _fetch_all
self._prefetch_related_objects()
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-
packages/django/db/models/query.py", line 649, in

_prefetch_related_objects
prefetch_related_objects(self._result_cache,
self._prefetch_related_lookups)
File "/home/gagaro/.virtualenvs/django-test/local/lib/python2.7/site-

packages/django/db/models/query.py", line 1426, in
prefetch_related_objects
(through_attr, first_obj.__class__.__name__, lookup.prefetch_through))
AttributeError: Cannot find 'b' on B object, 'b__c' is an invalid
parameter to prefetch_related()
}}}

I tested with Django 1.7, 1.8, 1.8.2 and master (commit
24718b7dccd64dfa118fe8136e7b175babec679b).

--

--
Ticket URL: <https://code.djangoproject.com/ticket/24873#comment:2>

[Django]

#24873: Error while using Prefech on more than two levels
-------------------------------------+-------------------------------------
Reporter: Gagaro | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) |

Severity: Normal | Resolution:
Keywords: Prefetch | Triage Stage: Accepted
prefetch_related |
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Gagaro):

I did some more testing. All the tests actually fail when I try to access
the objects. I also added tests with m2m (which only fail when the objects
are accessed).

--
Ticket URL: <https://code.djangoproject.com/ticket/24873#comment:3>

[Django]

#24873: Error while using Prefech on more than two levels
-------------------------------------+-------------------------------------
Reporter: Gagaro | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) |

Severity: Normal | Resolution:
Keywords: Prefetch | Triage Stage: Accepted
prefetch_related |
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Gagaro):

I found and corrected the bug. I'm working on the pull request (it may
take a little bit of time as it is my first and I still need to add
regression tests).

--
Ticket URL: <https://code.djangoproject.com/ticket/24873#comment:4>

[Django]

#24873: Error while using Prefech on more than two levels
-------------------------------------+-------------------------------------

Reporter: Gagaro | Owner: Gagaro
Type: Bug | Status: assigned

Component: Database layer | Version: master
(models, ORM) |

Severity: Normal | Resolution:
Keywords: Prefetch | Triage Stage: Accepted
prefetch_related |
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Gagaro):

* owner: nobody => Gagaro
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/24873#comment:5>

[Django]

#24873: Error while using Prefech on more than two levels
-------------------------------------+-------------------------------------

Reporter: Gagaro | Owner: Gagaro
Type: Bug | Status: assigned

Component: Database layer | Version: master
(models, ORM) |

Severity: Normal | Resolution:
Keywords: Prefetch | Triage Stage: Accepted
prefetch_related |
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Gagaro):

Pull request: https://github.com/django/django/pull/4723

--
Ticket URL: <https://code.djangoproject.com/ticket/24873#comment:6>

[Django]

#24873: Error while using Prefech on more than two levels
-------------------------------------+-------------------------------------

Reporter: Gagaro | Owner: Gagaro
Type: Bug | Status: assigned

Component: Database layer | Version: master
(models, ORM) |

Severity: Normal | Resolution:
Keywords: Prefetch | Triage Stage: Ready for
prefetch_related | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by timgraham):

* has_patch: 0 => 1
* stage: Accepted => Ready for checkin

Comment:

Would be nice for someone familiar with the prefetch stuff to double check
this.

--
Ticket URL: <https://code.djangoproject.com/ticket/24873#comment:7>

[Django]

#24873: Error while using Prefech on more than two levels
-------------------------------------+-------------------------------------

Reporter: Gagaro | Owner: Gagaro
Type: Bug | Status: closed

Component: Database layer | Version: master
(models, ORM) |

Severity: Normal | Resolution: fixed

Keywords: Prefetch | Triage Stage: Ready for
prefetch_related | checkin
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tim Graham <timograham@…>):

* status: assigned => closed
* resolution: => fixed

Comment:

In [changeset:"74261bc593bcf9411e7e25a4731dd9471499819e" 74261bc5]:
{{{
#!CommitTicketReference repository=""
revision="74261bc593bcf9411e7e25a4731dd9471499819e"
Fixed #24873 -- Prevented nested Prefetch objects from being overwritten.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/24873#comment:8>