[Django] #29908: Blog access in b.entry_set.all()[i].blog should not hit the database

[Django]

#29908: Blog access in b.entry_set.all()[i].blog should not hit the database
-------------------------------------+-------------------------------------
Reporter: Carsten | Owner: nobody
Fuchs |
Type: Bug | Status: new
Component: Database | Version: 1.11
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Using the example models Blog and Entry from
https://docs.djangoproject.com/en/2.1/topics/db/queries/#related-objects,
this code:

{{{
#!python
b = Blog.objects.get(name="...")
for e in b.entry_set.all():
print(e.blog) # e.blog == b
}}}
should not trigger an access to the database, because `e.blog == b`.
Instead, for ''each'' `e`, the access to `e.blog` unexpectedly causes a
query to fetch the blog object.

(Originally posted at https://groups.google.com/d/msg/django-
users/Jwk9yd8Mikc/DhaEf8I5BgAJ)

--
Ticket URL: <https://code.djangoproject.com/ticket/29908>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#29908: Blog access in b.entry_set.all()[i].blog should not hit the database
-------------------------------------+-------------------------------------

Reporter: Carsten Fuchs | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham):

This test doesn't fail for me. Can you provide a failing test?
{{{
blog = Blog.objects.create(name='Beatles Blog')
Entry.objects.create(blog=blog)
Entry.objects.create(blog=blog)
with self.assertNumQueries(1):
for e in blog.entry_set.all():
self.assertEqual(e.blog.name, 'Beatles Blog')
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/29908#comment:1>

[Django]

#29908: Blog access in b.entry_set.all()[i].blog should not hit the database
-------------------------------------+-------------------------------------

Reporter: Carsten Fuchs | Owner: nobody
Type: Bug | Status: new

Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Carsten Fuchs):

Hi Tim,

it seems that this is triggered when the ForeignKey's `to_field` parameter
is used.
The test fails with the following model `EntryB`:

{{{
#!python
from django.db import models

class Blog(models.Model):
key = models.CharField(max_length=10, unique=True)
name = models.CharField(max_length=100)

def __str__(self):
return self.name

class EntryA(models.Model):
blog = models.ForeignKey(Blog, on_delete=models.CASCADE)
headline = models.CharField(max_length=255)

class EntryB(models.Model):
blog = models.ForeignKey(Blog, on_delete=models.CASCADE,
to_field='key') # this triggers the problem
headline = models.CharField(max_length=255)
}}}

Test cases:
{{{
#!python
from django.test import TestCase
from TestApp.models import Blog, EntryA, EntryB

class TestBlogs(TestCase):

def test_A(self):

blog = Blog.objects.create(name='Beatles Blog')

EntryA.objects.create(blog=blog)
EntryA.objects.create(blog=blog)

with self.assertNumQueries(1): # succeeds
for e in blog.entrya_set.all():
self.assertEqual(e.blog.name, 'Beatles Blog')

def test_B(self):

blog = Blog.objects.create(name='Beatles Blog')

EntryB.objects.create(blog=blog)
EntryB.objects.create(blog=blog)

with self.assertNumQueries(1): # This fails!
for e in blog.entryb_set.all():
self.assertEqual(e.blog.name, 'Beatles Blog')
}}}

Test results:
{{{
$ ./manage.py test
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.F
======================================================================
FAIL: test_B (TestApp.tests.TestBlogs)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/carsten/xxxTestProject/TestApp/tests.py", line 23, in test_B
self.assertEqual(e.blog.name, 'Beatles Blog')
File "/home/carsten/.virtualenvs/Zeiterfassung/lib/python3.6/site-
packages/django/test/testcases.py", line 88, in __exit__
query['sql'] for query in self.captured_queries
AssertionError: 3 != 1 : 3 queries executed, 1 expected
Captured queries were:
SELECT "TestApp_entryb"."id", "TestApp_entryb"."blog_id",
"TestApp_entryb"."headline" FROM "TestApp_entryb" WHERE
"TestApp_entryb"."blog_id" = ''
SELECT "TestApp_blog"."id", "TestApp_blog"."key", "TestApp_blog"."name"
FROM "TestApp_blog" WHERE "TestApp_blog"."key" = ''
SELECT "TestApp_blog"."id", "TestApp_blog"."key", "TestApp_blog"."name"
FROM "TestApp_blog" WHERE "TestApp_blog"."key" = ''

----------------------------------------------------------------------
Ran 2 tests in 0.003s

FAILED (failures=1)
Destroying test database for alias 'default'...
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/29908#comment:2>

[Django]

#29908: Foreign key isn't set on object after related set access if ForeignKey uses
to_field

-------------------------------------+-------------------------------------
Reporter: Carsten Fuchs | Owner: nobody

Type: Bug | Status: new

Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tim Graham):

* stage: Unreviewed => Accepted

Comment:

Confirmed the issue on master at 3d4d0a25b299a97314582156a0d63d939662d310.

--
Ticket URL: <https://code.djangoproject.com/ticket/29908#comment:3>

[Django]

#29908: Foreign key isn't set on object after related set access if ForeignKey uses
to_field
-------------------------------------+-------------------------------------
Reporter: Carsten Fuchs | Owner: nobody

Type: Bug | Status: new

Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Simon Charette):

* needs_better_patch: 0 => 1
* has_patch: 0 => 1

Comment:

https://github.com/django/django/pull/10595

--
Ticket URL: <https://code.djangoproject.com/ticket/29908#comment:4>

[Django]

#29908: Foreign key isn't set on object after related set access if ForeignKey uses
to_field
-------------------------------------+-------------------------------------
Reporter: Carsten Fuchs | Owner: nobody

Type: Bug | Status: new

Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Simon Charette):

* needs_better_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/29908#comment:5>

[Django]

#29908: Foreign key isn't set on object after related set access if ForeignKey uses
to_field
-------------------------------------+-------------------------------------
Reporter: Carsten Fuchs | Owner: nobody

Type: Bug | Status: closed

Component: Database layer | Version: 1.11
(models, ORM) |

Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tim Graham <timograham@…>):

* status: new => closed
* resolution: => fixed

Comment:

In [changeset:"75dfa92a05c7161edd0ba7bc9ceab9b54d3383db" 75dfa92a]:
{{{
#!CommitTicketReference repository=""
revision="75dfa92a05c7161edd0ba7bc9ceab9b54d3383db"
Fixed #29908 -- Fixed setting of foreign key after related set access if
ForeignKey uses to_field.

Adjusted known related objects handling of target fields which relies on
from and to_fields and has the side effect of fixing a bug bug causing
N+1 queries when using reverse foreign objects.

Thanks Carsten Fuchs for the report.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/29908#comment:6>

[Django]

#29908: Foreign key isn't set on object after related set access if ForeignKey uses
to_field
-------------------------------------+-------------------------------------
Reporter: Carsten Fuchs | Owner: nobody
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette <charette.s@…>):

In [changeset:"0cf85e6b074794ac91857aa097f0b3dc3e6d9468" 0cf85e6b]:
{{{
#!CommitTicketReference repository=""
revision="0cf85e6b074794ac91857aa097f0b3dc3e6d9468"
Refs #29908 -- Optimized known related objects assignment.

Since CPython implements a C level attrgetter(*attrs) it even outperforms
the
most common case of a single known related object since the resulting
attribute
values tuple is built in C.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/29908#comment:7>