[Django] #31420: Regression when using SimpleLazyObject in a subquery
Django
-------------------------------------+-------------------------------------
Reporter: JEphron | Owner: nobody
Type: Bug | Status: new
Component: Database | Version: 3.0
layer (models, ORM) | Keywords: simplelazyobject,
Severity: Normal | queryset, subquery
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Prior to 35431298226165986ad07e91f9d3aca721ff38ec it was possible to use a
SimpleLazyObject in a queryset as demonstrated below. This appears to be a
regression.
== Models
{{{
from django.contrib.auth.models import User
from django.db import models
class A(models.Model):
pass
class B(models.Model):
a = models.ForeignKey(A, on_delete=models.CASCADE)
class C(models.Model):
owner = models.ForeignKey(User, on_delete=models.CASCADE)
}}}
== TestCase
{{{
from django.contrib.auth.models import User
from django.db.models import OuterRef, Subquery
from django.test import TestCase
from django.utils.functional import SimpleLazyObject
from ..models import A, B, C
class BugTestCase(TestCase):
def test_bug(self):
owner_user = (
B.objects.filter(a=OuterRef("pk"))
.annotate(owner_user=Subquery(C.objects.values("owner")))
.values("owner_user")
)
user = SimpleLazyObject(lambda:
User.objects.create_user("testuser"))
A.objects.annotate(owner_user=Subquery(owner_user)).filter(
owner_user=user
)
}}}
Sorry for the somewhat arbitrary testcase, hopefully it's sufficient to
repro this issue.
== Results
{{{
Traceback (most recent call last):
File
"/Users/u/PycharmProjects/django_debug/dingus/tests/test_bullshit.py",
line 20, in test_bug
owner_user=user
File
"/Users/u/.virtualenvs/django_debug/src/django/django/db/models/query.py",
line 881, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File
"/Users/u/.virtualenvs/django_debug/src/django/django/db/models/query.py",
line 899, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File
"/Users/u/.virtualenvs/django_debug/src/django/django/db/models/sql/query.py",
line 1297, in add_q
clause, _ = self._add_q(q_object, self.used_aliases)
File
"/Users/u/.virtualenvs/django_debug/src/django/django/db/models/sql/query.py",
line 1325, in _add_q
split_subq=split_subq, simple_col=simple_col,
File
"/Users/u/.virtualenvs/django_debug/src/django/django/db/models/sql/query.py",
line 1214, in build_filter
condition = self.build_lookup(lookups, reffed_expression, value)
File
"/Users/u/.virtualenvs/django_debug/src/django/django/db/models/sql/query.py",
line 1123, in build_lookup
lookup = lookup_class(lhs, rhs)
File
"/Users/u/.virtualenvs/django_debug/src/django/django/db/models/lookups.py",
line 20, in __init__
self.rhs = self.get_prep_lookup()
File
"/Users/u/.virtualenvs/django_debug/src/django/django/db/models/lookups.py",
line 70, in get_prep_lookup
return self.lhs.output_field.get_prep_value(self.rhs)
File
"/Users/u/.virtualenvs/django_debug/src/django/django/db/models/fields/__init__.py",
line 968, in get_prep_value
return int(value)
TypeError: int() argument must be a string, a bytes-like object or a
number, not 'SimpleLazyObject'
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31420>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: JEphron | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: simplelazyobject, | Triage Stage:
queryset, subquery | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> class A(models.Model):
> pass
>
New description:
class A(models.Model):
pass
A.objects.annotate(owner_user=Subquery(owner_user)).filter(
owner_user=user
)
File "/Users/u/PycharmProjects/django_debug/foo/tests/test_bug.py", line
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/31420#comment:1>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: simplelazyobject, | Triage Stage:
queryset, subquery | Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> class A(models.Model):
> pass
>
> File "/Users/u/PycharmProjects/django_debug/foo/tests/test_bug.py",
New description:
Prior to 35431298226165986ad07e91f9d3aca721ff38ec it was possible to use a
SimpleLazyObject in a queryset as demonstrated below. This new behavior
class A(models.Model):
pass
A.objects.annotate(owner_user=Subquery(owner_user)).filter(
owner_user=user
)
File "/Users/u/PycharmProjects/django_debug/foo/tests/test_bug.py", line
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/31420#comment:2>
Django
-------------------------------------+-------------------------------------
Reporter: Jordan Ephron | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 3.0
(models, ORM) |
Keywords: simplelazyobject, | Triage Stage: Accepted
queryset, subquery |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Simon Charette (added)
* severity: Normal => Release blocker
* stage: Unreviewed => Accepted
Comment:
I agree that this behavior was changed in
35431298226165986ad07e91f9d3aca721ff38ec, but you can easily create a test
with `SimpleLazyObject()` (which is not a public API) and `Subquery()`
that always fails with `TypeError`, e.g.
{{{
def test_subquery_filter_by_lazy(self):
test_user = SimpleLazyObject(lambda:
get_user_model().objects.get(username='test'))
qs = Company.objects.annotate(
related=Subquery(Employee.objects.filter(lastname=OuterRef('ceo__lastname')).values('user')),
).filter(related=test_user)
self.assertEqual(qs.get(), self.foobar_ltd)
}}}
This is somehow related with nested subqueries.
--
Ticket URL: <https://code.djangoproject.com/ticket/31420#comment:3>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: simplelazyobject, | Triage Stage: Accepted
queryset, subquery |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):
Alright so here's what's happening here.
When the most outer `filter(owner_user=user)` call is made the lookup
logic tries to resolve `owner_user`'s `output_field` in order to
`get_lookup('exact')` on it. `owner_user` points to `Subquery(owner_user)`
and it's `.output_field` is `self.query.output_field`. Since
`.values('owner_user')` refers to `Subquery(C.objects.values('owner'))`
which refers to `self.query.output_field` which refers to a column mapping
to a `Col` referencing `C.owner` the actual outer most
`filter(owner_user=user)` left hand side output field is whatever
`sql.Query.output_field` returns for a selected field.
This happens to be `Col.field`
https://github.com/django/django/blob/89032876f427a77ab4de26493190280377567d1c/django/db/models/sql/query.py#L235-L236
Now `Col.field` is actually `Expression.field` which is actually an alias
for `.output_field` (not sure why) but in order to allow the special
related field lookup conversion of `SimpleLazyObject` to model instance
(e.g. needed for stuff like `.filter(user=request.user)` where
`request.user` is a `SimpleLazyObject`) it's the `Col.target` that should
be used as `output_field`.
{{{#!diff
diff --git a/django/db/models/sql/query.py b/django/db/models/sql/query.py
index 78c4f47b5b..8ad9d139f9 100644
--- a/django/db/models/sql/query.py
+++ b/django/db/models/sql/query.py
@@ -233,7 +233,7 @@ class Query(BaseExpression):
@property
def output_field(self):
if len(self.select) == 1:
- return self.select[0].field
+ return self.select[0].target
elif len(self.annotation_select) == 1:
return
next(iter(self.annotation_select.values())).output_field
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31420#comment:4>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: simplelazyobject, | Triage Stage: Accepted
queryset, subquery |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by felixxm):
Confirmed. Thanks, Simon! It also fixes issue that never worked (see
[https://code.djangoproject.com/ticket/31420#comment:3 comment]).
--
Ticket URL: <https://code.djangoproject.com/ticket/31420#comment:5>
Django
-------------------------------------+-------------------------------------
| Ramezani
Type: Bug | Status: assigned
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: simplelazyobject, | Triage Stage: Accepted
queryset, subquery |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => Hasan Ramezani
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/31420#comment:6>
Django
-------------------------------------+-------------------------------------
Reporter: Jordan Ephron | Owner: Hasan
| Ramezani
Type: Bug | Status: assigned
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: simplelazyobject, | Triage Stage: Accepted
queryset, subquery |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Jordan Ephron):
Thanks for the quick triage folks,
--
Ticket URL: <https://code.djangoproject.com/ticket/31420#comment:7>
Django
-------------------------------------+-------------------------------------
Reporter: Jordan Ephron | Owner: Hasan
| Ramezani
Type: Bug | Status: assigned
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: simplelazyobject, | Triage Stage: Accepted
queryset, subquery |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/31420#comment:8>
Django
-------------------------------------+-------------------------------------
Reporter: Jordan Ephron | Owner: Hasan
| Ramezani
Component: Database layer | Version: 3.0
(models, ORM) |
Keywords: simplelazyobject, | Triage Stage: Accepted
queryset, subquery |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"4237050684427db45ea834fe89d9e11c0520201e" 4237050]:
{{{
#!CommitTicketReference repository=""
revision="4237050684427db45ea834fe89d9e11c0520201e"
Fixed #31420 -- Fixed crash when filtering subquery annotation against a
SimpleLazyObject.
Thanks Simon Charette for the solution and analysis.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31420#comment:9>
Django
-------------------------------------+-------------------------------------
Reporter: Jordan Ephron | Owner: Hasan
| Ramezani
Type: Bug | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Release blocker | Resolution: fixed
Keywords: simplelazyobject, | Triage Stage: Accepted
queryset, subquery |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"22a2e97fc30487af89d7c34b753853e7b510083d" 22a2e97f]:
{{{
#!CommitTicketReference repository=""
revision="22a2e97fc30487af89d7c34b753853e7b510083d"
[3.0.x] Fixed #31420 -- Fixed crash when filtering subquery annotation
against a SimpleLazyObject.
Thanks Simon Charette for the solution and analysis.
Backport of 4237050684427db45ea834fe89d9e11c0520201e from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31420#comment:10>