[Django] #30687: GIS distance lookups fail within subqueries using OuterRef
Django
-------------------------------------+-------------------------------------
Reporter: Andrew | Owner: nobody
Brown |
Type: Bug | Status: new
Component: Database | Version: 2.2
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
I discovered this when trying to make a query of this form:
{{{#!python
from django.db import models
from django.contrib.gis.db.models.fields import PointField, RasterField
class ModelA(models.Model):
pointA = PointField()
class ModelB(models.Model):
pointB = PointField()
def query1():
return ModelA.objects.annotate(
has_value=Exists(ModelB.objects.filter(
pointB__dwithin=(OuterRef("pointA"), 10)
))
).filter(has_value=True)
}}}
This fails with "ValueError: This queryset contains a reference to an
outer query and may only be used in a subquery"
I dug into things, and while I may not fully understand how queries are
processed, I think I've identified the problem in
`Query.resolve_lookup_value()`:
{{{#!python
def resolve_lookup_value(self, value, can_reuse, allow_joins, simple_col):
if hasattr(value, 'resolve_expression'):
kwargs = {'reuse': can_reuse, 'allow_joins': allow_joins}
if isinstance(value, F):
kwargs['simple_col'] = simple_col
value = value.resolve_expression(self, **kwargs)
elif isinstance(value, (list, tuple)):
# The items of the iterable may be expressions and therefore need
# to be resolved independently.
for sub_value in value:
if hasattr(sub_value, 'resolve_expression'):
if isinstance(sub_value, F):
sub_value.resolve_expression(
self, reuse=can_reuse, allow_joins=allow_joins,
simple_col=simple_col,
)
else:
sub_value.resolve_expression(self, reuse=can_reuse,
allow_joins=allow_joins)
return value
}}}
This resolves the value passed in as the rhs of a filter. For single
objects, it calls `resolve_expression()` and then returns the result. But
for multiple objects in a list or tuple, it calls `resolve_expression()`
on each but doesn't return the resolved objects. Rather it returns the
original list.
As a consequence, during the call to filter, the passed in `OuterRef`
object doesn't get resolved to a `ResolvedOuterRef` object, since the
`__dwithin` lookup takes a tuple of `(value, distance)`, triggering that
second code path above. Later during the call to annotate, the `OuterRef`
does resolve to a `ResolvedOuterRef` but when the query is compiled,
`ResolvedOuterRef.as_sql()` is called which raises the error.
I modified `resolve_lookup_value()` to return the list or tuple of
resolved values, and wrote a test for the query above, which I will submit
in a PR shortly.
--
Ticket URL: <https://code.djangoproject.com/ticket/30687>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 2.2
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
PR submitted: https://github.com/django/django/pull/11634
--
Ticket URL: <https://code.djangoproject.com/ticket/30687#comment:1>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* version: 2.2 => master
* stage: Unreviewed => Accepted
Comment:
OK, thanks for the report and the test case Andrew.
--
Ticket URL: <https://code.djangoproject.com/ticket/30687#comment:2>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
Comment:
I left a few comments for improvement but I'm marking as '''RFC''' because
they are really minor and could be addressed by the committer.
Thanks for the patch and tests Andrew, that was a breeze to review.
--
Ticket URL: <https://code.djangoproject.com/ticket/30687#comment:3>
Django
-------------------------------------+-------------------------------------
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"8a281aa7fe76a9da2284f943964a9413697cff1f" 8a281aa7]:
{{{
#!CommitTicketReference repository=""
revision="8a281aa7fe76a9da2284f943964a9413697cff1f"
Fixed #30687 -- Fixed using of OuterRef() expressions in distance lookups.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30687#comment:4>