[Django] #24615: ordering queryset by distance is broken
Django
-----------------------------+--------------------
Reporter: MattBlack85 | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.8
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------
I was ordering my queryset by distance after calling .distance() on it, it
was working well with django 1.7 but after upgrading to django 1.8 I am
now getting this error:
{{{
File "/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/query.py", line 162, in __iter__
self._fetch_all()
File "/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/query.py", line 965, in _fetch_all
self._result_cache = list(self.iterator())
File "/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/query.py", line 1217, in iterator
for row in compiler.results_iter():
File "/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/sql/compiler.py", line 783, in results_iter
results = self.execute_sql(MULTI)
File "/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/sql/compiler.py", line 818, in execute_sql
sql, params = self.as_sql()
File "/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/sql/compiler.py", line 367, in as_sql
extra_select, order_by, group_by = self.pre_sql_setup()
File "/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/sql/compiler.py", line 49, in pre_sql_setup
order_by = self.get_order_by()
File "/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/sql/compiler.py", line 276, in get_order_by
field, self.query.get_meta(), default_order=asc))
File "/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/sql/compiler.py", line 552, in
find_ordering_name
if field.rel and path and opts.ordering and name != field.attname:
AttributeError: 'DistanceField' object has no attribute 'rel'
}}}
Is there anything I missed upgrading to 1.8?
--
Ticket URL: <https://code.djangoproject.com/ticket/24615>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: MattBlack85 | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.8
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
It seems likely to be a bug. Could you write a regression test for
Django's test suite or at least tell us how we can reproduce the error?
--
Ticket URL: <https://code.djangoproject.com/ticket/24615#comment:1>
Django
Reporter: MattBlack85 | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.8
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by MattBlack85):
After digging a little bit more into this I discovered this is not
strictly related to ordering, in fact normal ordering still works:
{{{
In [1]: from django.contrib.gis.geos import Point
In [2]: qs = GeoData.objects.all()
In [3]: nqs = qs.distance(Point(3, 3)).order_by('distance')
In [4]: nqs[0].distance
Out[5]: Distance(m=157177.768212)
}}}
but when iterating the queryset ordered by distance I get the error:
{{{
In [1]: from django.contrib.gis.geos import Point
In [2]: qs = GeoData.objects.all()
In [3]: nqs = qs.distance(Point(3, 3)).order_by('distance')
In [4]: vlqs = nqs.values_list('wisp', flat=True)
In [5]: id_list = [id for id in vlqs]
---------------------------------------------------------------------------
AttributeError Traceback (most recent call
last)
<ipython-input-5-7fa41993bc21> in <module>()
----> 1 id_list = [id for id in vlqs]
/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/query.pyc in __iter__(self)
160 - Responsible for turning the rows into model
objects.
161 """
--> 162 self._fetch_all()
163 return iter(self._result_cache)
164
/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/query.pyc in _fetch_all(self)
963 def _fetch_all(self):
964 if self._result_cache is None:
--> 965 self._result_cache = list(self.iterator())
966 if self._prefetch_related_lookups and not
self._prefetch_done:
967 self._prefetch_related_objects()
/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/query.pyc in iterator(self)
1215 compiler = self.query.get_compiler(self.db)
1216 if self.flat and len(self._fields) == 1:
-> 1217 for row in compiler.results_iter():
1218 yield row[0]
1219 elif not self.query.extra_select and not
self.query.annotation_select:
/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/sql/compiler.pyc in results_iter(self, results)
781 converters = None
782 if results is None:
--> 783 results = self.execute_sql(MULTI)
784 fields = [s[0] for s in self.select[0:self.col_count]]
785 converters = self.get_converters(fields)
/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/sql/compiler.pyc in execute_sql(self,
result_type)
816 result_type = NO_RESULTS
817 try:
--> 818 sql, params = self.as_sql()
819 if not sql:
820 raise EmptyResultSet
/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/sql/compiler.pyc in as_sql(self, with_limits,
with_col_aliases, subquery)
365 refcounts_before = self.query.alias_refcount.copy()
366 try:
--> 367 extra_select, order_by, group_by =
self.pre_sql_setup()
368 if with_limits and self.query.low_mark ==
self.query.high_mark:
369 return '', ()
/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/sql/compiler.pyc in pre_sql_setup(self)
47 """
48 self.setup_query()
---> 49 order_by = self.get_order_by()
50 extra_select = self.get_extra_select(order_by,
self.select)
51 group_by = self.get_group_by(self.select + extra_select,
order_by)
/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/sql/compiler.pyc in get_order_by(self)
274 # '-field1__field2__field', etc.
275 order_by.extend(self.find_ordering_name(
--> 276 field, self.query.get_meta(),
default_order=asc))
277 else:
278 if col not in self.query.extra_select:
/home/matt/repos/uwncom/lib/python2.7/site-
packages/django/db/models/sql/compiler.pyc in find_ordering_name(self,
name, opts, alias, default_order, already_seen)
550 # append the default ordering for that model unless the
attribute name
551 # of the field is specified.
--> 552 if field.rel and path and opts.ordering and name !=
field.attname:
553 # Firstly, avoid infinite loops.
554 if not already_seen:
AttributeError: 'DistanceField' object has no attribute 'rel'
}}}
the model I use is very simple:
{{{
import uuid
from django.contrib.gis.db import models
class GeoData(models.Model):
id = UUIDField(primary_key=True, editable=False, default=uuid.uuid4)
coverage = models.MultiPolygonField(null=True, blank=True)
objects = models.GeoManager()
def __unicode__(self):
return "%s" % self.coverage # pragma: no cover
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24615#comment:2>
Django
Reporter: MattBlack85 | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.8
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by miki725):
* cc: miroslav@… (added)
* severity: Normal => Release blocker
* stage: Unreviewed => Accepted
Comment:
Was able to reproduce the same traceback in Djagno1.8. Django1.7 this
passes. Also Django1.9 (master branch) traceback is different. Attached is
the patch with the test which reproduces the issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/24615#comment:3>
Django
Reporter: MattBlack85 | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.8
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by miki725):
* Attachment "ticket_24615.diff" added.
reproduction test case
Django
Reporter: MattBlack85 | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.8
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
Bisected the issue to 0c7633178fa9410f102e4708cef979b873bccb76
Anssi, does the traceback give you a hint about a possible
cause/resolution?
--
Ticket URL: <https://code.djangoproject.com/ticket/24615#comment:4>
Django
Reporter: MattBlack85 | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.8
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
Also note if we simply add a `rel = None` attribute to geometry
`BaseField`, the same `find_ordering_name` method fails a bit later with:
{{{
File "/home/claude/virtualenvs/djangogit/local/lib/python2.7/site-
packages/django/db/models/sql/compiler.py", line 261, in get_order_by
default_order=asc))
File "/home/claude/virtualenvs/djangogit/local/lib/python2.7/site-
packages/django/db/models/sql/compiler.py", line 540, in
find_ordering_name
return [(t.get_col(alias), order, False) for t in targets]
AttributeError: 'RawSQL' object has no attribute 'get_col'
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24615#comment:5>
Django
Reporter: MattBlack85 | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.8
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by akaariai):
I'm a bit busy right now. I'll take a look by Thursday.
--
Ticket URL: <https://code.djangoproject.com/ticket/24615#comment:6>
Django
Reporter: MattBlack85 | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.8
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by akaariai):
* stage: Accepted => Ready for checkin
Comment:
The problem turns out to be that order_by references to expressions which
are masked out of the SELECT clause (by .values() for example) weren't
resolved correctly.
--
Ticket URL: <https://code.djangoproject.com/ticket/24615#comment:7>
Django
Reporter: MattBlack85 | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.8
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by akaariai):
* needs_better_patch: 0 => 1
* stage: Ready for checkin => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/24615#comment:8>
Django
---------------------------------+------------------------------------
Reporter: MattBlack85 | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.8
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
--
Ticket URL: <https://code.djangoproject.com/ticket/24615#comment:9>
Django
Reporter: MattBlack85 | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 1.8
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by akaariai):
* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/24615#comment:10>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: GIS | Version: 1.8
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz <claude@…>):
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"fb5c7748daa25a40bfc62834acb5731c71a6b713" fb5c7748]:
{{{
#!CommitTicketReference repository=""
revision="fb5c7748daa25a40bfc62834acb5731c71a6b713"
Fixed #24615 -- ordering by expression not part of SELECT
Fixed queries where an expression was used in order_by() but the
expression wasn't in the query's select clause (for example the
expression could be masked by .values() call)
Thanks to Trac alias MattBlack85 for the report.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24615#comment:11>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: GIS | Version: 1.8
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Claude Paroz <claude@…>):
In [changeset:"70ff455a35066a1b6e2db03f211f56c7b38285eb" 70ff455]:
{{{
#!CommitTicketReference repository=""
revision="70ff455a35066a1b6e2db03f211f56c7b38285eb"
[1.8.x] Fixed #24615 -- ordering by expression not part of SELECT
Fixed queries where an expression was used in order_by() but the
expression wasn't in the query's select clause (for example the
expression could be masked by .values() call)
Thanks to Trac alias MattBlack85 for the report.
Backport of fb5c7748da from master.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24615#comment:12>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: GIS | Version: 1.8
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by MattBlack85):
Awesome, thanks guys!
--
Ticket URL: <https://code.djangoproject.com/ticket/24615#comment:13>