[Django] #31910: Django geo model aggregate union fails with distinct
Django
-----------------------------------------+------------------------
Reporter: Eran Keydar | Owner: nobody
Type: Uncategorized | Status: new
Component: GIS | Version: 3.1
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
The following code works ok on django 2.7, and fails on django 3 and 3.1
(without the distinct if works ok)
{{{
class PolyModel(models.Model):
name = models.CharField(max_length=50)
geom = models.PolygonField()
def run_example():
PolyModel.objects.all().delete()
for x in range(1, 10):
PolyModel.objects.create(
name="poly_{}".format(x),
geom=Point(34+x, 34+x).buffer(0.01)
)
print(PolyModel.objects.distinct().aggregate(u=Union('geom')))
}}}
Output:
{{{
<ipython-input-5-c368ff40e50b> in <module>
----> 1 PolyModel.objects.distinct().aggregate(u=Union('geom'))
~/.virtualenvs/bug1/lib/python3.6/site-packages/django/db/models/query.py
in aggregate(self, *args, **kwargs)
396 if not query.annotations[alias].contains_aggregate:
397 raise TypeError("%s is not an aggregate
expression" % alias)
--> 398 return query.get_aggregation(self.db, kwargs)
399
400 def count(self):
~/.virtualenvs/bug1/lib/python3.6/site-
packages/django/db/models/sql/query.py in get_aggregation(self, using,
added_aggregate_names)
503
504 converters =
compiler.get_converters(outer_query.annotation_select.values())
--> 505 result = next(compiler.apply_converters((result,),
converters))
506
507 return dict(zip(outer_query.annotation_select, result))
~/.virtualenvs/bug1/lib/python3.6/site-
packages/django/db/models/sql/compiler.py in apply_converters(self, rows,
converters)
1096 value = row[pos]
1097 for converter in convs:
-> 1098 value = converter(value, expression,
connection)
1099 row[pos] = value
1100 yield row
~/.virtualenvs/bug1/lib/python3.6/site-
packages/django/contrib/gis/db/backends/postgis/operations.py in
converter(value, expression, connection)
385
386 def converter(value, expression, connection):
--> 387 return None if value is None else
GEOSGeometryBase(read(value), geom_class)
388 return converter
389
~/.virtualenvs/bug1/lib/python3.6/site-
packages/django/contrib/gis/geos/prototypes/io.py in read(self, wkb)
151 return wkb_reader_read(self.ptr, wkb_s, len(wkb_s))
152 elif isinstance(wkb, (bytes, str)):
--> 153 return wkb_reader_read_hex(self.ptr, wkb, len(wkb))
154 else:
155 raise TypeError
~/.virtualenvs/bug1/lib/python3.6/site-
packages/django/contrib/gis/geos/libgeos.py in __call__(self, *args)
150
151 def __call__(self, *args):
--> 152 return self.func(*args)
153
154 @cached_property
~/.virtualenvs/bug1/lib/python3.6/site-
packages/django/contrib/gis/geos/prototypes/threadsafe.py in
__call__(self, *args)
45 # Call the threaded GEOS routine with the pointer of the
context handle
46 # as the first argument.
---> 47 return self.cfunc(self.thread_context.handle.ptr, *args)
48
49 def __str__(self):
ArgumentError: argument 3: <class 'TypeError'>: wrong type
}}}
Looking at the generated query, gives this:
{{{
{'sql': 'SELECT ST_Union("__col1") FROM (SELECT DISTINCT
"core_polymodel"."id" AS Col1, "core_polymodel"."name" AS Col2,
"core_polymodel"."geom" AS Col3, "core_polymodel"."geom" AS "__col1" FROM
"core_polymodel") subquery',
'time': '0.001'}
}}}
So it seems that the union is done on the wrong column.
Thanks
--
Ticket URL: <https://code.djangoproject.com/ticket/31910>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Eran Keydar | Owner: nobody
Type: Uncategorized | Status: new
Component: GIS | Version: 3.1
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Eran Keydar:
Old description:
New description:
The following code works ok on django 2.7, and fails on django 3 and 3.1
(without the distinct if works ok)
Database is postgis
Output:
}}}
}}}
Thanks
--
--
Ticket URL: <https://code.djangoproject.com/ticket/31910#comment:1>
Django
Reporter: Eran Keydar | Owner: nobody
Type: Uncategorized | Status: new
Component: GIS | Version: 3.1
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Eran Keydar:
Old description:
> The following code works ok on django 2.7, and fails on django 3 and 3.1
> (without the distinct if works ok)
>
> Database is postgis
>
New description:
The following code works ok on django 2.7, and fails on django 3 and 3.1
(without the distinct if works ok)
Database is postgis
Output:
}}}
Thanks
--
--
Ticket URL: <https://code.djangoproject.com/ticket/31910#comment:2>
Django
Reporter: Eran Keydar | Owner: nobody
Type: Uncategorized | Status: new
Component: GIS | Version: 3.1
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Old description:
> The following code works ok on django 2.7, and fails on django 3 and 3.1
> (without the distinct if works ok)
>
> Database is postgis
>
> }}}
>
> Thanks
New description:
The following code works ok on django 2.2, and fails on django 3 and 3.1
(without the distinct if works ok)
Database is postgis
Output:
}}}
Thanks
--
Comment (by Eran Keydar):
These are the queries from django 2.2 and django 3.1
--
Ticket URL: <https://code.djangoproject.com/ticket/31910#comment:3>
Django
Reporter: Eran Keydar | Owner: nobody
Type: Uncategorized | Status: new
Component: GIS | Version: 3.1
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Eran Keydar):
Same error with the following expression (ok on 2.2)
{{{
PolyModel.objects.annotate(c=BoundingCircle('geom')).aggregate(u=Union('c'))
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31910#comment:4>
Django
Reporter: Eran Keydar | Owner: Simon
| Charette
Type: Bug | Status: assigned
Component: GIS | Version: 3.1
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Simon Charette):
* owner: nobody => Simon Charette
* status: new => assigned
* type: Uncategorized => Bug
* severity: Normal => Release blocker
* stage: Unreviewed => Accepted
Comment:
Confirmed regression caused by fff5186d3215e0ba06e47090226169f2230786b0
--
Ticket URL: <https://code.djangoproject.com/ticket/31910#comment:5>
Django
Reporter: Eran Keydar | Owner: Simon
| Charette
Type: Bug | Status: assigned
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Simon Charette):
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/31910#comment:6>
Django
Reporter: Eran Keydar | Owner: Simon Charette
Type: Bug | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by felixxm):
* severity: Release blocker => Normal
Comment:
[https://github.com/django/django/pull/13332 PR]
This is a regression introduced in Django 3.0 which is in extended
support, so it's not a release blocker.
--
Ticket URL: <https://code.djangoproject.com/ticket/31910#comment:7>
Django
Reporter: Eran Keydar | Owner: Simon Charette
Type: Bug | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by felixxm):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/31910#comment:8>
Django
Reporter: Eran Keydar | Owner: Simon Charette
Type: Bug | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Simon Charette):
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/31910#comment:9>
Django
Reporter: Eran Keydar | Owner: Simon Charette
Type: Bug | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/31910#comment:10>
Django
Reporter: Eran Keydar | Owner: Simon Charette
Component: GIS | Version: 3.1
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"c2d4926702045e342a668057f0a758eec9db9436" c2d49267]:
{{{
#!CommitTicketReference repository=""
revision="c2d4926702045e342a668057f0a758eec9db9436"
Fixed #31910 -- Fixed crash of GIS aggregations over subqueries.
Regression was introduced by fff5186 but was due a long standing issue.
AggregateQuery was abusing Query.subquery: bool by stashing its
compiled inner query's SQL for later use in its compiler which made
select_format checks for Query.subquery wrongly assume the provide
query was a subquery.
This patch prevents that from happening by using a dedicated
inner_query attribute which is compiled at a later time by
SQLAggregateCompiler.
Moving the inner query's compilation to SQLAggregateCompiler.compile
had the side effect of addressing a long standing issue with
aggregation subquery pushdown which prevented converters from being
run. This is now fixed as the aggregation_regress adjustments
demonstrate.
Refs #25367.
Thanks Eran Keydar for the report.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31910#comment:11>
Django
Reporter: Eran Keydar | Owner: Simon Charette
Type: Bug | Status: closed
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by GitHub <noreply@…>):
In [changeset:"13b6fff11703a694e155b84d41d02822bbc0aaa0" 13b6fff]:
{{{
#!CommitTicketReference repository=""
revision="13b6fff11703a694e155b84d41d02822bbc0aaa0"
Refs #31910 -- Fixed GeoQuerySetTest.test_geoagg_subquery() test on Oracle
18c.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31910#comment:12>