[Django] #31450: Querying on ArrayField(PointField) returns string instead of list
Django
-------------------------------------+-------------------------------------
Reporter: nsangwan | Owner: (none)
Type: Bug | Status: new
Component: | Version: 3.0
contrib.postgres | Keywords: ArrayField,
Severity: Normal | PointField
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
The following code
{{{
from django.db import models
from django.contrib.postgres.fields import ArrayField
from django.contrib.gis.db.models import PointField
class Foo(models.Model):
bar = ArrayField(models.TextField)
baz = ArrayField(PointField())
}}}
has this wrong behaviour:
{{{
x = Foo.objects.get(id=1)
x.bar # => ['Foo', 'Bar']
x.baz # => '{HexRepresentationofPoint1,HexRepresentationofPoint2}'
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31450>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: ArrayField, | Triage Stage:
PointField | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* Attachment "postgis_tests.zip" added.
Reproduce test case for #31450
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
PointField |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
Comment:
Thanks for the report. Reproduced at
578c03b276e435bcd3ce9eb17b81e85135c2d3f3 with the attached test case.
--
Ticket URL: <https://code.djangoproject.com/ticket/31450#comment:1>
Django
-------------------------------------+-------------------------------------
| Ramezani
Type: Bug | Status: assigned
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: ArrayField, | Triage Stage: Accepted
PointField |
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: (none) => Hasan Ramezani
* needs_better_patch: 0 => 1
* has_patch: 0 => 1
* status: new => assigned
Comment:
[https://github.com/django/django/pull/12732 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/31450#comment:2>
Django
-------------------------------------+-------------------------------------
| Ramezani
Type: Bug | Status: assigned
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: ArrayField, | Triage Stage: Accepted
PointField |
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Hasan Ramezani):
Comment from the PR:
Thanks @Simon for your suggestion.
I've tried to create and register a new array type for `geometry` based on
your suggestion at #8453. here is the code:
{{{
def get_geometry_oids(connection_alias):
with connections[connection_alias].cursor() as cursor:
cursor.execute("SELECT typarray FROM pg_type WHERE typname =
'geometry'")
return tuple(row[0] for row in cursor)
...
geometry_oids = get_geometry_oids(connection.alias)
new_type = psycopg2.extensions.new_type(geometry_oids, "geometry",
caster) # caster is function to cast hex to geometry object
array_type = psycopg2.extensions.new_array_type(geometry_oids,
'geometry[]', new_type)
psycopg2.extensions.register_type(array_type, None)
}}}
The problem is that in `caster` function, I get all items in array not one
item. something like this:
`0101000020E610000000000000000018400000000000003840:0101000020E610000000000000000014400000000000003740`
which is represent two points that seprated by `:`.
I've checed the column value in postgres console. for an array of points I
got this:
`{0101000020E6100000000000000000F03F0000000000000040:0101000020E6100000000000000000F03F0000000000000040}`
While, I got below value for an array of chars:
`{foo,bar}`
the difference between two cases is for an array of points I got `:` as a
separator, but for an array of chars I got `,`.
I also tried to insert data from console to an array of points column by:
`insert into test values (ARRAY[st_geomFromText('POINT(1 1)', 4326),
st_geomFromText('POINT(1 1)', 4326)]);`
the result was the same(`:` as separator).
Do you have any idea?
--
Ticket URL: <https://code.djangoproject.com/ticket/31450#comment:3>
Django
-------------------------------------+-------------------------------------
| Ramezani
Type: Bug | Status: assigned
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: ArrayField, | Triage Stage: Accepted
PointField |
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Simon Charette (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/31450#comment:4>
Django
-------------------------------------+-------------------------------------
| Ramezani
Type: Bug | Status: assigned
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: ArrayField, | Triage Stage: Accepted
PointField |
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):
I spent a few hours trying to figure it out but it looks like there might
be something wrong with `psycopg2`. When registering an array type for
`geometry` from the retrieved oid it simply doesn't get invoked for
columns of type `geometry(POINT, srid)[]` behind
`ArrayField(PointField())`.
Hasan your code is registering an array type and a scalar type for the
same `oid`, you want to register a type for the scalar oid and an array
type for array oid. Look at how `get_hstore_oids` and
`psycopg2.extras.register_hstore` do it.
--
Ticket URL: <https://code.djangoproject.com/ticket/31450#comment:5>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: ArrayField, | Triage Stage: Accepted
PointField |
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: Hasan Ramezani => (none)
* status: assigned => new
--
Ticket URL: <https://code.djangoproject.com/ticket/31450#comment:6>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: assigned
Severity: Normal | Resolution:
Keywords: ArrayField, | Triage Stage: Accepted
PointField |
Needs tests: 0 | Patch needs improvement: 1
-------------------------------------+-------------------------------------
* owner: (none) => Saurabh
Ticket URL: <https://code.djangoproject.com/ticket/31450#comment:7>
Django
-------------------------------------+-------------------------------------
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: ArrayField, | Triage Stage: Accepted
PointField |
Needs tests: 0 | Patch needs improvement: 1
-------------------------------------+-------------------------------------
* owner: Saurabh => (none)
--
Ticket URL: <https://code.djangoproject.com/ticket/31450#comment:8>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: assigned
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: ArrayField, | Triage Stage: Accepted
PointField |
Needs tests: 0 | Patch needs improvement: 1
-------------------------------------+-------------------------------------
I would like to work on improving the existing patch for this ticket. I
will investigate the issue and submit an updated PR if appropriate.
--
Ticket URL: <https://code.djangoproject.com/ticket/31450#comment:9>
Django
-------------------------------------+-------------------------------------
| Prasad
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: ArrayField, | Triage Stage: Accepted
PointField |
Needs tests: 0 | Patch needs improvement: 1
-------------------------------------+-------------------------------------
* owner: (none) => Lakshya Prasad
--
Ticket URL: <https://code.djangoproject.com/ticket/31450#comment:10>
Django
-------------------------------------+-------------------------------------
| Prasad
Type: Bug | Status: assigned
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: ArrayField, | Triage Stage: Accepted
PointField |
Needs tests: 0 | Patch needs improvement: 1
-------------------------------------+-------------------------------------
Hi, I worked on this ticket and opened a pull request.
PR: https://github.com/django/django/pull/20925
The patch ensures that `ArrayField` falls back to `base_field.to_python()`
when `from_db_value()` is not implemented by the base field (e.g.
`PointField`), which prevents the value from being returned as a string.
Feedback welcome. Thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/31450#comment:11>
Django
-------------------------------------+-------------------------------------
| Prasad
Type: Bug | Status: assigned
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: ArrayField, | Triage Stage: Accepted
PointField |
Needs tests: 0 | Patch needs improvement: 1
-------------------------------------+-------------------------------------
@Australia19-11-2023
Hi Lakshya,
I’ve been investigating #31450 and have reproduced the issue. I noticed
that you’re currently working on it and have opened PR #20925.
I’ve found a possible approach involving the PostgreSQL/PostGIS geometry
array caster and would like to investigate it further. Would you be okay
with me working on an alternative approach or helping improve the existing
PR?
Here is my implementation:
https://github.com/otmaniredha02/django/commit/963f6cb50c0643ffedf897835059aeb10356c091
Thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/31450#comment:12>