[Django] #25091: Array field equality lookup fails with ProgrammingError
Django
----------------------------------+-----------------
Reporter: unklphil | Owner:
Type: Bug | Status: new
Component: contrib.postgres | Version: 1.8
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+-----------------
Note: This bug report is mostly based on a reddit post by Glueon:
http://redd.it/38j43l
With the model:
{{{
class MyModel(models.Model):
emails = ArrayField(models.EmailField())
}}}
When trying to fetch a row with a list of emails:
{{{
MyModel.objects.filter(emails=['te...@test.com'])
}}}
the following error occurs:
{{{
ProgrammingError: operator does not exist: character varying[] = text[]
HINT: No operator matches the given name and argument type(s). You might
need to add explicit type casts.
}}}
Because the resulting query is:
{{{
SELECT * FROM some_table WHERE emails = ARRAY['te...@test.com'];
}}}
By default type of `ARRAY['te...@test.com']` is `text []` while Django
stores it as a `varchar []`.
Glueon suggests that casting the array to `varchar []` using raw sql
solves the problem:
{{{
SELECT * FROM some_table WHERE emails = ARRAY['te...@test.com']::varchar[];
}}}
I did post some other workarounds in the comments of the reddit post, but
there are some situations in which the workarounds are not possible, such
as when using `get_or_create` or `update_or_create`.
--
Ticket URL: <https://code.djangoproject.com/ticket/25091>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: unklphil | Owner:
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by charettes):
* needs_docs: => 0
* needs_better_patch: => 0
* version: 1.8 => master
* needs_tests: => 0
* stage: Unreviewed => Accepted
Comment:
This seems related to #25091.
--
Ticket URL: <https://code.djangoproject.com/ticket/25091#comment:1>
Django
Reporter: unklphil | Owner:
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by BertrandBordage):
Here’s a ready-to-use workaround for `get_or_create`, as described
[https://www.reddit.com/r/django/comments/38j43l/django_orm_postgresql_and_array_field_equality/crw7y4i
in this redit post].
{{{#!python
def bugfree_get_or_create(manager, **kwargs):
"""
This is workaround for https://code.djangoproject.com/ticket/25091
"""
model = manager.model
defaults = kwargs.pop('defaults', {})
for k, array in kwargs.items():
if isinstance(model._meta.get_field(k), ArrayField):
for i, item in enumerate(array):
kwargs[k + '__%d' % i] = item
kwargs[k + '__len'] = len(array)
del kwargs[k]
try:
return manager.get(**kwargs)
except model.DoesNotExist:
kwargs.update(defaults)
return manager.create(**kwargs)
}}}
Instead of writing
`YourModel.objects.get_or_create(your_array_field=['abc', 'def'],
other_kwargs=…)`, write `bugfree_get_or_create(YourModel.objects,
your_array_field=['abc', 'def'], other_kwargs=…)`.
--
Ticket URL: <https://code.djangoproject.com/ticket/25091#comment:2>
Django
Reporter: unklphil | Owner:
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by charettes):
I would like to add a notice that the code above is not even close to be
as bullet proof as `get_or_create` and you should expect race conditions
if you were to use it.
--
Ticket URL: <https://code.djangoproject.com/ticket/25091#comment:3>
Django
Reporter: unklphil | Owner:
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by calvingiles):
I have found two simple workarounds to this, but they may have some issues
that I am not aware of. The first is to pass the array to the get or
get_or_create method not as a list, but as a string formatted using
{{{
'{1.0, 2.0}'
}}}
instead of
{{{
ARRAY[1.0, 2.0]
}}}
which causes postgres to handle an implicit cast from numeric to real
(which I needed in this case).
The other is to perform this conversion in `get_db_prep_value()` by
overriding the ArrayField class. For my specific case, this appears to
work, but it may not be robust to arrays of different types (especially
strings):
{{{
class StrFormatArrayField(ArrayField):
"""
Override the array format to use "'{1.0, 2.0, 3.0}'" instead of
"ARRAY([1.0, 2.0, 3.0])"
to get around an issue with implicit type conversion.
"""
def get_db_prep_value(self, value, connection, prepared=False):
if isinstance(value, list) or isinstance(value, tuple):
prepped = [self.base_field.get_db_prep_value(i, connection,
prepared) for i in value]
joined = ', '.join([str(v) for v in prepped])
return '{' + joined + '}'
else:
return value
}}}
Any advice on improving upon this or issues I may face would be greatly
appreciated.
--
Ticket URL: <https://code.djangoproject.com/ticket/25091#comment:4>
Django
Reporter: unklphil | Owner:
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Stranger6667):
* has_patch: 0 => 1
Comment:
PR: https://github.com/django/django/pull/5575
--
Ticket URL: <https://code.djangoproject.com/ticket/25091#comment:5>
Django
Reporter: unklphil | Owner:
Type: Bug | Status: closed
Component: contrib.postgres | Version: master
Severity: Normal | Resolution: duplicate
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by charettes):
* status: new => closed
* resolution: => duplicate
Comment:
#25666 is a duplicate with a patch.
--
Ticket URL: <https://code.djangoproject.com/ticket/25091#comment:6>