[Django] #24709: New contrib.postgres ArrayField doesn't work quite as expected with .update () and F() objects
Django
() and F() objects
-------------------------------------+-------------------------------------
Reporter: ris | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.8
(models, ORM) | Keywords: postgresql contrib
Severity: Normal | arrayfield update
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
On a model along the lines of
{{{
class ModelA ( models.Model ):
name = TextField ()
name_array = ArrayField ( models.TextField () )
}}}
If you perform
{{{
>>> ModelA.objects.update ( name_array = [ F ( "name" ) ] )
}}}
You end up with
{{{
>>> ModelA.objects.all ()[0].name_array
[u'F(name)']
}}}
I see what you've done there django, but that's not quite what I meant.
--
Ticket URL: <https://code.djangoproject.com/ticket/24709>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: ris | Owner: nobody
Component: contrib.postgres | Version: 1.8
Severity: Normal | Resolution:
Keywords: postgresql contrib | Triage Stage: Accepted
arrayfield update |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: => 0
* component: Database layer (models, ORM) => contrib.postgres
* needs_tests: => 0
* needs_docs: => 0
* type: Bug => New feature
* stage: Unreviewed => Accepted
Comment:
I am not sure that F expressions can be supported like that, but if not,
it might be worth documenting the restriction and/or throwing a better
error message rather than silently coercing to string.
--
Ticket URL: <https://code.djangoproject.com/ticket/24709#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: ris | Owner: nobody
Component: contrib.postgres | Version: 1.8
Severity: Normal | Resolution:
Keywords: postgresql contrib | Triage Stage: Accepted
arrayfield update |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by mjtamlyn):
The silent coercion to string is normal behaviour for Django actually in
many cases - we actually would like `ModelA.objects.update(name_array=[1,
2])` to work (as it would if it were a `CharField`.
Supporting F objects like that is a very nice idea, but I'm not sure how
possible it is. If we do implement it, it should work for Hstore and Json
as well. In these cases there are more complex options - for example you
can update just a single key of the Hstore without affecting the rest,
which would be a very useful addition.
A simpler option could be to have a distinct API for it - something like
`ModelA.objects.update(name_array=FArray(0, F(name))` which would generate
an update statement like `UPDATE app_modela SET name_array[0]=F(name)`.
This API could also be a bit clever, and also support
`ModelA.objects.update(name_array=FArray([F('name')]))`. We would then
have similar FJSON and FHstore objects. I'm pretty sure these could be
supported without changes to the core ORM by making them look sufficiently
like F objects.
--
Ticket URL: <https://code.djangoproject.com/ticket/24709#comment:2>
Django
-------------------------------------+-------------------------------------
Reporter: ris | Owner: nobody
Component: contrib.postgres | Version: 1.8
Severity: Normal | Resolution:
Keywords: postgresql contrib | Triage Stage: Accepted
arrayfield update |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by adamchainz):
I have written a similar separate F API for the the comma-separated fields
in Django-MySQL - see http://django-
mysql.readthedocs.org/en/latest/model_fields.html#listf-expressions . I
made a `ListF` class for doing single atomic add/remove operations on
either end of the list, which is about all that can be done with MySQL's
comma-separated-list parsing capabilities.
It is obviously much less involved than Postgres' arrays, but maybe it
will inspire. I think add/remove operations are quite useful to consider
on top of the 'set element at position X to Y' and 'set array field to
[Y]' operations.
--
Ticket URL: <https://code.djangoproject.com/ticket/24709#comment:3>
Django
-------------------------------------+-------------------------------------
Reporter: ris | Owner: nobody
Component: contrib.postgres | Version: 1.8
Severity: Normal | Resolution:
Keywords: postgresql contrib | Triage Stage: Accepted
arrayfield update |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by jarshwah):
This seems like a very similar problem to how F() and Sum() evolved
separately when they could have been the same with a little (lot..) more
work. Marc, could you require that the inputs to Array/HStore be
expressions (with an appropriate _parse_expression() to wrap basic values
in Value())? If you provide support for expressions only, then getting
basic values in comes fairly naturally too.
I haven't looked into the implementation of these fields though, so I
don't know how feasible it would be to support.
--
Ticket URL: <https://code.djangoproject.com/ticket/24709#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: ris | Owner: nobody
Component: contrib.postgres | Version: 1.8
Severity: Normal | Resolution:
Keywords: postgresql contrib | Triage Stage: Accepted
arrayfield update |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by codingjoe):
There is more broken here:
{{{
ModelA.objects.update(name_array=F("name_array") + 'some item'])
}}}
This should result in:
{{{
...
SET "myapp_modela"."name_arry" = array_append("myapp_modela"."name_arry",
'some item')
...
}}}
not in:
{{{
...
SET "myapp_modela"."name_arry" = ("myapp_modela"."name_arry" + 'some
item')
...
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24709#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: ris | Owner: nobody
Component: contrib.postgres | Version: 1.8
Severity: Normal | Resolution:
Keywords: postgresql contrib | Triage Stage: Accepted
arrayfield update |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: info@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/24709#comment:6>
Django
-------------------------------------+-------------------------------------
Reporter: ris | Owner: nobody
Component: contrib.postgres | Version: 1.8
Severity: Normal | Resolution:
Keywords: postgresql contrib | Triage Stage: Accepted
arrayfield update |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Frederik Creemers):
I'm not that familiar with the internals, so I don't know if my use case
helps flesh out ideas for how to implement this, but here's what I have:
I have a score_weights list, which is an array of floating point
numbers.Every User has a "scores" field, containing an array with the same
number of items. I'd like to sort users by a weighted sum of these scores,
with the weights coming from this score_weights list. In other words, the
users are ordered by the dot product of their scores array and the
score_weights array.
{{{
order_expression = -1 * sum([
F("scores__%s" % i) * score_weights[i]
for i in xrange(len(crit.score_weights))
])
}}}
Currently, this raises this error:
{{{
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/query.py", line 256, in __iter__
self._fetch_all()
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/query.py", line 1087, in _fetch_all
self._result_cache = list(self.iterator())
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/query.py", line 54, in __iter__
results = compiler.execute_sql()
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/sql/compiler.py", line 824, in execute_sql
sql, params = self.as_sql()
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/sql/compiler.py", line 369, in as_sql
extra_select, order_by, group_by = self.pre_sql_setup()
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/sql/compiler.py", line 47, in
pre_sql_setup
order_by = self.get_order_by()
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/sql/compiler.py", line 308, in
get_order_by
self.query, allow_joins=True, reuse=None)
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/expressions.py", line 209, in
resolve_expression
for expr in c.get_source_expressions()
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/expressions.py", line 404, in
resolve_expression
c.rhs = c.rhs.resolve_expression(query, allow_joins, reuse, summarize,
for_save)
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/expressions.py", line 403, in
resolve_expression
c.lhs = c.lhs.resolve_expression(query, allow_joins, reuse, summarize,
for_save)
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/expressions.py", line 403, in
resolve_expression
c.lhs = c.lhs.resolve_expression(query, allow_joins, reuse, summarize,
for_save)
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/expressions.py", line 403, in
resolve_expression
c.lhs = c.lhs.resolve_expression(query, allow_joins, reuse, summarize,
for_save)
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/expressions.py", line 403, in
resolve_expression
c.lhs = c.lhs.resolve_expression(query, allow_joins, reuse, summarize,
for_save)
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/expressions.py", line 404, in
resolve_expression
c.rhs = c.rhs.resolve_expression(query, allow_joins, reuse, summarize,
for_save)
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/expressions.py", line 403, in
resolve_expression
c.lhs = c.lhs.resolve_expression(query, allow_joins, reuse, summarize,
for_save)
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/expressions.py", line 463, in
resolve_expression
return query.resolve_ref(self.name, allow_joins, reuse, summarize)
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/sql/query.py", line 1462, in resolve_ref
self.get_initial_alias(), reuse)
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/sql/query.py", line 1402, in setup_joins
names, opts, allow_many, fail_on_missing=True)
File "/Users/frederikcreemers/.virtualenvs/employebla/lib/python2.7
/site-packages/django/db/models/sql/query.py", line 1370, in names_to_path
" not permitted." % (names[pos + 1], name))
FieldError: Cannot resolve keyword u'0' into field. Join on 'scores' not
permitted.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/24709#comment:7>
Django
-------------------------------------+-------------------------------------
Reporter: ris | Owner: nobody
Component: contrib.postgres | Version: 1.8
Severity: Normal | Resolution: duplicate
Keywords: postgresql contrib | Triage Stage: Accepted
arrayfield update |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => duplicate
Comment:
Duplicate of #26355
(even though this ticket is older, the other one has a pull request which
is why I'm closing this one)
--
Ticket URL: <https://code.djangoproject.com/ticket/24709#comment:8>