[Django] #30826: Backwards-incompatible change to querying data within arrays in JSONFields using `contains` in 1.11.25
Django
using `contains` in 1.11.25
-------------------------------------+-------------------------------------
Reporter: souldeux | Owner: nobody
Type: Bug | Status: new
Component: Database | Version: 1.11
layer (models, ORM) |
Severity: Normal | Keywords: JSONField, contains
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Consider a model that looks like this:
{{{
class Event(models.Model):
data = JSONField(default=dict)
}}}
Create an instance of this model with a nested array:
{{{
data = {
"things": ["thing1", "thing2", "thing3"]
}
Event.objects.create(data=data)
}}}
The following lookup works in 1.11.24, but returns an error (traceback
below) in 1.11.25:
{{{
Event.objects.filter(data__things__contains="thing1")
}}}
1.11.25 Traceback:
{{{
TypeError Traceback (most recent call
last)
/usr/lib/python3.7/site-packages/django/db/models/query.py in
__repr__(self)
224
225 def __repr__(self):
--> 226 data = list(self[:REPR_OUTPUT_SIZE + 1])
227 if len(data) > REPR_OUTPUT_SIZE:
228 data[-1] = "...(remaining elements truncated)..."
/usr/lib/python3.7/site-packages/django/db/models/query.py in
__iter__(self)
248 - Responsible for turning the rows into model
objects.
249 """
--> 250 self._fetch_all()
251 return iter(self._result_cache)
252
/usr/lib/python3.7/site-packages/django/db/models/query.py in
_fetch_all(self)
1119 def _fetch_all(self):
1120 if self._result_cache is None:
-> 1121 self._result_cache = list(self._iterable_class(self))
1122 if self._prefetch_related_lookups and not
self._prefetch_done:
1123 self._prefetch_related_objects()
/usr/lib/python3.7/site-packages/django/db/models/query.py in
__iter__(self)
51 # Execute the query. This will also fill compiler.select,
klass_info,
52 # and annotations.
---> 53 results =
compiler.execute_sql(chunked_fetch=self.chunked_fetch)
54 select, klass_info, annotation_col_map = (compiler.select,
compiler.klass_info,
55
compiler.annotation_col_map)
/usr/lib/python3.7/site-packages/django/db/models/sql/compiler.py in
execute_sql(self, result_type, chunked_fetch)
874 result_type = NO_RESULTS
875 try:
--> 876 sql, params = self.as_sql()
877 if not sql:
878 raise EmptyResultSet
/usr/lib/python3.7/site-packages/django/db/models/sql/compiler.py in
as_sql(self, with_limits, with_col_aliases)
439 # (see docstring of get_from_clause() for
details).
440 from_, f_params = self.get_from_clause()
--> 441 where, w_params = self.compile(self.where) if
self.where is not None else ("", [])
442 having, h_params = self.compile(self.having) if
self.having is not None else ("", [])
443 result = ['SELECT']
/usr/lib/python3.7/site-packages/django/db/models/sql/compiler.py in
compile(self, node, select_format)
371 sql, params = vendor_impl(self, self.connection)
372 else:
--> 373 sql, params = node.as_sql(self, self.connection)
374 if select_format is FORCE or (select_format and not
self.query.subquery):
375 return node.output_field.select_format(self, sql,
params)
/usr/lib/python3.7/site-packages/django/db/models/sql/where.py in
as_sql(self, compiler, connection)
77 for child in self.children:
78 try:
---> 79 sql, params = compiler.compile(child)
80 except EmptyResultSet:
81 empty_needed -= 1
/usr/lib/python3.7/site-packages/django/db/models/sql/compiler.py in
compile(self, node, select_format)
371 sql, params = vendor_impl(self, self.connection)
372 else:
--> 373 sql, params = node.as_sql(self, self.connection)
374 if select_format is FORCE or (select_format and not
self.query.subquery):
375 return node.output_field.select_format(self, sql,
params)
/usr/lib/python3.7/site-packages/django/contrib/postgres/lookups.py in
as_sql(self, qn, connection)
9 lhs, lhs_params = self.process_lhs(qn, connection)
10 rhs, rhs_params = self.process_rhs(qn, connection)
---> 11 params = lhs_params + rhs_params
12 return '%s %s %s' % (lhs, self.operator, rhs), params
13
TypeError: can only concatenate tuple (not "list") to tuple
}}}
This appears to have been introduced in 1.11.25 when addressing this
ticket: https://code.djangoproject.com/ticket/30769
While it is entirely possible that this is an incorrect use of `contains`
in this context to begin with, this is a breaking change for projects
using 1.11.* with this syntax.
--
Ticket URL: <https://code.djangoproject.com/ticket/30826>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: Nic Perry | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: JSONField, contains | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
* severity: Normal => Release blocker
Comment:
Thanks for this report, yes it is a regression in the newest release.
Regression in 6c3dfba89215fc56fc27ef61829a6fff88be4abb.
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:1>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* Attachment "test-30826.diff" added.
Regression test.
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Simon Charette (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:2>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Massimo Costa (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:3>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Carlton Gibson):
[https://github.com/django/django/pull/11863 Pull request]
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:4>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by rsichny):
getting the same with chained `__has_key` lookup (and i can assume any
other chained lookups will crash)
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: Nic Perry | Owner:
Type: Bug | Status: assigned
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Massimo Costa):
Replying to [comment:5 Roman Sichny]:
> getting the same with chained `__has_key` lookup (and i can assume any
other chained lookups will crash)
just to confirm that my initial report (now closed as duplicate of this
one) was about `__has_key`
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:7>
Django
-------------------------------------+-------------------------------------
Reporter: Nic Perry | Owner:
| louiseGrandjonc
Type: Bug | Status: assigned
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => assigned
* owner: nobody => louiseGrandjonc
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:6>
Django
-------------------------------------+-------------------------------------
| Grandjonc
Type: Bug | Status: assigned
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
| checkin
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:8>
Django
-------------------------------------+-------------------------------------
Reporter: Nic Perry | Owner: Louise
| Grandjonc
Type: Bug | Status: assigned
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: JSONField, contains | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Shaheed Haque (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:9>
Django
-------------------------------------+-------------------------------------
Reporter: Nic Perry | Owner: Louise
| Grandjonc
Component: Database layer | Version: 1.11
(models, ORM) |
Keywords: JSONField, contains | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"cf7ef5d2589c1ca68189202f4c1f947e6a1ae53a" cf7ef5d2]:
{{{
#!CommitTicketReference repository=""
revision="cf7ef5d2589c1ca68189202f4c1f947e6a1ae53a"
[3.0.x] Fixed #30826 -- Fixed crash of many JSONField lookups when one
hand side is key transform.
Regression in 6c3dfba89215fc56fc27ef61829a6fff88be4abb.
Backport of 7d1bf29977bb368d7c28e7c6eb146db3b3009ae7 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:10>
Django
-------------------------------------+-------------------------------------
Reporter: Nic Perry | Owner: Louise
| Grandjonc
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution: fixed
Keywords: JSONField, contains | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"7d1bf29977bb368d7c28e7c6eb146db3b3009ae7" 7d1bf299]:
{{{
#!CommitTicketReference repository=""
revision="7d1bf29977bb368d7c28e7c6eb146db3b3009ae7"
Fixed #30826 -- Fixed crash of many JSONField lookups when one hand side
is key transform.
Regression in 6c3dfba89215fc56fc27ef61829a6fff88be4abb.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:11>
Django
-------------------------------------+-------------------------------------
Reporter: Nic Perry | Owner: Louise
| Grandjonc
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution: fixed
Keywords: JSONField, contains | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"323467e286787684de18d2731776c71667a296be" 323467e2]:
{{{
#!CommitTicketReference repository=""
revision="323467e286787684de18d2731776c71667a296be"
[2.2.x] Fixed #30826 -- Fixed crash of many JSONField lookups when one
hand side is key transform.
Regression in 6c3dfba89215fc56fc27ef61829a6fff88be4abb.
Backport of 7d1bf29977bb368d7c28e7c6eb146db3b3009ae7 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:12>
Django
-------------------------------------+-------------------------------------
Reporter: Nic Perry | Owner: Louise
| Grandjonc
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution: fixed
Keywords: JSONField, contains | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"522af9d6737550ef035a173c08a8276028b68917" 522af9d6]:
{{{
#!CommitTicketReference repository=""
revision="522af9d6737550ef035a173c08a8276028b68917"
[2.1.x] Fixed #30826 -- Fixed crash of many JSONField lookups when one
hand side is key transform.
Regression in 6c3dfba89215fc56fc27ef61829a6fff88be4abb.
Backport of 7d1bf29977bb368d7c28e7c6eb146db3b3009ae7 from master.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:13>
Django
-------------------------------------+-------------------------------------
Reporter: Nic Perry | Owner: Louise
| Grandjonc
Type: Bug | Status: closed
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution: fixed
Keywords: JSONField, contains | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"a843a9ba8da40ffae49c9ec13e7b5c41d5e40ca5" a843a9ba]:
{{{
#!CommitTicketReference repository=""
revision="a843a9ba8da40ffae49c9ec13e7b5c41d5e40ca5"
[1.11.x] Fixed #30826 -- Fixed crash of many JSONField lookups when one
hand side is key transform.
Regression in 6c3dfba89215fc56fc27ef61829a6fff88be4abb.
Backport of 7d1bf29977bb368d7c28e7c6eb146db3b3009ae7 from master.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30826#comment:14>