[Django] #32203: Error with values with JSON path lookup in Sqlite when value is numeric.
Django
------------------------------------------+------------------------
Reporter: Gordon Wrigley | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.1
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------+------------------------
This Django 3.1.3, I only see this with Sqlite, it works fine with MySQL
and Postgres.
When I do a path lookup with values like
`Bob.objects.values("my_json_field__position")` if there is an integer,
float or bool in "position" then I get a JSON decode error.
Strings, nones, dicts and lists all work, fetching the top level dict
works and filtering on the path lookup works.
`TypeError: the JSON object must be str, bytes or bytearray, not float`
{{{
..\..\.venv\data_browser\lib\site-packages\django\db\models\query.py:287:
in __iter__
self._fetch_all()
..\..\.venv\data_browser\lib\site-packages\django\db\models\query.py:1308:
in _fetch_all
self._result_cache = list(self._iterable_class(self))
..\..\.venv\data_browser\lib\site-packages\django\db\models\query.py:111:
in __iter__
for row in compiler.results_iter(chunked_fetch=self.chunked_fetch,
chunk_size=self.chunk_size):
..\..\.venv\data_browser\lib\site-
packages\django\db\models\sql\compiler.py:1100: in apply_converters
value = converter(value, expression, connection)
..\..\.venv\data_browser\lib\site-
packages\django\db\models\fields\json.py:79: in from_db_value
return json.loads(value, cls=self.decoder)
}}}
Queries that produce strings or None's seem to work fine.
--
Ticket URL: <https://code.djangoproject.com/ticket/32203>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Gordon Wrigley | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.1
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by Gordon Wrigley:
Old description:
> This Django 3.1.3, I only see this with Sqlite, it works fine with MySQL
> and Postgres.
>
> When I do a path lookup with values like
> `Bob.objects.values("my_json_field__position")` if there is an integer,
> float or bool in "position" then I get a JSON decode error.
> Strings, nones, dicts and lists all work, fetching the top level dict
> works and filtering on the path lookup works.
>
> `TypeError: the JSON object must be str, bytes or bytearray, not float`
>
> {{{
> ..\..\.venv\data_browser\lib\site-packages\django\db\models\query.py:287:
> in __iter__
> self._fetch_all()
> ..\..\.venv\data_browser\lib\site-
> packages\django\db\models\query.py:1308: in _fetch_all
> self._result_cache = list(self._iterable_class(self))
> ..\..\.venv\data_browser\lib\site-packages\django\db\models\query.py:111:
> in __iter__
> for row in compiler.results_iter(chunked_fetch=self.chunked_fetch,
> chunk_size=self.chunk_size):
> ..\..\.venv\data_browser\lib\site-
> packages\django\db\models\sql\compiler.py:1100: in apply_converters
> value = converter(value, expression, connection)
> ..\..\.venv\data_browser\lib\site-
> packages\django\db\models\fields\json.py:79: in from_db_value
> return json.loads(value, cls=self.decoder)
> }}}
>
> Queries that produce strings or None's seem to work fine.
New description:
This Django 3.1.3, I only see this with Sqlite, it works fine with MySQL
and Postgres.
When I do a path lookup with values like
`Bob.objects.values("my_json_field__position")` if there is an integer,
float or bool in "position" then I get a JSON decode error.
Strings, nones, dicts and lists all work, fetching the top level dict
works and filtering on the path lookup works.
`TypeError: the JSON object must be str, bytes or bytearray, not float`
{{{
..\..\.venv\data_browser\lib\site-packages\django\db\models\query.py:287:
in __iter__
self._fetch_all()
..\..\.venv\data_browser\lib\site-packages\django\db\models\query.py:1308:
in _fetch_all
self._result_cache = list(self._iterable_class(self))
..\..\.venv\data_browser\lib\site-packages\django\db\models\query.py:111:
in __iter__
for row in compiler.results_iter(chunked_fetch=self.chunked_fetch,
chunk_size=self.chunk_size):
..\..\.venv\data_browser\lib\site-
packages\django\db\models\sql\compiler.py:1100: in apply_converters
value = converter(value, expression, connection)
..\..\.venv\data_browser\lib\site-
packages\django\db\models\fields\json.py:79: in from_db_value
return json.loads(value, cls=self.decoder)
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/32203#comment:1>
Django
Reporter: Gordon Wrigley | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.1
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by sage):
* cc: sage (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/32203#comment:2>
Django
Reporter: Gordon Wrigley | Owner: sage
Type: Bug | Status: assigned
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by sage):
* owner: nobody => sage
* status: new => assigned
* type: Uncategorized => Bug
* component: Uncategorized => Database layer (models, ORM)
* stage: Unreviewed => Accepted
Comment:
This is because SQLite's `JSON_EXTRACT` returns deserialized values. A
simple solution is just to add `TypeError`
[https://github.com/django/django/blob/master/django/db/models/fields/json.py#L80
here], which will also partially "solve" problems with people who use the
JSON data type on PostgreSQL. Another option is to wrap the value in
`JSON_QUOTE`... which I think is the better option.
I'm guessing there's also a bug in querying `__key` on `{"key":
"\"value\""}`, which will return `value` rather than `"value"`. I
''think'' wrapping the value in `JSON_QUOTE` should fix this as well.
--
Ticket URL: <https://code.djangoproject.com/ticket/32203#comment:3>
Django
Reporter: Gordon Wrigley | Owner: sage
Type: Bug | Status: assigned
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by sage):
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/13691 PR]. I opted to catch
`TypeError` instead. Need opinions on how to proceed.
--
Ticket URL: <https://code.djangoproject.com/ticket/32203#comment:4>
Django
Reporter: Gordon Wrigley | Owner: sage
Type: Bug | Status: assigned
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Gordon Wrigley):
I should add that I see this on both Windows and Linux
--
Ticket URL: <https://code.djangoproject.com/ticket/32203#comment:5>
Django
Reporter: Gordon Wrigley | Owner: sage
Type: Bug | Status: assigned
Component: Database layer | Version: 3.1
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* severity: Normal => Release blocker
--
Ticket URL: <https://code.djangoproject.com/ticket/32203#comment:6>
Django
Reporter: Gordon Wrigley | Owner: sage
Component: Database layer | Version: 3.1
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"fe6e5824218bab7cf47dee112d68325b338f9947" fe6e5824]:
{{{
#!CommitTicketReference repository=""
revision="fe6e5824218bab7cf47dee112d68325b338f9947"
Fixed #32203 -- Fixed QuerySet.values()/values_list() crash on key
transforms with non-string values on SQLite.
Thanks Gordon Wrigley for the report.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32203#comment:8>
Django
Reporter: Gordon Wrigley | Owner: sage
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Release blocker | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"7408c4cd157ae18971302a2baf7aacce7c204a09" 7408c4cd]:
{{{
#!CommitTicketReference repository=""
revision="7408c4cd157ae18971302a2baf7aacce7c204a09"
Refs #32203 -- Added tests for QuerySet.values()/values_list() on key
transforms with non-trivial values.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32203#comment:7>
Django
Reporter: Gordon Wrigley | Owner: sage
Component: Database layer | Version: 3.1
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"a7935fe9424fed9401e4c0a2a54e234007cf30da" a7935fe]:
{{{
#!CommitTicketReference repository=""
revision="a7935fe9424fed9401e4c0a2a54e234007cf30da"
[3.1.x] Fixed #32203 -- Fixed QuerySet.values()/values_list() crash on key
transforms with non-string values on SQLite.
Thanks Gordon Wrigley for the report.
Backport of fe6e5824218bab7cf47dee112d68325b338f9947 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32203#comment:10>
Django
Reporter: Gordon Wrigley | Owner: sage
Type: Bug | Status: closed
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"a2abeb3de76624fe34d7f046183b94c98a7bd09f" a2abeb3]:
{{{
#!CommitTicketReference repository=""
revision="a2abeb3de76624fe34d7f046183b94c98a7bd09f"
[3.1.x] Refs #32203 -- Added tests for QuerySet.values()/values_list() on
key transforms with non-trivial values.
Backport of 7408c4cd157ae18971302a2baf7aacce7c204a09 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32203#comment:9>