[Django] #36371: JSONField.from_db_value crashes when DB returns parsed JSON despite KeyTransform guard

[Django]

#36371: JSONField.from_db_value crashes when DB returns parsed JSON despite
KeyTransform guard
-------------------------------------+-------------------------------------
Reporter: Mason Pitts | Type: Bug
Status: new | Component: Database
| layer (models, ORM)
Version: 5.2 | Severity: Normal
Keywords: jsonfield, | Triage Stage:
from_db_value, double-decoding, | Unreviewed
psycopg3, cx_oracle, python- |
oracledb |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
In Django 5.2, the default implementation of JSONField.from_db_value()
only skips double-decoding when the ORM expression is a KeyTransform on a
non-string. However, many modern database drivers (e.g. PostgreSQL
psycopg3, Oracle DB_TYPE_JSON via cx_Oracle 8.1+/python-oracledb) will
automatically deserialize JSON columns into native Python types (dict,
list) before Django sees them. Since from_db_value() still unconditionally
calls json.loads() in most cases, you get:


{{{
TypeError: the JSON object must be str, bytes or bytearray, not list
}}}

even though the value is already a valid Python object.

Here is the current code below as of 5/6/2025.


{{{
def from_db_value(self, value, expression, connection):
if value is None:
return value
# Some backends (SQLite at least) extract non-string values in
their
# SQL datatypes.
if isinstance(expression, KeyTransform) and not isinstance(value,
str):
return value
try:
return json.loads(value, cls=self.decoder)
except json.JSONDecodeError:
return value
}}}


Here is a potential solution that attempts to return value if it is a
Python type.


{{{
def from_db_value(self, value, expression, connection):
if value is None:
return None

# Decode binary data first.
if isinstance(value, (bytes, bytearray)):
value = value.decode()

# If value isn’t a string at this point, the driver already gave
us
# a native Python type (dict, list, bool, int, float, ...).
if not isinstance(value, str):
return value

try:
return json.loads(value, cls=self.decoder)
except json.JSONDecodeError:
return value
}}}


Steps to reproduce:

1. Define a model with a models.JSONField().

2. Use a database and driver combination that natively decodes JSON
columns—for example:

- PostgreSQL with psycopg3

- Oracle 21c+ JSON column type with cx_Oracle 8.1+ or python-
oracledb in thin mode

* I encountered this problem using oracle_db in thin mode.

3. Query the model in the Django admin or via MyModel.objects.all().

4. Observe the traceback raising a TypeError when json.loads() is fed a
list or dict.


Version information:

Django: 5.2

Python: 3.12.10

Affected drivers/backends:

PostgreSQL with psycopg3

Oracle 21c+ with cx_Oracle 8.1+ / python-oracledb in thin mode
--
Ticket URL: <https://code.djangoproject.com/ticket/36371>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36371: JSONField.from_db_value crashes when DB returns parsed JSON despite
KeyTransform guard
-------------------------------------+-------------------------------------

Reporter: Mason Pitts | Owner: (none)
Type: Bug | Status: closed
Component: Database layer | Version: 5.2
(models, ORM) | Resolution:
Severity: Normal | worksforme

Keywords: jsonfield, | Triage Stage:
from_db_value, double-decoding, | Unreviewed
psycopg3, cx_oracle, python- |
oracledb |

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Natalia Bidart):

* easy: 1 => 0
* has_patch: 1 => 0
* resolution: => worksforme
* status: new => closed

Comment:

Hello Mason Pitts, thank you for taking the time to create this report. I
can't reproduce what you are describing: I'm using Django `main`,
PostgreSQL 16.8 and psycopg version 3.2.3. Given this model:
{{{#!python
class VulnerableModel(models.Model):
data = models.JSONField()
}}}
And this test case:
{{{#!python
class Ticket36371TestCase(TestCase):

def test_try(self):
from django.db import connection
print(connection.connection)

from .models import VulnerableModel
self.assertEqual(VulnerableModel.objects.all().count(), 0)

VulnerableModel.objects.create(data={"test": 1})
self.assertEqual(VulnerableModel.objects.all().count(), 1)

VulnerableModel.objects.create(data=["my list", 1, 2])
self.assertEqual(VulnerableModel.objects.all().count(), 2)

VulnerableModel.objects.create(data="This is a plain string")
self.assertEqual(VulnerableModel.objects.all().count(), 3)

items = [i.data for i in VulnerableModel.objects.all()]
self.assertEqual(items, [{"test": 1}, ["my list", 1, 2], "This is
a plain string"])
}}}
Running the tests shows:
{{{
$ python -Wall manage.py test testapp.tests.Ticket36371TestCase
Found 1 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
<psycopg.Connection [INTRANS] (user=nessita database=test_djangotest) at
0x754ee7715f90>
.
----------------------------------------------------------------------
Ran 1 test in 0.003s

OK
Destroying test database for alias 'default'...
}}}

Given this, I think this report seems better suited to be a support
request. The best place to get answers to your issue is using any of the
user support channels from [https://docs.djangoproject.com/en/dev/faq/help
/#how-do-i-do-x-why-doesn-t-y-work-where-can-i-go-to-get-help this link].

Since the goal of this issue tracker is to track issues about Django
itself, and your issue seems, at first, to be located in your custom code,
I'll be closing this ticket following the
[https://docs.djangoproject.com/en/dev/internals/contributing/triaging-
tickets/#closing-tickets ticket triaging process]. If, after debugging,
you find out that this is indeed a bug in Django, please re-open with the
specific details and please be sure to include a small Django project to
reproduce or a failing test case.
--
Ticket URL: <https://code.djangoproject.com/ticket/36371#comment:1>

[Django]

#36371: JSONField.from_db_value crashes when DB returns parsed JSON despite
KeyTransform guard
-------------------------------------+-------------------------------------
Reporter: Mason Pitts | Owner: (none)
Type: Bug | Status: closed
Component: Database layer | Version: 5.2
(models, ORM) | Resolution:
Severity: Normal | worksforme
Keywords: jsonfield, | Triage Stage:
from_db_value, double-decoding, | Unreviewed
psycopg3, cx_oracle, python- |
oracledb |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

Likely related to tickets such as #31973, #32542.

You possibly have a field declared as `json` on Postgres instead of
`jsonb` which
[https://github.com/django/django/blob/f7d97dd11819be8996798a7197c5695a317334a0/django/db/backends/postgresql/base.py#L335-L341
the ORM explicitly request not to be deserialized by the backend] to
support the `JSONField.decoder` feature.
--
Ticket URL: <https://code.djangoproject.com/ticket/36371#comment:2>