[Django] #32542: Native postgres json field support is broken
Django
-------------------------------------+-------------------------------------
Reporter: Artem | Owner: nobody
Type: Bug | Status: new
Component: Database | Version: 3.1
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 1
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
In Django 3.1.1, there was a refactoring of encoders/decoders of DB fields
which affected postgres json/jsonb field support.
Before:
Postgres **json** and **jsonb** field values are encoded/decoded into
python native format by **psycopg2**. **JSONField** had it's own decoder
but if was omitted if db has native json support.
After:
**Jsonb** field is no longer decoded by **psycopg2** (an empty stub is
used instead of `json.joads`), all decoding stuff is delegated to
`JSONField.from_db_value(...)`.
But postgres **json** field is still decoded by **psycopg2**, which causes
'TypeError' while trying to decode it again by `JSONField`.
Breaking commit:
https://github.com/django/django/commit/0be51d2226fce030ac9ca840535a524f41e9832c
Example:
{{{
class CustomJSONField(models.JSONField):
def db_type(self, *args, **kwargs):
# stored as-is, keys order is preserved
return 'json'
class TestModel(models.Model)
json_data = CustomJSONField(default=dict)
TestModel(json_data={"foo": "bar"}).save()
TestModel.objects.last()
}}}
**Expected behavior:** code snippet works
**Actual behavior:**
{{{
TypeError: the JSON object must be str, bytes or bytearray, not dict
}}}
The issue was mentioned in few tickets (#31973, #31956) but eventually
their authors found workarounds (e.g. by switching to **jsonb** field) or
still using Django <= 3.1. I think support of postgres json (not only
jsonb) is important too.
The possible fix is to add the decoder stub for **json** field too:
Before
(https://github.com/django/django/blob/main/django/db/backends/postgresql/base.py#L218):
{{{
psycopg2.extras.register_default_jsonb(conn_or_curs=connection,
loads=lambda x: x)
}}}
After:
{{{
psycopg2.extras.register_default_json(conn_or_curs=connection,
loads=lambda x: x)
psycopg2.extras.register_default_jsonb(conn_or_curs=connection,
loads=lambda x: x)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32542>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: Artem | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
New description:
In Django 3.1.1, there was a refactoring of encoders/decoders of DB fields
which affected postgres json/jsonb field support.
Before:
Postgres **json** and **jsonb** field values are encoded/decoded into
python native format by **psycopg2**. **JSONField** had it's own decoder
but if was omitted if db has native json support.
After:
**Jsonb** field is no longer decoded by **psycopg2** (an empty stub is
used instead of `json.joads`), all decoding stuff is delegated to
`JSONField.from_db_value(...)`.
But postgres **json** field is still decoded by **psycopg2**, which causes
'TypeError' while trying to decode it again by `JSONField`.
Breaking commit:
https://github.com/django/django/commit/0be51d2226fce030ac9ca840535a524f41e9832c
Example:
{{{
class CustomJSONField(models.JSONField):
def db_type(self, *args, **kwargs):
# stored as-is, keys order is preserved
return 'json'
class TestModel(models.Model)
json_data = CustomJSONField(default=dict)
TestModel(json_data={"foo": "bar"}).save()
TestModel.objects.last()
}}}
**Expected behavior:** code snippet works (as in Django 3.1).
**Actual behavior:**
{{{
TypeError: the JSON object must be str, bytes or bytearray, not dict
}}}
The issue was mentioned in few tickets (#31973, #31956) but eventually
their authors found workarounds (e.g. by switching to **jsonb** field) or
still using Django <= 3.1. I think support of postgres json (not only
jsonb) is important too.
The possible fix is to add the decoder stub for **json** field too:
Before
(https://github.com/django/django/blob/main/django/db/backends/postgresql/base.py#L218):
{{{
psycopg2.extras.register_default_jsonb(conn_or_curs=connection,
loads=lambda x: x)
}}}
After:
{{{
psycopg2.extras.register_default_json(conn_or_curs=connection,
loads=lambda x: x)
psycopg2.extras.register_default_jsonb(conn_or_curs=connection,
loads=lambda x: x)
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/32542#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: Artem | Owner: nobody
Component: Database layer | Version: 3.1
(models, ORM) |
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* has_patch: 1 => 0
* resolution: => duplicate
Comment:
Duplicate of #32135. Using `json` data type was never officially
supported.
--
Ticket URL: <https://code.djangoproject.com/ticket/32542#comment:2>