[Django] #25995: Support serialization to `django.contrib.postgres.fields.jsonb.py`
Django
----------------------------------------------+--------------------
Reporter: jimgraham | Owner: nobody
Type: New feature | Status: new
Component: Database layer (models, ORM) | Version: 1.9
Severity: Normal | Keywords: JSONB
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------------------+--------------------
It would be great if the `jsonb` field could support serialization of
objects to the JSON field. The `DjangoJSONEncoder` would be helpful here
for `datetime`s, `Decimal`s etc that are in JSON blobs.
I've worked on a similar serializer if it would help to come up with a
pull request.
--
Ticket URL: <https://code.djangoproject.com/ticket/25995>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: jimgraham | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: JSONB | Triage Stage: Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* cc: mjtamlyn (added)
* needs_better_patch: => 0
* component: Database layer (models, ORM) => contrib.postgres
* needs_tests: => 0
* needs_docs: => 0
Comment:
I think the problem here is that we couldn't also deserialize that data
since we have no way to know if the user originally passed a string or
some other Python type. For example, if we see '1' in the JSONField, do we
deserialize it to 1 (integer) or leave it as a string? Using
`DjangoJSONEncoder` for model serialization is different because we have
the model field type hint to use for deserialization.
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:1>
Django
Reporter: jimgraham | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Easy pickings: 0 | UI/UX: 0
Comment (by jimgraham):
I think serializing would be be optional at the model level, and the user
could decide how they wanted objects serialized/deserialize. The default
would continue to be no serialization.
Is it worthwhile coming up with a PR for comments, or is this a non-
starter?
Thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:2>
Django
Reporter: jimgraham | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
Could you be a bit more explicit about what the behavior would look like,
e.g. with some code examples?
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:3>
Django
Reporter: jimgraham | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Easy pickings: 0 | UI/UX: 0
Comment (by mjtamlyn):
Such a field would actually be quite different in implementation to the
current field. We currently lever psycopg2's serialization and while this
can be customised[1], it is most easily customisable at the field level.
There is a possibility of providing an alternative to `Json()` for
adaption (into the db), but coming back out can only be done at the
connection level.
A JSON field which does all of its de-/serialization in the field would be
possible of course, one such implementation is
https://github.com/bradjasper/django-jsonfield/
[1] http://initd.org/psycopg/docs/extras.html#json-adaptation
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:4>
Django
Reporter: jimgraham | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
Do you think there are any action items for this ticket then? If not,
perhaps we could document the restriction and note that alternative fields
exist.
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:5>
Django
Reporter: jimgraham | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Easy pickings: 0 | UI/UX: 0
Comment (by jimgraham):
Based on what you've pointed out, I think pointing to the external
projects is the best thing to do.
Thanks for your time in reviewing this.
Regards.
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:6>
Django
----------------------------------+--------------------------------------
Reporter: jimgraham | Owner: nobody
Type: New feature | Status: closed
Component: contrib.postgres | Version: 1.9
Severity: Normal | Resolution: wontfix
Keywords: JSONB | Triage Stage: Unreviewed
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* resolution: => wontfix
Comment:
[https://github.com/django/django/pull/5928 Doc patch]. I avoided
recommending a particular third-party field since we avoid that as it's
difficult to curate a list of packages that doesn't go stale.
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:7>
Django
----------------------------------+--------------------------------------
Type: New feature | Status: new
Severity: Normal | Resolution:
Easy pickings: 0 | UI/UX: 0
Changes (by aaugustin):
* status: closed => new
* resolution: wontfix =>
Comment:
I hit the same problem: https://groups.google.com/d/msg/django-
developers/upg9pgGvaUs/LMN8Xfq8EQAJ
If Django doesn't provide an escape hatch, I suspect I won't be the only
one to resort to monkey-patching.
Right now I don't have time to dig into this issue but I don't think we
can leave it there altogether.
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:8>
Django
----------------------------------+------------------------------------
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: JSONB | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* version: 1.9 => master
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:9>
Django
----------------------------------+------------------------------------
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: JSONB | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by zachborboa):
* cc: zachborboa@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:10>
Django
----------------------------------+------------------------------------
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: JSONB | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"c432dd40bde37667bfe6bb59eaff0a14c50cd27b" c432dd40]:
{{{
#!CommitTicketReference repository=""
revision="c432dd40bde37667bfe6bb59eaff0a14c50cd27b"
Refs #25995 -- Documented that JSONField doesn't handle sophisticated
serialization.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:11>
Django
----------------------------------+------------------------------------
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: JSONB | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"3503b816cf449c7c94766c8f3f890a52083db392" 3503b81]:
{{{
#!CommitTicketReference repository=""
revision="3503b816cf449c7c94766c8f3f890a52083db392"
[1.9.x] Refs #25995 -- Documented that JSONField doesn't handle
sophisticated serialization.
Backport of c432dd40bde37667bfe6bb59eaff0a14c50cd27b from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:12>
Django
----------------------------------+------------------------------------
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: JSONB | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by debanshuk):
We could use a custom data type representation scheme to represent dates,
decimals etc in JSON.
The scheme could be similar to what MongoDB uses for it's extended JSON
(https://docs.mongodb.com/manual/reference/mongodb-extended-json/#bson-
data-types-and-associated-representations).
Eg: a date object would be stored as {{{ {"$date": <ISO-8601 date string>}
}}}
Or,
It could be something more sophisticated and generic like
{{{
{
"$djExtJson": {
"$type": <type string>,
"$args": <args array>,
"$kwargs": <kwargs object>
}
}
}}}
Where '$type' would be the full path of a class (eg, {{{ datetime.date }}}
or {{{ decimal.Decimal }}}) and '$args' and '$kwargs' would be the
arguments for constructor of the class.
So a date object would be stored as:
{{{
{"$djExtJson": {"$type": "datetime.date", "$args": ["2016", "7", "6"]}}
}}}
and a decimal object as:
{{{
{"$djExtJson": {"$type": "decimal.Decimal", "$args": ["21.5"]}}
}}}
Using this format, one would be able to store any class's object in JSON.
Eg:
{{{
# my_utils.py
class ABC():
def __init__(self, dt: date):
self.date = dt
}}}
An object of above class could be stored as:
{{{
{"$djExtJson": {
"$type": "my_utils.ABC",
"$args": [{"$djExtJson": {"$type": "datetime.date", "$args": ["2016",
"7", "6"]}}]
}}
}}}
Deserialisation of objects using above format would be very simple, as we
would have type information stored withing the object with all the
arguments required to call constructor of that type, so we would be able
to call the constructor and get the object.
For serialisation, it won't be possible to get arguments which need to be
passed to a class's constructor just by looking at a object of that class.
So for that, we can have custom code in Django for supporting
serialisation of common standard library objects and builtins. And for
user defined classes, we can created a mixin (we may call it {{{
DjExtJsonSerialisable }}}) which would simply stores arguments (and
keyword arguments) passed to the constructor of a user defined class in
some attribute, when a object of a class inheriting from at mixin is
created (attribute can {{{ __dj_ext_json_data__ }}}).
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:13>
Django
----------------------------------+------------------------------------
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: JSONB | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by debanshuk):
* cc: debanshuk2007@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:14>
Django
----------------------------------+------------------------------------
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: JSONB | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by claudep):
* has_patch: 0 => 1
Comment:
This [https://github.com/django/django/pull/7071 PR] adds custom encoding
support.
The decoding part is another story. The patch currently recommends using
`from_db`/`from_db_value` hooks. If anyone can come with a concrete
proposal with another method, please propose it.
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:15>
Django
Reporter: jimgraham | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by berkerpeksag):
* stage: Accepted => Ready for checkin
Comment:
[https://github.com/django/django/pull/7071 PR #7071] looks good to me. I
left a comment about the type of the `encoding` parameter in the pull
request.
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:16>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: closed
Component: contrib.postgres | Version: master
Severity: Normal | Resolution: fixed
Keywords: JSONB | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz <claude@…>):
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"13c3e5d5a05e9c358d212d154addd703cac3bc66" 13c3e5d]:
{{{
#!CommitTicketReference repository=""
revision="13c3e5d5a05e9c358d212d154addd703cac3bc66"
Fixed #25995 -- Added an encoder option to JSONField
Thanks Berker Peksag and Tim Graham for the reviews.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25995#comment:17>