Extending JSONField serialization

[Aymeric Augustin]

Hello,

I’m using the JSONField provided by django.contrib.postgres for logging arbitrary data to PostgreSQL.

For this use case, I’d like my data to be JSON serialized with as little fuss as possible. Unfortunately lots of things (dates, datetimes, decimals, etc.) aren’t natively JSON-serializable and django.contrib.postgres doesn’t seem to provide a way to override the serializer.

I can wire Django’s JSON encoder, which provides reasonable encoding rules for all data types I need, by monkey-patching JSONField:


# Monkey-patch django.contrib.postgres to use a smarter JSON serializer.

import json

from django.contrib.postgres.fields import jsonb
from django.core.serializers.json import DjangoJSONEncoder


class DjangoJson(jsonb.Json):

def dumps(self, obj):
return json.dumps(obj, cls=DjangoJSONEncoder)


jsonb.Json = DjangoJson


Is this a common problem? Should JSONField accept additional kwargs to customize the encoder and the decoder?

Best regards,

--
Aymeric.

[Tim Graham]

This came up in a ticket a couple days ago: https://code.djangoproject.com/ticket/25995

[Tom Christie]

> Should JSONField accept additional kwargs to customize the encoder and the decoder?

Quick take here:

That sounds like a bit too much "cleverness" to me. The most obvious issue it'd cause is putting values of one type into the field, but getting objects of a different type back. (eg datetime getting coerced into a string on the way in, and left as a string on the way out).
I'd rather see the field closely map to what the underlying database field *actually* provides.

[Aymeric Augustin]

> On 5 janv. 2016, at 18:37, Tom Christie <christ...@gmail.com> wrote:
>
>> Should JSONField accept additional kwargs to customize the encoder and the decoder?
>
> Quick take here:
>
> That sounds like a bit too much "cleverness" to me. The most obvious issue it'd cause is putting values of one type into the field, but getting objects of a different type back. (eg datetime getting coerced into a string on the way in, and left as a string on the way out).

Yes, I understand how that could surprise a developer.

A smart deserializer that would attempt to convert some strings back to their original type, based on their content, would create the opposite risk: a string that matches the format of a date could be accidentally returned as a date.

I wouldn’t do this for mission-critical data — but then I wouldn’t store it in a JSON field either. Django projects should only use a JSON field for data that isn’t worth normalizing into actual fields. Writing a schema to map keys to types defeats the point; if you’re writing a schema, just express it with traditional model fields.

I don’t think Django should (de)serialize non-native JSON types by default, but it should make it possible through public APIs, as this is a common requirement. For my use case, logging, the convenience of being able to store dates, datetimes and decimals without resorting the heavy guns (DRF serializers) helps a lot.

--
Aymeric.

[Michael Manfre]

It should be configurable and I like the kwargs idea. I've also had to monkey patch JSONField in this way for datetimes.

Regards,

Michael Manfre

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/B995F2B7-E1D1-4F82-8032-E502EA11680A%40polytechnique.org.

For more options, visit https://groups.google.com/d/optout.

--

GPG Fingerprint: 74DE D158 BAD0 EDF8

keybase.io/manfre

[Dwight Gunning]

It's interesting that you say JSON Fields shouldn't be used for mission critical data. Is that widely recognised?

I feel like there are genuine uses cases for using JSON Fields to store mission critical data. For instance, a Javascript single-page-app style client with a set of user preferences to adjust the UI look and feel. These preferences need to be persisted between sessions but don't really need to be normalised or separated into individual fields by Django, aside from perhaps light validation.

Datetime encoding/decoding to integrate with third party systems is a regular headache for me. I typically writ DRF serializers to take care of datetime formatting, but a more general solution closer to the data layer would be nice.

Regards,

Dwight

--

[Aymeric Augustin]

Hello Dwight,

I was trying to express the fact that JSONField is appropriate for storing data that won’t be used for joining other tables, filtering, aggregating, etc. but rather just for reading.

“Not mission-critical” was a simplification. That said, data that meets the criteria above tends not to be the most important data in an application.

In your example, at worst, if your app fails to read what it wrote previously to the JSON field, the user will just have to set their preferences again.

-- 

Aymeric.

-- 
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a2868ea8-c559-4240-aaba-dbf1f6efbe64%40googlegroups.com.

[Tom Christie]

Customizing the encoder (or even using DjangoJSONEncoder by default) isn't so bad.

I'm less convinced about the usefulness of customizing the decoder - once you've encoded the data into JSON any additional type information is lost, so casting back to python primitives is always going to need to be handled explicitly.

Being able to set field values that contain eg datetimes *could* be a useful shortcut, but it only makes sense to me if we make it clear that values passed into the JSON field will only ever be stored/retrieved as their JSON primitive representations.

Cheers,

  Tom

[Michael Manfre]

On Wed, Jan 6, 2016 at 9:58 AM, Tom Christie <christ...@gmail.com> wrote:

Customizing the encoder (or even using DjangoJSONEncoder by default) isn't so bad.

I'm less convinced about the usefulness of customizing the decoder - once you've encoded the data into JSON any additional type information is lost, so casting back to python primitives is always going to need to be handled explicitly.

Whether or not a configurable decoder seems useful for the situations we can think of, only allowing users to configure half of the encode/decode cycle seems odd to me. Allowing both to be configurable requires a trivial amount of extra effort to implement and maintain.

Regards,

Michael Manfre

[Dwight Gunning]

Aymeric, thanks for clarifying. I take your point for joins although I'm a bit surprised you don't feel JSONFields are appropriate for filtering.

I haven't tried yet but I am interested in the ability to efficiently index into the JSONField (thanks to use of Postgres' jsonb fields). That is, to design models that are a hybrid of our traditional, schema-oriented approach, but also include a JSON Field and offer the benefits of the more recent schemaless/nosql approach.

Sorry if that's all a bit off topic for the django-dev list. Happy to switch over to django-users if that's more appropriate.

Dwight 

--

Dwight Gunning

Twitter me: @dwightgunning

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/upg9pgGvaUs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/E507FE42-D738-410B-B447-D152A67797A3%40polytechnique.org.

[Raphaël Barrois]

Indeed, this is only helpful if the user is able to customize both encoding and decoding of its messages.
For instance, on one of our projects, we use a custom serializer that encodes complex types as {"__type__": "foo", "value": "bar"} dicts.
Our deserializer can then handle the decoding of such fields to a native Python datastructure.

This is possible with "jsonfield" (https://pypi.python.org/pypi/jsonfield) which allows to set a custom encoder/decoder.

Coming back to Aymeric's question, I'd love to be able to set a custom decoder/encoder for a JSONField.
Being able to only specify the encoding is error-prone, since my code will then have to handle the distinction
between "this is a value set pre-save, so it's in a native type" and "this time, it's from the database, so I have to cast it back".


Regards,

--
Raphaël Barrois

[Shai Berger]

Pet peeve: Please ignore if you don't care about database technologies.

On Wednesday 06 January 2016 17:12:34 Dwight Gunning wrote:
> design models that are a hybrid of our traditional, schema-oriented
> approach, but also include a JSON Field and offer the benefits of the more
> recent schemaless/nosql approach.
>

Actually, nosql brought new ideas in terms of database distribution; as far as
data management goes, the whole schemaless movement is one big regression. The
"more recent" approach is, in fact, older, and the Relational Model was mostly
a solution to its problems.

Shai.

[Tom Christie]

Let's stay on track, here, and not get distracted by the relative merits of document DBs vs relational DBs. :)

I don't have any serious objections to including option encoder/decoder arguments, but I'm not overly keen either, so color me a -0.

[Marcin Nowak]

On Tuesday, January 5, 2016 at 5:29:14 PM UTC+1, Tim Graham wrote:

This came up in a ticket a couple days ago: https://code.djangoproject.com/ticket/25995

May I ask why this ticket was resolved just by adding some information to the documentation?

I do not understand why some 3rd party package has some significance here.

Let's add encoder_class as an optional kwarg of JSONField and pass it to loads/dumps calls.

Bad idea?

Marcin

[Daniele Procida]

On Wed, Jun 22, 2016, Marcin Nowak <marcin....@gmail.com> wrote:

>On Tuesday, January 5, 2016 at 5:29:14 PM UTC+1, Tim Graham wrote:
>>
>> This came up in a ticket a couple days ago:
>> https://code.djangoproject.com/ticket/25995

>May I ask why this ticket was resolved just by adding some information to
>the documentation?

It's not resolved - the ticket is open, and Accepted.

>I do not understand why some 3rd party package has some significance here.

While it remains unresolved, that's useful information.

Daniele

[Marcin Nowak]

It's not resolved - the ticket is open, and Accepted.

Thanks. I've seen comment with status set to closed and I didn't notice the current.

My fault.

Marcin