[Django] #22821: DjangoJSONEncoder no longer supports simplejson

Django

unread,

Jun 12, 2014, 10:48:12 AM6/12/14

to django-...@googlegroups.com

#22821: DjangoJSONEncoder no longer supports simplejson
-----------------------------------------+--------------------
Reporter: Keryn Knight <django@…> | Owner: nobody
Type: New feature | Status: new
Component: Core (Serialization) | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------------+--------------------
For better or worse, `DjangoJSONEncoder`, despite living in the
`core.serializers` package (and to my knowledge not being public API), is
useful to support the few extra datatypes in standard json usage, and
doesn't have any negative side affects. So it gets used.

Previously, in Django 1.4, `core.serializers.json` relied upon
`simplejson`, but starting with 1.5 only used `json`.

Since then, `simplejson` has moved forward and has a slightly different
API that prevents using `DjangoJSONEncoder`, because it subclasses
`json.JSONEncoder` rather than `simplejson.JSONEncoder`, thus this worked
in 1.4:
{{{
import simplejson as json
from django.core.serializers.json import DjangoJSONEncoder
json.dumps("{}", cls=DjangoJSONEncoder)
}}}
but in 1.5+, it yields a TypeError, with `simplejson==3.5.2`:
{{{
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/path/to/python2.7/site-packages/simplejson/__init__.py", line
382, in dumps
**kw).encode(obj)
TypeError: __init__() got an unexpected keyword argument
'namedtuple_as_object'
}}}
I think it ought to be possible to decouple the implementation such that
people wanting to continue using simplejson could still benefit from the
implementation of default, by doing something like (untested, pseudo-
code):
{{{
class RichEncoder(object):
def default(self, o):
...
return (RichEncoder, self).default(o)

class DjangoJSONEncoder(RichEncoder, json.JSONEncoder): pass
DateTimeAwareJSONEncoder = DjangoJSONEncoder
}}}
meanwhile, userland implementations would replace the `json.JSONEncoder`
part with `simplejson.JSONEncoder`, I guess.

This came up and bit someone in the IRC channel, and I'm reporting it
because I ''think'' it's supportable, though it'd benefit me nought.

--
Ticket URL: <https://code.djangoproject.com/ticket/22821>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Jun 12, 2014, 11:29:08 AM6/12/14

to django-...@googlegroups.com

#22821: DjangoJSONEncoder no longer supports simplejson

-------------------------------------+-------------------------------------
Reporter: Keryn Knight | Owner: nobody
<django@…> | Status: new
Type: New feature | Version: master
Component: Core | Resolution:
(Serialization) | Triage Stage:
Severity: Normal | Unreviewed
Keywords: | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by aaugustin):

* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0

Comment:

See #18023 for why this is hard.

If I remember correctly, the short version is that simplejson broke both
backwards and forwards compatibility with itself, making it impossible to
be compatible with arbitrary versions of simplejson.

--
Ticket URL: <https://code.djangoproject.com/ticket/22821#comment:1>

Django

unread,

Jun 12, 2014, 1:25:18 PM6/12/14

to django-...@googlegroups.com

#22821: DjangoJSONEncoder no longer supports simplejson

Comment (by Keryn Knight <django@…>):

Perhaps I'm not understanding the nuances of the problem (it wouldn't be
the first time), but I'm not convinced that solving this particular
problem is difficult (periphery around it, perhaps)
The following code-bomb works for me using `simplejson` versions (the last
''PATCH'' releases of every ''MINOR'' version):
* 2.1.6
* 2.6.2
* 3.0.9
* 3.3.3
* 3.4.1
* 3.5.2
and the `json` module in Python 2.7.6 (I'm guessing/hoping the least Py3K
compatible part of this is using `xrange`, and `print` as a keyword)
{{{
import simplejson
import json
import decimal
import datetime

# So this is taken straight out of the DjangoJSONEncoder implementation
# in master as of 7548aa8ffd46eb6e0f73730d1b2eb515ba581f95

class RichEncoder(object):
def default(self, o):

# See "Date Time String Format" in the ECMA-262 specification.
if isinstance(o, datetime.datetime):
r = o.isoformat()
if o.microsecond:
r = r[:23] + r[26:]
if r.endswith('+00:00'):
r = r[:-6] + 'Z'
return r
elif isinstance(o, datetime.date):
return o.isoformat()
elif isinstance(o, datetime.time):
if is_aware(o):
raise ValueError("JSON can't represent timezone-aware
times.")
r = o.isoformat()
if o.microsecond:
r = r[:12]
return r
elif isinstance(o, decimal.Decimal):
return str(o)
else:
return super(RichEncoder, self).default(o)

# this line is what would end up as DjangoJSONEncoder
# and also aliased as DateTimeAwareJSONEncoder
class stdlibJSONEncoder(RichEncoder, json.JSONEncoder): pass

# this would by necessity be a userland implementation, but is obviously
less
# code to copy-paste & maintain.
class simpleJSONEncoder(RichEncoder, simplejson.JSONEncoder): pass

data = {'decimal': decimal.Decimal('4.011'), 'datetime':
datetime.datetime.today()}

print "Naive:"
# naive, stdlib
try:
json.dumps(data)
except TypeError as e:
print e
# naive, simplejson
try:
simplejson.dumps(data)
except TypeError as e:
print e

print "\n\nEncoding from Django:"
print json.dumps(data, cls=stdlibJSONEncoder)
print simplejson.dumps(data, cls=simpleJSONEncoder)
# allow decimals to be strings
print simplejson.dumps(data, cls=simpleJSONEncoder, use_decimal=False)

class CustomJSONEncoder(object):
def default(self, o):
# contrived example ...
if hasattr(o, '__iter__'):
return tuple(o)
return super(CustomJSONEncoder, self).default(o)

# so you need to encode *other* things, but you want the date/decimal
handling
# both of these would obviously be userland implementations
class stdlibCustomJSONEncoder(CustomJSONEncoder, stdlibJSONEncoder): pass
class simpleCustomJSONEncoder(CustomJSONEncoder, simpleJSONEncoder): pass
# long-winded version.
class simpleCustomJSONEncoder2(CustomJSONEncoder, RichEncoder,
simplejson.JSONEncoder): pass

more_data = {'xrange': xrange(0, 10), 'datetime':
datetime.datetime.today()}

print "\n\nEncoding from Django, + extra stuff:"
print json.dumps(more_data, cls=stdlibCustomJSONEncoder)
print simplejson.dumps(more_data, cls=simpleCustomJSONEncoder)
# long-winded version, as above
print simplejson.dumps(more_data, cls=simpleCustomJSONEncoder2)

print "\n\nWithout our custom encoder:"
last_data = {'xrange': xrange(0, 10)}
try:
json.dumps(last_data)
except TypeError as e:
print e
try:
simplejson.dumps(last_data)
except TypeError as e:
print e
}}}
None of these exhibit the `namedtuple_as_object` issue presented in the
ticket itself, because their encoders match their expectations.

If `simplejson` is still buggy, 30-40 releases later, as the mentioned
ticket indicated, that's fine; that's for an end-user to deal with, or for
`simplejson` to handle in future releases, but as long as the contract for
a JSONEncoder instance's default is just accept `o` and return a
serializable version of it, I'd think the above would continue to work.

By separating the custom `default` object introspection out of the encoder
itself, it at least grants the possibility of users getting the best of
both Django's code and simplejson's, if that is their desire. The danger
of passing a `json.JSONEncoder` to `simplejson.dumps()` as that ticket
describes, is then circumventable, though left as an exercise to the user.

--
Ticket URL: <https://code.djangoproject.com/ticket/22821#comment:2>

Django

unread,

Jun 24, 2014, 8:39:05 AM6/24/14

to django-...@googlegroups.com

#22821: DjangoJSONEncoder no longer supports simplejson

Comment (by timo):

If this broke in 1.5 and hasn't been raised since then, I am thinking it
may not be worth it. I'm also not sure Django should get further into the
serializer business.

Consider the the maintenance burden of adding simplejson to the test
dependencies. Also the latest version doesn't support Python 3.2 so we'd
have to skip the tests there.

--
Ticket URL: <https://code.djangoproject.com/ticket/22821#comment:3>

Django

unread,

Jul 7, 2014, 6:52:30 PM7/7/14

to django-...@googlegroups.com

#22821: DjangoJSONEncoder no longer supports simplejson

-------------------------------------+-------------------------------------
Reporter: Keryn Knight | Owner: nobody

<django@…> | Status: closed

Type: New feature | Version: master

Component: Core | Resolution: wontfix