Django, Simplejson and speedups

[Johan Bergström]

Simplejson has been getting some version increments lately and Django
hasn't been including them (there are tickets), which is - in my
opinion - okay. Constantly upgrading the included simplejson library
is an never ending rabbit chase where required effort doesn't motivate
possible benefits (read: speedups) if no serious bugs should surface.

I still wonder if there's a better solution for somehow allowing users
to drop in their own simplejson without resorting to edit
django.utils.simplejson or overriding sys.modules. This would reduce
the need for upgrading the bundled package more than once each year
something as well as allowing C-speedups come into play. The current
code for C-speedups looks a bit messed up as well.

Including native simplejson libraries on the fly feels a bit scary
since it allows for messing up both tests and running code, so that's
out of the question. I know I'm short of solutions here, but the
current behavior could improve. Any ideas?

[Ludvig Ericson]

On Nov 28, 2008, at 10:56, Johan Bergström wrote:
> I still wonder if there's a better solution for somehow allowing users
> to drop in their own simplejson without resorting to edit
> django.utils.simplejson or overriding sys.modules. This would reduce
> the need for upgrading the bundled package more than once each year
> something as well as allowing C-speedups come into play. The current
> code for C-speedups looks a bit messed up as well.

I've been having these concerns lately as well. Especially the speedups
part.

The Django-bundled simplejson code runs:

from django.utils.simplejson._speedups import
encode_basestring_ascii as c_encode_basestring_ascii

This, obviously, was an unintended search-and-replace artifact as there
is no possible way to get that speedup module there unless you're
really trying. (read: hack up sys.modules.)

So, my suggestion as far as speedups goes is to just leave it at:

from simplejson._speedups import encode_basestring_ascii as
c_encode_basestring_ascii

(And likewise for decoder.py, I can make a patch if wanted.)

Obviously that races the concern that `simplejson._speedups` might be
version 8737 and not the desired revision we expect, but I'm willing to
bet it's safe, and if the API should change dramatically (unlikely),
we'll notice fairly quickly by means of tests failing.

- Ludvig

[Malcolm Tredinnick]

On Fri, 2008-11-28 at 01:56 -0800, Johan Bergström wrote:
> Simplejson has been getting some version increments lately and Django
> hasn't been including them (there are tickets), which is - in my
> opinion - okay. Constantly upgrading the included simplejson library
> is an never ending rabbit chase where required effort doesn't motivate
> possible benefits (read: speedups) if no serious bugs should surface.

At 1.0 time we had the latest release installed (or we may have only
been one behind). We'll update to a later version over time in trunk,
certainly before 1.1. I have a note to do this at some point in the
future, but it always takes a little bit of time to do the merge and
check that there are no breakages (particularly, as Ludvig notes, we
didn't quite get this right last time). There's no point doing an update
for every single release unless there's some major bug.

I'm not at all worried about being a little behind between major
releases.

>
> I still wonder if there's a better solution for somehow allowing users
> to drop in their own simplejson without resorting to edit
> django.utils.simplejson or overriding sys.modules. This would reduce
> the need for upgrading the bundled package more than once each year
> something as well as allowing C-speedups come into play. The current
> code for C-speedups looks a bit messed up as well.

I dislike this. We should just ship a recent release. I don't see a huge
win for allowing the relatively rare case of wanting to run your own
simplejson, given that it introduces yet another setting (or importing
both the Django and the local version and doing a version comparison).

Malcolm

[flo...@gmail.com]

I think there's actually a big win in favoring a system-installed
simplejson over Django's included version. The system-installed
package is actually orders of magnitude faster than Django's included
version. That's reason enough to do some small code changes. In my
opinion, this could all be solved by a bit of code:

try:
import simplejson
except ImportError:
try:
import json as simplejson
except ImportError:
from django.utils import simplejson

Django already does this in various parts of the codebase.
Considering that simplejson is now part of Python2.6, compatibility
concerns should be much less of a problem, since newer releases of
Python libraries are generally backwards compatible with older
versions. (That and simplejson has a long track record of preserving
backwards compatibility.)

Once/if Django deprecates Python 2.5 compatibility, the
django.utils.simplejson version could simply be removed in favor of a
system-installed simplejson or the one in the standard library.

Thanks,
Eric Florenzano

On Nov 28, 6:10 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:

[Malcolm Tredinnick]

On Fri, 2008-11-28 at 19:05 -0800, flo...@gmail.com wrote:
> I think there's actually a big win in favoring a system-installed
> simplejson over Django's included version. The system-installed
> package is actually orders of magnitude faster than Django's included
> version.

That's a bit of a universal claim and not really supportable. You don't
know what version the "system installed" one is for every system out
there. In some cases, the installed version will be slower. And, as I
mentioned originally, I think we should endeavour to keep up with the
latest release certainly for each released major version of Django, and
more frequently as time permits.

Maybe you meant that preference for the C-compiled extension should be
preferred. I address that below.

> That's reason enough to do some small code changes. In my
> opinion, this could all be solved by a bit of code:
>
> try:
> import simplejson
> except ImportError:
> try:
> import json as simplejson
> except ImportError:
> from django.utils import simplejson

As I mentioned in my original post (without laying out all the details)
and above, this would require version checking, since there's absolutely
no guarantee that the "installed" version is going to be more recent
than Django's version.

I wouldn't mind testing for the C version and importing that
conditionally, since Bob Ippolito does mention 10 - 150x speed ups for
string processing there (quite believable for Python vs. C with
intensive string processing), but it's not always going to be at the top
end for us, since simplejson is just one component in the serialisation
pipeline and we still have to go through the C-Python barrier to get to
the __unicode__ or __str__ method, since our internal serialisation is
mostly operating on objects.

>
> Django already does this in various parts of the codebase.
> Considering that simplejson is now part of Python2.6, compatibility
> concerns should be much less of a problem,

What is in Python 2.6 is a consideration only. It's not the only version
of Python being used.

Regards,
Malcolm

[flo...@gmail.com]

On Nov 28, 7:31 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:

> That's a bit of a universal claim and not really supportable. You don't
> know what version the "system installed" one is for every system out
> there. In some cases, the installed version will be slower.

Very good point. Thanks for hitting me with the "everyone is not you"
stick :)

> I wouldn't mind testing for the C version and importing that
> conditionally, since Bob Ippolito does mention 10 - 150x speed ups for
> string processing there (quite believable for Python vs. C with
> intensive string processing)

This is really the central point I was trying to go for. I think the
massive potential speedups are worth the slightly uglier code.

> What is in Python 2.6 is a consideration only. It's not the only version
> of Python being used.

Of course. That's why in my code snippet I imported it
conditionally. My point was that the modules in the Python standard
library (or rather, modules chosen for inclusion in the standard
library) tend to remain relatively backwards compatible after their
inclusion. So while it always pays to be diligent, in all likelihood
the API won't change dramatically going forward.

Thanks,
Eric Florenzano

[Malcolm Tredinnick]

On Fri, 2008-11-28 at 20:49 -0800, flo...@gmail.com wrote:
[...]

> > I wouldn't mind testing for the C version and importing that
> > conditionally, since Bob Ippolito does mention 10 - 150x speed ups for
> > string processing there (quite believable for Python vs. C with
> > intensive string processing)
>
> This is really the central point I was trying to go for. I think the
> massive potential speedups are worth the slightly uglier code.

But the devil is in the details: trying to avoid multiple imports to
check if the C accessory is available.

In any case, I've got a patch that fixes things up: updates Django to
latest version (2.0.5), prefers system version if it's of an equal or
later version or has the C speedups and prefers the python 2.6 json
module over the Django version after that.

However, since James has started a "let's deprecate simplejson" thread,
I'm going to wait until that comes to some conclusion before doing
anything, otherwise things will just look confusing.

Regards,
Malcolm