JSON serializer and UTF-8

[olive]

Hello,

I'm trying to send som UTF-8 data this way:

json = serializers.serialize("json", myQuerySet, ensure_ascii=False)
return HttpResponse(json, mimetype="text/javascript; charset=UTF-8")

But the accented characters are not displayed properly in the browser.

But if I put this on the first line of my views.py:

# -*- coding: UTF-8 -*-

and I try to send hard coded data this way:

json = '[{"pk": "1", "model": "body.datacontent", "fields":
{"content": "Heading à 1", "type": "hdg"}},{"pk": "2", "model":
"body.datacontent", "fields": {"content": "Para 1", "type": "par"}}]'

Then "Heading à 1" is properly displayed in the browser.

Question: am I using serializers.serialize the right way or is it
broken ?

Note: my database is UTF-8 encoded too (I have tested that by
extracting an accented string from it and displaying it in the
browser).

Olive

[patrickk]

I never got that to work either - here´s how I´m dealing with it:

import django.utils.simplejson as simplejson

data = []
...
try:
school_list = School.objects.filter(...)
for school in school_list:
school_info = school.name.decode('utf-8') + " / " +
school.address.decode('utf-8')
data.append([school.id, school_info])
...
jsonList = simplejson.dumps((data))
...
return HttpResponse(jsonList)

patrick

[olive]

.. and this the way I've done it in the mean time:

json = '['
for co in section.contentsorder_set.all().iterator():
if json != '[':
json += ','
json += '{"content": "'+co.datacontent.content+'", "type":
"'+co.datacontent.type+'"}'
json += ']'

Time to write an helper fn !

thanks anyway,

Olivier

[Gábor Farkas]

olive wrote:
> Hello,
>
> I'm trying to send som UTF-8 data this way:
>
> json = serializers.serialize("json", myQuerySet, ensure_ascii=False)
> return HttpResponse(json, mimetype="text/javascript; charset=UTF-8")
>
> But the accented characters are not displayed properly in the browser.
>
>
> But if I put this on the first line of my views.py:
>
> # -*- coding: UTF-8 -*-
>
> and I try to send hard coded data this way:
>
> json = '[{"pk": "1", "model": "body.datacontent", "fields":
> {"content": "Heading à 1", "type": "hdg"}},{"pk": "2", "model":
> "body.datacontent", "fields": {"content": "Para 1", "type": "par"}}]'
>
> Then "Heading à 1" is properly displayed in the browser.
>
>
> Question: am I using serializers.serialize the right way or is it
> broken ?

i think you're using it the right way..

actually the problem lies somewhere in simplejson (the library that the
json-serializer uses).

example:

a = u'gábor'
b = a.encode('utf-8')

from django.utils.simplejson import dumps

In [37]: dumps(a,ensure_ascii=False)
Out[37]: u'"g\xe1bor"'

In [38]: dumps(a,ensure_ascii=True)
Out[38]: '"g\\u00e1bor"'

In [39]: dumps(b,ensure_ascii=False)
Out[39]: '"g?\xa1bor"'

In [40]: dumps(b,ensure_ascii=True)
Out[40]: '"g\\u00c3\\u00a1bor"'

so the unicode-string versions work fine, but with the
bytecode-versions, it just doesn't work.

gabor

[olive]

Thank you Patrick and Gábor,

Here is my final view (tested with Django HTTP server, FireFox 2.0 and
IE6.0):

from django.shortcuts import HttpResponse
from wabe.body.models import Section
from django.utils.simplejson import dumps
def loadcontent(request):
post = request.POST.copy()
section = Section.objects.get(pk=post['section'])
contents = []
for co in section.contentsorder_set.all().iterator():
contents.append({'content': co.datacontent.content,
'type': co.datacontent.type})
json = dumps(contents, ensure_ascii=False)
return HttpResponse(json, mimetype="text/json;
charset=UTF-8")