Safari and XMLHttpRequest encoding

[Jonathan LaCour]

Seems like I am the only one busy with email today :)

Has anyone come up with a workaround to this problem:

http://trac.turbogears.org/turbogears/ticket/186

... its currently causing me great pain! I filed a bug against
WebKit's bugzilla, but I would love to find a workaround, if possible.

--
Jonathan LaCour
http://cleverdevil.org

[Bob Ippolito]

On Mar 7, 2006, at 2:50 PM, Jonathan LaCour wrote:

>
> Seems like I am the only one busy with email today :)
>
> Has anyone come up with a workaround to this problem:
>
> http://trac.turbogears.org/turbogears/ticket/186
>
> ... its currently causing me great pain! I filed a bug against
> WebKit's bugzilla, but I would love to find a workaround, if possible.

Isn't this irrelevant for JSON since simplejson escapes everything
but ASCII?

-bob

[Jonathan LaCour]

Well, I am not entirely sure. Its possible that the bug I mentioned
isn't my problem, but I have a controller method that returns some
unicode strings inside a dictionary. This controller method is being
called by loadJSONDoc() on the client side, and the unicode string is
being placed into a textarea. On Safari, I see escaped text. On
Firefox, I see proper unicode strings.

So, it seems like this may be the bug I mentioned, but I don't know
for sure.

[Bob Ippolito]

Write a test case so that other people can see the issue.

-bob

[Hiroki Ohtani]

Hi all,

This is popular problem when we handle non-ascii charactor with AJAX or
json.
Safari and KDE Konqueror always mishanlde json charactor encoding.

Workaround is :
urlescape json string at server side
and unescape json string at client side.

[Bob Ippolito]

simplejson already escapes everything outside of ASCII to \uXXXX
literals, which are certainly going to be interpreted correctly on
the client side -- at least when using eval("(" + json + ")") or any
other correct JSON parser.

I want to see a test case because the problem definitely isn't where
the original poster thinks it is... unless he is using a pre-
simplejson TurboGears, in which case the bug was fixed when switching
from json-py to simplejson.

-bob

[Jonathan LaCour]

> Write a test case so that other people can see the issue.

Okay, I created a simple test case as a TurboGears 0.9a1
application. It works fine in Firefox, but doesn't work properly in
Safari. You can download it at:

http://cleverdevil.org/testcase.tar.gz

I definitely could be doing something wrong, but if it works in one
browser and not another, that leads me to believe that its a bug in
the browser.

Thanks for your help.

[Bob Ippolito]

On Mar 8, 2006, at 6:49 AM, Jonathan LaCour wrote:

>
>> Write a test case so that other people can see the issue.
>
> Okay, I created a simple test case as a TurboGears 0.9a1
> application. It works fine in Firefox, but doesn't work properly in
> Safari. You can download it at:
>
> http://cleverdevil.org/testcase.tar.gz
>
> I definitely could be doing something wrong, but if it works in one
> browser and not another, that leads me to believe that its a bug in
> the browser.

Looks like it's a bug in TG somewhere.. simplejson does the right thing:

>>> import simplejson
>>> print simplejson.dumps(dict(text=u'\u00bfHabla espa\u00f1ol?'))
{"text":"\u00bfHabla espa\u00f1ol?"}

but TG does not:

>>> import urllib
>>> print urllib.urlopen('http://127.0.0.1:8080/test').read()
{"tg_flash":null, "text":"¿Habla español?"}

This is apparently Kevin's fault according to svn blame:

370 kevin class GenericJSON(JSONEncoder):
829 kevin def __init__(self):
829 kevin super(GenericJSON, self).__init__
(ensure_ascii = False)

In this revision:
http://trac.turbogears.org/turbogears/changeset/829

Supposedly this "helps" non-ascii characters through JSON, but it
doesn't have the intended result.

The right fix for #430 would've been to ensure unicode strings from
browser input, instead of ensuring potential garbage makes its way
into the JSON output. In this case, it would be a change to the way
URLs (at least the query string) are handled.

This could be fixed by making cherrypy.lib.parseQueryString look at
the resultant dict for keys that are outside of ASCII and decoding
them to UTF-8... or it could be done as a filter like the formencode
NestedVariablesFilter in startup.

-bob

[Kevin Dangoor]

On 3/8/06, Bob Ippolito <b...@redivi.com> wrote:
> The right fix for #430 would've been to ensure unicode strings from
> browser input, instead of ensuring potential garbage makes its way
> into the JSON output. In this case, it would be a change to the way
> URLs (at least the query string) are handled.
>
> This could be fixed by making cherrypy.lib.parseQueryString look at
> the resultant dict for keys that are outside of ASCII and decoding
> them to UTF-8... or it could be done as a filter like the formencode
> NestedVariablesFilter in startup.

You're correct about the diagnosis and the solution. Thanks for
looking at this! I've reopened #430 and rolled back the original
patch.

Kevin

[Jonathan LaCour]

> You're correct about the diagnosis and the solution. Thanks for
> looking at this! I've reopened #430 and rolled back the original
> patch.

I patched my TG to reflect the change, and my application now works
perfectly.

Thanks for tracking this down!