Internationalization and Javascript issue

[zaf]

Hi everyone,
I started my web app in french, which means there's a fairly good
amount of strings that contain non-ascii characters. While this isn't
a problem in my python files or templates (all I have to do is write
the string inside
_(u' ') and it works fine), it is a problem when I have accents in a _
(' ') inside a JS file.

Basically, the following js code (using jquery) :

$('<li class="delivery_req" style="font-style:italic"><a>'+_("Vous
devez sélectionner au moins une version")+'</a></li>').appendTo($
(await_ul));

provokes the following error when doing tg-admin collect :

Working on cinego/static/javascript-lib/jquery.progressbar.js
Traceback (most recent call last):
File "/usr/bin/tg-admin", line 8, in <module>
load_entry_point('TurboGears==1.0.8', 'console_scripts', 'tg-
admin')()
File "/usr/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/
turbogears/command/base.py", line 400, in main
command.run()
File "/usr/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/
turbogears/command/i18n.py", line 137, in run
self.scan_source_files()
File "/usr/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/
turbogears/command/i18n.py", line 310, in scan_source_files
self.scan_js_files(tmp_potfile, js_files)
File "/usr/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/
turbogears/command/i18n.py", line 414, in scan_js_files
text = normalize(text.decode().encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
12: ordinal not in range(128)

Is there anyway to get the internationalization module to accept non-
ascii characters ? I really don't want to have to look for every
string an correct them.
I tried using the -a option of tg-admin collect but I still get the
same error.
Thanks in advance.
Tom

[Diez B. Roggisch]

zaf schrieb:

This comes from a Python-file I presume? If yes, why don't you prefix
the _("Vous... with a u to make it a unicode literal?

However, you must be aware that this only works if the code in question
is executed at *runtime*, not at import-time. A bit more context would
be useful.

Diez

[zaf]

Actually it doesn't come from a python file, it comes from a
Javascript file, so I can't use the u prefix (I tried).
I've use the u prefix in all my python file.
but there is also a TG internationalization method _( ) for javascript
strings and they won't allow my non-ascii characters apparently.
Tom

[Diez B. Roggisch]

zaf schrieb:

> Actually it doesn't come from a python file, it comes from a
> Javascript file, so I can't use the u prefix (I tried).
> I've use the u prefix in all my python file.
> but there is also a TG internationalization method _( ) for javascript
> strings and they won't allow my non-ascii characters apparently.

Ah, ok, now I see. This is obviously a bug. The decode should take an
encoding-parameter, but that must be passed on the commandline then.

As a workaround, you should be able to use english or simplified
messages & map them to the translations.

I the meantime, you might just patch the thing yourself, passing the
encoding of your choice.

Diez

[zaf]

> Ah, ok, now I see. This is obviously a bug. The decode should take an
> encoding-parameter, but that must be passed on the commandline then.
>
> As a workaround, you should be able to use english or simplified
> messages & map them to the translations.

I really don't want to have to go through all my files and change the
strings that are the problem.

>
> I the meantime, you might just patch the thing yourself, passing the
> encoding of your choice.

Can you be a little more specific ? I don't understand what you mean
by passing the encoding of my choice. Where do I do that ? And what
encoding should be used ?

>
> Diez

[zaf]

okay so I patched it this way for me and it looks like it's working (I
have to check further though).
In /usr/lib/python2.5/site-packages/TurboGears-1.0.8-py2.5.egg/
turbogears/command/i18n.py line 414, I replaced :
text = normalize(text.encode('utf-8'))
with :
text = normalize(text)

and on line 182 I replaced :
key = unicode(key)
with :
key = unicode(key.decode('utf-8'))
Now I'm still kinda of confused when it comes to uncide and encoding
so I don't really know why this works and the previous way didn't.
Could this be a patch for TG ?
Tom

[Christoph Zwerschke]

zaf schrieb:

> Could this be a patch for TG ?

Yes, please create a trac ticket so we can sort this out and fix it in
TG 1.x.

-- Christoph