TurboGears has problems with utf-8 flash messages

[forgems]

turbogears.flash(u"Nieprawidlowe haslo lub login")
raise cherrypy.HTTPRedirect("/login")

When i have something like this, turbogears raises exception.
Here is stacktrace:
Traceback (most recent call last):
File
"/usr/lib/python2.4/site-packages/CherryPy-2.1.0_betadev-py2.4.egg/cherrypy/_cphttptools.py",
line 276, in run
main()
File
"/usr/lib/python2.4/site-packages/CherryPy-2.1.0_betadev-py2.4.egg/cherrypy/_cphttptools.py",
line 498, in main
body = page_handler(*args, **cherrypy.request.paramMap)
File
"/usr/lib/python2.4/site-packages/TurboGears-0.5.1-py2.4.egg/turbogears/controllers.py",
line 93, in newfunc
output = func(self, *args, **kw)
File "/home/forge/tgportal/tgportal/controllers.py", line 58, in
login
turbogears.flash(u"Nieprawidlowe haslo lub login")
File
"/usr/lib/python2.4/site-packages/TurboGears-0.5.1-py2.4.egg/turbogears/controllers.py",
line 103, in flash
cherrypy.response.simpleCookie['turbogearsflash'] = message
File "/usr//lib/python2.4/Cookie.py", line 580, in __setitem__
rval, cval = self.value_encode(value)
File "/usr//lib/python2.4/Cookie.py", line 667, in value_encode
strval = str(val)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
9-10: ordinal not in range(128)

Is there a way to have non ascii characters in flash message ?

[forgems]

Ohh. If i remove 'u' before flash message. I have weird characters
instead of unicode string.
My python source encoding is UTF-8

[Kevin Dangoor]

Your Python source might be UTF-8, but I think you need a
sitecustomize file if you want str(foo) to automatically use UTF-8
encoding.

On 9/20/05, forgems <for...@gmail.com> wrote:
>
> turbogears.flash(u"Nieprawidlowe haslo lub login")
> raise cherrypy.HTTPRedirect("/login")

can you try
turbogears.flash(u"Nieprawidlowe haslo lub login".encode("utf8"))

I'm pretty sure that will work. I'm also quite certain that it would
be nicer for the flash function to do that for you.

Kevin

[forgems]

Hi, still doesn't work :(
Now it ouputs sth like that
<DIV CLASS="flash">Nieprawid&#196;&#185;&#194;&#130;owe
has&#196;&#185;&#194;&#130;o lub login</DIV>
Instead of Unicode characters i have html codes.
For one letter it's 4-codes.

[Kevin Dangoor]

On 9/20/05, forgems <for...@gmail.com> wrote:

> Now it ouputs sth like that
> <DIV CLASS="flash">Nieprawid&#196;&#185;&#194;&#130;owe
> has&#196;&#185;&#194;&#130;o lub login</DIV>
> Instead of Unicode characters i have html codes.
> For one letter it's 4-codes.

Yuck. Sounds like we need a test case. Are you running Safari, by any chance?

Kevin

--
Kevin Dangoor
Author of the Zesty News RSS newsreader

email: k...@blazingthings.com
company: http://www.BlazingThings.com
blog: http://www.BlueSkyOnMars.com

[forgems]

Nope, I'm running firefox 1.06 under linux.
I think that's problem with Kid.

[Kevin Dangoor]

On 9/20/05, forgems <for...@gmail.com> wrote:
>

> Nope, I'm running firefox 1.06 under linux.
> I think that's problem with Kid.

I'm pretty sure that Kid handles unicode just fine.

The flash message is stored in a cookie. I'd be more inclined to debug
down the cookie path and see that everything is happening well there.

I haven't tried UTF-8 encoding something that's already UTF-8. I'm
assuming you get the same string out, right? If so, Kid really
shouldn't be the problem.

[bon...@gmail.com]

could it be viewer issue or missing some encoding info some where in
the header ?

[forgems]

Browser has the right encoding set. This means it's utf-8.
I think that Kevin has right, and the problem lies somewhere in
cookies. I'm gonna check this tommorow at work.

[Ian Bicking]

Kevin Dangoor wrote:
> I haven't tried UTF-8 encoding something that's already UTF-8. I'm
> assuming you get the same string out, right? If so, Kid really
> shouldn't be the problem.

When you do:

orig = u'\u1000'
enc = orig.encode('UTF-8')
enc2 = enc.encode('UTF-8')

when calling enc.encode, it decodes the unicode string using the default
encoding (ASCII) and then re-encodes to UTF-8. So re-encoding will
typically cause Unicode errors. But not until a non-American uses it --
we can just blithely do whatever we want in our small ASCII world :-/

I haven't used it yet (I have good Unicode intentions, but not good
Unicode actions ;), but I made this function to help me build test cases
with Unicode: http://svn.colorstudy.com/home/ianb/makeuni.py (I'm not
apt to actually write Unicode literals otherwise).

--
Ian Bicking / ia...@colorstudy.com / http://blog.ianbicking.org

[Bob Ippolito]

On Sep 20, 2005, at 9:42 AM, Kevin Dangoor wrote:

>
> On 9/20/05, forgems <for...@gmail.com> wrote:
>
>>
>> Nope, I'm running firefox 1.06 under linux.
>> I think that's problem with Kid.
>>
>
> I'm pretty sure that Kid handles unicode just fine.
>
> The flash message is stored in a cookie. I'd be more inclined to debug
> down the cookie path and see that everything is happening well there.
>
> I haven't tried UTF-8 encoding something that's already UTF-8. I'm
> assuming you get the same string out, right? If so, Kid really
> shouldn't be the problem.

If you try str.encode with a unicode encoding it will decode the str
with the default encoding and then encode it with the given
encoding. Not what you want. You should only encode unicode
instances (except for the dumb str<->str encodings like zlib, base64,
etc.).

-bob

[Kevin Dangoor]

On 9/20/05, Bob Ippolito <b...@redivi.com> wrote:
> If you try str.encode with a unicode encoding it will decode the str
> with the default encoding and then encode it with the given
> encoding. Not what you want. You should only encode unicode
> instances (except for the dumb str<->str encodings like zlib, base64,
> etc.).

So the answer may be as simple as encoding the string on the way into
the cookie and decoding it on its way back out of the cookie.

[Jeff Watkins]

Kevin Dangoor wrote:
> On 9/20/05, Bob Ippolito <b...@redivi.com> wrote:
>
>>If you try str.encode with a unicode encoding it will decode the str
>>with the default encoding and then encode it with the given
>>encoding. Not what you want. You should only encode unicode
>>instances (except for the dumb str<->str encodings like zlib, base64,
>>etc.).
>
>
> So the answer may be as simple as encoding the string on the way into
> the cookie and decoding it on its way back out of the cookie.
>
> Kevin
>

Kevin, I don't suppose there's any standard Javascript kicking around
TurboGears that updates the flash message after Ajax requests? Is it
even possible to get the cookie out of the XMLHttpRequest object (portably)?

Jeff

(Good stuff this TurboGears. Thanks.)

[Kevin Dangoor]

On 9/20/05, Jeff Watkins <je...@metrocat.org> wrote:
> Kevin, I don't suppose there's any standard Javascript kicking around
> TurboGears that updates the flash message after Ajax requests? Is it
> even possible to get the cookie out of the XMLHttpRequest object (portably)?

If you're doing a JSON request, the hash that comes back with have a
"turbogearsflash" entry. Just grab that and put it wherever you are
putting your status messages in your app.

> (Good stuff this TurboGears. Thanks.)

You're welcome! And thanks for giving a go!

[forgems]

I have found temporary solution for this, instead of :
<div py:if="turbogearsflash" class="flash"
py:content="turbogearsflash"></div>
I use:
<div py:if="turbogearsflash" class="flash"
py:content="XML(turbogearsflash)"></div>,
And my output is ok, with unicode letters :)