myghty: Error(UnicodeDecodeError): 'ascii' codec can't decode byte 0xc3

[Chris Shenton]

Using Pylons-0l9.4.1 and Myghty-1.1. I can't seem to set encoding to
utf-8 for Myghty or mod_python so that Pylons gets it.

My app was working well but just started bombing, likely because one
of my techs entered a name into a form which had non-ascii chars, and
then the rendering in the display failed. When I run with debug
enabled, I get a bit more info:

Error: Error(UnicodeDecodeError): 'ascii' codec can't decode byte 0xc3
in position 15: ordinal not in range(128)
File: /usr/local/clientproj/er/templates/vendor_show.myt line 11
Context: 8:
<tr><th>Client Name</th><th>Updated</th><th>Created</th></tr>
9: % for client in c.clients:
10: <tr>
11: <td><a href="<%h.url_for('client_show', client_id=client.client_id)%>"><%client.name%></a></td>
12: <td><%client.ts_updated%></td>
13: <td><%client.ts_created%></td>
14: </tr>

Error - myghty.exception.Error: Error(UnicodeDecodeError?):
'ascii' codec can't decode byte 0xc3 in position 15: ordinal not
in range(128) at
/usr/local/lib/python2.4/site-packages/Myghty-1.1-py2.4.egg/myghty/requestbuffer.py
line 367

I've read up on this and found suggestions in

http://www.myghty.org/docs/params.myt#parameters_output_encoding

to set encoding in config/environment.py, which I've done:

myghty = {}
myghty['log_errors'] = True
myghty['escapes'] = dict(l=webhelpers.auto_link, s=webhelpers.simple_format)

myghty['output_encoding'] = 'utf-8'
myghty['encoding_errors'] = 'htmlentityreplace'

return pylons.config.Config(myghty, map, paths)

but this didn't help. The error continued to complain about "ascii"
codec so it looks like this setting above isn't getting seen by
myghty. It also suggested setting this in the mod_python section of
my Apache config, which I also tried:

<Location /er>
SetHandler mod_python
Pythonhandler mod_python.wsgi
PythonPath "['/usr/local/clientjob] + sys.path"
PythonOption wsgi.application startup-bluecoat::app
PythonOption SCRIPT_NAME /er
# 2007-04-05 setting in config/environment.py isnt' working?
PythonOption MyghtyOutputEncoding "utf-8"
PythonOption MyghtyEncodingErrors "htmlentityreplace"
</Location>

But this also had no effect, same complaint about "ascii".

http://pylonshq.com/docs/internationalization.html said I could do the
following, again in config/environment.py:

tmpl_options['myghty.output_encoding'] = 'UTF-8'

but that file didn't know about the "tmpl_options" dict.

I've seen similar suggestion to the above in a #pylons IRC transcript
but none of these are working for me. Why is it still using an
"ascii" codec when I've set it in two different places?

I've tried removing the template cache .py and .pyc files but that had
no effect.

I was finally able to get things to work by creating
.../site-packages/sitecustomize.py:

import sys
sys.setdefaultencoding('utf-8')

and saw that one of my clients names was an O-umlaut. (Should I now
expect surprises from other things, like the fact that I'm not using
unicode strings in my python?)

I'd really like to do this in Pylons or Myghty instead than site-wide
and don't understand why the documented mechanisms don't work for me.

Any suggestions? Thanks.

[Philip Jenvey]

Setting your default system encoding to utf-8 will work -- but it's
incredibly inconvenient, as you've noticed, especially when deploying
your app to new environments.

The output encoding setting is how the final, rendered string Myghty
returns will be encoded to. The UnicodeDecodeError you're seeing is
due to the incoming data passed to Myghty, it occurs when Myghty
converts everything to unicode internally. It defaults to the
system's default encoding (ascii) when converting inputted strings to
unicode.

The correct way to solve this problem is to pass Myghty unicode
objects. The other option is to disable unicode support in Myghty,
via the global 'disable_unicode' option, or on a template by template
basis by passing disable_unicode=True to Pylons' render function.

--
Philip Jenvey

[ste...@gmail.com]

Actually, yes, this flag has no effect when i was trying.
But, we have resolved it. We save the .myt file into encoding utf-8,
and add a line to the top of the myt file:

# encoding: utf-8

The Myghty will add the line into the translated .myt.py file like
this:

# File: templates|base.myt CompilerID: Myghty.Lexer|Myghty.Compiler|
(5, {'disable_unicode': False}) Timestamp: Wed May 16 16:12:52 2007
# -*- encoding: utf-8 -*-

so, everything is ok.

> http://pylonshq.com/docs/internationalization.htmlsaid I could do the

[voltron]

Hi Steve, this is good information as it is a good workaround for the
problem that I have been having with my Mako templates with
Umlauts( see thread http://groups.google.com/group/pylons-discuss/browse_thread/thread/4d278dee35c62519)

I´m just worried that it seems like a hack and not the official way to
set things in Pylons.

> >http://pylonshq.com/docs/internationalization.htmlsaidI could do the

[Jens Hoffrichter]

On 5/16/07, voltron <nhy...@googlemail.com> wrote:
>
> Hi Steve, this is good information as it is a good workaround for the
> problem that I have been having with my Mako templates with
> Umlauts( see thread http://groups.google.com/group/pylons-discuss/browse_thread/thread/4d278dee35c62519)
>
> I´m just worried that it seems like a hack and not the official way to
> set things in Pylons.

Maybe it is not officially declared in Pylons, but it is for sure the
correct way to set the encoding for a Python file, in general.

But working with UTF-8 and Unicode can sometimes be a real pain in
Python. I resorted to setting the site.py on all sites I manage to
UTF-8, but it is a very cumbersome solution.

Jens

[voltron]

Oh I did not know about that, thanks Jens. Excuse my newbiness, but
could you explain your method further? where is site.py?

Thanks

On May 16, 1:13 pm, "Jens Hoffrichter" <jens.hoffrich...@gmail.com>
wrote:

> On 5/16/07, voltron <nhy...@googlemail.com> wrote:
>
> > Hi Steve, this is good information as it is a good workaround for the
> > problem that I have been having with my Mako templates with

> > Umlauts( see threadhttp://groups.google.com/group/pylons-discuss/browse_thread/thread/4d...)

[Jens Hoffrichter]

Hey Voltron,

On 5/16/07, voltron <nhy...@googlemail.com> wrote:
>
> Oh I did not know about that, thanks Jens. Excuse my newbiness, but
> could you explain your method further? where is site.py?

The declaration for working with encodings in Python files is
described here, in PEP 263:

http://www.python.org/dev/peps/pep-0263/

The site.py is normally located in your base library directory, for me
on Debian Linux it is
/usr/lib/python2.4, or maybe even /etc/python2.4/site.py

Search for the string "ascii" in the file, and then you can see why it
is very hard to change the default encoding after this file is
executed. And site.py is automagically executed everytime you start
python ;)

But, as I said, it is cumbersome, because you must have access to the
site.py file, and this isn't guaranteed, especially for a shared
hoster.

There exists another solution, at least for normal python applications:

You can place a file called "sitecustomize.py" in the directory of
your application (or maybe in the first library path for it, I'm not
sure, but that is normally the application path), where you can put in
something like that:

---- snip ----
import sys

sys.setdefaultencoding("UTF-8")
---- snip ----

This works as well. I normally don't use it, and especially haven't
used it with pylons (because I'm actually on this list because I had a
Myghty caching question, and somehow stayed ;) ), but this is the only
solution python applications compiled with py2exe, as you have no
other method of setting the site.py there.

But, as I said, I don't know how it would work in pylons, maybe
someone with a broader knowledge in pylons can look at it.

Jens

[voltron]

Cool, thanks!

On May 16, 2:26 pm, "Jens Hoffrichter" <jens.hoffrich...@gmail.com>
wrote:

[Shannon -jj Behrens]

By the way, do you know about this trick:

reload(sys)
sys.setdefaultencoding(encoding)

It gets around the site.py problem.

Best Regards,
-jj

--
http://jjinux.blogspot.com/