having issue with cherrypy with mako templates and ASCII chars greater than 127

[Tony Caduto]

Hi,

I have had a cherrypy app in production for over a year and recently I have been getting errors when someone uploads a file 

with a name that contains extended ascii chars like the EM Dash etc

What happens is this error is returned when the app tries to display a grid with the file names:

Invalid Request! 

500 Internal Server Error

The server encountered an unexpected condition which prevented it from fulfilling the request. 

cherrypy version = 3.2.0

I have # -*- coding: utf-8 -*-

at the top of my code.

Is there a way to prevent this error when a file name being display has one of these odd chars in it?

[Eugene Van den Bulke]

Hi Tony,

Sounds like you want to wacth Ned Batchelder's talk at Pycon earlier
this year: http://pyvideo.org/video/948/pragmatic-unicode-or-how-do-i-stop-the-pain

The encoding in your source file relates to the source not the
"strings" manipulated by your application.

Looks like mako expect bytes and you are sending it a unicode string.
Figure out what the string is encoded in (probably utf-8) and encode
before sending it to Mako.

>>> filename = u'file\u2014name'
>>> print filename
file—name
>>> filename.encode('utf-8')
'file\xe2\x80\x94name'

Hope this help,

--
EuGeNe -- follow me http://twitter.com/3kwa

[Tony Caduto]

Hi,

Actually that line at the top is not just for the source code.

If I change it to:

# -*- coding: cp1252 -*-

The output is correct in a test app with the filename in question, with UTF8 the extended chars are showing in the rendered template as ?

unfortunately make is still puking on the filename even when I change the magic comment :-)

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To post to this group, send email to cherryp...@googlegroups.com.
To unsubscribe from this group, send email to cherrypy-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cherrypy-users?hl=en.

[Eugene Van den Bulke]

Could you share the exact traceback?

Looking very quickly at the Mako documentation it expects unicode and
should deal with it well so you must be sending it bytes instead of
unicode at some stage, see the traceback below.

>>> unicode_filename = u'file\u2014name'
>>> bytes_filename = unicode_filename.encode('utf-8')
>>> print Template("filename : ${filename}").render(filename=unicode_filename)
filename : file—name
>>> print Template("filename : ${filename}").render(filename=bytes_filename)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/eugene/Envs/mako/lib/python2.7/site-packages/mako/template.py",
line 397, in render
return runtime._render(self, self.callable_, args, data)
File "/Users/eugene/Envs/mako/lib/python2.7/site-packages/mako/runtime.py",
line 764, in _render
**_kwargs_for_callable(callable_, data))
File "/Users/eugene/Envs/mako/lib/python2.7/site-packages/mako/runtime.py",
line 796, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "/Users/eugene/Envs/mako/lib/python2.7/site-packages/mako/runtime.py",
line 822, in _exec_template
callable_(context, *args, **kwargs)
File "memory:0x1007700d0", line 22, in render_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
4: ordinal not in range(128)

It is very important to know what is the encoding used in your application.

Cheers,

[Tony Caduto]

HI, 

I got it working with the odd chars in the file names.

Did this:

import sys

reload(sys)

sys.setdefaultencoding('utf-8')

template_lookup = TemplateLookup(directories=['{0}templates'.format(ppath)],output_encoding='cp1252')

If I had the mako encoding set to utf-8 it would render and not error out but the offending char would be all

add looking, changing it to cp1252 did the trick and the add chars display and the server does not error out.

It also would not work if I simply set the python default encoding to cp1252, not entirely sure why it works

with this combo, but whatever.

Thanks for the help.

Tony