UTF-8 output trouble in Mako with Pyramid

[Michael Gimbel]

Hi all.

I'm encountering MakoRenderingException and UnicodeDecodeError
exceptions throughout my application when trying to render utf8 data
which was originally user input.

There don't seem to be Pyramid configuration options for Mako's
output_encoding='utf-8', encoding_errors='replace'.

Python/Mako magic doesn't seem to be doing it either, ## -*- coding:
utf-8 -*-

I've tried using a mako filter, which helped: mako.default_filters =
decode.utf8 (in .ini file)
But I still have problems anywhere I'm using filters or functions in
my templates: ${var_with_utf8_chars | custom_filter}
This seems to fix it but I don't want to have to do this throughout
the entire application: ${var_with_utf8_chars.decode('utf8') }
custom_filter}

I was thinking if the above is the only way, the best place to do it
would be at the model level when the utf8 data is loaded out of the
database. But I was hoping for a better, more magical solution.

Anyone have any ideas?

Thanks,
Mike

[Arndt Droullier]

You could try to set the python encoding to utf-8 or check if it is already set.

If set python will use utf-8 by default. 

It's just one line in sitecustomize.py:

e.g. in site-packages/sitecustomize.py
--------------------------------------------------------------------
import sys
sys.setdefaultencoding('utf-8')

-------------------------------------------------------------------- 

Arndt.

2012/4/17 Michael Gimbel <michael...@rightbrainnetworks.com>

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To post to this group, send email to pylons-...@googlegroups.com.
To unsubscribe from this group, send email to pylons-discus...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.

--
DV Electric / Arndt Droullier / www.dvelectric.de

[Marius Gedminas]

Decoding UTF-8 data into Unicode objects at the model level _is_ the
better solution. E.g. SQLAlchemy can do that seamlessly.

Marius Gedminas
--
Never trust a smiling Gates.

[Mike]

Thanks very much guys.  Next app I plan on using SQLAlchemy for my first time.

Marius, would it be better to decode to utf8 during __setattr__() vs __getattribute__()?

It seems to me that they'd give the same results.  Whichever is accesses less would be a more efficient place to put the decode (which is probably application dependent).


-Mike

[Marius Gedminas]

On Fri, Apr 20, 2012 at 08:43:33AM -0700, Mike wrote:
> Thanks very much guys. Next app I plan on using SQLAlchemy for my first
> time.
>
> Marius, would it be better to decode to utf8 during __setattr__() vs
> __getattribute__()?

Neither?

I can't answer without knowing what your code is all about.

The general rule is: encode/decode at your application boundaries.
Pyramid already converts your HTTP form data into Unicode. You need
to do the same at the points where you're talking to the database.

What database is it?

Marius Gedminas
--
Vi has excellent support for the PC speaker. Nearly every key combination will
invoke the PC speaker to make use of this.
-- G_Morgan on reddit