Unicode error (once again) with Genshi and ToscaWidget

[Philippe Lemarre]

Hello every body

I found a problem with unicode somewhere between ToscaWidget, Genshi and
maybe sqlalchemy.
(I 'm using TG2 (1.9.7a1) on Ubuntu 8.04 /python2.5)

I 've a controller fetching data in the DB and passing the result as a
query object, ie:

@expose(template="myproject.templates.welcome")
def index(self):
mydata = DBSession.query(MyData).filter(...somefilter...)
return dict(mydata = mydata, page= u'Accueil')

With this controller, unicode letters (I mean non-ASCII) are rendered by
genshi --no problems

I have another controller fetching the SAME data and passing the result
to a template containing a toscaWidget form ie :
@expose(template="myproject.templates.edition")
def edit(self, data_id, **kw):
mydata = DBSession.query(MyData).get(data_id)
modelname = u'Modification du message'
pylons.c.form = mydata_form
return dict(page = modelname,
mydataform = mydata)

The widget
class MyDataForm(TableForm):
class fields(WidgetsList):
data_id = HiddenField()
textarea = TextArea(validator = UnicodeString(not_empty =
True, strip = True))

mydata_form = MyDataForm(action = "/save",
submit_text = "Enregistrer")

With this controller and this widget, I get the well known genshi Error
"UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in
position 40: ordinal not in range(128)"
Naturally, if the data contain only ASCII, it works fine

Notice that this controller pass the result as MyData object, not a
Query object.
I've tried to pass the data as a list using query.all(), or a query
object. It does'nt work , even with ASCII.

What's wrong with that ?
Any help?

Thx in advance

Philippe

[Alberto Valverde]

Is this UnicodeString the one from FormEncode or the one in tw.forms?
IIRC, tw.forms had its own version so it doesn't encode on from_python()
since templates now expect unicode all way through until finally served.
The rule of thumb is: Always use unicode internally. Tell SA to decode
when it reads from the db, make validators decode when they validate
from the web so all your app deals with internally are nicely decoded
unicode objects. TG will take care of encoding them at the last moment
before sending the response to the client.

Alberto

P.S: CC-ing toscawidgets-discuss since other TW users might benefit from
this thread.

[Philippe Lemarre]

Alberto Valverde a écrit :

OK thanks a lot it works.

It was the UnicodeString from formencode

Instead of

from formencode.validators import UnicodeString

i use
from tw.forms.validators import UnicodeString

and Genshi seems really happier.

Philippe