Unicode error when a TextField with UnicodeString validator gets a non-ascii value

[Cito]

The following code throws a UnicodeDecodeError for me:

from formencode import validators
from tw import forms

field = forms.TextField('test',
validator=validators.UnicodeString)

print field.display(u'käse')

This is the exact error I get:

for event in stream:
File "...\genshi\template\base.py", line 516, in _eval
new_attrs.append((name, u''.join(value)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
1: ordinal not in range(128)

This seems to be a pretty basic usage for me, so I wonder why nobody
stumbled over this problem so far.

[Alberto Valverde]

Use the UnicodeString validator from tw.forms.validators. The problem
with the one in FE (unless it has changed in a recent revision) is that
it returns an encoded str from from_python() instead of an unicode obect
which is what most template languages expect.

Alberto

[Christoph Zwerschke]

Alberto Valverde wrote:
> Use the UnicodeString validator from tw.forms.validators. The problem
> with the one in FE (unless it has changed in a recent revision) is that
> it returns an encoded str from from_python() instead of an unicode obect
> which is what most template languages expect.

Ok, I see. It's even mentioned in the TW docs, but not in the TG2 docs
(not yet - I will add a hint about this problem now).

Obviously this is still necessary; FormEncode's UnicodeString always
encodes the output. Actually you can even set the outputEncoding, but
supressing it with outputEncoding=None is currently not possible. I'll
suggest a patch to allow that. Then we will have

tw.forms.validators.UnicodeString()
== formencode.validators.UnicodeString(outputEncoding=None)

-- Christoph

[Christoph Zwerschke]

Christoph Zwerschke schrieb:
> tw.forms.validators.UnicodeString()
> == formencode.validators.UnicodeString(outputEncoding=None)

This feature is now in the FormEncode trunk and will be available in
formencode versions > 1.2.2.

-- Christoph