error handling and validators again (i18n issue)

[Max Ischenko]

Hi folks,

Looks like it was too premature to declare the yesterday's problem gone. This time I tried to be as detailed as possible; this error is a real show-stopper for me.

Here is my setup:

class UserRegistrationFormFields(widgets.WidgetsDeclaration):
email = widgets.TextField(label=_(u'E-mail'),
validator=EmailValidator(not_empty=True))
password = widgets.PasswordField(
validator=validators.UnicodeString(max=15, min=3, not_empty=True))
nickname = widgets.TextField(
validator=validators.UnicodeString(not_empty=False, min=3, max=25))
foo = widgets.TextField(validator=validators.Int())
user_regisration_form = TableForm(submit_text=_('Register'),
fields=UserRegistrationFormFields())

# Note the 'foo' field above -- I added it to verify that validator (Int) is actually called

Here is my controller:

@tg.expose(html=".templates.register")
def register(self, tg_errors=None, **form_data):
if tg_errors:
print 'FORM ERRORS', tg_errors
print 'REGISTER FORM', form_data
return dict(form=user_regisration_form, data=form_data)

@tg.expose()
@tg.validate(form=user_regisration_form)
@tg.error_handler(register)
def do_register(self, **form_data):
print 'REGISTER COMPLETE', form_data
raise tg.redirect('/')

If I fill-in all required values and there are no errors all proceeds just fine. However, if there is an error in nickname field I got those nasty UnicodeDecodeError. The error only appears if nickname contains non-ascii characters.

I instrumented the code and validator (shown below) with debug statements. Here is console output I get:

calling _to_python('\xd0\x96') yields u'\u0416'
FORM ERRORS {'password': <formencode.api.Invalid instance at 0x01490058>, 'nickname': <formencode.api.Invalid instance at 0x014900D0>, 'email': <formencode.api.Invalid instance at 0x01490148>}
REGISTER FORM {'foo': 123, 'tg_exceptions': None, 'tg_source': <function do_register at 0x013F5EB0>, 'password': '', 'nickname': '\xd0\x96', 'email': ''}
Traceback
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0

Note that while UnicodeString.to_python is called the form data contains original encoded string, not unicode string. Note also that 'foo' field in the form data does contain parsed integer (123) not the original string.

Here is the output I got when the form is processed OK (e.g. when there were no errors):

calling _to_python('\xd0\x96\xd0\x9e\xd0\x9f\xd0\x90') yields u'\u0416\u041e\u04
1f\u0410'
REGISTER COMPLETE {'password': u'456', 'nickname': u'\u0416\u041e\u041f\u0410',
'email': 'al...@oo.net', 'foo': 123}
127.0.0.1 - - [14/Feb/2006:12:42:55] "POST /do_register HTTP/1.1" 302 145

Note again that 'nickname' field is unicode string now.

At last, here is the validator I use:

def trace(f):
name = f.__name__
def wrapper(self, value, state):
v = f(self, value, state)
print 'calling %s(%r) yields %r' % (name, value, v)
return v
return wrapper

class UnicodeString(String):
encoding = 'utf-8'
@trace
def _to_python(self, value, state):
if value:
if isinstance(value, unicode):
return value
if hasattr(value, '__unicode__'):
return unicode(value)
return unicode(value, self.encoding)
return u''
@trace
def _from_python(self, value, state):
if hasattr(value, '__unicode__'):
value = unicode(value)
if isinstance(value, unicode):
return value.encode(self.encoding)
return str(value)