formish error on unicode filenames

[Wichert Akkerman]

I am seeing errors on a website when people upload a filename with a
UTF8 encoded filename. It looks like this breaks when formish is
building a dotted dictionary and WebOb tries to decode the filename for
the uploaded file. This is the traceback:

File
"/srv/software/buildout/eggs/formish-0.9.1-py2.6.egg/formish/forms.py",
line 1111, in validate
self.request = request
File
"/srv/software/buildout/eggs/formish-0.9.1-py2.6.egg/formish/forms.py",
line 1081, in _set_request
request_data = _unflatten_request_data(request_data)
File
"/srv/software/buildout/eggs/formish-0.9.1-py2.6.egg/formish/forms.py",
line 1284, in _unflatten_request_data
container_factory=container_factory)
File
"/srv/software/buildout/eggs/dottedish-0.6-py2.6.egg/dottedish/api.py",
line 78, in unflatten
for (key, value) in l:
File
"/srv/software/buildout/eggs/formish-0.9.1-py2.6.egg/formish/forms.py",
line 1283, in <genexpr>
return unflatten(((key, request_data.getall(key)) for key in keys),
File
"/srv/software/buildout/eggs/WebOb-0.9.8-py2.6.egg/webob/multidict.py",
line 330, in getall
return [self._decode_value(v) for v in
self.multi.getall(self._encode_key(key))]
File
"/srv/software/buildout/eggs/WebOb-0.9.8-py2.6.egg/webob/multidict.py",
line 300, in _decode_value
self.errors)
File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode characters in posit

Looking at the formish code there is some suspicious logic in
_set_request. My guess is that WebOb can not handle decoding twice
twice, and formish does that anyway when it gets the decoded multidict
from the WebOb request and then turns it into a new UnicodeMultiDict
which will happily try to decode again.

Wichert.

[Tim Parkin]

Hi Wichert,

I agree that there is likely a problem here - I'll try and take a look
at it in the next couple of days..

Tim

[Wichert Akkerman]

Hi Tim,

I was wondering if you've had a chance to look at this?

Wichert.

--
Wichert Akkerman <wic...@wiggy.net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.

[Wichert Akkerman]

Hi Tim,

On 8/11/10 08:56 , Wichert Akkerman wrote:
> Hi Tim,
>
> On 2010-8-3 10:28, Tim Parkin wrote:
>> On Jul 30, 9:02 am, Wichert Akkerman<wich...@wiggy.net> wrote:
>>> Looking at the formish code there is some suspicious logic in
>>> _set_request. My guess is that WebOb can not handle decoding twice
>>> twice, and formish does that anyway when it gets the decoded multidict
>>> from the WebOb request and then turns it into a new UnicodeMultiDict
>>> which will happily try to decode again.
>>>
>>
>> Hi Wichert,
>>
>> I agree that there is likely a problem here - I'll try and take a look
>> at it in the next couple of days..
>
> I was wondering if you've had a chance to look at this?

any news on this? I just got another few bugreports related to this and
was hoping you'ld have had a chance to look into this.

Wichert.

[Wichert Akkerman]

Hi Tim,

is this still on your radar?

Wichert.