Unicode Error accessing request.params

[Brian Brinegar]

I have a pyramid 1.2 application which processes donations made via
PayPal using a service called PayPal Instant Payment Notification.
Today, we had a donation from an international user with non-ascii
characters in their last name. PayPal makes a post request to our
server with information about the pyramid view that handles these the
request failed when I tried to access the last name from
request.params.get('last_name')

I believe requests are encoded as windows-1252, not UTF-8, which is
causing this error. Unfortunately, I do not have the option of
changing the encoding method on PayPal's end.


File "/home/mario/webapps/mario_dyn/mario/mario/views/donate.py",
line 241, in donate_ipn
value = request.params.get(key)
File "/home/mario/webapps/mario_dyn/lib/python2.7/UserDict.py", line
168, in get
return self[key]
File "/home/mario/webapps/mario_dyn/lib/python2.7/webob/multidict.py",
line 312, in __getitem__
return self._decode_value(self.multi.__getitem__(self._encode_key(key)))
File "/home/mario/webapps/mario_dyn/lib/python2.7/webob/multidict.py",
line 301, in _decode_value
value = value.decode(self.encoding, self.errors)
File "/home/mario/webapps/mario_dyn/lib/python2.7/encodings/utf_8.py",
line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 1:
invalid continuation byte

So, my question is, how do I tell the the request that the request is
encoded windows-1252 not utf-8, or convert it to utf-8 prior to
accessing the params MultiDict.

Thank you,
Brian

[Blaise Laflamme]

Hi Brian,

yes you can change the encoding on paypal's end. Look here:

Profile > My Selling Tools > Paypal button language encoding > More options

And select utf8 on both popups.

[Brian Brinegar]

Blaise,

Unfortunately, I do not have access to the Paypal encoding settings.
I'm tracking donations made to an account I do not own.

So, I need to be able to accept windows-1252 encoding in my pyramid app.

Is there a way I can decode and re-encode the request.body before it's
parsed into the params MultiDict?

Brian

> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/pylons-discuss/-/TGRcs0t_dkAJ.
> To post to this group, send email to pylons-...@googlegroups.com.
> To unsubscribe from this group, send email to
> pylons-discus...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/pylons-discuss?hl=en.

[Michael Merickel]

From my unaccepted SO answer.. AFAIK it's correct though.

http://stackoverflow.com/a/11061308/704327

Pyramid supports a request factory. You can use this to decode the request.

def request_factory(environ):
req = pyramid.request.Request(environ)
return req.decode(charset='gbk')

config.set_request_factory(request_factory)

This means that the request passed around within pyramid will be using
the gbk charset. Failure to decode the request with this charset will
cause UnicodeDecodeError exceptions so you may want to add extra logic
to your request factory to handle these things.