Identity bug? Passing non-ASCII characters as kwargs to a handler with @identity.require(identity.not_anonymous()) gives internal server error

[Thomas Wittek]

Hi!

I just came across this problem.
Suppose I have this (minimal) controller:

class Root(controllers.RootController):
@expose()
def test(self, *args, **kwargs):
return "bla"

@expose()
@identity.require(identity.not_anonymous())
def test2(self, *args, **kwargs):
return "bla2"

When I call `/test?a=ä`, all works as expected. But when I call the
second handler `/test2?a=ä`, I get an error:

500 Internal error

The server encountered an unexpected condition which prevented it from
fulfilling the request.

Page handler: 'ordinal not in range(128)'
Traceback (most recent call last):
File "c:\programme\python25\lib\site-packages\cherrypy-2.2.1-
py2.5.egg\cherrypy\_cphttptools.py", line 103, in _run
applyFilters('before_main')
File "c:\programme\python25\lib\site-packages\cherrypy-2.2.1-
py2.5.egg\cherrypy\filters\__init__.py", line 151, in applyFilters
method()
File "c:\programme\python25\lib\site-packages\cherrypy-2.2.1-
py2.5.egg\cherrypy\filters\decodingfilter.py", line 31, in before_main
self.decode(enc)
File "c:\programme\python25\lib\site-packages\cherrypy-2.2.1-
py2.5.egg\cherrypy\filters\decodingfilter.py", line 50, in decode
decodedParams[key] = value.decode(enc)
File "c:\programme\python25\lib\encodings\utf_8.py", line 16, in
decode
return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in
position 0: ordinal not in range(128)

I guess that this is not supposed to throw this error, is it?
I'm using TG 1.0.4b2.

Thanks for any advice!

[Florent Aide]

On Oct 30, 2007 12:05 PM, Thomas Wittek <strea...@googlemail.com> wrote:
>
> Hi!
>
> I just came across this problem.
> Suppose I have this (minimal) controller:
>
> class Root(controllers.RootController):
> @expose()
> def test(self, *args, **kwargs):
> return "bla"
>
> @expose()
> @identity.require(identity.not_anonymous())
> def test2(self, *args, **kwargs):
> return "bla2"
>
> When I call `/test?a=ä`, all works as expected. But when I call the
> second handler `/test2?a=ä`, I get an error:
>
> 500 Internal error

Good catch!

Could you create a ticket on our trac system and ping us back here.
Please try to attach some example project so we can reproduce easily
the error.

Thanks,
Florent.

[Thomas Wittek]

On Oct 30, 3:04 pm, "Florent Aide" <florent.a...@gmail.com> wrote:

> Could you create a ticket on our trac system and ping us back here.

Great, my ticket got rejected: "Submission rejected as potential spam
(Akismet says content is spam)"

Text:

==== 8< ====

Suppose I have this (minimal) controller:

{{{
class Root(controllers.RootController):
@expose()
def test(self, *args, **kwargs):
return "bla"

@expose()
@identity.require(identity.not_anonymous())
def test2(self, *args, **kwargs):
return "bla2"
}}}

When I call `/test?a=ä`, all works as expected.
But when I call the second handler `/test2?a=ä`, I get an error:

{{{
500 Internal error
[..]
File "[..]python25\lib\site-packages\cherrypy-2.2.1-py2.5.egg

\cherrypy\filters\decodingfilter.py", line 50, in decode
decodedParams[key] = value.decode(enc)

File "[..]python25\lib\encodings\utf_8.py", line 16, in decode

return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in
position 0: ordinal not in range(128)
}}}

The full test project (actually, only the controllers.py has been
modified after the quickstart) is attached.

==== >8 ====

> Please try to attach some example project so we can reproduce easily
> the error.

Can be downloaded here: http://gedankenkonstrukt.de/files/tg-id-utf-bug.zip

Cheers

[Thomas Wittek]

On Oct 30, 12:05 pm, Thomas Wittek <streawkc...@googlemail.com> wrote:
> When I call `/test?a=ä`, all works as expected. But when I call the

> second handler `/test2?a=ä`, I get an error: [..] 500 Internal error

Found another, maybe related bug:
`/test2?a.b=b`

500 Internal error

The server encountered an unexpected condition which prevented it from
fulfilling the request.

Traceback (most recent call last):

File "c:\programme\python25\lib\site-packages\cherrypy-2.2.1-
py2.5.egg\cherrypy\_cphttptools.py", line 103, in _run
applyFilters('before_main')
File "c:\programme\python25\lib\site-packages\cherrypy-2.2.1-
py2.5.egg\cherrypy\filters\__init__.py", line 151, in applyFilters
method()
File "c:\programme\python25\lib\site-packages\cherrypy-2.2.1-
py2.5.egg\cherrypy\filters\decodingfilter.py", line 31, in before_main
self.decode(enc)
File "c:\programme\python25\lib\site-packages\cherrypy-2.2.1-
py2.5.egg\cherrypy\filters\decodingfilter.py", line 50, in decode
decodedParams[key] = value.decode(enc)

AttributeError: 'dict' object has no attribute 'decode'

Seems like Identity and CP are battling each other in decoding the
request params...

[Florent Aide]

On Oct 30, 2007 4:06 PM, Thomas Wittek <strea...@googlemail.com> wrote:
>
> On Oct 30, 3:04 pm, "Florent Aide" <florent.a...@gmail.com> wrote:
>
> > Could you create a ticket on our trac system and ping us back here.
>
> Great, my ticket got rejected: "Submission rejected as potential spam
> (Akismet says content is spam)"

We just created the spam filtering but now we trained it a little and
also tweaked the parameters, so it should accept you ticket. Could you
try again to create it ? And sorry for the inconvenience.

Florent.

[Thomas Wittek]

On Oct 30, 9:33 pm, "Florent Aide" <florent.a...@gmail.com> wrote:

> On Oct 30, 2007 4:06 PM, Thomas Wittek <streawkc...@googlemail.com> wrote:
> > Great, my ticket got rejected: "Submission rejected as potential spam
> > (Akismet says content is spam)"
>
> We just created the spam filtering but now we trained it a little and
> also tweaked the parameters, so it should accept you ticket. Could you
> try again to create it ? And sorry for the inconvenience.

Ok, works.
Filed as #1598 under http://trac.turbogears.org/ticket/1598

[Adi]

Hi

Struggling with the same error, here's the workaround I found
that seems to be working for me. I'm using TG 1.0.2.2

IN turbogears\visit\api.py
IN class VisitFilter(BaseFilter):
IN def before_main(self):

adding at the end of before_main (line 165 for TG 1.0.0.2):

for k,v in cherrypy.request.params.iteritems():
cherrypy.request.params[k]=v.encode('utf-8')

seems to do the job. However as I'm not using the latest TG version
and I'm not too confident with all this I'd rather let the
knowleadgeable team check whether this is ok or not! My 2 cents...

[Thomas Wittek]

On Nov 15, 11:19 am, Adi <fanet...@gmail.com> wrote:
> IN turbogears\visit\api.py
> IN class VisitFilter(BaseFilter):
> IN def before_main(self):
>
> adding at the end of before_main (line 165 for TG 1.0.0.2):
>
> for k,v in cherrypy.request.params.iteritems():
> cherrypy.request.params[k]=v.encode('utf-8')

Doesn't work for me (TG 1.0.4b2).
Maybe I inserted the code at the wrong position?
I added it before:

try:
for plugin in _plugins:
plugin.record_request(visit)

[Adi]

Here's where I have put the 2 extra lines: at the end of the method:

# Inform all the plugins that a request has been made for the
current
# visit. This gives plugins the opportunity to track click-
path or
# retrieve the visitor's identity.

try:
for plugin in _plugins:
plugin.record_request( visit )

except cherrypy.InternalRedirect, e:
# Can't allow an InternalRedirect here because CherryPy is
dumb,
# instead change cherrypy.request.object_path to the url
desired.
cherrypy.request.object_path= e.path

for k,v in cherrypy.request.params.iteritems():
cherrypy.request.params[k]=v.encode('utf-8')

By the way In my case I'm not decorating a method by using
@identity.require(identity.not_anonymous()) but using
require = identity.in_group("mygroup") at the start of a
SecureResource controller class.But I did have te same kind of error
before modifying turbogears\visit\api.py

[Thomas Wittek]

On Nov 16, 10:38 am, Adi <fanet...@gmail.com> wrote:
> Here's where I have put the 2 extra lines: at the end of the method:

Oh, great. Seems to work for me!
Thank you very much.

I added your patch to the ticket system.
Maybe someone can investigate and integrate it.

[Thomas Wittek]

Doesn't work with nested request params (i.e. foo.bar.baz) as they get
converted into nested dicts, so the value `v` may be a `dict` that
doesn't have a method named `encode`: AttributeError: 'dict' object
has no attribute 'encode'.

To fix your fix, you have to do do a recursive conversion:

def encode_utf8(params):
for k, v in params.items():
if type(v) is dict:
encode_utf8(v)
else:
params[k] = v.encode('utf-8')

encode_utf8(cherrypy.request.params)

Funnily, you'll then get an error from CP:

File "c:\programme\python25\lib\site-packages\cherrypy-2.2.1-

py2.5.egg\cherrypy\filters\decodingfilter.py", line 50, in decode
decodedParams[key] = value.decode(enc)

AttributeError: 'dict' object has no attribute 'decode'

But I've got a feeling that the real problems are located in another
place, deeper in the system. But unfortunately I don't know much about
TG's/CP's internals.
Maybe this is ultimately a bug in CP. But I really don't know.