Problem with TG 1.0.4b3
Fred C
After upgrade to 1.0.4b3 I am getting this error.
Any idea why?
500 Internal error
The server encountered an unexpected condition which prevented it from
fulfilling the request.
Traceback (most recent call last):
File "/usr/pkg/lib/python2.5/site-packages/CherryPy-2.2.1-py2.5.egg/
cherrypy/_cphttptools.py", line 103, in _run
applyFilters('before_main')
File "/usr/pkg/lib/python2.5/site-packages/CherryPy-2.2.1-py2.5.egg/
cherrypy/filters/__init__.py", line 151, in applyFilters
method()
File "/usr/pkg/lib/python2.5/site-packages/TurboGears-1.0.4b3-
py2.5.egg/turbogears/visit/api.py", line 223, in before_main
cherrypy.request.params = encode_utf8(cherrypy.request.params)
File "/usr/pkg/lib/python2.5/site-packages/TurboGears-1.0.4b3-
py2.5.egg/turbogears/visit/api.py", line 187, in encode_utf8
res[k] = v.encode('utf-8')
File "/usr/pkg/lib/python2.5/cgi.py", line 548, in __getattr__
raise AttributeError, name
AttributeError: encode
Florent Aide
>
>
> After upgrade to 1.0.4b3 I am getting this error.
> Any idea why?
Yep. Could you give us the request you try to do (url with full args).
I suspect the "before_main" filter to try to encode something which is
not a 'string'.
Give us your request url and we may find a way to fix this :)
Florent.
Florent Aide
BTW: Could you create a ticket in our trac system so we don't forget
this issue ?
Best regards,
Florent.
Roger Demetrescu
On Dec 6, 2007 3:08 PM, Florent Aide <floren...@gmail.com> wrote:
>
> On Dec 6, 2007 5:51 PM, Fred C <bsd...@gmail.com> wrote:
> >
> >
> > After upgrade to 1.0.4b3 I am getting this error.
> > Any idea why?
>
> Yep. Could you give us the request you try to do (url with full args).
> I suspect the "before_main" filter to try to encode something which is
> not a 'string'.
Agree...
Maybe we should change that code to something like:
if isinstance(v, dict):
res[k] = encode_utf8(v)
elif isinstance(v, list):
res[k] = [vv.encode('utf-8') for vv in v]
elseif isinstance(v, basestring):
res[k] = v.encode('utf-8')
else:
raise Exception("some warning about this unsupported type")
That would ease the work to identify some non-trivial use cases...
[]s
Roger
Florent Aide
>
> Hei Florent !
Hallo Roger!
>
> Maybe we should change that code to something like:
>
>
> if isinstance(v, dict):
> res[k] = encode_utf8(v)
> elif isinstance(v, list):
> res[k] = [vv.encode('utf-8') for vv in v]
> elseif isinstance(v, basestring):
> res[k] = v.encode('utf-8')
> else:
> raise Exception("some warning about this unsupported type")
I had the same idea in my small head :). I must also add some test
case to make sure we don't get a regression on this part (Last time I
committed this encode thingy I added some tests so it should be pretty
easy to add more).
Looking at that part of the code I see that my monkey patch for the
CP's before_main is technically correct but quite ugly... :( using
self as an argument in a free function could be considered bad
behavior...
I'll need to fix this too just for the cleansing.
Regards,
Florent.
Roger Demetrescu
> On Dec 6, 2007 6:24 PM, Roger Demetrescu <roger.de...@gmail.com> wrote:
> > Maybe we should change that code to something like:
> >
> >
> > if isinstance(v, dict):
> > res[k] = encode_utf8(v)
> > elif isinstance(v, list):
> > res[k] = [vv.encode('utf-8') for vv in v]
> > elseif isinstance(v, basestring):
> > res[k] = v.encode('utf-8')
> > else:
> > raise Exception("some warning about this unsupported type")
>
> I had the same idea in my small head :). I must also add some test
> case to make sure we don't get a regression on this part (Last time I
> committed this encode thingy I added some tests so it should be pretty
> easy to add more).
I would also do some performance improvements... like inverting the order
of the IF's.... basestring's seem to be the most common case here,
followed by dicts and lists....
if isinstance(v, basestring):
res[k] = v.encode('utf-8')
elif isinstance(v, dict):
res[k] = encode_utf8(v)
elif isinstance(v, list):
res[k] = [vv.encode('utf-8') for vv in v]
else:
raise Exception("some warning about this unsupported type")
[]s
Roger
Fred C
I don't have the problem anymore. After my first email I ran "tg-admin
update" on my project and everything seems to works now.
Since my production environment is 1.0.4b2 I have to test if my
updated project works fine with that environment.
I have also one more question. Why "tg-admin upgrade" try to mess up
the files like controllers.py and welcome.kid. If you are not very
carreful you can simply end up erasing these files (replace them with
the bootstrap version). I usually avoid running this command because
it can be dangerous and painful, if you don't have a recent backup.
-fred-
Florent Aide
>
>
> I don't have the problem anymore. After my first email I ran "tg-admin
> update" on my project and everything seems to works now.
I am still interested to see what was the request you tried that made
the "before_main" filter to bork-out... Could you give us more info
about this?
As per the upgrade command I am curious to know what it changed that
corrected the problem... do you have any idea? A diff between before
and now?
Regards,
Florent.
Fred C
I cand send you the request. The call came from a form with a
fileupload field. I have tried to reproduce the bug to send you more
information but I couldn't. The only thing I can tell you is that the
method send was never executed. TG broke before.
If I have some time this evening I'll get the previews version from a
backup and send you some more info on that problem.
-fred-
offmessage
I too am having the same problem, from a form where I am uploading an
image.
On Dec 6, 9:10 pm, "Florent Aide" <florent.a...@gmail.com> wrote:
> I am still interested to see what was the request you tried that made
> the "before_main" filter to bork-out... Could you give us more info
> about this?
itself, in a FieldStorage instance.
>>> repr(v)
FieldStorage('newimage', 'image.jpg', '.........')
Running 'tg-admin update' has not fixed it for me.
if I add apply the following diff to
/branches/1.0/turbogears/visit/api.py r3807
11a12
> from cherrypy._cpcgifs import FieldStorage
186a188,189
> elif isinstance(v, FieldStorage):
> res[k] = v
My problem goes away. Is this a valid patch, or am I missing
something?
Cheers
Andy
Florent Aide
>
> Hi there
>
> I too am having the same problem, from a form where I am uploading an
> image.
[...]
> if I add apply the following diff to
> /branches/1.0/turbogears/visit/api.py r3807
>
> 11a12
> > from cherrypy._cpcgifs import FieldStorage
> 186a188,189
> > elif isinstance(v, FieldStorage):
> > res[k] = v
>
> My problem goes away. Is this a valid patch, or am I missing
> something?
we will make this more robust by excluding everthing that does not
have an encode() method but this is the idea...
Thanks for the report.
Florent.
Fred C
On Dec 6, 2007, at 1:10 PM, Florent Aide wrote:
>
> On Dec 6, 2007 9:44 PM, Fred C <bsd...@gmail.com> wrote:
>>
>>
>> I don't have the problem anymore. After my first email I ran "tg-
>> admin
>> update" on my project and everything seems to works now.
>
> I am still interested to see what was the request you tried that made
> the "before_main" filter to bork-out... Could you give us more info
> about this?
Ok Florent, I am back with that problem.
On one of my production machines I have that problem and I don't know
how to fix it.
The same programs works fine now on my dev machine but fail with that
error on my prod machine.
Here is the traceback I get.
2007-12-11 08:40:06,874 cherrypy.msg INFO HTTP: Traceback (most recent
call last):
File "/usr/local/lib/python2.5/site-packages/CherryPy-2.2.1-
py2.5.egg/cherrypy/_cphttptools.py", line 103, in _run
applyFilters('before_main')
File "/usr/local/lib/python2.5/site-packages/CherryPy-2.2.1-
py2.5.egg/cherrypy/filters/__init__.py", line 151, in applyFilters
method()
File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.4b3-
py2.5.egg/turbogears/visit/api.py", line 223, in before_main
cherrypy.request.params = encode_utf8(cherrypy.request.params)
File "/usr/local/lib/python2.5/site-packages/TurboGears-1.0.4b3-
py2.5.egg/turbogears/visit/api.py", line 187, in encode_utf8
res[k] = v.encode('utf-8')
File "/usr/local/lib/python2.5/cgi.py", line 548, in __getattr__
raise AttributeError, name
AttributeError: encode
Here is the beginning of the method, but TG fail before it gets to
call it.
@expose(template="caf.templates.sent")
@validate(validators=TXfileValidator())
@error_handler(index)
def send(self, **kw):
# save the filename
filelen = int(request.wsgi_environ['HTTP_CONTENT_LENGTH'])
if filelen > MAX_FILESIZE:
return dict(tg_template="caf.templates.message",
msg_class="error",
message=toobig + " (%s)" % MAX_FILESIZE )
uf = kw['file']
.......
Christopher Arndt
> On one of my production machines I have that problem and I don't know
> how to fix it.
Can you try applying the fix from changeset
http://trac.turbogears.org/changeset/3807 or just install the recent
version from SVN [1]?
Chris
Roger Demetrescu
No, it won't work... :)
In Fred's first message, he paste the traceback, which ended with:
"
File "/usr/pkg/lib/python2.5/cgi.py", line 548, in __getattr__
raise AttributeError, name
AttributeError: encode
"
Looking at my python2.5 cgi.py, it is clear that he hit the following
method from cgi.FieldStorage:
def __getattr__(self, name):
if name != 'value':
raise AttributeError, name # <===== here it is
if self.file:
self.file.seek(0)
value = self.file.read()
self.file.seek(0)
elif self.list is not None:
value = self.list
else:
value = None
return value
Florent had the idea of checking if the object has an encode
attribute. Something like:
================
def encode_utf8(params):
'''
will recursively encode to utf-8 all values in a dictionnary
'''
res = dict()
for k, v in params.items():
if isinstance(v, dict):
res[k] = encode_utf8(v)
elif isinstance(v, list):
res[k] = [vv.encode('utf-8') for vv in v]
elif hasattr(v, 'encode'):
res[k] = v.encode('utf-8')
else:
res[k] = v
return res
================
Didn't test the code... but this should do the trick...
[]s
Roger
Fred C
I have this fix and it doesn't work when there is a FileStorage object
in the params.
I have added this few lines and it works.
elif isinstance(v, types.StringTypes):
res[k] = v.encode('utf-8')
else
res[k] = v
I don't understand why you try to encode everything into utf-8. Only
strings should be encoded.
Or I am wrong ?
-fred-
Roger Demetrescu
> Florent had the idea of checking if the object has an encode
> attribute. Something like:
>
> ================
> def encode_utf8(params):
> '''
> will recursively encode to utf-8 all values in a dictionnary
> '''
> res = dict()
> for k, v in params.items():
> if isinstance(v, dict):
> res[k] = encode_utf8(v)
> elif isinstance(v, list):
> res[k] = [vv.encode('utf-8') for vv in v]
> elif hasattr(v, 'encode'):
> res[k] = v.encode('utf-8')
> else:
> res[k] = v
>
> return res
> ================
>
> Didn't test the code... but this should do the trick...
>
> []s
> Roger
>
Forget to mention that the line:
elif hasattr(v, 'encode'):
could be changed to:
elif isinstance(v, basestring):
Don't think yet which solution we should adopt... ;-)
Christoph Zwerschke
>> ================
>> def encode_utf8(params):
>> '''
>> will recursively encode to utf-8 all values in a dictionnary
>> '''
>> res = dict()
>> for k, v in params.items():
>> if isinstance(v, dict):
>> res[k] = encode_utf8(v)
>> elif isinstance(v, list):
>> res[k] = [vv.encode('utf-8') for vv in v]
>> elif hasattr(v, 'encode'):
>> res[k] = v.encode('utf-8')
>> else:
>> res[k] = v
>>
>> return res
>> ================
>>
>> Didn't test the code... but this should do the trick...
>
> elif hasattr(v, 'encode'):
> could be changed to:
> elif isinstance(v, basestring):
I think we should also handle the case that we have a list of dicts or
FileStorages or other non-basestrings, so
res[k] = [vv.encode('utf-8') for vv in v]
should become
res[k] = map(encode_utf8, v)
-- Christoph
Roger Demetrescu
> Roger Demetrescu schrieb:
> > Forget to mention that the line:
> > elif hasattr(v, 'encode'):
> > could be changed to:
> > elif isinstance(v, basestring):
>
> I think we should also handle the case that we have a list of dicts or
> FileStorages or other non-basestrings, so
>
> res[k] = [vv.encode('utf-8') for vv in v]
>
> should become
>
> res[k] = map(encode_utf8, v)
Your "patch" is quite elegant... and I am +1 for it ;)
But I'm just curious.. how can we get a list of dicts ? I recall I
did some tests with cherrypy query string and couldn't get these kind
of objects in my controller methods.
[]s
Roger
Christoph Zwerschke
> But I'm just curious.. how can we get a list of dicts ? I recall I
> did some tests with cherrypy query string and couldn't get these kind
> of objects in my controller methods.
That's quite possible: url?a-1.b=1&a-2.b=2
This does not come from CherryPy, but from FormEncode, see:
http://formencode.org/Validator.html#http-html-form-input
And we should also cover this case in the unit tests.
-- Christoph
Florent Aide
I have some code on my machine that is still pretty messed-up but
which should cover nearly all the cases... I need to do some more
testing and then I'll ask for some review because I don't have enough
testing material to make sure of every corner by myself.
Florent.
Roger Demetrescu
>
> Roger Demetrescu wrote:
> > But I'm just curious.. how can we get a list of dicts ? I recall I
> > did some tests with cherrypy query string and couldn't get these kind
> > of objects in my controller methods.
>
> That's quite possible: url?a-1.b=1&a-2.b=2
>
> This does not come from CherryPy, but from FormEncode, see:
> http://formencode.org/Validator.html#http-html-form-input
Thanks... it is clear to me now... :)
> And we should also cover this case in the unit tests.
Yeaph....
[]s
Roger
Christoph Zwerschke
>> And we should also cover this case in the unit tests.
>
> I have some code on my machine that is still pretty messed-up but
> which should cover nearly all the cases... I need to do some more
> testing and then I'll ask for some review because I don't have enough
> testing material to make sure of every corner by myself.
I have already checked in what I believe to be a complete fix and unit
test coverage for this issue, because I urgently needed it for an app in
production. Can you have a look at it? If you think it is ok, then can
you make a 1.04b4 release since this issue is breaking a lot of apps and
people are creating new tickets for this all of the time.
Also, I noticed that this whole thing has been handled only in the 1.0
branch so far, and not been merged to the 1.1 branch. Shall I go for it
or is there a reason why we don't want/need to have this in 1.1?
-- Christoph
Florent Aide
>
> Florent Aide wrote:
> >> And we should also cover this case in the unit tests.
[...]
> I have already checked in what I believe to be a complete fix and unit
> test coverage for this issue, because I urgently needed it for an app in
> production. Can you have a look at it? If you think it is ok, then can
> you make a 1.04b4 release since this issue is breaking a lot of apps and
> people are creating new tickets for this all of the time.
>
> Also, I noticed that this whole thing has been handled only in the 1.0
> branch so far, and not been merged to the 1.1 branch. Shall I go for it
> or is there a reason why we don't want/need to have this in 1.1?
As soon as the thing is solid again we shall put it in 1.1. I'll take
a look at this tonight and release b4... :)
Thanks for the work Chris.
Florent