text outside of utf-8?
Dave Horner
When testing, I put ♠♣♥♦ entities on the
html page by a text form element for chat entry (named 'say').
One of my friends, copied the displayed spades,clubs,hearts,and
diamonds text and pasted it into the chat entry text element...and
sent.
Here's the output from my console when it happens....
headers: {u'username': u'dave', u'body':
u'{"handle":"dave","say":"\u2660\u2663\u2665\u2666","geofix_lat":"","geofix_long":""}',
u'content-length': u'71', u'content-encoding': u'utf-8',
u'destination': u'/topic/chatroom-1234', u'content-type':
u'text/plain'}
processing send : /topic/chatroom-1234
sending data: {'body':
u'{"handle":"dave","say":"\u2660\u2663\u2665\u2666","geofix_lat":"","geofix_long":""}'}
MorbidQ Error: problem connecting to server: "http://localhost:5000/send"
09/13/09 14:51:38:461 ACCESS connection closed from
173.50.198.254:61836 to localhost:61613
Lost connection. Reason: [Failure instance: Traceback (failure with
no frames): <class 'twisted.internet.error.ConnectionDone'>:
Connection was closed cleanly.
]
Unhandled error in Deferred:
Traceback (most recent call last):
File "/opt/python2.5/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/web/client.py",
line 152, in handleResponse
self.factory.page(response)
File "/opt/python2.5/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/web/client.py",
line 304, in page
self.deferred.callback(page)
File "/opt/python2.5/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py",
line 243, in callback
self._startRunCallbacks(result)
File "/opt/python2.5/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py",
line 312, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/opt/python2.5/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py",
line 328, in _runCallbacks
self.result = callback(self.result, *args, **kw)
File "/opt/python2.5/lib/python2.5/site-packages/morbid-0.8.7.3-py2.5.egg/morbid/morbid.py",
line 112, in frame_send
def frame_send(self, (headers, body)):
exceptions.TypeError: unpack non-sequence
So I guess I can strip out the bad characters on the browser side, but
it does still cause morbidq to stop working when someone sends
characters outside of utf-8... Is there something that can be added
to morbidq to handle these characters gracefully?
Thanks,
--dave
http://dave.thehorners.com/
Whitesell, Ken
you asking to simply filter them out from the body, or are you looking to
have them be handled as valid?
(I believe the first is going to be *much* easier than the second.)
Ken
Dave Horner
requests. So filtering them out is a fine
solution to me.
Thanks,
--dave
http://dave.thehorners.com/
Whitesell, Ken
Disclaimers:
1) I'm not used to working with unicode
2) I'm looking at a slightly older version (0.8.6.1)
Superficially, it looks to me like the error message being thrown
would be caused by the restq handler not handling those characters.
Specifically, the error message at the bottom of the exception:
> >> line 112, in frame_send
> >> def frame_send(self, (headers, body)):
> >> exceptions.TypeError: unpack non-sequence
> >>
This is telling me that whatever is calling frame_send isn't passing
something that can be divided into a 'headers' and a 'body'.
If I test this little code snippet:
class A:
def fs(self, (headers, body)):
print self, headers, body
with the following sequence:
a = A()
a.fs(("some headers here", u'\u2660\u2663\u2665\u2666'))
no error is generated.
If you're not using a restq handler, I'll try to recreate your symptoms
in my environment.
If you are, I might suggest taking a closer look at what your handler
is returning in this situation.
Ken
> -----Original Message-----
> From: mor...@googlegroups.com
> [mailto:mor...@googlegroups.com] On Behalf Of Dave Horner
> Sent: Monday, September 14, 2009 7:46 AM
> To: mor...@googlegroups.com
> Subject: Re: text outside of utf-8?
>
>
> Ya, mostly I just don't want the morbidq server to stop
> handling requests. So filtering them out is a fine solution to me.
>
> Thanks,
> --dave
> >
Dave Horner
Yes, you are looking at output from the restq handler. The code
derives from the sample in tree.
heres the code relating to send in my restq handler....
class DummyLeaf(object):
def __init__(self, data):
self.data = data
def render(self, request):
return self.data
def wrap(data):
print "sending data:",data
return DummyLeaf(encode(data))
class RestQListner(resource.Resource):
def onSend(self,headers):
username = headers.get('username',None)
newBody = headers.get('body','').replace("apples","bananas")
return wrap({"body":newBody})
def getChild(self, path, request):
if not path or path == "/":
return wrap(cbUrls)
headers = decode(request.content.read())
print '-------------------'
print 'headers:',headers
global restqUsers
username = headers['username']
destination = headers.get("destination",None)
if path in cbUrls:
print "processing ",path,":",destination
if path == "connect":
return self.onConnect(headers)
global restqUsers
if username not in restqUsers:
return wrap({"allow":"no"})
if path == "send":
return self.onSend(headers)
elif path == "subscribe":
return self.onSubscribe(headers)
elif path == "unsubscribe":
return self.onUnsubscribe(headers)
elif path == "disconnect":
return self.onDisconnect(headers)
return wrap({"allow":"no"})
In the output you can see that the 'sending data:' line is called and
then my restq handler seems to go away or something...
Maybe the encode is no good? DummyLeaf(encode(data)) I tried doing
unicode( data, "utf-8" ), but this doesn't work as it comes in as a
dict...
I myself an not familiar with unicode specifics in python either.
Been getting in a bit of an education lately... :D
processing send : /topic/chatroom-1234
sending data: {'body':
u'{"handle":"dave","say":"\u2660\u2663\u2665\u2666","geofix_lat":"","geofix_long":""}'}
MorbidQ Error: problem connecting to server: "http://localhost:5000/send"
Lost connection. Reason: [Failure instance: Traceback (failure with
no frames): <class 'twisted.internet.error.ConnectionDone'>:
Connection was closed cleanly.
]
09/14/09 23:07:51:335 ACCESS connection closed from
173.50.198.254:57589 to localhost:61613
Unhandled error in Deferred:
Traceback (most recent call last):
File "/opt/python2.5/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/web/client.py",
line 152, in handleResponse
self.factory.page(response)
File "/opt/python2.5/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/web/client.py",
line 304, in page
self.deferred.callback(page)
File "/opt/python2.5/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py",
line 243, in callback
self._startRunCallbacks(result)
File "/opt/python2.5/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py",
line 312, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/opt/python2.5/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/defer.py",
line 328, in _runCallbacks
self.result = callback(self.result, *args, **kw)
File "/opt/python2.5/lib/python2.5/site-packages/morbid-0.8.7.3-py2.5.egg/morbid/morbid.py",
line 112, in frame_send
def frame_send(self, (headers, body)):
exceptions.TypeError: unpack non-sequence
> Superficially, it looks to me like the error message being thrown
> would be caused by the restq handler not handling those characters.
Could very well be. I'm a little confused myself. :)
The trace does show the exception in morbid.py though....yes, agree my
handler may not be returning a good frame...not sure what the
offending code is yet.
File "/opt/python2.5/lib/python2.5/site-packages/morbid-0.8.7.3-py2.5.egg/morbid/morbid.py",
line 112, in frame_send
def frame_send(self, (headers, body)):
exceptions.TypeError: unpack non-sequence
> If you are, I might suggest taking a closer look at what your handler
> is returning in this situation.
Thank you for your time and thoughts.