Parse non-ascii char-ref raise an error

5 views
Skip to first unread message

deemonster

unread,
Jul 4, 2009, 6:24:08 PM7/4/09
to beautifulsoup
BeautifulSoup.__version__ == 3.0.1

This script produces a bug:
import BeautifulSoup
BeautifulSoup.BeautifulSoup('<html attr="aa&#160;bb"></html>')

Traceback (most recent call last):
File "./t2.py", line 5, in <module>
BeautifulSoup.BeautifulSoup('<html attr="aa&#160;bb"></html>')
File "/var/lib/python-support/python2.5/BeautifulSoup.py", line
1236, in __init__
BeautifulStoneSoup.__init__(self, *args, **kwargs)
File "/var/lib/python-support/python2.5/BeautifulSoup.py", line 900,
in __init__
self._feed()
File "/var/lib/python-support/python2.5/BeautifulSoup.py", line 925,
in _feed
SGMLParser.feed(self, markup)
File "/usr/lib/python2.5/sgmllib.py", line 100, in feed
self.goahead(0)
File "/usr/lib/python2.5/sgmllib.py", line 134, in goahead
k = self.parse_starttag(i)
File "/usr/lib/python2.5/sgmllib.py", line 286, in parse_starttag
self._convert_ref, attrvalue)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position
0: ordinal not in range(128)

This fix helps correct it:
--- sgmllib.py 2009-07-05 02:21:02.000000000 +0400
+++ /usr/lib/python2.5/sgmllib.py 2009-07-05 02:21:16.000000000 +0400
@@ -397,7 +397,10 @@ class SGMLParser(markupbase.ParserBase):
return self.convert_codepoint(n)

def convert_codepoint(self, codepoint):
- return chr(codepoint)
+ if 0 <= codepoint <= 127:
+ return chr(codepoint)
+ else:
+ return unichr(codepoint)

def handle_charref(self, name):
"""Handle character reference, no need to override."""
Reply all
Reply to author
Forward
0 new messages