A patch to BeautifulSoup 3.0.7a and 3.1.0 for encoding errors

5 views
Skip to first unread message

Natim

unread,
Nov 6, 2009, 10:10:26 PM11/6/09
to beautifulsoup
Hello

I did a patch to BeautifulSoup 3.0.7a and 3.1.0

When trying to parse http://www.presse-citron.net/ a bug occurs since
it should be in utf-8 but some characters are in ISO-8859-1.

I fixed it like that :

1818,1820c1818,1822
< # print "That didn't work!"
< # print e
< return None
---
> #print "That didn't work!"
> #print e
> self.markup = markup.decode('latin1', 'ignore')
> self.originalEncoding = proposed
>

Enjoy using BeautifulSoup.

Aaron DeVore

unread,
Nov 6, 2009, 11:39:55 PM11/6/09
to beauti...@googlegroups.com
There were several errors that came up in the Beautiful Soup test
suite when I ran it on your patch. I know almost nothing about
Unicode/encodings, so I'm only guessing about what's happening. In
several of the tests, some of the more exotic strings get messed up.
Run BeautifulSoupTests.py to get an idea of what I'm talking about.

I started a branch on the Git Hub-hosted fork of Beautiful Soup. The
branch name is "natim-iso-8859-1". I haven't used Git branching so I'm
trying it out whenever possible. :)

-Aaron DeVore

Natim

unread,
Nov 6, 2009, 10:42:46 PM11/6/09
to beautifulsoup
Hello,

After some tests, it looks better like that :

--- /tmp/BeautifulSoup.py 2009-11-07 11:04:25.000000000 +0800
+++ BeautifulSoup.py 2009-11-07 11:14:22.000000000 +0800
@@ -1815,9 +1815,11 @@
self.markup = u
self.originalEncoding = proposed
except Exception, e:
- # print "That didn't work!"
- # print e
- return None
+ #print "That didn't work!"
+ #print e
+ self.markup = markup.decode('utf-8', 'replace')
+ self.originalEncoding = 'utf-8'
+
#print "Correct encoding: %s" % proposed
return self.markup

In this case we assume utf-8 to be the default encoding, and we
replace unknown characters with '?'

Regards

Rémy Hubscher

On 7 nov, 11:10, Natim <hubscher.r...@gmail.com> wrote:
> Hello
>
> I did a patch to BeautifulSoup 3.0.7a and 3.1.0
>
> When trying to parsehttp://www.presse-citron.net/a bug occurs since

Aaron DeVore

unread,
Nov 7, 2009, 3:58:08 PM11/7/09
to beauti...@googlegroups.com
Natim,
I'm still getting errors. Here's what shows up:

FAIL: testDontConvertSmartQuotesWhenAlsoConvertingEntities (__main__.EncodeRed)
----------------------------------------------------------------------
Traceback (most recent call last):
File "BeautifulSoupTests.py", line 839, in
testDontConvertSmartQuotesWhenAlsoConvertingEntities
'Il a dit, &lsaquo;Sacr&eacute; bl&#101;u!&rsaquo;')
AssertionError: 'Il a dit, Sacr&eacute; bl&#101;u!' != 'Il a dit,
&lsaquo;Sacr&eacute; bl&#101;u!&rsaquo;'

=======================================
FAIL: testRewrittenMetaTag (__main__.EncodeRed)
----------------------------------------------------------------------
Traceback (most recent call last):
File "BeautifulSoupTests.py", line 823, in testRewrittenMetaTag
isolatin.replace("ISO-Latin-1", "utf-8").replace("\xe9", "\xc3\xa9"))
File "BeautifulSoupTests.py", line 18, in assertSoupEquals
self.assertEqual(str(c(toParse)), rep)
AssertionError: '<html><meta http-equiv="Content-type"
content="text/html; charset=utf-8" />Sacrleu!</html>' != '<html><meta
http-equiv="Content-type" content="text/html; charset=utf-8"
/>Sacr\xc3\xa9 bleu!</html>'

======================================================================
FAIL: testRewrittenXMLHeader (__main__.EncodeRed)
----------------------------------------------------------------------
Traceback (most recent call last):
File "BeautifulSoupTests.py", line 761, in testRewrittenXMLHeader
self.assertSoupEquals(old_text, new_text)
File "BeautifulSoupTests.py", line 18, in assertSoupEquals
self.assertEqual(str(c(toParse)), rep)
AssertionError: "<?xml version='1.0' encoding='utf-8'?><foo></foo>" !=
"<?xml version='1.0' encoding='utf-8'?><foo>&rsquo;</foo>"

========================================
FAIL: testSmartQuotesNotSoSmartAnymore (__main__.EncodeRed)
----------------------------------------------------------------------
Traceback (most recent call last):
File "BeautifulSoupTests.py", line 833, in testSmartQuotesNotSoSmartAnymore
'&lsquo;Foo&rsquo; <!--blah-->')
File "BeautifulSoupTests.py", line 18, in assertSoupEquals
self.assertEqual(str(c(toParse)), rep)
AssertionError: 'Foo <!--blah-->' != '&lsquo;Foo&rsquo; <!--blah-->'

==========================================
FAIL: testUnicodeDammitStandalone (__main__.EncodeRed)
----------------------------------------------------------------------
Traceback (most recent call last):
File "BeautifulSoupTests.py", line 707, in testUnicodeDammitStandalone
self.assertEquals(dammit.unicode, "<foo>&#x2019;</foo>")
AssertionError: u'<foo></foo>' != '<foo>&#x2019;</foo>'

======================================
FAIL: testFindAllText (__main__.FollowThatTag)
----------------------------------------------------------------------
Traceback (most recent call last):
File "BeautifulSoupTests.py", line 82, in testFindAllText
[u'\xbb'])
AssertionError: [] != [u'\xbb']
Reply all
Reply to author
Forward
0 new messages