soup.findAll returns UnicodeEncodeError with 3.1.0.1, but not with 3.0.7a
33 views
Skip to first unread message
toffer
Nov 23, 2009, 8:52:24 AM11/23/09
to beautifulsoup
For a simple test case, I get a UnicodeEncodeError with
BeautifulSoup-3.1.0.1 for certain methods, but I don't get those
errors with 3.0.7a.
Here's the test HTML I am using: "<html><body>Price: £2.99</body></
html>"
Using this HTML, I can 'print soup.find('body')' with 3.1.0.1, but I
can't 'print soup.findAll('body'). But, both find and findAll work
fine with 3.0.7a.
I've attached two interactive python shell transcripts to demonstrate.
I've read that the parser with 3.0.7a handles parsing bad HTML better
than 3.1.0.1, but should I expect to get more UnicodeEncodeErrors with
3.1.0.1 as well? Or, am I doing something wrong? (Always quite
likely!)
Thanks,
Tom
Running python shell on a Mac (Leopard: OSX 10.5.8) using Python
2.5.1.
##
## Test using BeautifulSoup 3.1.0.1
##
$ python
Python 2.5.1 (r251:54863, Jun 17 2009, 20:37:34)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import BeautifulSoup3101
>>> BeautifulSoup3101.__version__
'3.1.0.1'
>>> unicode_html = u'<html><body>Price: \xa32.99</body></html>'
>>> unicode_html
u'<html><body>Price: \xa32.99</body></html>'
>>> print unicode_html
<html><body>Price: £2.99</body></html>
>>> utf8_html = unicode_html.encode('utf-8')
>>> type(utf8_html)
<type 'str'>
>>> utf8_html
'<html><body>Price: \xc2\xa32.99</body></html>'
>>> print utf8_html
<html><body>Price: £2.99</body></html>
>>> soup = BeautifulSoup3101.BeautifulSoup(utf8_html, fromEncoding='utf-8')
>>> soup
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in
position 19: ordinal not in range(128)
>>> soup.prettify()
'<html>\n <body>\n Price: \xc2\xa32.99\n </body>\n</html>'
>>> print soup.prettify()
<html>
<body>
Price: £2.99
</body>
</html>
>>> soup.find('body')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in
position 13: ordinal not in range(128)
>>> print soup.find('body')
<body>Price: £2.99</body>
>>> print soup.findAll('body')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in
position 13: ordinal not in range(128)
##
## Test using BeautifulSoup 3.0.7a
##
$ python
Python 2.5.1 (r251:54863, Jun 17 2009, 20:37:34)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import BeautifulSoup307a
>>> BeautifulSoup307a.__version__
'3.0.7a'
>>> unicode_html = u'<html><body>Price: \xa32.99</body></html>'
>>> unicode_html
u'<html><body>Price: \xa32.99</body></html>'
>>> print unicode_html
<html><body>Price: £2.99</body></html>
>>> utf8_html = unicode_html.encode('utf-8')
>>> type(utf8_html)
<type 'str'>
>>> utf8_html
'<html><body>Price: \xc2\xa32.99</body></html>'
>>> print utf8_html
<html><body>Price: £2.99</body></html>
>>> soup = BeautifulSoup307a.BeautifulSoup(utf8_html, fromEncoding='utf-8')
>>> soup
<html><body>Price: £2.99</body></html>
>>> soup.prettify()
'<html>\n <body>\n Price: \xc2\xa32.99\n </body>\n</html>'
>>> print soup.prettify()
<html>
<body>
Price: £2.99
</body>
</html>
>>> soup.find('body')
<body>Price: £2.99</body>
>>> print soup.find('body')
<body>Price: £2.99</body>
>>> print soup.findAll('body')
[<body>Price: £2.99</body>]
BeautifulSoup-3.1.0.1 for certain methods, but I don't get those
errors with 3.0.7a.
Here's the test HTML I am using: "<html><body>Price: £2.99</body></
html>"
Using this HTML, I can 'print soup.find('body')' with 3.1.0.1, but I
can't 'print soup.findAll('body'). But, both find and findAll work
fine with 3.0.7a.
I've attached two interactive python shell transcripts to demonstrate.
I've read that the parser with 3.0.7a handles parsing bad HTML better
than 3.1.0.1, but should I expect to get more UnicodeEncodeErrors with
3.1.0.1 as well? Or, am I doing something wrong? (Always quite
likely!)
Thanks,
Tom
Running python shell on a Mac (Leopard: OSX 10.5.8) using Python
2.5.1.
##
## Test using BeautifulSoup 3.1.0.1
##
$ python
Python 2.5.1 (r251:54863, Jun 17 2009, 20:37:34)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import BeautifulSoup3101
>>> BeautifulSoup3101.__version__
'3.1.0.1'
>>> unicode_html = u'<html><body>Price: \xa32.99</body></html>'
>>> unicode_html
u'<html><body>Price: \xa32.99</body></html>'
>>> print unicode_html
<html><body>Price: £2.99</body></html>
>>> utf8_html = unicode_html.encode('utf-8')
>>> type(utf8_html)
<type 'str'>
>>> utf8_html
'<html><body>Price: \xc2\xa32.99</body></html>'
>>> print utf8_html
<html><body>Price: £2.99</body></html>
>>> soup = BeautifulSoup3101.BeautifulSoup(utf8_html, fromEncoding='utf-8')
>>> soup
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in
position 19: ordinal not in range(128)
>>> soup.prettify()
'<html>\n <body>\n Price: \xc2\xa32.99\n </body>\n</html>'
>>> print soup.prettify()
<html>
<body>
Price: £2.99
</body>
</html>
>>> soup.find('body')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in
position 13: ordinal not in range(128)
>>> print soup.find('body')
<body>Price: £2.99</body>
>>> print soup.findAll('body')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in
position 13: ordinal not in range(128)
##
## Test using BeautifulSoup 3.0.7a
##
$ python
Python 2.5.1 (r251:54863, Jun 17 2009, 20:37:34)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import BeautifulSoup307a
>>> BeautifulSoup307a.__version__
'3.0.7a'
>>> unicode_html = u'<html><body>Price: \xa32.99</body></html>'
>>> unicode_html
u'<html><body>Price: \xa32.99</body></html>'
>>> print unicode_html
<html><body>Price: £2.99</body></html>
>>> utf8_html = unicode_html.encode('utf-8')
>>> type(utf8_html)
<type 'str'>
>>> utf8_html
'<html><body>Price: \xc2\xa32.99</body></html>'
>>> print utf8_html
<html><body>Price: £2.99</body></html>
>>> soup = BeautifulSoup307a.BeautifulSoup(utf8_html, fromEncoding='utf-8')
>>> soup
<html><body>Price: £2.99</body></html>
>>> soup.prettify()
'<html>\n <body>\n Price: \xc2\xa32.99\n </body>\n</html>'
>>> print soup.prettify()
<html>
<body>
Price: £2.99
</body>
</html>
>>> soup.find('body')
<body>Price: £2.99</body>
>>> print soup.find('body')
<body>Price: £2.99</body>
>>> print soup.findAll('body')
[<body>Price: £2.99</body>]
Aaron DeVore
Nov 23, 2009, 8:04:37 PM11/23/09
to beauti...@googlegroups.com
Tom,
I don't have a great understanding of how Beautiful Soup handles
encodings, but I'll give it a shot. The system for serialization in
Beautiful Soup 3.0.x is different than in 3.1.x. Specifically, the
behavior of __repr__ is slightly different. __repr__ is the hook used
by the interactive interpreter to implicitly print out the results of
some expressions to the terminal.
In BS 3.0.x, __repr__ is identical to __str__. In BS 3.1.x, the
behavior of __repr__ when it comes to encoding is just oh so slightly
different than __str__. The specific code starts at line 615 in
BeautifulSoup.py, version 3.1.0.1. For your code, the difference
apparently will cause a problem with __repr__, but not __str__. The
problem comes when unicode.encode is called with 'utf-8' (the default
for __repr__) instead of None (used by __str__).
unicode.encode('utf-8') chocks on the Euro symbol, but
unicode.encode(None) does not.
The reason that you're running into trouble with "print
soup.findAll('body')" as opposed to "print soup.find('body')" is
caused by how print interacts with list.__str__. From what I can tell,
list.__str__ goes through its contents and uses the C equivalent to
repr(...) on them. If you were to manual loop through the list and
call str(...) on the individual resulting elements you wouldn't get
any UnicodeEncodeErrors. "print soup.find('body')" directly calls
__str__.
As an aside, if you ever need to print out something as a list you can
use a list comprehension to explicitly run __unicode__.
[unicode(element) for element in soup.findAll('tag-name')]
Cheers!
Aaron DeVore
> --
>
> You received this message because you are subscribed to the Google Groups "beautifulsoup" group.
> To post to this group, send email to beauti...@googlegroups.com.
> To unsubscribe from this group, send email to beautifulsou...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/beautifulsoup?hl=.
>
>
>
I don't have a great understanding of how Beautiful Soup handles
encodings, but I'll give it a shot. The system for serialization in
Beautiful Soup 3.0.x is different than in 3.1.x. Specifically, the
behavior of __repr__ is slightly different. __repr__ is the hook used
by the interactive interpreter to implicitly print out the results of
some expressions to the terminal.
In BS 3.0.x, __repr__ is identical to __str__. In BS 3.1.x, the
behavior of __repr__ when it comes to encoding is just oh so slightly
different than __str__. The specific code starts at line 615 in
BeautifulSoup.py, version 3.1.0.1. For your code, the difference
apparently will cause a problem with __repr__, but not __str__. The
problem comes when unicode.encode is called with 'utf-8' (the default
for __repr__) instead of None (used by __str__).
unicode.encode('utf-8') chocks on the Euro symbol, but
unicode.encode(None) does not.
The reason that you're running into trouble with "print
soup.findAll('body')" as opposed to "print soup.find('body')" is
caused by how print interacts with list.__str__. From what I can tell,
list.__str__ goes through its contents and uses the C equivalent to
repr(...) on them. If you were to manual loop through the list and
call str(...) on the individual resulting elements you wouldn't get
any UnicodeEncodeErrors. "print soup.find('body')" directly calls
__str__.
As an aside, if you ever need to print out something as a list you can
use a list comprehension to explicitly run __unicode__.
[unicode(element) for element in soup.findAll('tag-name')]
Cheers!
Aaron DeVore
>
> You received this message because you are subscribed to the Google Groups "beautifulsoup" group.
> To post to this group, send email to beauti...@googlegroups.com.
> To unsubscribe from this group, send email to beautifulsou...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/beautifulsoup?hl=.
>
>
>
Tom Offermann
Nov 24, 2009, 1:09:44 PM11/24/09
to beauti...@googlegroups.com
Aaron,
Thanks for investigating this.
I'm curious why this was changed from 3.0.x to 3.1.x. It really makes
iteratively developing a scraper much more challenging when you can't
print out the results!
For now, I'll stick with 3.0.7a, since that's worked well for me so far.
Tom
Thanks for investigating this.
I'm curious why this was changed from 3.0.x to 3.1.x. It really makes
iteratively developing a scraper much more challenging when you can't
print out the results!
For now, I'll stick with 3.0.7a, since that's worked well for me so far.
Tom
Aaron DeVore
Nov 24, 2009, 3:46:58 PM11/24/09
to beauti...@googlegroups.com
It's not really that big of a deal. Just make sure that __str__ or
__unicode__ get called instead of __repr__.
-Aaron DeVore
>
>
>
__unicode__ get called instead of __repr__.
-Aaron DeVore
> --
>
> You received this message because you are subscribed to the Google Groups "beautifulsoup" group.
> To post to this group, send email to beauti...@googlegroups.com.
> To unsubscribe from this group, send email to beautifulsou...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/beautifulsoup?hl=en.
>
> You received this message because you are subscribed to the Google Groups "beautifulsoup" group.
> To post to this group, send email to beauti...@googlegroups.com.
> To unsubscribe from this group, send email to beautifulsou...@googlegroups.com.
>
>
>
Reply all
Reply to author
Forward
0 new messages