Unicode error with web scraping

[LJ]

Hi. I am writing some quick-n-dirty code to scrape together a RSS feed
from a section of the ITV website, and am stuck with this persistent
error. If I try to open the resultant xml file in Firefox, it errors
out and is unable to recognize the text pulled from the <h1> tags of
http://www.itv-f1.com/Feature.aspx?Type=Gravel_Trap&id=41916 --
'Itâ&euro;&trade;s a hard lifeâ&euro;'. I am not sure what encoding
this is, the webpage doesnt mention it, and a query of
handle.headers.getencoding() gives '7bit'.

Here is the code:

import urllib2
from BeautifulSoup import BeautifulSoup

def getcontents(article_link):
""" Given the URL of a gravel trap article, this will return the
title of that article. """
handle = urllib2.urlopen(article_link)
soup = BeautifulSoup(handle.read())
article_heading = str(soup('h1')) # get the only <H1> tag in the
page
return article_heading

if __name__ == '__main__':
rssfile = open('C:/Users/xyz/Desktop/graveltraprss.xml','wb')
article_url = 'http://www.itv-f1.com/Feature.aspx?
Type=Gravel_Trap&id=41916'
contents = getcontents(article_url)
rssfile.write(
"""<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">
<channel>
<title>Gravel Trap - ITV.com/F1</title>
<link>http://www.itv-f1.com/Feature.aspx?
Type=Gravel_Trap</link>
<description>The light hearted side of F1.</description>
<language>en-us</language>
<pubDate>Tue, 10 Jun 2003 04:00:00 GMT</pubDate>
<lastBuildDate>Tue, 10 Jun 2003 09:41:01 GMT</
lastBuildDate>
<docs>http://blogs.law.harvard.edu/tech/rss</docs>
<generator>Web scrapped by Python &amp; BeautifulSoup</
generator>
<managingEditor>x...@yahoo.com</managingEditor>
<webMaster>x...@yahoo.com</webMaster>""")
rssfile.write(
""" <item>
<title>""" + contents + """</title>
<link>%s</link>
<description>blank</description>
<pubDate>Tue, 20 May 2003 08:56:02 GMT</pubDate>
<guid>http://liftoff.msfc.nasa.gov/
2003/05/20.html#item570</guid>
</item>""" % article_url)
rssfile.write('</channel></rss>')
rssfile.close()



I have tried a million things: tried appending a 'u' before the
strings, opening the file using codecs.open(), pickle, binary & the
default file opening modes ... nothing seems to work.

Please help.

Thanks

- LJ

[Kent Johnson]

LJ wrote:
> Hi. I am writing some quick-n-dirty code to scrape together a RSS feed
> from a section of the ITV website, and am stuck with this persistent
> error. If I try to open the resultant xml file in Firefox, it errors
> out and is unable to recognize the text pulled from the <h1> tags of
> http://www.itv-f1.com/Feature.aspx?Type=Gravel_Trap&id=41916 --

> 'Itā&euro;&trade;s a hard lifeā&euro;'. I am not sure what encoding

> this is, the webpage doesnt mention it, and a query of
> handle.headers.getencoding() gives '7bit'.

BS seems to be mis-interpreting the encoding of the web page and
escaping non-ascii characters as named entities.

The actual encoding of the page is utf-8; BS converts it as Windows-1252.

Try this:
handle = urllib2.urlopen(article_link)
data = handle.read()
soup = BeautifulSoup(data, fromEncoding='utf-8')

Kent

[LJ]

Yep that solved the problem. Big thanks.
Any hint on how you detected the encoding of the page to be UTF-8?

Thanks

On Apr 12, 4:10 pm, Kent Johnson <ken...@tds.net> wrote:
> LJ wrote:
> > Hi. I am writing some quick-n-dirty code to scrape together a RSS feed
> > from a section of the ITV website, and am stuck with this persistent
> > error. If I try to open the resultant xml file in Firefox, it errors
> > out and is unable to recognize the text pulled from the <h1> tags of
> >http://www.itv-f1.com/Feature.aspx?Type=Gravel_Trap&id=41916--

> > 'Itâ&euro;&trade;s a hard lifeâ&euro;'. I am not sure what encoding

[Kent Johnson]

- Firefox gets it right - look under View / Character encoding
- Lots of experience looking at UTF-8 text rendered as latin-1 - it
tends to show a lot of accented 'a' characters and euro signs.

Kent