Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Python HTML parser chokes on UTF-8 input

266 views
Skip to first unread message

Johannes Bauer

unread,
Oct 9, 2008, 4:54:59 PM10/9/08
to
Hello group,

I'm trying to use a htmllib.HTMLParser derivate class to parse a website
which I fetched via
httplib.HTTPConnection().request().getresponse().read(). Now the problem
is: As soon as I pass the htmllib.HTMLParser UTF-8 code, it chokes. The
code is something like this:

prs = self.parserclass(formatter.NullFormatter())
prs.init()
prs.feed(website)
self.__result = prs.get()
prs.close()

Now when I take "website" directly from the parser, everything is fine.
However I want to do some modifications before I parse it, namely UTF-8
modifications in the style:

website = website.replace(u"föö", u"bär")

Therefore, after fetching the web site content, I have to convert it to
UTF-8 first, modify it and convert it back:

website = website.decode("latin1")
website = website.replace(u"föö", u"bär")
website = website.encode("latin1")

This is incredibly ugly IMHO, as I would really like the parser to just
accept UTF-8 input. However when I omit the reecoding to latin1:

File "CachedWebParser.py", line 13, in __init__
self.__process(website)
File "CachedWebParser.py", line 55, in __process
prs.feed(website)
File "/usr/lib64/python2.5/sgmllib.py", line 99, in feed
self.goahead(0)
File "/usr/lib64/python2.5/sgmllib.py", line 133, in goahead
k = self.parse_starttag(i)
File "/usr/lib64/python2.5/sgmllib.py", line 285, in parse_starttag
self._convert_ref, attrvalue)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 0:
ordinal not in range(128)

Annoying, IMHO, that the internal html Parser cannot cope with UTF-8
input - which should (again, IMHO) be the absolute standard for such a
new language.

Can I do something about it?

Regards,
Johannes

--
"Meine Gegenklage gegen dich lautet dann auf bewusste Verlogenheit,
verlästerung von Gott, Bibel und mir und bewusster Blasphemie."
-- Prophet und Visionär Hans Joss aka HJP in de.sci.physik
<48d8bf1d$0$7510$5402...@news.sunrise.ch>

Terry Reedy

unread,
Oct 9, 2008, 6:03:37 PM10/9/08
to pytho...@python.org
Johannes Bauer wrote:
> Hello group,
>
> I'm trying to use a htmllib.HTMLParser derivate class to parse a website
> which I fetched via
> httplib.HTTPConnection().request().getresponse().read(). Now the problem
> is: As soon as I pass the htmllib.HTMLParser UTF-8 code, it chokes. The
> code is something like this:

I believe you are confusing unicode with unicode encoded into bytes with
the UTF-8 encoding. Having a problem feeding a unicode string, not
'UFT-8 code', which in Python can only mean a UTF-8 encoded byte string.


>
> prs = self.parserclass(formatter.NullFormatter())
> prs.init()
> prs.feed(website)
> self.__result = prs.get()
> prs.close()
>
> Now when I take "website" directly from the parser, everything is fine.
> However I want to do some modifications before I parse it, namely UTF-8
> modifications in the style:
>
> website = website.replace(u"föö", u"bär")
>
> Therefore, after fetching the web site content, I have to convert it to
> UTF-8 first, modify it and convert it back:
>

> website = website.decode("latin1") # produces unicode
> website = website.replace(u"föö", u"bär") #remains unicode
> website = website.encode("latin1") # produces byte string in the latin-1 encoding


>
> This is incredibly ugly IMHO, as I would really like the parser to just
> accept UTF-8 input.

To me, code that works is prettier than code that does not.

In 3.0, text strings are unicode, and I believe that is what the parser
now accepts.

>However when I omit the reecoding to latin1:
>
> File "CachedWebParser.py", line 13, in __init__
> self.__process(website)
> File "CachedWebParser.py", line 55, in __process
> prs.feed(website)
> File "/usr/lib64/python2.5/sgmllib.py", line 99, in feed
> self.goahead(0)
> File "/usr/lib64/python2.5/sgmllib.py", line 133, in goahead
> k = self.parse_starttag(i)
> File "/usr/lib64/python2.5/sgmllib.py", line 285, in parse_starttag
> self._convert_ref, attrvalue)
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 0:
> ordinal not in range(128)

When you do not bother to specify some other encoding in an encoding
operation, sgmllib or something deeper in Python tries the default
encoding, which does not work. Stop being annoyed and tell the
interpreter what you want. It is not a mind-reader.

> Annoying, IMHO, that the internal html Parser cannot cope with UTF-8
> input - which should (again, IMHO) be the absolute standard for such a
> new language.

The first version of Python came out in 1989, I believe, years before
unicode. One of the features of the new 3.0 version is that is uses
unicode as the standard for text.

Terry Jan Reedy

Johannes Bauer

unread,
Oct 9, 2008, 6:13:36 PM10/9/08
to
Terry Reedy schrieb:

> Johannes Bauer wrote:
>> Hello group,
>>
>> I'm trying to use a htmllib.HTMLParser derivate class to parse a website
>> which I fetched via
>> httplib.HTTPConnection().request().getresponse().read(). Now the problem
>> is: As soon as I pass the htmllib.HTMLParser UTF-8 code, it chokes. The
>> code is something like this:
>
> I believe you are confusing unicode with unicode encoded into bytes with
> the UTF-8 encoding. Having a problem feeding a unicode string, not
> 'UFT-8 code', which in Python can only mean a UTF-8 encoded byte string.

I also believe I am. Could you please elaborate further?

Do I understand correctly when saying that type 'str' has no associated
default encoding, but type 'unicode' does? Does this mean that really
the only way of coping with that stuff is doing what I've been doing?

>> This is incredibly ugly IMHO, as I would really like the parser to just
>> accept UTF-8 input.
>
> To me, code that works is prettier than code that does not.
>
> In 3.0, text strings are unicode, and I believe that is what the parser
> now accepts.

Well, yes, I suppose working code is nicer than non-working code.
However I am sure you will agree that explicit encoding conversions are
cumbersome and error-prone.

>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 0:
>> ordinal not in range(128)
>
> When you do not bother to specify some other encoding in an encoding
> operation, sgmllib or something deeper in Python tries the default
> encoding, which does not work. Stop being annoyed and tell the
> interpreter what you want. It is not a mind-reader.

How do I tell the interpreter to parse the strings I pass to it as
unicode? The way I did or is there some better way?

>> Annoying, IMHO, that the internal html Parser cannot cope with UTF-8
>> input - which should (again, IMHO) be the absolute standard for such a
>> new language.
>
> The first version of Python came out in 1989, I believe, years before
> unicode. One of the features of the new 3.0 version is that is uses
> unicode as the standard for text.

Hmmm. I suppose you're right there. Python 3.0 really sounds quite nice,
do you know when will approximately be ready?

Jerry Hill

unread,
Oct 9, 2008, 7:54:04 PM10/9/08
to pytho...@python.org
On Thu, Oct 9, 2008 at 4:54 PM, Johannes Bauer <dfnson...@gmx.de> wrote:
> Hello group,

>
> Now when I take "website" directly from the parser, everything is fine.
> However I want to do some modifications before I parse it, namely UTF-8
> modifications in the style:
>
> website = website.replace(u"föö", u"bär")

That's not utf-8, that's unicode. Even if your file is saved as
utf-8, you're telling python to convert those from utf-8 encoded bytes
to unicode characters, by prefixing them with 'u'.

> Therefore, after fetching the web site content, I have to convert it to
> UTF-8 first, modify it and convert it back:

You have to convert it to unicode if and only if you are doing
manipulation with unicode stings.

> website = website.decode("latin1")
> website = website.replace(u"föö", u"bär")
> website = website.encode("latin1")
>
> This is incredibly ugly IMHO, as I would really like the parser to just
> accept UTF-8 input. However when I omit the reecoding to latin1:

You could just use the precise Latin-1 byte strings you'd like to replace:

website = website.replace("f\xf6\xf6", "b\xe4r")

Or, you could set the encoding of your source file to Latin-1, by
putting the following on the first or second line of your source file:

# -*- coding: Latin-1 -*-

Then use the appropriate literals in your source code, making sure
that you save it as Latin-1 in your editor of choice.

Truthfully, though, I think your current approach really is the right
one. Decode to unicode character strings as soon as they come into
your program, manipulate them as unicode, then select your preferred
encoding when you write them back out. It's explicit, and only takes
two lines of code.

--
Jerry

Terry Reedy

unread,
Oct 9, 2008, 11:08:40 PM10/9/08
to pytho...@python.org
Johannes Bauer wrote:
> Terry Reedy schrieb:
>> Johannes Bauer wrote:
>>> Hello group,
>>>
>>> I'm trying to use a htmllib.HTMLParser derivate class to parse a website
>>> which I fetched via
>>> httplib.HTTPConnection().request().getresponse().read(). Now the problem
>>> is: As soon as I pass the htmllib.HTMLParser UTF-8 code, it chokes. The
>>> code is something like this:
>> I believe you are confusing unicode with unicode encoded into bytes with
>> the UTF-8 encoding. Having a problem feeding a unicode string, not
>> 'UFT-8 code', which in Python can only mean a UTF-8 encoded byte string.
>
> I also believe I am. Could you please elaborate further?

I am a unicode neophyte. My source of info is the first 3 or so
chapters of the unicode specification.
http://www.unicode.org/versions/Unicode5.1.0/
I recommend that or other sites for other questions. It took me more
than one reading of the same topics in different texts to pretty well
'get it'

> Do I understand correctly when saying that type 'str' has no associated
> default encoding, but type 'unicode' does?

I am not sure what you mean. Unicode strings in Python are internally
stored in USC-2 or UCS-4 format.

> Does this mean that really
> the only way of coping with that stuff is doing what I've been doing?

Having two text types in 2.x was necessary as a transition strategy but
has also been something of a mess. You did it one way. Jerry gave you
an alternative that I could not have explained. Your choice. Or use 3.0.

..


> Hmmm. I suppose you're right there. Python 3.0 really sounds quite nice,
> do you know when will approximately be ready?

For my current purposes, it is ready enough. Developers *really* hope
to get 3.0 final out by mid-December. The schedule was pushed back
because a) the outside world has not completely and cleanly switched to
unicode text and b) some people who just started with the release
candidate have found import bugs that earlier testers did not. It still
needs more testing from more different users (hint, hint).

Terry Jan Reedy

Marc 'BlackJack' Rintsch

unread,
Oct 10, 2008, 3:03:08 AM10/10/08
to
On Fri, 10 Oct 2008 00:13:36 +0200, Johannes Bauer wrote:

> Terry Reedy schrieb:


>> I believe you are confusing unicode with unicode encoded into bytes
>> with the UTF-8 encoding. Having a problem feeding a unicode string,
>> not 'UFT-8 code', which in Python can only mean a UTF-8 encoded byte
>> string.
>
> I also believe I am. Could you please elaborate further?
>
> Do I understand correctly when saying that type 'str' has no associated
> default encoding, but type 'unicode' does?

`str` doesn't know an encoding. The content could be any byte data
anyway. And `unicode` doesn't know an encoding either, it is unicode
characters. How they are represented internally is not the business of
the programmer. If you want operate with unicode characters you have to
decode a byte string (`str`) with the appropriate encoding. If you want
feed `unicode` to something that expects bytes and not unicode characters
you have to encode again.

>>> This is incredibly ugly IMHO, as I would really like the parser to
>>> just accept UTF-8 input.

It accepts UTF-8 input but not `unicode` objects.

> However I am sure you will agree that explicit encoding conversions are
> cumbersome and error-prone.

But implicit conversions are impossible because the interpreter doesn't
know which encoding to use and refuses to guess. Implicit and guessed
conversions are error prone too.

Ciao,
Marc 'BlackJack' Rintsch

John Nagle

unread,
Oct 17, 2008, 11:55:03 AM10/17/08
to
Johannes Bauer wrote:
> Hello group,
>
> I'm trying to use a htmllib.HTMLParser derivate class to parse a website
> which I fetched via
> httplib.HTTPConnection().request().getresponse().read(). Now the problem
> is: As soon as I pass the htmllib.HTMLParser UTF-8 code, it chokes. The
> code is something like this:

Try BeautifulSoup. It actually understands how to detect the encoding
of an HTML file (there are three different ways that information can be
expressed), and will shift modes accordingly.

This is an ugly problem. Sometimes, it's necessary to parse part of
the file, discover that the rest of the file has a non-ASCII encoding,
and restart the parse from the beginning. BeautifulSoup has the
machinery for that.

John Nagle

0 new messages