bs4 parses webpage incorrectly while lxml parses it fine

[Mark Grandi]

So, i'm running python3 and i have lxml installed, and ive noticed
that bs4 is parsing this webpage incorrectly and its causing problems
because a lot of the elements are malformed.

I run this:

Python 3.2.2 (v3.2.2:137e45f15c0b, Sep 3 2011, 17:28:59)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> thefile = open("/Users/markgrandi/Desktop/langref/mx/effects/Iris.html", "r")
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(thefile)
>>> thefile.close()
>>> outputfile = open("/Users/markgrandi/Desktop/bs4.txt", "w")
>>> outputfile.write(soup.prettify())
104520
>>> outputfile.close()


>>> import lxml.html
>>> thefile = open("/Users/markgrandi/Desktop/langref/mx/effects/Iris.html", "r")
>>> tree = lxml.html.parse(thefile)
>>> thefile.close()
>>> output = open("/Users/markgrandi/Desktop/lxml.txt", "w")
>>> output.write(lxml.html.tostring(tree, pretty_print=True).decode("utf-8"))
81037
>>> output.close()


this is the output:
bs4: http://pastebin.com/KV38maCY
lxml: http://pastebin.com/ZfWS11Dg

page that i am loading (iris.html): http://pastebin.com/L0NFdRJf (or
http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/mx/effects/Iris.html)

you can see at line 2768 in the bs4 output, it pretty much dies,
because there is a <script> block that starts out with a <!-- -->, but
the lxml output (line 1139) handles it fine.

any idea on why bs4 is getting tripped up?

[Thomas Kluyver]

I've also just encountered this bug; it seems to be a regression since
4.0.1, probably caused by this change:

http://bazaar.launchpad.net/~leonardr/beautifulsoup/bs4/revision/195

In my case, it's breaking somewhere around byte 512, which would be
the end of the first chunk fed to lxml. I've replicated this on Linux
and OS X, with Python 3.2.

It looks like this user is seeing the same problem with Python 2.7:
http://groups.google.com/group/beautifulsoup/browse_thread/thread/f64417237f8ac867

Thomas

> page that i am loading (iris.html):http://pastebin.com/L0NFdRJf(orhttp://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/mx...)

[Leonard Richardson]

Thanks, I can duplicate this now. I've filed a bug and fixed the problem:

https://bugs.launchpad.net/beautifulsoup/+bug/972466

I'm not convinced the other recent problem is related. I'm looking into it now.

Leonard

> --
> You received this message because you are subscribed to the Google Groups "beautifulsoup" group.
> To post to this group, send email to beauti...@googlegroups.com.
> To unsubscribe from this group, send email to beautifulsou...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/beautifulsoup?hl=en.
>

[福田尚三]

Thanks, Thomas

I've encountered same bug in HTML parsing with lxml.

It is on Windows7, with Python 2.7.

bs 4.3.2 still has this bug in feed method of builder.lxml.LXMLTreeBuilder.

In my case, I remove LXMLTreeBuilder.feed() to be able to call LXMLTreeBuilderForXML.feed() which you fixed. It seems OK.

Shozo Fukuda

2012年4月3日火曜日 21時42分06秒 UTC+9 Thomas Kluyver: