Does not parse engadget.com or youtube

[mclovin]

Hey guys, I was trying to extract some stuff and it turns out that
beautiful soup will not successfully parse a couple of things...
(everything is done with the latest version of beautiful soup on Linux
and on Vista)

The first is Engadget. It only parses up to the header, and then that
is it. It stops in the middle of a list in the header gets to the
middle of the list and then just stops w/o error. I can successfully
browse the parse tree, but obviously it isnt there. You can try to
generate the files yourself and see what it gives you:

url = "http://engadget.com/"
print "downloading", url
html = u.urlopen(url)
print "parsing..."
parsed = BeautifulSoup(html)


f = open('engadget.html', 'w')
f.write(u.urlopen(url).read())
f.close()

f = open('engadget_after_soup.html', 'w')
f.write(parsed.prettify())
f.close()


For me, engadget.html was about 160kb while engadget_after_soup.html
was only 9 kb. I tried a number of parsers bundled with soup and
nothing worked.

Also, youtube doesnt work. It fails on error.
http://pastebin.com/medc8950

It fails when parsing the javascript.


Any suggestions?

[mclovin]

Hey guys, nvrm, everything works great in 3.0.7a

> Also, youtube doesnt work. It fails on error.http://pastebin.com/medc8950