BeautifulSoup failure parsing MLB.com pages

[Wells]

Parsing this URL causes an exception to be thrown:
http://www.mlb.com/stats/individual_player_gamebygamelog.jsp?playerID=407812

The exception information:

Traceback (most recent call last):
File "./go.py", line 6, in <module>
p = player.Player(407812)
File "/home/wells/py-mlb/lib/player.py", line 26, in __init__
parser = BeautifulSoup.BeautifulSoup(result)
File "/usr/local/lib/python2.6/dist-packages/BeautifulSoup.py", line
1499, in __init__
BeautifulStoneSoup.__init__(self, *args, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/BeautifulSoup.py", line
1230, in __init__
self._feed(isHTML=isHTML)
File "/usr/local/lib/python2.6/dist-packages/BeautifulSoup.py", line
1263, in _feed
self.builder.feed(markup)
File "/usr/lib/python2.6/HTMLParser.py", line 108, in feed
self.goahead(0)
File "/usr/lib/python2.6/HTMLParser.py", line 150, in goahead
k = self.parse_endtag(i)
File "/usr/lib/python2.6/HTMLParser.py", line 314, in parse_endtag
self.error("bad end tag: %r" % (rawdata[i:j],))
File "/usr/lib/python2.6/HTMLParser.py", line 115, in error
raise HTMLParseError(message, self.getpos())
HTMLParser.HTMLParseError: bad end tag: u'</scr"+"ipt>', at line 118,
column 114

Is there some way to get around this without using regular
expressions? Thanks.

[Aaron DeVore]

Go for Beautiful Soup version 3.0.7a. That should work.

[Wells]

Thanks. 3.0.7a does indeed work. Is there a a reason why it's broken
in the newer version, and will I lose this functionality in newer
versions of BeautifulSoup?

On Oct 26, 4:05 pm, Aaron DeVore <aaron.dev...@gmail.com> wrote:
> Go for Beautiful Soup version 3.0.7a. That should work.
>
> On Mon, Oct 26, 2009 at 11:02 AM, Wells <we...@submute.net> wrote:
>
> > Parsing this URL causes an exception to be thrown:

> >http://www.mlb.com/stats/individual_player_gamebygamelog.jsp?playerID...

[sste...@gmail.com]

On Oct 26, 2009, at 5:16 PM, Wells wrote:

>
> Thanks. 3.0.7a does indeed work. Is there a a reason why it's broken
> in the newer version, and will I lose this functionality in newer
> versions of BeautifulSoup?

Long story short: Python dropped support for the lax parser that made
Beautiful Soup able to handle much of the icky HTML that it did.

I have proposed a branch containing that icky code, but don't have
time to do it myself.

There is a 3.0.7a floating around on bitbucket or github with some
improvements and still using the old parser; I'd suggest that anyone
needing 3.0.7 functionality migrate to that and let's move it forward.

I, for one, cannot use the new version at all as the new "improved"
parser doesn't handle much in the way of invalid HTML.

S