Funky Soup Results

[Daniel W]

I'm using the python selenium driver (http://pypi.python.org/pypi/
selenium) with the method browser.page_source to get the page text.
The text itself looks pretty normal to me.
When I try to BeautifulSoup the results, it just returns one giant
blob, and at the end is this:

></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></head></html>>

so it's clearly a parsing issue, because that's not what's at the end
of the text I'm handing to the BeautifulSoup() function.

Any tips?

[Jim Tittsler]

On Wed, Feb 1, 2012 at 14:10, Daniel W <dweitz...@gmail.com> wrote:
[...]

> The text itself looks pretty normal to me.
> When I try to BeautifulSoup the results, it just returns one giant
> blob, and at the end is this:
>
>></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></head></html>>
>
> so it's clearly a parsing issue, because that's not what's at the end
> of the text I'm handing to the BeautifulSoup() function.

Are you sure the page is valid HTML? </head> certainly suggests that
there are closing tags missing much earlier in your source.

[Daniel W]

Thanks for the reply.
See, it sure ought to be valid HTML - it's from a major website,
pulled via a widely used package.
I've found that stripping scripts:
stripped = re.sub(r'<script.+/>', '', text)
helps some.
I'm not completely sure why, because the soup still ends with all the
</divs>.
If you're curious, I can send email you the source text, I'd be much
obliged, because my solution feels fragile.

On Feb 2, 4:37 pm, Jim Tittsler <jtitts...@gmail.com> wrote: