I've changed Beautiful Soup so that the default tree builder (which as
you're all aware uses the not-very-good HTMLParser) can be swapped out
for one that uses a different parser. I've written a tree builder for
lxml that's twice as fast as the HTMLParser. I'll also be writing one
for html5lib.
Writing a TreeBuilder is very easy. The TreeBuilder's job is to run a
document through a parser and generate "start tag", "end tag", and
"data" events for Beautiful Soup's consumption. All information about
how the underlying parser works is encapsulated in the tree builder
code. So there are many ways to build a Beautiful Soup tree, and once
you have the tree you can use all the Beautiful Soup methods on it.
I've created a small compatibility test suite for TreeBuilders in
tests/treebuilder.py. If this test suite passes with your tree
builder, you probably implemented the tree builder correctly. Every
tree builders will interpret bad HTML differently, but they should all
interpret valid HTML the same way.
I've also moved Beautiful Soup to zc.buildout for build and
deployment. This won't affect you if you just use Beautiful Soup, but
it changes where the source files live in the package.
The code is at the trunk of the beautifulsoup project on Launchpad:
http://code.launchpad.net/beautifulsoup
This will probably become Beautiful Soup 4.0. I'd like to hear
feedback about the TreeParser design and how the compatibility suite
can be improved.
Leonard
