The Beautiful Soup 4 beta is here!
Leonard Richardson
When Beautiful Soup was first released in 2004, the state of HTML
parsing in Python was appalling. Over the past eight years, things
have improved so dramatically that Beautiful Soup's HTML parser is no
longer a competitive advantage. I don't want to duplicate other
peoples', work, so I'm getting Beautiful Soup out of the parser
businesss. Beautiful Soup's job is now to provide a Pythonic
screen-scraping API on top of a data structure created by a
third-party parser.
This will be Beautiful Soup 4, and I've been planning it for
years. With help from Thomas Kluyver and Ezio Melotti, I've now met
the three main goals of Beautiful Soup 4:
1. Make a single codebase that works under Python 2 and Python 3.
2. Stop using SGMLParser (removed in Python 3) and make it possible to
swap out one parser for another.
3. Support two major Python parsers (lxml and html5lib) as well as
Python's (not currently very good) batteries-included parser,
html.parser.
The first version of BS4 is almost ready for release, and I'd like you
to test it out, if you haven't already. I still to fix some things, in
particular some performance problems. But, note that even with the
performance problems, BS4 is faster than BS3 across the board.
On Python 2 or Python 3 you can install the BS4 beta with this command:
easy_install beautifulsoup4
You can also get the source tarball:
http://www.crummy.com/software/BeautifulSoup/download/4.x/beautifulsoup4-4.0.0b3.tar.gz
The documentation has been completely rewritten and is available here:
http://www.crummy.com/software/BeautifulSoup/doc/
You may find the section on porting BS3 code to BS4 especially
interesting:
http://www.crummy.com/software/BeautifulSoup/doc/#porting-code-to-bs4
There are three major things I'd like your feedback on before
completing the release.
= Hall of Fame =
The BS3 documentation lists open-source projects that use Beautiful
Soup. I stopped maintaining this list many years ago because there are
hundreds of these projects, and since most of them are
screen-scrapers, they're pretty ephemeral.
I'd like to bring this feature back as a "hall of fame", featuring
applications of Beautiful Soup that grab a reader's attention. People
who used Beautiful Soup in a high-profile way or to tackle a big
issue. Projects that are interesting to hear about even if the
software doesn't work anymore, or uses an old version of Beautiful
Soup, or if Beautiful Soup was used internally and the public only saw
the results.
My bias is towards projects having to do with space, science,
journalism, politics and social justice. Here are some examples so you
know the kind of thing I'm thinking of:
"Movable Type", a work of digital art on display in the lobby of the
New York Times building, uses Beautiful Soup to scrape New York Times
feeds. (http://www.nytimes.com/2007/10/25/arts/design/25vide.html)
Alexander Harrowell uses Beautiful Soup to track the businesss
activities of an arms merchant.
(http://www.harrowell.org.uk/viktormap.html)
The Lawrence Journal-World used Beautiful Soup in 2006 and 2010 to
gather election results.
(http://www.b-list.org/weblog/2010/nov/02/news-done-broke/)
The NOAA's Forecast Applications Branch uses Beautiful Soup in
TopoGrabber, a script for downloading "high resolution USGS datasets."
(http://laps.noaa.gov/topograbber/)
If you did anything of this sort, or know of someone who did, I'd
like to hear about it.
= Do you prefer lxml or html5lib? =
Right now, the parser ranking goes lxml, html5lib, html.parser. I like
lxml because it's incredibly fast and it can parse anything. But I'd
like to see what you think of the trees it generates. Would html5lib,
with its web-browser-like heuristics, be a better default?
= substitute_html_entities =
BS3 had a number of overlapping and inconsistent ways of turning
HTML/XML entities into Unicode characters, and possibly turning
Microsoft smart quotes into HTML entities at the same time. In BS4,
all this stuff is gone. HTML and XML entities are *always* converted
into Unicode characters.
This is great but there's one problem: output. If you want to turn
those Unicode characters back into entities when outputting as a
string, you need to call soup.encode(substitute_html_entities=True),
which is a little clunky. I'm thinking of adding an
'output_html_entities' attribute that you can set on a soup or tag to
control whether this substitution happens. Do you like this idea?
I think I also need to ensure that characters like "&" and "<" are
*always* converted to XML entities on output, even though this will
hurt performance a bit.
= Conclusion =
What you install with "easy_install beautifulsoup4" is a beta
release. If I hear of a problem soon, there's still time to fix it,
even if it means a major change to the API. So please try it out and
give me feedback.
Leonard
Bruce Eckel
(should be "changes").
-- Bruce Eckel
www.Reinventing-Business.com
www.MindviewInc.com
--
You received this message because you are subscribed to the Google Groups "beautifulsoup" group.
To post to this group, send email to beauti...@googlegroups.com.
To unsubscribe from this group, send email to beautifulsou...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beautifulsoup?hl=en.
Derek Litz
= Do you prefer lxml or html5lib? =
dealing
with bad html. However, most stuff would probably be handled well
enough
by lxml so why not go for lean and mean? If I needed a slower
specialty
parser for exceedingly bad html, then I'd go looking for one, or just
use
BeautifulSoup3 :)
As an aside:
Do you believe the BeautifulSoup3 parser to be sub par to the
competition
now even for small snippets of exceedingly bad html? (A more common
use
case then one would think!) What's in store for the future of BS3?
= substitute_html_entities =
I think that you should have substituting the html entities on by
default.
The reason being the intended purpose of str is to be the nice
printable
version. Another good reason is it seems like this would be more
backwards
compatible, correct?
Also, I think you can avoid the variable altogether by having repr
return
the version without substituting the html entities. This seems fine
because
a truly canonical representation of the object itself would not be all
that
useful compared to the unicode representation. Of course, this breaks
anything
that may have been using repr interchangeably with str. IMHO too bad
so sad.
Thanks for this update! I was honestly thinking I'd love a
BeautifulSoup front
end to lxml because it's the API I remember how to use without having
to refer to
documentation. And speaking of documentation, great work!
Derek Litz
> BS4 with no external dependencies will be comparable to BS3, but
> slightly faster.
Great!
> I think this might be an ASCII-centric (and thus, America-centric)
> view of things. For me, é is a strange character and I'd prefer to>
> represent it as the ASCII string é in an ASCII document. But
> for someone from France, it's a normal character they can type on
> their keyboard, and I suspect they want to represent it as é in a
> UTF-8 document.
Good point. Perhaps, a point of view change is required to
solve this problem, since trying to make this nice for every
region seems like a challenge that should be solved by regional
developers. Perhaps, a nice clean way to add new representations
to bs4, and then can be contributed back to the bs4 project?
Now, let's change to the point of view of a developer, which is
quite possibly your primary audience :). As a developer I'd like
the default str representation to be identical to the source
string.
One use case I had for bs3 was to parse snippets and by comparing
the parsed str to the original, and if there were any changes I
could conclude that there was something wrong with the tags. This
had the downside that the source string had to be formatted in
a particular way.
I think we could possibly improve upon what bs3 offered by
staying true to the source string without any arbitrary
influences. Simply add/remove tags where needed to make
it a valid tree.
The above requirement, however, may not be feasible across
parsers so a more relaxed, but nearly as useful constraint
would be to keep the str representation of soup identical
only if there were no changes required to make the tree.
Once changes are required some arbitrary changes may creep
in, but keeping html entities/unicode true to source would
be ideal.
I'm not sure how difficult either of these would be, and
if they are difficult and potentially error prone as well,
perhaps the best default would be one you could guarantee,
implies good performance, is simple across parsers, and
well documented.
If you go with this you could leave other views up to the
bs4 community, and possibly popular demand.
> I could replace every single non-ASCII character with an appropriate
> HTML escape, and that might be useful in some cases, but I don't think
> it's the right thing to do by default. It seems like the same kind of
> mistake to escape the non-ASCII characters that happen to have named
> HTML entities defined for them. I don't know--I'm still trying to
> figure this out.
Yeah, this goes back to my previous point. This seems like too big of
a problem to solve completely. A Pythonic way of adding and calling
new views seems the most appropriate as well as documentation to aid
developers who wish to contribute.
Really there are two choices, have it be a method on the soup object
or function call that takes a soup object. We could do both by
reusing the special method Python idiom.
bs4.views.ascii_style_html(soup)
which could mean:
soup.__view__('ascii_style_html')
Either way really don't like the idea of having it be a side effect
based on an attribute of the object. I think your current 'clunky'
way is actually better :)