Unwanted conversion of HTML entities

[peppergrower]

How can I get BeautifulSoup to not convert HTML entities when
accessing attributes that contain them? I'm currently retrieving
certain links and then asking for the 'href' attribute. The
interesting thing is that when I simply print out the entire link
object, the entities are intact. When I print out just the 'href'
attribute, I get a version with converted HTML entities. Here's some
example code:

from BeautifulSoup import BeautifulSoup
text = """<a href="index.php?task=view&amp;id=1">link text</a>"""
soup = BeautifulSoup(text,convertEntities=None)
links = soup.findAll('a')
print links
for item in links:
print item['href']

When I run this, I get the following output:

[<a href="index.php?option=com_content&amp;task=view&amp;id=1">link
text</a>]
index.php?option=com_content&task=view&id=1

Note the difference between printing the list of links and printing
the 'href' attribute of each link (well, only one in this example).
This is even when I explicitly state that convertEntities should be
'None'. Why? Is this a bug? (If it is, I can look at patching it.)
If it's expected behavior, why, and how can I work around it?

This matters for what I'm doing because in my actual code, I'm
extracting just the links (using parseOnlyThese), processing them, and
doing a search and replace in the original text to change those
links. However, the search and replace does nothing because it's
searching for the version without HTML entities, which doesn't exist
in the original.

From a purely logical standpoint, the behavior of the two outputs
should be the same, and from a use standpoint, it ought to be possible
to disable entity conversion because sometimes you do want them in the
parsed output.

Thanks,
peppergrower

[Aaron DeVore]

The current behavior is like that for specific reasons. When Beautiful
Soup is serializing, as in str(element) or element.prettify(), it
needs to convert to a valid XML/XHTML document. That involves escaping
the href attribute, including the '&'.

Someone directly accessing the href attribute, on the other hand,
would typically need the unescaped value. That way it could be used
to, for example, make an HTTP request for the document that is linked
to.

convertEntities is intended specifically for parsing, not
serialization. Technically, the behavior could be changed to work with
serialization as well. However, the way the Beautiful Soup
parsing/serialization is designed would make that a non-trivial
change. There could also be issues with extract() not correctly
pulling Tags out of the tree.

The first fix that comes to mind is doing an extra pass to convert
&amp; to &. A simple

result.replace("&amp;", "&")

would probably do unless you have some &amp; in the rest of the document.

-Aaron DeVore

> --
>
> You received this message because you are subscribed to the Google Groups "beautifulsoup" group.
> To post to this group, send email to beauti...@googlegroups.com.
> To unsubscribe from this group, send email to beautifulsou...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/beautifulsoup?hl=en.
>
>
>

[peppergrower]

Hm. So neither of them actually leaves the original untouched, I just
happened to have a fully-escaped link in my test case, so I didn't
realize it would have escaped it otherwise when calling str().

Well, darn. Unfortunately I don't see an elegant way around this.
I'll look at trying something semi-kludgy. Maybe a regex...I realize
those are horrible for HTML (I tried them before discovering the
difficulties), but I'm looking for a very particular type of string
that is very consistent. (I'm trying very hard to leave the rest of
the document untouched, so running a replace on all the escaped
entities in the entire document isn't going to work. Plus, that would
break any XHTML compliance.)

I realize that BeautifulSoup is mainly used to clean up bad HTML, so
maybe this isn't a common enough use case to justify changes. In
fact, what I'm trying to do might go against the whole philosophy
behind BeautifulSoup (which I respect as a very nice, useful tool).
However, I might as well ask: how hard would it be to create an
extract-only mode that doesn't modify any of the original document,
but could parse it and then find and return certain elements
(unmodified)?

Perhaps a better question: do you know of any robust HTML parsers
written in Python that already do this?

[Aaron DeVore]

Can you modify the links in the tree before you serialize?

for a in soup.findAll('a', href=re.compile("..."):
...modify a['href'] here...

unicode(soup)

- Aaron DeVore

[peppergrower]

Hm. The thing is, I'm not actually parsing the whole document--I'm
just pulling out the links (using parseOnlyThese). When I tried
setting the href attribute of these links, it didn't seem to do
anything. (I'd rather not run the whole page through BeautifulSoup at
this point, just because I want to only change this one little bit--
not have the whole page prettified.)

Perhaps there's a better way to do this that would accomplish both
your suggestion and my goal? (I'm only at a very basic level with
BeautifulSoup, as I'm sure you can tell.)

[Aaron DeVore]

peppergrower,
I'm confused. Are you trying to use parseOnlyThese because you just
want to change the 'a' tags, but leave the rest of the document the
same?

-Aaron DeVore

[peppergrower]

Partly that, and partly for a little speed boost. Plus, I just didn't
care about the rest of the document; right now I'm trying to fix a
bunch of broken links that all follow the same pattern, and I'd rather
leave the rest of the document alone. (That's not to say that the
rest couldn't use some improvement too, but until I know exactly what
kind of changes BeautifulSoup would make I'd rather not blindly trust
it. The pages do work right now, after all.)

[Aaron DeVore]

And now for a little more trying to tease out a solution...

At some point in your program, are you going to have something like this?:

document = document.replace(beforeATagText, afterATagText)

In this example, beforeATagText is the text that was in the original
document and afterATagText is the text from the Beautiful Soup tag
after manipulation.

-Aaron DeVore