ampersands

[Aron Griffis]

Hi Leonard,

WebTest uses BeautifulSoup to parse a page and find forms, then
stringifies each of them. See
https://github.com/Pylons/webtest/blob/master/webtest/response.py#L75

The problem is that entity-encoded entities aren't preserved
through this process, because BeautifulSoup makes a guess and
gets it wrong. It seems to me that BeautifulSoup should be able
to assume that text nodes need to be escaped without guessing,
since they were almost certainly creating as a result of parsing.

I'm currently using the monkey patch below in my application to
work around the problem. Is this something you'd consider
changing in BeautifulSoup4 proper?

Thanks,
Aron


# Patch BeautifulSoup4 to escape XML entities properly in textarea
#
# A browser will decode HTML entities on display in <textarea>, then
# submit the content back without encoding. This asymmetrical behavior
# serves a purpose: It allows the browser to parse where the <textarea>
# starts and ends without worrying about the content of the textarea
# itself. And then on submit, the browser assumes the form handler needs
# no special treatment, because it is not parsing HTML at that point.
#
# WebTest relies on BeautifulSoup4 to find forms in the document, then
# stringifies each form prior to parsing out the fields. Unfortunately
# BeautifulSoup4 tries to be smart in its serialization, so it sees
# &amp; and declines to re-escape the ampersand. Then the second parse
# turns what started as "&amp;amp;" into "&".
#
# Since the content was previously parsed by bs4, the ampersand should
# clearly be re-encoded regardless.

import re
from bs4.dammit import EntitySubstitution

if EntitySubstitution.substitute_xml('&') == EntitySubstitution.substitute_xml('&amp;'):
EntitySubstitution.BARE_AMPERSAND_OR_BRACKET = re.compile(r'[<>&]')
assert EntitySubstitution.substitute_xml('&') != EntitySubstitution.substitute_xml('&amp;')

[Leonard Richardson]

> WebTest uses BeautifulSoup to parse a page and find forms, then
> stringifies each of them. See
> https://github.com/Pylons/webtest/blob/master/webtest/response.py#L75
>
> The problem is that entity-encoded entities aren't preserved
> through this process, because BeautifulSoup makes a guess and
> gets it wrong. It seems to me that BeautifulSoup should be able
> to assume that text nodes need to be escaped without guessing,
> since they were almost certainly creating as a result of parsing.
>
> I'm currently using the monkey patch below in my application to
> work around the problem. Is this something you'd consider
> changing in BeautifulSoup4 proper?

Aron,

I don't understand your problem. Can you run me through it with some
example markup and code?

I've gone through a number of simple experiments, trying to understand
your problem, without getting any behavior I didn't expect. The one
unusual thing is that html.parser and lxml parse the contents of a
<textarea> tag as markup, not as text. (html5lib parses it as text.)
But that doesn't seem to be your problem, and there's nothing I can do
about that.

Leonard

[Aron Griffis]

Leonard Richardson wrote: [Wed May 15 2013, 05:24:48PM EDT]

> I don't understand your problem. Can you run me through it with some
> example markup and code?

Here you go. Sorry I didn't include a simple test case earlier!

soup = bs4.BeautifulSoup('<form><textarea>&amp;amp;</textarea></form>')
print str(soup.find('form'))

<form><textarea>&amp;</textarea></form>

Notice that the original &amp;amp; has been changed to &amp;
because bs4 guesses that it's already properly encoded. It
doesn't matter what parser you use because the bug is in the
serialization. The textarea also doesn't matter, except that it's
a more likely place to find entity-encoded HTML.

Hopefully my original email and monkey patch make more sense now.

Thanks,
Aron

[Aron Griffis]

Hi Leanard,

Did this test case and explanation make sense? Just making sure this didn't slip through the cracks.

Thanks,

Aron

[Leonard Richardson]

Yeah, I'm going to work on it today.

Leonard

> --
> You received this message because you are subscribed to the Google Groups
> "beautifulsoup" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to beautifulsou...@googlegroups.com.
> To post to this group, send email to beauti...@googlegroups.com.
> Visit this group at http://groups.google.com/group/beautifulsoup?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Leonard Richardson]

I filed a bug for this and fixed it.

https://bugs.launchpad.net/beautifulsoup/+bug/1182183

Leonard

[Aron Griffis]

Thanks Leonard!  I'll be curious to see the code change once you've pushed it to the repository.

Regards,
Aron