Bs4 xml parse losing content. Bug or parse configuration issue?

115 views

Skip to first unread message

Isaac Egglestone

unread,

Feb 12, 2021, 9:02:13 PM2/12/21

to beautifulsoup

Hi I'm posting here before raising an issue report as I'm unsure if there is some other problem with what I'm doing rather than an actual bug. Also as the problem appears to be between BeautifulSoup and lxml parser somewhere I need to have a reference link of having raised this on the forum before actually providing it to possibly both groups of maintainers.

Context: I'm processing some xml contained within an ebook for further distribution.

The actual problem is simple, after running BS4 diagnose on a file the output is missing content. Specifically the "opf" text before the colon. Now the BS4 maintainers indicate in their docs that this means there is a problem the my parser. That being lxml. However when I run a similar basic test with the xml parser I don't see the loss occurring.

The original guntenburg document contains:

<?xml version="1.0" encoding="utf-8"?>

<dc:rights>Public domain in the USA.</dc:rights>

<dc:identifier opf:scheme="URI" id="id">http://www.gutenberg.org/2600</dc:identifier>

<dc:creator opf:file-as="Tolstoy, Leo, graf">graf Leo Tolstoy</dc:creator>

The lxml parser via BS4 diagnose produces the following output with opf missing:

?xml version="1.0" encoding="utf-8"?>

<dc:rights>

Public domain in the USA.

</dc:rights>

<dc:identifier :scheme="URI" id="id">

http://www.gutenberg.org/ebooks/1502

</dc:identifier>

<dc:creator :file-as="Shakespeare, William">

William Shakespeare

</dc:creator>

<dc:title>

The History of King Henry the Sixth, Third Part

</dc:title>

Running the same basic test with lxml however shows clean output with opf intact:

<dc:rights>Public domain in the USA.</dc:rights>

<dc:identifier id="id" opf:scheme="URI">http://www.gutenberg.org/ebooks/1502</dc:identifier>

<dc:creator opf:file-as="Shakespeare, William">William Shakespeare</dc:creator>

<dc:title>The History of King Henry the Sixth, Third Part</dc:title>

<dc:language xsi:type="dcterms:RFC4646">en</dc:language>

To reproduce put all these files in the same dir and run the following two commands:

docker build

docker-compose run util python3 /test/testparse.py

requirements.txt:

beautifulsoup4

lxml

Dockerfile:

FROM python:3. 7

COPY requirements.txt requirements.txt

RUN pip install -r requirements.txt --upgrade

docker-compose.yml:

version: "3"

services:

util:

build: .

volumes:

- ./:/test

testparse.py:

#!/usr/bin/env python3

from bs4 import BeautifulSoup

from bs4.diagnose import diagnose

from lxml import etree

content_full_path = "test/content.opf"

def get_soup(file_to_parse):

"Gets a soup object for a given file"

with open(file_to_parse, "r") as f:

file = f.read()

diagnose(file)

soup = BeautifulSoup(file, "xml")

return soup

def check_untouched_parse(file_to_parse):

"Test clean xml save"

soup = get_soup(file_to_parse)

print(soup)

# Beautiful soup diagnose test:

print("Beautiful soup diagnose test:")

print("==============================================================================")

nothing_var = check_untouched_parse(content_full_path)

# lxml test

print("Etree version of parse")

print("==============================================================================")

parser = etree.XMLParser(remove_blank_text=True)

root = etree.parse(content_full_path, parser)

# Print the loaded XML

print(etree.tostring(root, encoding="unicode", pretty_print=True))

content.opf:

<?x

ml version='1.0' encoding='UTF-8'?>

<dc:rights>Public domain in the USA.</dc:rights>

<dc:identifier opf:scheme="URI" id="id">http://www.gutenberg.org/2600</dc:identifier>

<dc:creator opf:file-as="Tolstoy, Leo, graf">graf Leo Tolstoy</dc:creator>

<dc:contributor opf:role="trl" opf:file-as="Maude, Aylmer">Aylmer Maude</dc:contributor>

<dc:contributor opf:role="trl" opf:file-as="Maude, Louise">Louise Maude</dc:contributor>

<dc:title>War and Peace</dc:title>

<dc:language xsi:type="dcterms:RFC4646">en</dc:language>

<dc:subject>Historical fiction</dc:subject>

<dc:subject>War stories</dc:subject>

<dc:subject>Napoleonic Wars, 1800-1815 -- Campaigns -- Russia -- Fiction</dc:subject>

<dc:subject>Russia -- History -- Alexander I, 1801-1825 -- Fiction</dc:subject>

<dc:subject>Aristocracy (Social class) -- Russia -- Fiction</dc:subject>

<dc:date opf:event="publication">2001-04-01</dc:date>

<dc:date opf:event="conversion">2020-04-02T07:55:23.696736+00:00</dc:date>

<dc:source>https://www.gutenberg.org/files/2600/2600-h/2600-h.htm</dc:source>

</metadata>

</manifest>

</spine>

<guide>

</guide>

</package>

leonardr

unread,

Feb 12, 2021, 10:12:42 PM2/12/21

to beautifulsoup

Hi, Isaac,

Thanks for asking about this, and taking the time to provide detailed instructions for reproducing the issue.

The input and output XML documents you give are not the same string, but I'm pretty sure they're equivalent XML. Making the output more closely resemble the input is a reasonable thing to want, and since lxml does it, it should be possible to do it in Beautiful Soup. Based on your instructions, I've filed issue #1915583 to track the work necessary to make this change.

The underlying issue is that the original XML defines two aliases for the "http://www.idpf.org/2007/opf" namespace URIs: first "opf", and then the default namespace:

Beautiful Soup keeps a mapping of namespace URIs to aliases, which it uses when converting the parsed document back to a string. When a single URI is mapped to multiple aliases, it looks like the last alias encountered--in this case the default namespace--is the one that gets used. All tags from that namespace go out with the last alias defined for that namespace, even if they came in with a different alias.

This behavior is understandably unexpected, and I think it'll be possible to get the more intuitive behavior that raw lxml exhibits, but I don't think the meaning of the XML document is changing.

Leonard

Isaac Egglestone

unread,

Feb 13, 2021, 9:11:11 AM2/13/21

to beauti...@googlegroups.com

First, thanks for responding so quickly and carefully to my post.

RE: This behavior is understandably unexpected, and I think it'll be possible to get the more intuitive behavior that raw lxml exhibits, but I don't think the meaning of the XML document is changing.

As my knowledge of the XML standard isn't at any kind of master level, I can't agree nor disagree with you there, although it does make sense in some sense.

However, the latest epubchecker (by the DAISY Consortium on behalf of the W3C) seems to consider the xml in this format Erroneous and unsuitable for Ebook readers.

Example errors:

'ERROR(RSC-005): /tests/pg1502.epub/OEBPS/content.opf(-1,-1): Error while parsing file: Attribute name "xmlns:xsi" associated with an element type "dc:identifier" must be followed by the ' = ' character. ERROR(OPF-001): /tests/gutenburgbookparser/pg1502.epub/OEBPS/content.opf(-1,-1): There was an error when parsing the EPUB version: Version not found. Check finished with errors Messages: 0 fatals / 2 errors / 0 warnings / 0 infos EPUBCheck completed' should be empty. The above error seems related to the dc:identifier being followed by a colon ":"

Original text: <dc:identifier id="id" opf:scheme="URI">http://www.gutenberg.org/ebooks/1502</dc:identifier>
Problem text:

<dc:identifier :scheme="URI" id="id"> http://www.gutenberg.org/ebooks/1502 </dc:identifier>

So thank you for opening the issue.

Do you think removing the default namespace via lxml directly and then re-adding it after parsing it with BS4 would be a feasible work around?

I'll also run some Epubcheck tests on not having a default namespace as well to see if that is also workable.

--
You received this message because you are subscribed to a topic in the Google Groups "beautifulsoup" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beautifulsoup/4j0dMYJ48pw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to beautifulsou...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beautifulsoup/61656384-5749-456c-aff3-1800f4226943n%40googlegroups.com.

leonardr

unread,

Feb 13, 2021, 10:04:38 AM2/13/21

to beautifulsoup

Isaac,

You're totally right; that is invalid XML. I got caught up in what was happening inside Beautiful Soup and I stopped looking at the output.

The good news is this issue is much easier to fix than the duplicate-prefix issue; I fixed it in revision 595.

Removing the default namespace should work. Since all your files come from a single source, you should also be able to use a regular expression to remove the "opf:" prefix from the markup wherever it appears, and fall back to the default prefix. I'm not sure how this problem came about, but this is the first time it's been reported, so I think it's related to the fact that a single URI is given two prefixes and the second one is the default (empty string) prefix.

Leonard

Isaac Egglestone

unread,

Feb 13, 2021, 10:05:50 PM2/13/21

to beauti...@googlegroups.com

A huge thanks!

Yeah I’ll try both approaches and see how it goes.

I’ll try out your revision as well and report back my findings for anyone else that ends up in a similar situation.

To view this discussion on the web visit https://groups.google.com/d/msgid/beautifulsoup/3ed710f4-af41-4a8c-afc7-9b41d23a2eb2n%40googlegroups.com.

Reply all

Reply to author

Forward

0 new messages