Xpact-core now clears 100,000+
real-world test files
I want to share a
milestone for Xpact-core, the native Eiffel XML
parser ported from libexpat's C source. A sweep of every
XML-family file I could find on a working Linux Mint desktop, a
Steam library, and also on a mounted Windows partition has now
cleared 132,305 files with zero failures.
Given that libexpat
sits underneath an enormous share of the world's XML processing —
it's not a stretch to say the files in this sweep were, at some
point, parsed by code that now runs behind billions of users.
Xpact-core is being held to that same bar.
The numbers
| Directory |
Extensions
tested |
Files
passed |
Failed |
/usr/share |
glade,
rng, policy, xsl, ui, docbook, xml, svg |
84,861 |
0 |
$ISE_EIFFEL |
xml,
eant, ecf |
2,946 |
0 |
$HOME/.steam |
manifest,
svg, xml |
5,822 |
0 |
$HOME/.es |
xml |
63 |
0 |
Windows
System32 |
xml |
872 |
0 |
Windows
WinSxS |
manifest |
37,741 |
0 |
| Total |
|
132,305 |
0 |
What
"passed" actually checks
Each file in the corpus is scored not by a single
pass/fail bit but by a fixed set of content-type checksums, one
CRC-32 per parse event category. The categories come straight from
the token-to-label table in Xpact-core:
Parse_data_types: HASH_TABLE [INTEGER, STRING]
once
create Result.make_from_iterable_tuples (
[Tok_text, "text"], -- text content
[Tok_cdata, "cdata"], -- CDATA text content
[Tok_comment, "comment"], -- comment
[Tok_tag, "tag"], -- tag name (open element)
[Tok_attribute, "attribute"], -- attribute value
[Tok_pi_name, "pi-name"], -- processing instruction name
[Tok_pi_value, "pi-data"] -- processing instruction data
>>)
end
A file only counts as passed when every one of these
seven running checksums, text, cdata, comment, tag, attribute,
pi-name, and pi-data, matches between Xpact-core and eXpat. The
mass-testing driver behind the 132,305-file sweep is file_tree_tests.e:
it walks the tree, parses each file first with Xpact-core, then a
second time by shelling out to the C reference parser, xml_crc_32.c,
and diffs the resulting checksums. Failures are written to a log
for later inspection.
For completeness, the one notable gap in that list is attribute_id,
which isn't checksummed separately yet. I haven't hit a failure
traceable to it so far, but it's on the list to close out rather
than something I'm treating as settled. Separately, there's also a
companion tool that dumps the parsed document back out in an
indented, human-readable form, useful for eyeballing structure
directly rather than only comparing checksums.
If both Xpact
and eXpat fail to parse a file that counts as a successful test,
as they both need to agree on what is well formed.
How I zoom in when something
does fail
The whole-corpus
run only tells me pass or fail per file. When I need to find out why
a file fails, the first step is to re-run just that one file
through an expanded, per-content-type CRC-32 comparison against
eXpat:
finnian@Lillemor:~/Dev/Eiffel/library/Xpact-core$ . scripts/compare_expat_crc_32.sh /usr/share/doc/libxml-parser-perl/examples/canontst.xml
Comparing CRC-32 Xpact and eXpat for canontst.xml
Type: attribute
Program -crc_32: Xpact-core XML tools (Eiffel)
Checksum for attribute: 2657696183
Program: eXpat XML CRC-32 parser (C lang)
Checksum for attribute: 2657696183
Type: cdata
Checksum for cdata: 0 / 0
Type: comment
Checksum for comment: 0 / 0
Type: tag
Checksum for tag: 602060393 / 602060393
Type: text
Checksum for text: 4294833632 / 4294833632
Type: pi-name
Checksum for pi-name: 754283652 / 754283652
Type: pi-data
Checksum for pi-data: 1560826922 / 4294173483 <-- mismatch
That last line is
the signal: attribute, cdata,
comment, tag, text, and pi-name all match, but pi-data diverges. That
immediately narrows the bug to processing-instruction data
handling, without touching the debugger yet.
Once a content
type is flagged, I go one level finer: a step-by-step CRC-32
trace, one running checksum per token, written to two parallel
logs (Xpact-core and eXpat). A small Python script walks both logs
in lockstep and reports the exact line where the checksums first
diverge. That line number points straight at the token, and from
there the EiffelStudio debugger takes over to isolate the exact
routine.
Coarse CRC
compare → per-type breakdown → step-by-step trace → log diff →
debugger. Each stage cuts the search space by an order of
magnitude before I ever set a breakpoint.
Edge cases this process has
surfaced
Running against a
real, messy, unsupervised filesystem rather than a curated
conformance suite keeps turning up cases the conformance suites
don't force:
- Default attribute values
declared in a DTD must be synthesized for elements that omit
that attribute.
- Processing instructions
need explicit, correct handling, not just pass-through.
- UTF-8 and UTF-16
byte-order marks must be detected and stripped correctly.
- UTF-16 encodings need
first-class decoding support, not just UTF-8 fast paths.
- Recursively expanded
entities have to resolve correctly without runaway expansion.
%R%N must be normalized to %N in every code path that
touches line endings, not just the obvious ones.
None of these
show up until you throw a large, unfiltered pile of real files at
the parser. That's the value of the 100K+ run: it's less a
conformance checkbox and more a fuzz corpus that happens to
already exist on disk.
On the performance side
The single most consequential decision came from
noticing something about the data flow rather than from optimising
any routine. Upstream of all actual parsing there was already a
point where incoming file chunks were copied into the parse
buffer. That existing copy turned out to be a much better place to
do the UTF-8 encoding than anywhere inside the parser itself.
Since the bytes were being touched and moved anyway, transcoding
UTF-16 (or any other supported encoding) to UTF-8 at that boundary
costs almost nothing extra, and everything downstream can then
assume it is reading UTF-8 and nothing else. This is a deliberate
departure from the original eXpat design, which carries the source
encoding all the way through tokenisation and instantiates its
scanner logic separately for each encoding, converting to UTF-8
only at token boundaries.
The consequences ran deeper than
expected. With a single internal encoding,
the pluggable encoding machinery in the
scanning path became redundant and many
classes were simply deleted, which greatly
simplified the entire architecture. It
also changed the character of the hot
loops: where the scanner previously had to
step through the buffer with index := advance (index)
to accommodate variable code unit widths,
it can now write index
:= index + 1.
Removing that indirection from the
innermost loops, multiplied across every
byte of every document, produced a
measurable improvement in overall
throughput.
A second gain came from
rethinking what it means to reset a parser
between documents. Originally, resetting
meant calling make
again: a hard reset that rebuilt the
entire parser object graph from scratch,
including buffers, caches and tables, and
left the previous graph behind as work for
the garbage collector. Profiling against
eXpat on small documents showed this fixed
per-parse cost dominating the actual
parsing work. The replacement is a soft
reset that simply restores attributes to
their default values and wipes out the
semantic tables, such as the entity table,
while every buffer, cache and table keeps
its allocated capacity for reuse. On small
files this improved performance, since a
few kilobytes of parsing no longer pays
for the reconstruction of the machinery
around it. The distinction that emerged is
a useful one in general: a reset must
clear semantic state but should preserve
capacity state.
In the attached performance
chart, the UTF-16 shows poor performance
because of it's small size of 12K. About
14 K size is needed to reach parity with
eXpat which has a faster setup time
between parses. In the megabyte ranges,
the Xpact excels.
Next goal: 200,000 files
132,305 is a good waypoint, not
the ceiling. I suspect I've barely
scratched the surface of what's sitting
inside a full Windows 11 install;
I also want to bring
LibreOffice's .odt,
.ods,
and .docx
files into the corpus. These are
zip containers with XML inside, so the
harness needs a small extension: unzip
each file first, then hand the extracted content.xml
(and friends) to the existing test
pipeline. That's a modest addition, not a
redesign. The source files themselves will
be personal documents out of my own Documents
folder, so this batch grows the corpus
with real, organically-messy office
documents rather than more system XML.
If anyone knows of other good
hunting grounds for large,
naturally-occurring XML corpora, on Linux,
Windows, or elsewhere, I'd welcome
pointers. Package manager metadata, IDE
project files, and application config
trees have all been productive so far;
suggestions for other rich veins are very
welcome.
Finnian
Repo: https://github.com/finnianr/Xpact-core
--
SmartDevelopersUseUnderScoresInTheirIdentifiersBecause_it_is_much_easier_to_read
