WeBWorK in the pre-processor

[Rob Beezer]

Mostly for Alex and Oscar, but perhaps of general interest if you have WeBWorK
problems in your project. Next two paragraphs are for everybody.

Organizing and sending WeBWorK problems to a WW server, getting back various
things (especially a static representation in PreTeXt syntax), and then putting
that all back into your source, was one of the first jobs for the pre-processor.
Maybe the very first. Alex and I have learned a lot and put a lot of effort
into making this behave. But we have also greatly expanded the capabilities of
the pre-processor since then and along the way. So the WW bits are long overdue
for a refactor.

So a heads-up for anybody working with WW and a report for those interested.
Authors and publishers (and developers) can safely stop reading here, you should
notice no changes, and extensive testing suggests this is definitely the case.
There is a plan for breaking changes, so look at the end if you are interested.

* With the exception of a little bit of Runestone-specific stuff (2 lines) all
the changes are in pretext-assembly.xsl. No change to the Python, no changes to
actual conversions. Output of just the pre-processor has only intended changes.
Thanks, Claude (and thanks to a careful separation and modularization).

* Line counts remain almost unchanged. So this is mostly rearrangement and
some streamlining.

* Short version: the modifications to your source, with representations from
the server, now happens in something like the sixth pass of the pre-processor
rather than earlier in something like the third pass. And this is more in
keeping with other such modifications (e.g. MyOpenMath, STACK, Runestone).

More details at: https://github.com/PreTeXtBook/pretext/pull/2806

Why now? There are two modernizations I'd like to make, which will mean authors
will need to re-generate their WW representations from the server, or the CLI
will need to do it for them.

1. We use, and still use, a crappy identification string for tracking WW on its
route through all these transformations. A bit cumbersome to compute and very
brittle if you don't author a string to use instead. Like "webwork-563". We
have a much more efficient and robust way of making ids, and we can now recycle
one we already make somewhere between pass 3 and pass 6. The move to later
passes makes good sense, but was motivated by this ID upgrade. So this would be
a breaking change if you have any of these crappy IDs in play.

2. webwork-representations.xml is one monolithic file. Everything we do that
is similar creates multiple files. So I propose one file per WW as it comes
back from the server and resides in your "generated" directory. Transparent
(after a one-time change) and two payofs (at least). We will stop repeatedly
opening the big file just to search for and isolate material for one problem.
And if the server sends back bad XML (we are already trapping this in the
Python), it'll be clear which problem lead to the failure to read.

Red flags, warnings, concerns, questions, approvals, all welcome.

[Rob Beezer]

And I should have mentioned, there is a new undocumented stringparam,
assembly.debug, that does what it says. Right now, it will check consistency of
the two IDs that are built during the pre-processor passes, one made early in
the process and one made later in the process. There needs to be some care
inbetween to make sure they continue to match. So this will be a helpful
diagnostic for future work.

[Rob Beezer]

> There are two modernizations I'd like to make, which will mean authors will need to re-generate their WW representations from the server, or the CLI will need to do it for them.

Proposal now posted at

https://github.com/PreTeXtBook/pretext/pull/2809