ReactJS-based Pretext Web Frontend

[Jason Siefken]

I have been experimenting with creating a new Javascript frontend for Pretext html books. My main goals are

  * Caching - I don't want the page to refresh when navigating between links; it should instead asynchronously load the new page.
  * Better Search - If pages are cached, full-text search could be implemented client-side, so no fancy server would be needed to host textbooks. This would also be better than the current google searches, which I find never turn up the information I want...

So far I have a proof of concept working: https://siefkenj.github.io/pretext-react/author-bio-TWJ.html
with source code at: https://github.com/siefkenj/pretext-react

I haven't taken the time to polish the UI at all, but caching and (non-animated) knowls work. Search is not yet implemented. (Also I only reworked the TOC and content area, so the pref/next buttons, etc. in the header won't work right).

Please let me know what you think! Also, I didn't make documentation for the source code that explains how it works (since it's just a prototype), but I'm happy to answer any questions.

    Jason

[Rob Beezer]

Dear Jason,

That *is* snappy!

Having native search (rather than a Google service) would be a huge improvement.
I've also never been too happy with what we have.

Thanks for your work on this.

Rob

> --
> You received this message because you are subscribed to the Google Groups
> "PreTeXt development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to pretext-dev...@googlegroups.com
> <mailto:pretext-dev...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pretext-dev/bd2fbd95-0d4a-5966-196f-c623f05697c9%40gmail.com
> <https://groups.google.com/d/msgid/pretext-dev/bd2fbd95-0d4a-5966-196f-c623f05697c9%40gmail.com?utm_medium=email&utm_source=footer>.

[Steven Clontz]

Oh, this is very nice. This kind of thing has been in the back of my head as well (although I found I prefer Svelte to React).

I'd been considering using hash-based routing to build a single index.html, but this would be a much better drop-in upgrade to preserve existing book URLs, and building (essentially identical) separate HTML files for each "page" would be significantly better for SEO support compared with hash-based routing. It's just a longer build process to make dozens of HTML files rather than just one I'd suppose.

I looked at your repo briefly. Am I right in surmising that your app takes an existing HTML build from PreTeXt, and uses Node to extract data from those HTML files to incorporate into the React app? I'd hesitate to add Node as a requirement to the PreTeXt build process (I lobby to get as much of it reliant on Python and web services as possible). However, we could (a) do the extraction of necessary data using Python into a JSON blob or something to be consumed by a static bundle.js build of the React app, or (b) rewrite pretext-html.xsl to inject the data of the full book on every HTML page to be picked up by a static bundle.js build of the React app. Short term, (a) is probably quicker.

Long-term... I'd really like a React/Svelte/other frontend framework app that ingests book.ptx XML directly and builds it into a book within the browser. This would be wonderful for projects like David Farmer's CAT that want to manipulate PreTeXt XML from within the browser and reflect those changes immediately. And it'd create the best way to get new authors writing PreTeXT: we can make sample article/book editable directly within the browser without any extra requirements beyond what's needed to read the book in the first place.

[Jason Siefken]

I'd been considering using hash-based routing to build a single index.html, but this would be a much better drop-in upgrade to preserve existing book URLs, and building (essentially identical) separate HTML files for each "page" would be significantly better for SEO support compared with hash-based routing. It's just a longer build process to make dozens of HTML files rather than just one I'd suppose.

I was thinking about a single page render too, but SEO might be important. I also experimented with single pages on my phone and it's pretty slow, so being broken into multiple pages helps mobile.

I looked at your repo briefly. Am I right in surmising that your app takes an existing HTML build from PreTeXt, and uses Node to extract data from those HTML files to incorporate into the React app?

Not quite. Not is only used to build the Javascript. Once the Javascript is loaded, it extracts the parts needed directly from the HTML that PreTeXt has output. Then the react components are created dynamically based on that extracted data. So, there is no node dependency for PreTeXt authors since they'd be using pre-built Javascript.

Long-term... I'd really like a React/Svelte/other frontend framework app that ingests book.ptx XML directly and builds it into a book within the browser. This would be wonderful for projects like David Farmer's CAT that want to manipulate PreTeXt XML from within the browser and reflect those changes immediately. And it'd create the best way to get new authors writing PreTeXT: we can make sample article/book editable directly within the browser without any extra requirements beyond what's needed to read the book in the first place.

This could be done! There are two challenges that jump out to me: 1) crossreferences. To get crossreferences to work, you really need to process the full content of every file, possibly multiple times. It's nice for that to be done once, statically, and have it be done with. 2) figures. Rendering tikz/latex in the browser is a continual struggle. The Tikzjax project is really cool: https://github.com/kisonecat/tikzjax and https://tikzjax-demo.glitch.me/  I tried to get my own build of it working, but I didn't succeed...

I learned last night that there is an `XSLTProcessor` object built into most Javascript runtimes. I'm not sure how many of the features of the current PreTeXt scripts it supports, but it's worth an experiment!

   Jason

[Steven Clontz]

Thanks for the correction - I'm still not 100% sure I'm following how it's implemented (I've starred the repo to investigate more when I can), but at the end of the day if this is just another bit of javascript folks can opt to include in the headers of their built HTML pages, this is pretty slick.

I'm currently writing new version of the Svelte app for https://checkit.clontz.org/ that will be a brave first all-client-side step at taking Mustache templates along with JSON data, converting them to PreTeXt-esque XML, which is then read using XSLTProcessor to populate a Svelte component to display in the browser. Once I finally get to that, I will share my successes/failures here (keeping in mind that the XML will be limited to something isomorphic to a very strict subset of PreTeXt, but it's a start).

I've been needling Jim (actually I haven't but I should) for a couple years now to get tikzjax packaged up for NPM/yarn installation. Human-authored diagrams that Just Work in both web and LaTeX contexts is an incredibly tough problem (spoken as the guy being yelled at for dragging his feet implementing diagrams in his random exercise authoring platform). I mentioned elsewhere that I wish someone would offer TikZ-as-a-service so I can use <img src="https://taas.example.org/?TIKZ_IN_BASE64"> or similar to view TikZ in my browser, but that's certainly a commitment.

[Rob Beezer]

On 1/24/22 12:00 PM, Jason Siefken wrote:
> I also experimented with single pages on my phone and it's pretty slow

The original motivation for knowls was to reduce MathJax processing on mobile.
That was 8 years ago, and seems less critical now.

> So, there is no node dependency for PreTeXt authors

Good! ;-)

> To get crossreferences to work, you really need to process the
> full content of every file, possibly multiple times. It's nice for that to be
> done once, statically

And remember that we can use XSL to output any kind of index/metadata/summary in
whatever form is best (JSON?).

> The Tikzjax project is really cool

Jim Fowler is known to us...

> I learned last night that there is an `XSLTProcessor` object built into most
> Javascript runtimes. I'm not sure how many of the features of the current
> PreTeXt scripts it supports, but it's worth an experiment!

When I first started down this path, I was using browsers to do the processing.
Chunking (and cross-references) broke that, iirc.

If `XSLTProcessor` builds on `libxslt` then differences are likely to be minor,
compared to `xsltproc` and Python's `lxml`.

https://en.wikipedia.org/wiki/Libxslt

Rob

[Steven Clontz]

For reference: https://developer.mozilla.org/en-US/docs/Web/API/XSLTProcessor (perhaps note the scary yellow warnings...)

But I realize I misspoke. My intention for CheckIt is not to use XSLT to transform PreTeXt-like XML into some other format. Rather, I plan to parse XML sources into https://developer.mozilla.org/en-US/docs/Web/API/XMLDocument objects, and then use those to serialize Svelte components used in my application. I'd imagine a similar approach would be more natural in a React context as well.

[David Farmer]

> The original motivation for knowls was to reduce MathJax processing on
> mobile. That was 8 years ago, and seems less critical now.

<historylesson>
Knowls were created for the LMFDB (lmfdb.org), and I recognized their
usefulness for something like Rob's First Course in Linear Algebra.
I directly converted Rob's very well-structured LaTeX to HTML,
creating something that was a small step toward what we have now.
This was just in time for the 2012 JMM (yes, more than 10 years ago).

The problem was that some sections would not render on an iPad
(current model then was iPad 2). Hiding all the proofs and examples
in knowls solved that problem. It turned out that people liked
it that way, which is why that is now the default.
</historylesson>

[David Farmer]

Looking at the dates on files, maybe I meant 2014 JMM?

> --
> You received this message because you are subscribed to the Google Groups
> "PreTeXt development" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to pretext-dev...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pretext-dev/alpine.LRH.2.21.2201241556050.14665%40li375-150.members.linode.com.
>

[Rob Beezer]

Thanks, Jason, for being a big part of today's Drop-In discussion.

Here's a demonstration of the incremental improvement in the HTML for ToC:

https://pretextbook.org/beta/css-refactor/derivatives.html


1. It'd be easy to add some data-* attributes to all this. Like division name
(your "level"), "codenumber", etc.

2. Titles are too complicated to risk putting into an attribute (math,
emphasis, quote marks, etc). It'd be easier for me to not put all this into
JSON, since I don't have a great tool to do the escaping (though I've done it
for Jupyter notebooks).


If we can iterate to better HTML, could we try that first? Can you suggest
improvements and additions to the demo above that would work well for your project?

Notice that each page's ToC is just a bit different. The ToC entry for the
(PreTeXt) division represented by the page has an "active" class that we have
used to highlight that entry in different ways. Of course, we can preserve
something similar if it is useful.

Thanks,
Rob

[Jason Siefken]

https://pretextbook.org/beta/css-refactor/derivatives.html

Lots of sub-subsections. That's a good test :-)

1.  It'd be easy to add some data-* attributes to all this.  Like division name (your "level"), "codenumber", etc.

2.  Titles are too complicated to risk putting into an attribute (math, emphasis, quote marks, etc).  It'd be easier for me to not put all this into JSON, since I don't have a great tool to do the escaping (though I've done it for Jupyter notebooks).

It sounds like maybe it'd be easiest to just store this in XML instead, then? Though it shouldn't be too hard to escape JSON strings. According to this post https://stackoverflow.com/questions/3020094/how-should-i-escape-strings-in-json the only things you need to escape are \ and ". Of course, literal newlines should also be replaced with \n...

I agree about attributes. There's no point in forcing non-attribute data into an attribute. Something like the following schema could be used:

<ul
        division="...">
    <li>
        <item
            codenumber="..."
            identifier="..."
            url="..."
        >The title</item>
        <!-- optional -->
        <ul division="...">...children...</ul>
    </li>
</ul>

or fully embrace XML and do

<ul division="...">
    <li>
        <title>...</title>
        <id>...</id>
        <url>...</url>
        <ul division="...">....
    </li>
</ul>

Basically, this would mimic a JSON format but in XML. It's close to but not equal to the html output. For instance, there's no <a /> tag. The thing that makes me nervous about reading the literal HTML is that the Javascript becomes fragile to changes in the HTML. Somebody might decide that they want to include small images or icons in the TOC and so modify the HTML output. If the Javascript was not ready for that, it would fail...

I guess, another way to state it is that I'm proposing a "data-only" API, whereby some information is embedded in the document for use by Javascript but that information never affects the rendering HTML of the code (unless Javascript decides to dynamically change the code).

If we can iterate to better HTML, could we try that first?  Can you suggest improvements and additions to the demo above that would work well for your project?

I notice there are empty `<ul></ul>` tags in (every?) <li>...</li>? The main thing that missing is the division. I think it makes most sense to put the division on the `ul`. This could
be done via a class `division-chapter` `division-subsection`, etc., or a `data-division="..."` attribute (or a `ptd-TOC-division-...` class to follow your existing naming scheme, though I think capital letters in class names are frowned upon).

The other thing that makes me nervous is the `data-scroll` attribute. If we are getting rid of the scroll, that should be gone too, but I use that information as a "division id".  Maybe it should be renamed/duplicated as `data-division-id`?

Notice that each page's ToC is just a bit different.  The ToC entry for the (PreTeXt) division represented  by the page has an "active" class that we have used to highlight that entry in different ways.  Of course, we can preserve something similar if it is useful.

In my current code, I process the TOC and then do a lookup based on the URL to determine what `division-id` is currently active. Then I add the `active` class to that element. It certainly wouldn't hurt to have `active-division-id` metadata embedded in the page somewhere.

    Jason

[David Farmer]

If the page has an attribute which identifies it, there is no
need to add the "active" class to the corresponding TOC entry
(assuming the TOC entry has the corresponding attribute).

The "active" class is used to scroll the TOC. But that is
something we want to do better.

> --
> You received this message because you are subscribed to the Google Groups
> "PreTeXt development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pretext-dev...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pretext-dev/447f4ce6-26e7-4d9b-196b-24dc996b2410%40ups.edu.
>

[Rob Beezer]

Thanks, Jason - that's all very helpful. Most of your nervousness is no longer
necessary or can be changed. What I sent you was serving slightly different
purposes.

XML would be great. But it'll need to be a different file, I think, since with
elements as suggested, an HTML page will not validate. (Using data-* attributes
is the way around that.) As David suggests, some sort of ID linkage should
allow us to accomplish that correspondence.

It'll still have *some* HTML, since the title itself is going to be authored in
PreTeXt and will need to be realized as HTML.

I've got a good starting place and will see what I can produce, perhaps over the
weekend. We can iterate from there.

Thanks,
Rob

> --
> You received this message because you are subscribed to the Google Groups
> "PreTeXt development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to pretext-dev...@googlegroups.com

> <mailto:pretext-dev...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pretext-dev/6fe4ed89-7c7a-34d1-90c6-54ac63f05e68%40gmail.com
> <https://groups.google.com/d/msgid/pretext-dev/6fe4ed89-7c7a-34d1-90c6-54ac63f05e68%40gmail.com?utm_medium=email&utm_source=footer>.

[Rob Beezer]

Dear Jason,

A rough-cut you might be able to work with - I gotta come back to this soon to
flesh out a few things.

https://pretextbook.org/beta/2022-01-29-react

1. Complete sample article, but you want to look at "toc-manifest.xml" which is
the only change.

2. Total XML style. Nested "division" elements. Each has several attributes.
- always followed by one "title" element which is PTX HTML
- then a list of children "division", possibly none
- child list could be inside a "children" element (?)

Nits:

* Missing front matter and back matter. Always a fiddle, but not insurmountable.

* One file, not per-page with old "active" class. We could make one file per
page? But that seems overkill.

* I plan to add a "debug.react" switch that will allow you and I, and anybody
else, experiment with a "drop in" of the new approach at any time.

* I can use a diff of what you've posted to see what JS to leave out and what
to add in. So nobody will need to fiddle. We'd just want your latest JS/React
in a (temporarily) stable location (github.io?). Send hints if you have time.

Rob

[Rob Beezer]

Dear Jason,

How about I evolve the HTML and the XML file, while you evolve the React/JS in a
stable location (previous message).

We concentrate on having full functionality with the left sidebar. Test Sage
Cells, knowls, JS interactives, WeBWorK, Runestone hosting, mobile experience -
everything and anything that might break due to aggressive caching.

We let others become familiar with the React code, and work on some documentation.

Save search for later, and anything else that might be a distinct improvement.

Questions, comments?

Thanks for your work integrating this into the status quo.

Rob

[Jeremy Sylvestre]

On Sat, Jan 29, 2022 at 1:25 PM Rob Beezer <bee...@ups.edu> wrote:

1.  Complete sample article, but you want to look at "toc-manifest.xml" which is
the only change.

How about a complete "doc-manifest.xml" down to the block level, including everything that can have a number or a title?

Then (eventually) you could have features like, for example, you right-click on  a TOC entry and it pops up a clickable mini-map of that division.

[Rob Beezer]

Thanks, Jeremy. Yes! Already gone there in my head. ;-)

Simple matter to recurse further and collect relevant items. Probably want to
then include something like the translations of words like "Example" and
"Theorem" for display purposes.

"Save search for later, and anything else that might be a distinct improvement."

But keep after us, I'm likely to forget.

Thanks,
Rob

> --
> You received this message because you are subscribed to the Google Groups
> "PreTeXt development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to pretext-dev...@googlegroups.com
> <mailto:pretext-dev...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pretext-dev/CALMYOnevi7hKQexiVrp_Yju1cDp-UD31Mmb1yDnnkAFGMFoejQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/pretext-dev/CALMYOnevi7hKQexiVrp_Yju1cDp-UD31Mmb1yDnnkAFGMFoejQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[David Farmer]

One issue is the overall structure of the page: how the TOC, header,
footer, etc, are arranged in the HTML source.

As Tom S was working on his new layout, it was determined that some
changes were needed. My next task is to determine that the current,
and the new, layout is possible with the proposed new HTML structure.
Then I will report back for others to check that the structure can
support what they might want to do.

The TOC will be the same, no matter what surrounds it. But I thought
others should know that the overall arrangement is in flux.

David

> email to pretext-dev...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pretext-dev/9d4db8a4-c1a6-d141-6304-1a1bcda44e68%40ups.edu.
>

[Jason Siefken]

On 1/29/22 3:25 PM, Rob Beezer wrote:

Dear Jason,

A rough-cut you might be able to work with - I gotta come back to this soon to flesh out a few things.

https://pretextbook.org/beta/2022-01-29-react

This is looking good! I notice the highest level is section? Is that right? I though there'd be a `part` or `chapter` there...

1.  Complete sample article, but you want to look at "toc-manifest.xml" which is the only change.

2.  Total XML style.  Nested "division" elements.  Each has several attributes.
    - always followed by one "title" element which is PTX HTML
    - then a list of children "division", possibly none
    - child list could be inside a "children" element (?)

Nits:

*  Missing front matter and back matter.  Always a fiddle, but not insurmountable.

Anything that should be in the TOC should be in the manifest...

*  One file, not per-page with old "active" class.  We could make one file per page?  But that seems overkill.

*  I plan to add a "debug.react" switch that will allow you and I, and anybody else, experiment with a "drop in" of the new approach at any time.

A debug flag would be very nice.

*  I can use a diff of what you've posted to see what JS to leave out and what to add in.  So nobody will need to fiddle.  We'd just want your latest JS/React in a (temporarily) stable location (github.io?).  Send hints if you have time.

For the React components (at least for development) all the <script> tags serving JS from pretext.org should be removed and replaced with       

        <script
            defer
            src="./static/js/bundle.js"
        ></script>
        <script
            defer
            src="./static/js/main.js"
        ></script>
        <link
            href="./static/css/main.css"
            rel="stylesheet"
        />

Note that the last item is a `css` file.

   Jason

[Rob Beezer]

On 1/29/22 5:10 PM, Jason Siefken wrote:
> This is looking good! I notice the highest level is section? Is that right? I
> though there'd be a `part` or `chapter` there...

Sample ARTICLE - just sections. ;-) And torture tests with edge cases.

I'm going to bifurcate (trifurcate?) the sample book into three versions that
will be better for testing parts and chapters.

> Anything that should be in the TOC should be in the manifest...

Will be, just ran out of time.

> For the React components (at least for development) all the <script> tags
> serving JS from pretext.org should be removed and replaced with

Thanks for the hints.

Would you be willing to put experimental work *also* at
https://siefkenj.github.io/pretext-react or similar?

We could have two flags: one would make relative/local href for development,
the other would let other PreTeXt developers build experimental versions with
their own projects for testing (not production).

I hope to have flags, complete ToC, and better sample books soon.

Rob

[Rob Beezer]

Pushed some changes:

1. Now called "doc-manifest.xml" (nee toc-manifest.xml) in response to Jeremy's
suggestion.

2. Temporary "stringparam" switch named "debug.react.global". Default is "no".
Using "yes" will get local references to React JS, drop principal PreTeXt JS
(maybe not enough?) and create file in (1).

3. Output will only be useful for Jason right now, but anybody can use the
switch to see the manifest file.

@Jason - I could post samples. But you can build your own. Holler if you need
guidance on using (2) - I'll be back in a few hours or others can help.

I'll try to spruce up the sample book later, as well.

Rob

[Jason Siefken]

2.  Temporary "stringparam" switch named "debug.react.global".  Default is "no".  Using "yes" will get local references to React JS, drop principal PreTeXt JS (maybe not enough?) and create file in (1).

Thanks!

@Jason - I could post samples.  But you can build your own.  Holler if you need guidance on using (2) - I'll be back in a few hours or others can help.

I have it almost working, but not quite. I created a `out/` subdirectory in `examples/sample-article` and then ran

xsltproc --stringparam debug.react.local yes --stringparam prefix "./" -xinclude ../../../xsl/pretext-html.xsl ../sample-article.xml

This produced several errors like:

PTX:ERROR:   the Asymptote diagram produced in "images/asymptote-lever.html" needs to be available relative to the primary source file, or if available it is perhaps ill-formed and its width cannot be determined (which you might report as a bug).  We might be able to proceed as if the diagram is square, but results can be unpredictable.


Should I be running the command differently? I also had to manually copy the contents of `media/` to the `out/` directory.

  Jason

[Jason Siefken]

It appears knowl.js is still being loaded. Can you remove that?

[Rob Beezer]

On 1/30/22 1:30 PM, Jason Siefken wrote:
> It appears knowl.js is still being loaded. Can you remove that?

Wasn't sure about that one. Will be gone now.

[Rob Beezer]

On 1/30/22 12:53 PM, Jason Siefken wrote:
> I have it almost working, but not quite. I created a `out/` subdirectory in
> `examples/sample-article` and then ran
>
> xsltproc --stringparam debug.react.local yes --stringparam prefix "./" -xinclude
> ../../../xsl/pretext-html.xsl ../sample-article.xml

"prefix" is not a "param" (a variable instead). You should not need it when you
use debug.react.local.

We have some better front-ends now, so with xsltproc you need to do some
things manually.

Copy "media" to your "out/" AND rename it to "external". Asymptote error should
go away.

Rob

>
> This produced several errors like:
>
> PTX:ERROR:   the Asymptote diagram produced in "images/asymptote-lever.html"
> needs to be available relative to the primary source file, or if available it is
> perhaps ill-formed and its width cannot be determined (which you might report as
> a bug). We might be able to proceed as if the diagram is square, but results can
> be unpredictable.
>
>
> Should I be running the command differently? I also had to manually copy the
> contents of `media/` to the `out/` directory.
>
>   Jason
>

> --
> You received this message because you are subscribed to the Google Groups
> "PreTeXt development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to pretext-dev...@googlegroups.com
> <mailto:pretext-dev...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pretext-dev/bb144683-a71a-5d36-03c4-362d767c77fb%40gmail.com
> <https://groups.google.com/d/msgid/pretext-dev/bb144683-a71a-5d36-03c4-362d767c77fb%40gmail.com?utm_medium=email&utm_source=footer>.

[Rob Beezer]

I've reogranized the "sample book" in the repository to make it clear thare are
three distinct versions:

1. with decorative parts
2. with structural parts
3. with no parts at all

@Jason - any one of these (and maybe all three) would be good for testing.

Rob

[Jason Siefken]

Okay. I have an updated version pushed to GitHub.

I do need another modification, though. I need the line

MathJax.startup.defaultReady()

removed from the MathJax startup code. It turns out there is an error when it comes to \labels in equations. When the document loads, MathJax typesets right away. Then the page is replaced with the React code and MathJax re-typesets the new page. If it sees the same \label again, it complains that the label is not unique. I tried, but failed, to have MathJax forget the labels when the page is reset, and then I figured it was probably better not to typeset anything until the actual page is ready. So, React now calls the MathJax startup when the page is ready.

Here are some other things that are fixable with more complicated logic, but maybe the solution is to rethink the HTML structure/CSS instead. Here are a bunch of questions/comments.

1. Knowl placement and classes. The CSS appears to be a bit of a mess. I'm not quite sure what classes/attributes knowls should have. This should be standardized.

The placement algorithm I currently use is to look for the knowl's parent, and place the knowl immediately after the parent. This is the right thing to do if the knowl is in a <p>, but it appears that if the knowl is in an <article> and *not* within a <p> then the knowl should be placed in that article? (There are cases where the <article> contains just the heading which is wrapped in a link.). Of course, the logic can be special-cased, but it'd be so much nicer not to. (I guess knowl placement in a table is also really complicated...)

If these do needed to be implemented client side, it would help to have a list of all the edge cases and the expected HTML structure.

2. In the TOC, it's very hard to know what should show a codeNumber and what shouldn't. I keep track of the level (i.e. depth) when I parse doc-manifest.xml, but there's not value that give the correct behavior. If I say level < 2, then appendices aren't numbered. If I use level < 3, then subsections are labeled. It would be nice to avoid a special case here.

3. A question: are all images included in figures? I.e., is the logic for the placement of the "zoom box" sophisticated? It would be very easy to do with static positioning as a modal, but then a zoomed in figure wouldn't scroll with the page. You'd have to click it again to make it "zoom out".

4. The placement of born-hidden footnote knowls is inconvenient...they're at the end of the document, so when they're naively "revealed", they won't show up in the currently visible page. What identifier do these have to indicate they should be moved before revealing?

5. MathJax race conditions...I have a feeling that we're going to encounter many, many MathJax race conditions. Basically, the page has two masters who are fighting...The easiest solution would be to take over and use MathJax static rendering. This would have the benefit that there would never be the "MathJax blink". However, issues like labels in equations are going to be really hard to solve that way. Perhaps Pretext should take over equation numbering, and then we can retypeset equations all we want without causing errors...

6. For some reason the \cancelto is not working in the React version. section-7.html#subsection-10   I don't know why...

7. Section 9 (graphics.html) causes a lot of trouble. I am not sure what it is doing or what scripts are trying to run...but I get a bunch of failed network requests which send the program into a loop somehow...Is another script trying to reload the page? (This appears to crash the page when running React in dev mode, but when it's compiled to production mode, it seems to work okay...)

8. Interesting sideffect of caching...if you start playing a video/sound and change pages, it keeps playing! That'll be fun to figure out :-).

9. Can we put the hash for the start of a section on the section-level blocks in the TOC? That way, clicking on a section would scroll to the top of the page without extra effort.

Let me know your thoughts and where I should focus my attention!

   Jason

[David Farmer]

> 1. Knowl placement and classes. The CSS appears to be a bit of a mess. I'm not quite sure what
> classes/attributes knowls should have. This should be standardized.
>
> The placement algorithm I currently use is to look for the knowl's parent, and place the knowl immediately after
> the parent. This is the right thing to do if the knowl is in a <p>, but it appears that if the knowl is in an
> <article> and *not* within a <p> then the knowl should be placed in that article? (There are cases where the
> <article> contains just the heading which is wrapped in a link.). Of course, the logic can be special-cased, but
> it'd be so much nicer not to. (I guess knowl placement in a table is also really complicated...)

Originally knowls were only for cross-references, and a reasonable
approximations was "after the nearest enclosing block, with list elements
and table entries as special cases". Then we started using knowls to
hide content like proofs and examples, so those were additional special
cases. See the 30 lines of code starting here:

https://github.com/PreTeXtBook/JS_lib/blob/main/knowl.js#L116

And also there can be knowls in math, see after this p:

https://pretextbook.org/examples/sample-article/html/section-7.html#p-299

which required a MathJax extension:

https://github.com/PreTeXtBook/JS_lib/blob/main/mathjaxknowl3.js

There is an added complication which may not actually be relevant
to any of the actual books written in PreTeXt: the knowl could have
a parameter that says where it should open (for example, a note in
the margin could open inside the body), or a knowl could be set to
replace existing content. One can imagine those use cases, I don't
think anyone is currently using either of those.

Both the Javacript and the CSS for knowls grew organically with many
changes over the years, so rethinking that would be good.

> 3. A question: are all images included in figures? I.e., is the logic for the placement of the "zoom box"
> sophisticated? It would be very easy to do with static positioning as a modal, but then a zoomed in figure
> wouldn't scroll with the page. You'd have to click it again to make it "zoom out".

Images do not need to be in figures. I think these days they are
always in a div.image-box (even when they are in a figure or side-by-side)

I suspect the logic to magnify an image is more complicated than
necessary, because it is trying to be backward-compatible with HTML
prior to enforcing the .image-box :

https://github.com/PreTeXtBook/JS_core/blob/main/pretext_add_on.js#L219

> 4. The placement of born-hidden footnote knowls is inconvenient...they're at the end of the document, so when
> they're naively "revealed", they won't show up in the currently visible page. What identifier do these have to
> indicate they should be moved before revealing?

I don't think that is specific to footnotes. Any knowl, born hidden
or not, uses its anchor to decide where its content is placed.
When you click, the content is fetched (either remotely or from
anywhere within the page) and then shown. It just so happens that
a recent change moved the hidden footnote source to the bottom
of the page (after I assured Rob that the knowls code did not care
where the hidden source was located).

> 5. MathJax race conditions...I have a feeling that we're going to encounter many, many MathJax race conditions.
> Basically, the page has two masters who are fighting...The easiest solution would be to take over and use
> MathJax static rendering. This would have the benefit that there would never be the "MathJax blink". However,
> issues like labels in equations are going to be really hard to solve that way. Perhaps Pretext should take over
> equation numbering, and then we can retypeset equations all we want without causing errors...

I think that is already the case (or it should be): the XSL determines
the equation numbering and inserts \tag{...} .

Should be a way to tell MathJax to not think about numbering?

> 6. For some reason the \cancelto is not working in the React version. section-7.html#subsection-10   I don't
> know why...

Maybe needs a MathJax extension?

David

[Rob Beezer]

On 1/30/22 9:00 PM, Jason Siefken wrote:
> I do need another modification, though. I need the line
>
> MathJax.startup.defaultReady()
>
> removed from the MathJax startup code.

Done. More in a bit about the rest.

[Rob Beezer]

Dear Jason,

David has you covered on lots of this. (Thanks, David.)


1. Section 9 graphics

The only dynamic figures I can think of:

(a) New 3D "sageplot" is in an iframe and loads threejs libraries from a CDN (I
believe). Figure 9.19.

(b) Asymptote 3D use WebGL, and therefore not in an iframe.

(c) unpkg.com is a trial use of a CDN that sometimes is unresponsive, it fetches
"prism" for syntax highlighting. Only. I should probably switch that one away.


-stringparam debug.chunk 2 will make subsections into pages, which should help
isolate the *type* of graphics.


2. Labels

Like David said, I believe we have taken over numbering from MathJax already,
and these may be vestigal. First thing I will investigate just now.


3. Codenumbers and Levels

Some items never have a number, like a "preface". Publishers can squelch
numbers at some depth, like effectively saying "Subsections and finer are not
numbered". These should have number="" in doc-manifest.xml.

"frontmatter" and "backmatter" are really containers in PTX source. There is no
"mainmatter". In hindsight, there probably should be 3 containers or zero. :-(
I am treating "frontmatter" and "backmatter" as if they should be entries in
the ToC. Debatable, but it means hierarchy in doc-manifest.xml is not uniform.
I suspect this is the source of your non-uniformity.

In existing HTML, I believe we have the "codemumber" class so a style can hide
numbers.

I can:

(a) not add a level/nesting for "frontmatter" and "backmatter" - they'd be
children of the root. Peers of "part" (if employed) or "chapter".

(b) As-is, but I write the level into each division's attributes.

(c) all of the above.


4. "Can we put the hash for the start of a section on the section-level blocks
in the TOC?" Not sure what you mean by "hash". And do you literally mean just
"section"? PTX, not HTML? I say "division" to mean *all* PTX elements that
define (traditional) document structure, leaving "section" for the one between
"chapter" and "subsection". ;-)


5. Unsure about "cancelto" - it can wait.


6. MathJax - we are in regular contact with the MJ devs. And/or the Google
Group "mathjax-users" is a pretty good support forum, you can mention you are
working on PreteXt. At a minimum, David and I read that one regularly.
Possible there is a good cure for avoiding race conditions? Cannot be the first
time React and MathJax have fought over a page?


7. Like David said - I just moved footnote content to the bottom of a page.
Solved a bug. But more important, a bug that was going to recur. It is a
little tricky to "hold" the footnote content from the middle of a paragraph (and
other oddball places we were missing and were unaware of) and "place" it outside
the paragraph.

8. Images and Zoom

Section 26 is chock full of (almost) every scenario, it'd be a good place to
test this sort of thing.


Rob

[Rob Beezer]

> It turns out there is an error when it comes to \labels in equations.

We number equations ourselves. And then place them in output for MathJax to see
via \tag{}. Good.

But if an equation has \label{foo}, then in all the new/extra structure that
MathJax generates, that equation gets an HTML id from MathJax with value
"mjx-eqn-foo". So consider a 10-equation display, and an author sets equation 6
to be numbered. Then we can make URL *that points to exactly that location*
within the display.

All is not lost - we can easily have a cross-reference to a numbered equation
point to (the top of) the entire 10-equation display. Not without precedent, a
knowl for such a cross-reference actually first displays the whole 10-equation
display as the knowl content, presuming context is important/necessary. Then it
is only that knowl's "in context" link that is so targeted to equation 6. We
have to do something similar for braille since "mjx-eqn-foo" does not survive
the conversion to Nemeth braille.

Let's assume this can be easily fixed (the change is very isolated), but only
after we are sure that we understand interactions (race conditions) with React
and MathJax. Likely we could get Davide C and Peter K to join us for Drop-In
some Friday (Peter is a sometime regular).

Rob

[David Farmer]

> But if an equation has \label{foo},

So this is yet another example of bad markup which should be banned
from PreTeXt?

[Rob Beezer]

On 1/31/22 9:54 AM, David Farmer wrote:
> So this is yet another example of bad markup which should be banned
> from PreTeXt?

Well, not exactly. pretext-html.xsl is creating the \label{} that are giving
Jason trouble.

But I guess if an author included such a thing, it would cause the same trouble.
A clever author *could* exploit the generated id, but we would turn off that
construction if the (internal) need went away.

Short answer: an occurence of "\label{" in math elements could be reported as a
bad thing by the pretext-validation-plus.xsl stylesheet.

[David Farmer]

> Well, not exactly. pretext-html.xsl is creating the \label{} that are
> giving Jason trouble.

Wow, I never noticed that. Seems easy enough to omit from the HTML.

[Rob Beezer]

On 1/31/22 10:20 AM, David Farmer wrote:
> Wow, I never noticed that.  Seems easy enough to omit from the HTML.

As Jason is reminding us, we've built up a lot of machinery.

There was a time early on where I carefully studied the MJ documentation and
made much greater use of this device. Past few years I've been weaning us off
of all that. This could be the death knell.

[Jason Siefken]

2.  Labels

Like David said, I believe we have taken over numbering from MathJax already, and these may be vestigal.  First thing I will investigate just now.

I see that is done :-). I am looking at `interesting-corollary.html` at "Second Version of FTC". I see there is a `\tag{..}` and a `\label{..}`. Is there a different mathjax command that just lets you inject HTML or surround the equation with a div/span with specified attributes? If so, we could do that and inject the ID ourselves instead of using a \label. Then MathJax would never complain.

3.  Codenumbers and Levels

Some items never have a number, like a "preface".  Publishers can squelch numbers at some depth, like effectively saying "Subsections and finer are not numbered".  These should have  number=""  in doc-manifest.xml.

"frontmatter" and "backmatter" are really containers in PTX source.  There is no "mainmatter".  In hindsight, there probably should be 3 containers or zero.  :-(   I am treating "frontmatter" and "backmatter" as if they should be entries in the ToC.  Debatable, but it means hierarchy in doc-manifest.xml is not uniform.  I suspect this is the source of your non-uniformity.

In existing HTML, I believe we have the "codemumber" class so a style can hide numbers.

I can:

(a) not add a level/nesting for "frontmatter" and "backmatter" - they'd be children of the root.  Peers of "part" (if employed) or "chapter".

(b) As-is, but I write the level into each division's attributes.

(c) all of the above.

I'd say the answer depends on what you want to be collapsible. If you can collapse the frontmatter/backmatter all together, then it should stay in a group. I normally think of "Appendices" as a chapter and "Appendix A", "Appendix B", etc. as sections, for what it's worth...

4.  "Can we put the hash for the start of a section on the section-level blocks in the TOC?"  Not sure what you mean by "hash".  And do you literally mean just "section"?  PTX, not HTML?  I say "division" to mean *all* PTX elements that define (traditional) document structure, leaving "section" for the one between "chapter" and "subsection".  ;-)

What I mean is this: Currently chapter 4 links to "interesting-corollary.html". I would like it to link to "interesting-corollary.html#interesting-corollary" (similarly to how the 4.1 link is to "interesting-corollary.html#subsection-second-FTC"

7.  Like David said - I just moved footnote content to the bottom of a page. Solved a bug.  But more important, a bug that was going to recur.  It is a little tricky to "hold" the footnote content from the middle of a paragraph (and other oddball places we were missing and were unaware of) and "place" it outside the paragraph.

Okay. The placement can all be done client side, but I could really use an exhaustive flowchart about how the placement should go. That way I don't have to hack the design for every new case I come across.

[Rob Beezer]

On 1/31/22 2:11 PM, Jason Siefken wrote:
> I see that is done :-). I am looking at `interesting-corollary.html` at "Second
> Version of FTC". I see there is a `\tag{..}` and a `\label{..}`.

You see the div.displaymath wrapping the LaTeX?

> Is there a
> different mathjax command that just lets you inject HTML or surround the
> equation with a div/span with specified attributes? If so, we could do that and
> inject the ID ourselves instead of using a \label. Then MathJax would never
> complain.

XSL can easily put an #id onto the div just mentioned, in lieu of the \label{}.
We just give up targeted "in context" cross-references from knowls into
(large) displays.

> I'd say the answer depends on what you want to be collapsible. If you can
> collapse the frontmatter/backmatter all together, then it should stay in a
> group. I normally think of "Appendices" as a chapter and "Appendix A", "Appendix
> B", etc. as sections, for what it's worth...

Technically, appendices are not even back matter. LaTeX has this wrong. :-(

We treat them as peers of chapters in a book, and peers of sections in an
article. For what it is worth... ;-) We don't have an "Appendices" grouping
in the HTML, but I think the PDF may have a decorative page just prior (maybe
just when there are parts?). I forget.

I think it would be good to collapse "frontmatter". Less certain about
"backmatter". Might even be easier that way (for me!).

> What I mean is this: Currently chapter 4 links to "interesting-corollary.html".
> I would like it to link to "interesting-corollary.html#interesting-corollary"
> (similarly to how the 4.1 link is to
> "interesting-corollary.html#subsection-second-FTC"

Aah. "hash" is overloaded. You want a Twitter hash.

A.com/B.html#C => "C" is a "fragment-identifier" ;-)

Yes, I think we can do this - I know where to look. Thanks for clarifying.

Rob

[David Farmer]

> 7.  Like David said - I just moved footnote content to the bottom of a page. Solved a bug.  But more
> important, a bug that was going to recur.  It is a little tricky to "hold" the footnote content from
> the middle of a paragraph (and other oddball places we were missing and were unaware of) and "place"
> it outside the paragraph.
>
> Okay. The placement can all be done client side, but I could really use an exhaustive flowchart about how the
> placement should go. That way I don't have to hack the design for every new case I come across.

I can write the code which places the knowl content. It is not
only about where it goes: when it goes in a table, you have to
wrap the knowl content so that it looks right in the place it goes.
(All the cases I know of are addressed in the existing knowls code,
at the link I posted earlier today. That is as close as we will get to
a flowchart. :)

And probably some cases will be different if we change how born-hidden
content is handled.

David

[Rob Beezer]

Appendices (including back-of-the-book "solutions" divisions) will be copacetic
if we

* always group "frontmatter"

* never group real back matter (references, index, colophon,...)

* "article": appendices are peers of "section"

* "book", no parts: appendices are peers of "chapter"

* "book", with parts: appendices get grouped into "Appendices" which is a peer
of the parts

Rob

[Rob Beezer]

> What I mean is this: Currently chapter 4 links to "interesting-corollary.html".
> I would like it to link to "interesting-corollary.html#interesting-corollary"
> (similarly to how the 4.1 link is to
> "interesting-corollary.html#subsection-second-FTC"

OK, all division/@url in the manifest will now have a fragment identifier
(hash!). But *only* in the manifest, this is not a universal change. (David
already does this with JS for the left margin click-to-a-link feature.)

I made a very small change to the values of division/@id, so that they are now
*really* the HTML id on the content. (Some are the value of @permid now, as
they should be. Long story.) Only two changes in sample article as a result.

Rob

[Rob Beezer]

Letting this percolate more. A "preface" is styled (in LaTeX) like a "chapter"
of a book. No such thing in an "article" (only an "abstract" is at all similar).

A "book" with no parts could have "preface", "acknowledgements", etc, "chapter",
"appendix", "index", etc, all as top-level peers in a sidebar ToC.

A "book" with parts could have top-level groupings, "frontmatter", "part",
Appendices, "backmatter", as peers. Which would be a place to correct the
mistake we made by following LaTeX and putting appendices into a "backmatter"
element. (I'm not suggesting changing the source markup. Yet.)

The tree implied by the Prev/Up/Next buttons might not match the ToC tree.
Which might be (a) no big deal, or (b) fixable.

Rob

[Rob Beezer]

On 1/30/22 9:00 PM, Jason Siefken wrote:
> I do need another modification, though. I need the line
>
> MathJax.startup.defaultReady()
>
> removed from the MathJax startup code. It turns out there is an error when it
> comes to \labels in equations. When the document loads, MathJax typesets right
> away. Then the page is replaced with the React code and MathJax re-typesets the
> new page. If it sees the same \label again, it complains that the label is not
> unique. I tried, but failed, to have MathJax forget the labels when the page is
> reset, and then I figured it was probably better not to typeset anything until
> the actual page is ready. So, React now calls the MathJax startup when the page
> is ready.

See post just now, \label{} is gone from HTML output.

Is it worth testing restoring this MathJax startup call to its natural location?

(I won't be surprised if the answer is "no". ;-) )

Rob

[Jason Siefken]

See post just now, \label{} is gone from HTML output.

That's great!

Is it worth testing restoring this MathJax startup call to its natural location?

No. It doesn't make sense to run MathJax until React has processed the page; just wasted cycles.

It also might be worth considering compiling MathJax into the react code instead of serving it from the MathJax website. It does do some dynamic loading, which complicates things, but it would be a step towards "private" hosting.

[Rob Beezer]

On 2/2/22 5:12 PM, Jason Siefken wrote:
> It also might be worth considering compiling MathJax into the react code instead
> of serving it from the MathJax website. It does do some dynamic loading, which
> complicates things, but it would be a step towards "private" hosting.

Yes, that could be a consideration. But lets wait - we've got our hands full
with MathJax as-is. ;-)

I put a reminder onto:

https://github.com/rbeezer/mathbook/issues/444

Thanks,
Rob