> Is it feasible to create custom output formats that would use the layout information generated by MathJax?
Yes. There are a variety of possibilities, depending on your needs. Could you be more specific about the format you are trying to produce? Is it HTML-like, or something completely different?
If it is HTML-like, then you might be able to subclass some of the CommonHTML output classes and modify them to do what you want. If it is completely different, then things would be a little more challenging. First, you would probably need to write a "DOM adaptor" that maps the various functionality needed by MathJax to the output format you are using in place of the DOM. See the ts/core/DOMadaptor.ts file, and the ones in ts/adaptors. The DOM adaptor abstracts that actions MathJax needs to be able to do (like creating elements, setting their attributes, and so on) so that they can be used on essentially any internal format, not just a browser DOM. See the LiteDOM implementation in the ts/adaptors directory, for example.
Then you would need to implement an output jax to generate whatever is needed in your output format. This takes the internal (MathML) format and generates the output in your format. The current output formats in v3 are HTML and SVG, and you can use these as examples, but you could generate whatever you want. These work by implementing object classes for each of the MathML node types that generate the output for the given type of node. So the class for <mfrac> generates the fraction output, and so on. The output jax are in the ts/output directory. The main jax is chtml.js or svg.js, and the rest of the code for that jax is in the ts/output/chtml or ts/oputput/svg subdirectories. The internal MathML nodes are implemented in ts/core/MmlTree, and the output jax has "wrappers" for these nodes that implement the output functions needed for each node. The base wrapper class is in Wrapper.ts in the output directory (e.g., ts/output/chtml/Wrapper.ts), and the individual node implementations are in the Wrappers sub-directory (e.g., ts/oputput/chtml/Wrappers). You could use either of the chtml or svg directories as a template for your own output format.
Much of the work done by the CommonHTML and SVG output jax is the same for both formats (e.g., generating the bounding box information, and other positioning information). That has all been abstracted out into a "common" output directory (ts/oputput/common). All the shared routines are there, and the two actual output jax subclass those. So the actual output jax are mostly just about laying out the HTML or SVG elements. That means your code could be mostly concerned with laying out whatever your format is, as most of the positioning and bounding box computations are already separated out in the common output base classes.
Depending on the output format you are trying to produce, you may find that the SVG output jax is easier to understand (except for tables with labeled rows).
The other important thing you will have to deal with is the fonts. The basic data for the fonts is in the common font classes, and the specific output jax add more information to them as needed. You may or may not have to add more information. The fonts are a pain, and the current implementation is using the old v2 fonts, which are not played out conveniently for the v3 font structures, so there is some complication there that is an unfortunate consequence of that. It will be straightened out when we rebuild the fonts for v3, but that hasn't happened yet.
That may give you an idea of what is required, and where to look.
> Great work with MathJax by the way - it's a very impressive bit of code!
Thanks for your kind words. We appreciate the support!