JFriCASSupport

[Ralf Hemmecke]

I would like to make enabeling FormatMathJax in the jFriCAS frontend
simpler. It should work like this:

setFormat!(FormatMathJax)
)set output formatted on

In fact, the attached package would do exactly that, BUT
I cannot call it like above.

(11) -> setFormat!(FormatMathJax)
There are 1 exposed and 0 unexposed library operations named
setFormat! having 1 argument(s) but none was determined to be
applicable. Use HyperDoc Browse, or issue
)display op setFormat!
to learn more about the available operations. Perhaps
package-calling the operation or using coercions on the arguments
will allow you to apply the operation.

Cannot find a definition or applicable library operation named
setFormat! with argument type(s)
Type

Perhaps you should use "@" to indicate the required return type,
or "$" to specify which version of the function you need.

(11) -> setFormat!(FormatMathJax pretend FormatterCategory)
Type: Void

(12) -> 1/sqrt(5)

+-+
\|5
(12) ----
5
--FORMAT:BEG:FormatMathJax
\[
\frac{\sqrt{5}}{5}
\]
--FORMAT:END:FormatMathJax

Of course, it is not very user-friendly to require a pretend. And it
makes this simple switch quite long.

Obviously, the interpreter does not recognise the type of FormatMathJax
as the category FormatterCategory, but only as Type.

So I programmed fmtjfricastype.spad.

Instead of

setFormat!: FormatterCategory -> Void

I now use

setFormat!: Type -> Void

with that

setFormat!(FormatMathJax)

works perfectly in a FriCAS session.

However, I find it a bit sad that in setFormat! I have to accept ALL
types. Can I at least check that the input is indeed of category
FormatterCategory even though I use

setFormat!: Type -> Void

otherwise it looks pretty type-unsafe.

Which way should I follow?


Second question.
At the same time as setFormat! is called, it should switch on this
format, i.e. do the equivalent of

)set output formatted on

How can I program this from within a .spad file.

Ralf

[Waldek Hebisch]

Yes, that is limitation of current interpreter. It is hard to
remove this limitation without substantial rewrite...

>
> So I programmed fmtjfricastype.spad.
>
> Instead of
>
> setFormat!: FormatterCategory -> Void
>
> I now use
>
> setFormat!: Type -> Void
>
> with that
>
> setFormat!(FormatMathJax)
>
> works perfectly in a FriCAS session.
>
> However, I find it a bit sad that in setFormat! I have to accept ALL
> types. Can I at least check that the input is indeed of category
> FormatterCategory even though I use
>
> setFormat!: Type -> Void
>
> otherwise it looks pretty type-unsafe.

Well, you can write 'setFormat!' like:

setFormat!(t : Type) : Void ==
if t has FormatterCategory then
setFormat!(t pretend FormatterCategory)$TV
else
error "t is not of FormatterCategory"

where TV is version of package accepting FormatterCategory.
With smart enough compiler the above would work without
'pretend', but I am affraid that ATM 'pretend' is needed.

>
> Which way should I follow?

Well, usual solution chosen by others was to have something like

setFormatMathJax()

Since 'setFormatMathJax' is handled by Spad compiler it should
work without 'pretend'...

>
> Second question.
> At the same time as setFormat! is called, it should switch on this
> format, i.e. do the equivalent of
>
> )set output formatted on
>
> How can I program this from within a .spad file.

There is 'systemCommand' in 'system.spad'.

--
Waldek Hebisch

[Ralf Hemmecke]

> Well, you can write 'setFormat!' like:
>
> setFormat!(t : Type) : Void ==
> if t has FormatterCategory then
> setFormat!(t pretend FormatterCategory)$TV
> else
> error "t is not of FormatterCategory"
>
> where TV is version of package accepting FormatterCategory.
> With smart enough compiler the above would work without
> 'pretend', but I am affraid that ATM 'pretend' is needed.

Thank you for this suggestion.

>> Which way should I follow?

> Well, usual solution chosen by others was to have something like
>
> setFormatMathJax()

Well, yes, possible, but unflexible.

Currently ")set output ... on" for different formats outputs always in
the order given by i-output.boot. Using setFormats!([F1, F2, F3]) would
allow the user to decide the order and also to compile his/her own
domain of type FormatterCategory and use it.

> There is 'systemCommand' in 'system.spad'.

Arrrhhh... I always forget 'systemCommand'. Thanks.

Ralf

[Waldek Hebisch]

On Sun, Sep 27, 2020 at 09:52:39PM +0200, Ralf Hemmecke wrote:
>
> Thank you for this suggestion.
>
> >> Which way should I follow?
>
> > Well, usual solution chosen by others was to have something like
> >
> > setFormatMathJax()
>
> Well, yes, possible, but unflexible.

Dedicated functions give easy inflexible option which should
satisfy needs of most users. Users needing more can use
general interface suffering full complexity and possible
uglyness (like 'pretend' above).

> Currently ")set output ... on" for different formats outputs always in
> the order given by i-output.boot. Using setFormats!([F1, F2, F3]) would
> allow the user to decide the order and also to compile his/her own
> domain of type FormatterCategory and use it.

Well, adding user-settable order to 'i-output.boot' would be
rather easy. Simply nobody bothered to implement this.
My plan was to switch to tabular description. More precisly
currently 'i-output.boot' contains bunch of variables for
each format. I want to change this to a record. Records
can be created at runtime so one could register new format
if desired. OTOH with well designed output system there
would be almost no need for user formats (useful formats
should be already included).

Those changes are currently suspended, as Formatted plays
its own tricks, at least partially incompatible with the design
above. ATM I am waiting for "dust to settle", that is
too see what exactly you want to change.

Repeating what I wrote some time ago: formatter should be
resposible for rendering OutputForm on a stream. Where
stream output goes should be due to separate managment
layer. If different formatters output on the same
stream it should be manager task to decide on order, etc.
AFAICS Formatted wants to manage such things, and this
is a problem: when there are two managers we get chaos
and lack of proper coordination.

<rant> There is quality called "cohesion" or "conceptual
integrity" meaning that parts of system follow common
design and compose into coherent whole. I could list
rather long list of design choices made in Axiom time
that I am not happy about. But I keep most of them,
because current choices play together reasonably well.
Change which improves one aspect can make other things
worse and balance may be negative. There is also
reality factor: some changes need a lot of work to
implement correctly and partial, incomplete change can
make things worse. Due to modularity of Spad incompatible
changes can coexist in different packages, but this is
suboptimal situation, leading to duplicate developement
effort and user confusion.
</rant>

--
Waldek Hebisch

[Ralf Hemmecke]

I would like to merge the attached file to support

setFormat! FormatMathJax

in the jfricas frontend for Fricas.

See
https://github.com/hemmecke/fricas/commits/jfricas

Ralf

[Ralf Hemmecke]

> Dedicated functions give easy inflexible option which should
> satisfy needs of most users. Users needing more can use
> general interface suffering full complexity and possible
> uglyness (like 'pretend' above).

I'm not completely against your suggestion

setFormatMathJaxOn()
setFormatMathJaxOff()

One could even achieve a certain order by the order in which these
commands are executed. But in my proposal just a minute ago, I only
added the "setFormats!" version, because that will stay anyway and I am
not yet completely decided about setFormatMathJaxOn().

>> Currently ")set output ... on" for different formats outputs always in
>> the order given by i-output.boot. Using setFormats!([F1, F2, F3]) would
>> allow the user to decide the order and also to compile his/her own
>> domain of type FormatterCategory and use it.

> Well, adding user-settable order to 'i-output.boot' would be
> rather easy. Simply nobody bothered to implement this.
> My plan was to switch to tabular description. More precisly
> currently 'i-output.boot' contains bunch of variables for
> each format. I want to change this to a record. Records
> can be created at runtime so one could register new format
> if desired. OTOH with well designed output system there
> would be almost no need for user formats (useful formats
> should be already included).

Well, in fact, I would like to get rid of all those variables and also
reduce i-output.boot for the output of types. Algebra output could be
done by Format2D. The only "bug" in Format2D is that it does not break
lines. I will now play around with it and see what *my* needs are and
perhaps other people also come forward with their wishes.

> Those changes are currently suspended, as Formatted plays
> its own tricks, at least partially incompatible with the design
> above. ATM I am waiting for "dust to settle", that is
> too see what exactly you want to change.

Not exactly clear what you mean here, but my wish was to bring more
stuff from BOOT to SPAD. Adding a new Formatter should be an action of
writing a .spad file and add register it in the Makefile, but not
additionally change something in setvart.boot, setvars.boot, msgdb.boot,
and i-output.boot. Certainly, that's not a very big deal, but that's
what I chose.

My first intention was to improve the TexFormat domain to be able to
compile book.pdf, but it ended in a compleltely different approach of
handling the OutputForm structure which in turn made it relatively easy
to come up with FormatMathJax, Format1D, and Format2D. Maybe some day I
spend some time to also put Texmacs and MathML formats into my scheme.
However, since I do not use those much, it may need some users to push me.

> Repeating what I wrote some time ago: formatter should be
> resposible for rendering OutputForm on a stream. Where
> stream output goes should be due to separate managment
> layer. If different formatters output on the same
> stream it should be manager task to decide on order, etc.
> AFAICS Formatted wants to manage such things, and this
> is a problem: when there are two managers we get chaos
> and lack of proper coordination.

Right, that it is a problem with two output managers.
However, I do not see my new framework as a big problem to "output on
different streams", because the main domain that is connected to actual
output is FormattedOutput. And, honestly, this is just my invention to
connect my formatter domains to actual output on a stream (the
"formatted" stream. I do not insist on this. This domain is pretty small
and we can certainly coordinate in removing this domain or adapting it
so that your "stream idea" gets realized. All the rest of the formatting
framework is independent of actual streams.

Ralf

[Waldek Hebisch]

On Sun, Sep 27, 2020 at 11:56:40PM +0200, Ralf Hemmecke wrote:
>
> >> Currently ")set output ... on" for different formats outputs always in
> >> the order given by i-output.boot. Using setFormats!([F1, F2, F3]) would
> >> allow the user to decide the order and also to compile his/her own
> >> domain of type FormatterCategory and use it.
>
> > Well, adding user-settable order to 'i-output.boot' would be
> > rather easy. Simply nobody bothered to implement this.
> > My plan was to switch to tabular description. More precisly
> > currently 'i-output.boot' contains bunch of variables for
> > each format. I want to change this to a record. Records
> > can be created at runtime so one could register new format
> > if desired. OTOH with well designed output system there
> > would be almost no need for user formats (useful formats
> > should be already included).
>
> Well, in fact, I would like to get rid of all those variables and also
> reduce i-output.boot for the output of types. Algebra output could be
> done by Format2D. The only "bug" in Format2D is that it does not break
> lines. I will now play around with it and see what *my* needs are and
> perhaps other people also come forward with their wishes.

I probably was not clear enough in what I wrote: my intent is
to handle output via Spad code. For compatibility, want
program logic to be close to current 'i-output.boot'. In
particular whan I wrote "records" I meant Spad Record
(doing records in Boot is ugly). I want to keep ')set output ...'
but Boot code implementing this should call Spad code for
real work.

> > Those changes are currently suspended, as Formatted plays
> > its own tricks, at least partially incompatible with the design
> > above. ATM I am waiting for "dust to settle", that is
> > too see what exactly you want to change.
>
> Not exactly clear what you mean here, but my wish was to bring more
> stuff from BOOT to SPAD. Adding a new Formatter should be an action of
> writing a .spad file and add register it in the Makefile, but not
> additionally change something in setvart.boot, setvars.boot, msgdb.boot,
> and i-output.boot. Certainly, that's not a very big deal, but that's
> what I chose.

As I wrote, I want to keep ')set' commands. However, "variable
parts" should be on Spad side, so after transition there would
be no need to change Boot code to add new format. Clearly,
there is a need to have registry of available formats. This
can be done as separtate domain. Theoreticaly each formatter
could do call to register itself. Such approach would be
elegant and allow adding formatters at runtime. Pragmaticaly,
fixed table of formatters may be better (one can find all
available formatters in the table and there is no need for
formatters to know about the table).

> My first intention was to improve the TexFormat domain to be able to
> compile book.pdf, but it ended in a compleltely different approach of
> handling the OutputForm structure which in turn made it relatively easy
> to come up with FormatMathJax, Format1D, and Format2D. Maybe some day I
> spend some time to also put Texmacs and MathML formats into my scheme.
> However, since I do not use those much, it may need some users to push me.

I admit that I do not see why you felt need for different structure.
AFAICS old structure can handle what you are doing and code
size seem to be comparable.

> > Repeating what I wrote some time ago: formatter should be
> > resposible for rendering OutputForm on a stream. Where
> > stream output goes should be due to separate managment
> > layer. If different formatters output on the same
> > stream it should be manager task to decide on order, etc.
> > AFAICS Formatted wants to manage such things, and this
> > is a problem: when there are two managers we get chaos
> > and lack of proper coordination.
>
> Right, that it is a problem with two output managers.
> However, I do not see my new framework as a big problem to "output on
> different streams", because the main domain that is connected to actual
> output is FormattedOutput. And, honestly, this is just my invention to
> connect my formatter domains to actual output on a stream (the
> "formatted" stream. I do not insist on this. This domain is pretty small
> and we can certainly coordinate in removing this domain or adapting it
> so that your "stream idea" gets realized. All the rest of the formatting
> framework is independent of actual streams.

No big problem. But getting code to work means that one has
to work out various little problems. More specifically,
I would like to have single output manager. I want this
manager to be compatible with current use and support
')set' commands. Note: I am not against having alternative
setup interface, but alternative should integrate cleanly
with ')set' commands.

Clearly, output manager should support what is needed by
JFriCAS. I admit that you surprised me by use of multiple
formats -- I assumed that JFriCAS will just use single
format. Concerning use of single stream versus multiple
streams: at low level we want single stream. HyperDoc
shows that with multiple low level streams one get
various synchronization problems. Such problems do not
appear when using single low level stream. At higer
level you clearly want multiple streams: the purpose
of markers that you emit is to allow demultiplexing
single low level stream into multiple logical streams.
Now, textual markers are simple but rather poor
method of demultiplexing. For HyperDoc I plan to
use simple binary scheme, using messages. Each
message would have fixed size header containing
id (identifying high level stream), and size
and a variable size payload. Such thing is rather
easy to implement. I think that it will work
well, but since it is still a plan we need to
wait to see how it will work in practice. Before
such thing is used for HyperDoc we need to do
several other changes. In fact, recent changes to
stream usage are part of preparations for HyperDoc
change.

Anyway, concerning JFriCASSupport: for current use it
may be included. However, I expect that it will
need substantial changes (or possibly vanish) in
near future.


--
Waldek Hebisch

[Ralf Hemmecke]

> I probably was not clear enough in what I wrote: my intent is to
> handle output via Spad code. For compatibility, want program logic
> to be close to current 'i-output.boot'. In particular whan I wrote
> "records" I meant Spad Record (doing records in Boot is ugly). I
> want to keep ')set output ...' but Boot code implementing this should
> call Spad code for real work.

OK, that makes it a bit clearer, but let me dream a bit.

I would not do it the way you describe. Rather, there should be a couple
of boot functions that can be called from Spad code.

The "formatter register domain" for the "record" you mentioned above,
would be a good idea.
This registry is filled at fricas start or even filled by hardcoding
with the existing formatters that are in the FriCAS code repository.
At runtime other formatters can be added.

Switching session output on/off or redirecting to a file is done via
)set output ...
and maybe other SPAD means.

So we need a BOOT function, that does the stuff that one currently has
to hardcode into servart.boot, setvars.boot and i-output.boot, i.e.,

(1) creation of a lisp stream
(2) creation of the setOutputXXX function
(3) creation of the describtOuputXXX function
(4) registering it in the "Table $setOptions".

Oh, but that would have the "record" in Lisp $setOptions and still all
the logic of handling ")set output ...".

Why not giving up ")set output"? Then from the Lisp/Boot side there
would be needed a function
(1) to create a lisp stream, open and close it or redirect it to a file.
That stream would be stored in a SPAD "FormatterRegistry" domain. And
all the rest would be done in SPAD. At registry time the Formatter
learns about where to actually send his output. Quite a couple of Boot
lines can be replaced by SPAD logic.

Only relying on the creation of a stream might also bring us closer to a
output logic similar to what TextWriter in Aldor does.

What do you think?

>> My first intention was to improve the TexFormat domain to be able
>> to compile book.pdf, but it ended in a compleltely different
>> approach of handling the OutputForm structure which in turn made it
>> relatively easy to come up with FormatMathJax, Format1D, and
>> Format2D. Maybe some day I spend some time to also put Texmacs and
>> MathML formats into my scheme. However, since I do not use those
>> much, it may need some users to push me.
>
> I admit that I do not see why you felt need for different structure.
> AFAICS old structure can handle what you are doing and code size seem
> to be comparable.

Well. Of course, there was no "need", but it happened that way for the
LaTeX formatter, because I wanted to give the user flexibility over what
latex commands are output (therefore this (changable) hashtable for each
OutputForm toplevel operator). It proved wonderfully easy to do the
other formatters in the same way.
Furthermore now it is also easy to modify OutputFormat and adapt the
formatters.
Additionally, my formatters do not know about output. They just
translate OuputForm into a String.

> No big problem. But getting code to work means that one has to work
> out various little problems. More specifically, I would like to have
> single output manager. I want this manager to be compatible with
> current use and support ')set' commands. Note: I am not against
> having alternative setup interface, but alternative should integrate
> cleanly with ')set' commands.

OK, we can have ")set output ..." and ONE manager.
I would also have to goal to get rid of some Boot code. If this means
abandoning ")set output", would not bother me too much.

You probably will program such a manager. But please do it at a branch
so that I can see the code while you work and we can discuss where I see
problems integrating my formatters.

Actually, we should also discuss for what such a manager should actually
work. We have

FortranFormat
HTMLFormat
MathMLFormat
TexFormat
TexmacsFormat
(OpenMath?)

TexFormat and HTMLFormat can be removed. They are buggy.
TexmacsFormat and MathML format can probably easily done in my framework.
So it remains FortranFormat and the 2D Algebra format from i-output.boot.

> Clearly, output manager should support what is needed by JFriCAS.

Thanks. If we coordinate, it is also possible to adjust jFriCAS to work
nicely together with what FriCAS provides. At the time it was released,
I had to take what was there. Convergence would be good.

> I admit that you surprised me by use of multiple formats -- I assumed
> that JFriCAS will just use single format.

No. As you probably noticed, when MathJax renders the output, it costs
usually quite some time. So people might want to switch to ordinary
ASCII-2D format. And maybe some day this slowness of MathJax might bring
me program FormatMathML.

> Concerning use of single stream versus multiple streams: at low level
> we want single stream. HyperDoc shows that with multiple low level
> streams one get various synchronization problems. Such problems do
> not appear when using single low level stream. At higer level you
> clearly want multiple streams: the purpose of markers that you emit
> is to allow demultiplexing single low level stream into multiple
> logical streams.

> Now, textual markers are simple but rather poor method of
> demultiplexing.

OK. But I must admit, that usage of markers for demultiplexing appeared
only in the last moment when I realised the problem with the order of
output for multi-command input.

> For HyperDoc I plan to use simple binary scheme, using messages.

Actually, I do not understand why you stick to HyperDoc. I cannot use
it, because on my 4K-display its contents is unreadably small.

Tutorials can be done with jFriCAS and published over the web or
provided with a FriCAS release.

The only thing that is not yet available otherwise is to find out by
which domain/package a certain function is implemented. I'm sure that
HyperDoc functionality can be brought into a package such that

implementedBy("func $ SomeDomain(X, Y)")

returns the actual constructor where the source code for "func" can be
found. I just don't know where exactly I find this HyperDoc
functionality in order to write such an "implementedBy" function.

Anyway, wouldn't it make sense to work towards removing HyperDoc,
replacing the graphics framework by something more modern and eventually
even remove sman and only have a single lisp image FRICASsys?
That would look like a goal.

> Anyway, concerning JFriCASSupport: for current use it may be
> included. However, I expect that it will need substantial changes
> (or possibly vanish) in near future.

OK. Thanks. I will first try to use it in action. I already realized
that it is not exposed so instead of

setFormat! FormatMathJax

I must call

setFormat!(FormatMathJax)$JFriCASSupport

still OK, but a bit cumbersome.

Ralf