Fwd: Adding TeXmacs symbols to FriCAS math-mode input

[Bill Page]

I think Andrey has a pretty good idea.

For the last few days I have been working with FriCAS in TeXmacs again
after not having looked at this option of a long time. A little more
than 10 years ago I converted a very useful Axiom beginner's tutorial
provided to me by Mike Dewar (originally prepared by NAG for the
commercial version of Axiom) to TeXmacs format.

http://download.savannah.gnu.org/releases/axiom/tutorial.pdf

(referenced here: http://axiom-wiki.newsynthesis.org/TeXmacs )

It used the Axiom interface to generate the Axiom output. I tried this
again and it (almost) works with FriCAS in the most recent version of
TeXmacs but it will need a clean up. I also noticed that for some
reason the TeXmacs tutorial.tgz source file distributed on the
axiom-wiki is only incompletely converted to TeXmacs while the pdf
file above actually seems complete. I am not sure what happened to
the fully converted source file. If anyone has a complete version in
their personally archives, please let me know.

I think the new version of FriCAS in TeXmacs is a lot better than the
old although still missing a few very important features.

On 17 March 2014 22:40, Andrey G. Grozin <A.G.G...@inp.nsk.su> wrote:
> ...
> Why not use TeXmacs as the main GUI? Currently,
> only mathemagix is doing so. It integrates with TeXmacs closely. For
> example, it has dynamic plots (and other dynamic objects). Say, I have a
> plot of a function depending on some auxilliary parameters somewhere in a
> TeXmacs window. I assign a new value to one of these parameters, and the
> plot immediately changes, without the need to re-run a plotting command.
> Such level of integration can be achieved in FriCAS, too, but this requires
> further work. ioHooks are invaluable for implementing interactions of FriCAS
> with external programs, such as TeXmacs and others.
>

I think it would be good to find out more about how to integrate
TeXmacs graphics with FriCAS. Where can I look to get started on
this? I tried using mathemagix for this but though I did manage to
compile it after some difficulty, I really cannot get it to do
anything more that add 1+1. :( It seems that the build system and
maybe also the source as distributed has some serious problems. Can
you suggest some other examples of plugins that create plot output in
TeXmacs?

> ...
> I'd say that making the user experience with TeXmacs (GUI) / FriCAS (engine)
> even better than it is today is very important. For example, TeXmacs can be
> an improved replacement for HyperDoc, with a modern and intuitive interface
> (HyperDoc definitely looks and feels old-fashioned). This can be achieved by
> doing only a moderate amount of work. This is especially important in
> today's world where programs without a nice and convenient GUI are
> considered old-fashioned, and are often not even considered by potential
> users.
>

Converting Hyperdoc htex files to TeXmacs format is probably quite
easy to achieve. Is there anyone who would like to join me in an
effort to do this?

> Andrey

I am not sure exactly what the best way might be for the FriCAS
project to support and encourage the further development of TeXmacs.
We can of course submit patches "up stream" to the TeXmacs project,
but perhaps we should consider other ways as well such as maintaining
a downstream version of the FriCAS plugin as part of the FriCAS
distribution?

Here is a patch to the FriCAS plugin that I just submitted to TeXmacs.
I have also attached a patch to the texmacs output formatter for
FriCAS to render the TeXmacs symbol <mapsto> for the FriCAS +-> arrow.

One problem that I am having right now is the display of SPAD compiler
output from the ')compile' command in TeXmacs. It seems like we may
not have the appropriate iohooks in place to demarc the output in such
a way that TeXmacs can synchronize and format it properly. Output
like this should also probably appear folded by default so as not to
clutter up the document. This sort of folding is done for example on
the axiom-wiki.

In fact it might be better to support the SPAD compiler in a more
integrated way as a new type of session in TeXmacs. A SPAD session
embedded in a TeXmacs document could be compiled instead of evaluated
as commands to the FriCAS interpreter. I understand that some work was
done on literate programming in TeXmacs and support for source code
syntax highlighting, etc. It might be nice to see how far we could go
with that but again I am lacking good information about this and/or
good examples.

Supporting folding is a nice idea but I am a bit unclear on how
folding affects sessions that are embedded in the folded sections.
When I am preparing a FriCAS page on the axiom-wiki for some specific
mathematically subject there are times when I really would like to
"hide" or otherwise relegate some preliminary FriCAS technicalities to
a kind of appendix. But of course for the proper execution of the
worksheet from the beginning it is necessary that these commands get
re-executed in the proper order. How could this be handled in a
TeXmacs-based FriCAS interface?

Regards,
Bill Page.

---------- Forwarded message ----------
From: Bill Page <bill...@newsynthesis.org>
Date: 24 March 2014 11:40
Subject: Adding TeXmacs symbols to FriCAS math-mode input
To: texmac...@texmacs.org, Texmacs-Dev <texma...@gnu.org>


The following patch is based on similar coding in mupad-input.scm It
encodes all TeXmacs symbols as FriCAS identifiers by introducing
escapes in FriCAS symbol names in such a way

[Waldek Hebisch]

Bill Page wrote:
>
> Here is a patch to the FriCAS plugin that I just submitted to TeXmacs.
> I have also attached a patch to the texmacs output formatter for
> FriCAS to render the TeXmacs symbol <mapsto> for the FriCAS +-> arrow.

It is an interesting hack. However, AFAICS the patch depends on
FriCAS doing no escaping in the output. Consider the following

s := id::String
#s
s(2)
"< some literal string from input file"
id([1, 2]) -- make scripted symbol

where 'id' is some special Texmacs symbol. Good input translation
make sure that things stay consistent, that is convertion
to string produces string containing given symbol, etc.
Producing for example TeX or MathML form inside Texmacs
session should give sensible results. Users expect
input file to work, and strings containing special Texmacs
characters should be properly escaped to produce equvalent
output in Texmacs and outside.

> One problem that I am having right now is the display of SPAD compiler
> output from the ')compile' command in TeXmacs. It seems like we may
> not have the appropriate iohooks in place to demarc the output in such
> a way that TeXmacs can synchronize and format it properly.

IIRC there are no iohooks specific to compiler.

--
Waldek Hebisch
heb...@math.uni.wroc.pl

[Bill Page]

On 24 March 2014 21:33, Waldek Hebisch <heb...@math.uni.wroc.pl> wrote:
> Bill Page wrote:
>>
>> Here is a patch to the FriCAS plugin that I just submitted to TeXmacs.
>> I have also attached a patch to the texmacs output formatter for
>> FriCAS to render the TeXmacs symbol <mapsto> for the FriCAS +-> arrow.
>
> It is an interesting hack. However, AFAICS the patch depends on
> FriCAS doing no escaping in the output. Consider the following
>
> s := id::String
> #s
> s(2)
> "< some literal string from input file"
> id([1, 2]) -- make scripted symbol
>
> where 'id' is some special Texmacs symbol.

An example from TeXmacs is attached to this email.

> Good input translation
> make sure that things stay consistent, that is conversion

> to string produces string containing given symbol, etc.

Yes perhaps that may be an issue. As you can see the current TeXmacs
interface does display the string as containing just one TeXmacs
symbol but of course indexing into the string produces characters from
the TeXmacs encoded name for the symbol. This might seem inconsistent
but it is quite easy to explain and understand. All we have to do is
remind people is that TeXmacs symbols are actually multi-character
symbols in FriCAS. This isn't such an odd idea to TeXmacs users since
they can in fact key the symbol in exactly this way. And practically
speaking how ofter do we really need that a symbol convert to a one
character string?

The only way I can see to prevent that is to actually re-code the
TeXmacs symbols into utf-8 characters (where that is possible). Since
we can handle utf-8 in FriCAS that might be a good idea. It is
something we can try but at first attempt I found TeXmacs rather
unfriendly to utf8 encoding. It might require someone with a little
more knowledge of TeXmacs internals.

> Producing for example TeX or MathML form inside Texmacs
> session should give sensible results.

??? What purpose would TeX and MathML serve inside TeXmacs?

> Users expect
> input file to work, and strings containing special Texmacs
> characters should be properly escaped to produce equvalent
> output in Texmacs and outside.

The output (see attached) looks mostly normal to me. There is a
problem with your example

> "< some literal string from input file"

as you can see TeXmacs is too aggressive in trying to interpret this
as a TeXmacs symbol since the string begins with <. This has nothing
to do with my proposed "hack". I think it should be considered either
a problem with the TeXmacs parsing of FriCAS output or maybe something
that should be handled by an escape added by the texmacs output
formatter.

>
>> One problem that I am having right now is the display of SPAD compiler
>> output from the ')compile' command in TeXmacs. It seems like we may
>> not have the appropriate iohooks in place to demarc the output in such
>> a way that TeXmacs can synchronize and format it properly.
>
> IIRC there are no iohooks specific to compiler.
>

How difficult would it be to add the same iohooks to the compiler
output that we already have in the interpreter output?

[Bill Page]

The TeXmacs code generated by

)set output texmacs on

in looks wrong or at least naive to me the case of strings :


(10) -> )set output texmacs on
(10) -> "<alpha>"

(10) "<alpha>"
scheme: (with "mode" "math"
(concat "<alpha>")
)

Type: String
(11) -> "< some other"

(11) "< some other"
scheme: (with "mode" "math"
(concat "< some other")
)

Type: String
(12) ->

I think we can do that better.


On 25 March 2014 10:06, Bill Page <bill...@newsynthesis.org> wrote:
> ...

[Waldek Hebisch]

Bill Page wrote:
>
> The TeXmacs code generated by
>
> )set output texmacs on
>
> in looks wrong or at least naive to me the case of strings :
>
>
> (10) -> )set output texmacs on
> (10) -> "<alpha>"
>
> (10) "<alpha>"
> scheme: (with "mode" "math"
> (concat "<alpha>")
> )
>
> Type: String
> (11) -> "< some other"
>
> (11) "< some other"
> scheme: (with "mode" "math"
> (concat "< some other")
> )
>
> Type: String
> (12) ->
>
> I think we can do that better.

Yes, string part is easy: we need properly escape special Texmacs
characters during output. But once we do this your input code
will be broken, because '<' and '>' in variables will be escaped
and you will see underlying string instead of Texmacs symbols you
want.

--
Waldek Hebisch
heb...@math.uni.wroc.pl

[Waldek Hebisch]

People build composite names attaching suffix to a stem. Sometimes
they need to go back to the stem. If stem consists of one
character they may just take first character of composite name
and assume that they get what they want.

> The only way I can see to prevent that is to actually re-code the
> TeXmacs symbols into utf-8 characters (where that is possible). Since
> we can handle utf-8 in FriCAS that might be a good idea. It is
> something we can try but at first attempt I found TeXmacs rather
> unfriendly to utf8 encoding. It might require someone with a little
> more knowledge of TeXmacs internals.

I am affraid that bigger problem is caused by fonts. AFAICS, you
have "the same" character coded in differnt fonst via appropriate
prefix. Currently FriCAS has no way to attach font information
to characters (or symbols). We could do such thing similarly
to "scripts": effectively via appropriate mangling of symbol
name. Alternatively, we could take advantage of Lisp property
list and use it to attach arbitrary information to symbols.
In this case we probably would be forced to use uninterned symbols,
which would require adjustment in other parts of FriCAS.
Anyway, at some moment we will have a way to represent faces
and then Texmacs interace will have to convert between Texmacs
and FriCAS form.

>
> > Producing for example TeX or MathML form inside Texmacs
> > session should give sensible results.
>
> ??? What purpose would TeX and MathML serve inside TeXmacs?

Why not? User may want programtically generate TeX or MathML
documents and if Texmacs aspires to be preferable interface
it should allow to do so.

> >> One problem that I am having right now is the display of SPAD compiler
> >> output from the ')compile' command in TeXmacs. It seems like we may
> >> not have the appropriate iohooks in place to demarc the output in such
> >> a way that TeXmacs can synchronize and format it properly.
> >
> > IIRC there are no iohooks specific to compiler.
> >
>
>
> How difficult would it be to add the same iohooks to the compiler
> output that we already have in the interpreter output?

Depends on what exactly you need. There is toplevel ')compile'
handling which may produce some error messages, but when
arguments are OK it dispatches to proper compiler. For
Spad top function is 'compileSpad2Cmd'. At start it prints
message like:

Compiling FriCAS source code from file
/mnt/lv3/fricas/axp19/pp1/pp4/OFTOOL.spad using old system
compiler.

we could easily add a hook before this message.
Compilation is split into phases: reading and tokenizing,
forming piles, parsing, proper Spad compilation and
Lisp compilation. Parsing and proper Spad compilation
works on single toplevel pile at a time (such pile
correspond to a single constructor). The reading and
tokenizing work on the whole file, but is lazy way:
once a pile is complete is is passed the next stage.
For each constructor we get one or two (in case of
category with default implementations) Lisp files
to compile.

We could easily add iohooks around existing stages
and hook at the end of whole compilation. There is
a glitch however: compilation may end up with an
error, then control is passed to toplevel. If we
add hooks inside normal program code, they will
be skipped in case of error. We can add hooks in
such a way that they will be run in error case too,
but in the past I noticed that construct that I used
to accomplish that interfered with debugging. So
there is some work to do to have precise hooks at
the end without lessening debugging capabilities.
It should be relatively easy to add a general hook
saying that there was error exit to toplevel.

--
Waldek Hebisch
heb...@math.uni.wroc.pl

[Bill Page]

On 27 March 2014 19:44, Waldek Hebisch <heb...@math.uni.wroc.pl> wrote:
> Bill Page wrote:

>> Type: String
>> (11) -> "< some other"
>>
>> (11) "< some other"
>> scheme: (with "mode" "math"
>> (concat "< some other")
>> )
>>
>> Type: String
>> (12) ->
>>
>> I think we can do that better.
>
> Yes, string part is easy: we need properly escape special Texmacs
> characters during output. But once we do this your input code
> will be broken, because '<' and '>' in variables will be escaped
> and you will see underlying string instead of Texmacs symbols you
> want.
>

Yes in "proper" generated TeXmacs the < and > characters would be
escaped as <less> and <gtr> and the output would be as you say. But
it does not really make good sense to me to treat strings as "math
mode" objects anyway. We should generated something like this

scheme: (with "mode" "math"
(text concat "_"" "< some other" "_""))
)

[Bill Page]

On 27 March 2014 21:21, Waldek Hebisch <heb...@math.uni.wroc.pl> wrote:
> Bill Page wrote:
>>
>> On 24 March 2014 21:33, Waldek Hebisch <heb...@math.uni.wroc.pl> wrote:
>> > Bill Page wrote:

>> ...

>> The only way I can see to prevent that is to actually re-code the
>> TeXmacs symbols into utf-8 characters (where that is possible). Since
>> we can handle utf-8 in FriCAS that might be a good idea. It is
>> something we can try but at first attempt I found TeXmacs rather
>> unfriendly to utf8 encoding. It might require someone with a little
>> more knowledge of TeXmacs internals.
>
> I am affraid that bigger problem is caused by fonts. AFAICS, you
> have "the same" character coded in differnt fonst via appropriate
> prefix. Currently FriCAS has no way to attach font information
> to characters (or symbols). We could do such thing similarly
> to "scripts": effectively via appropriate mangling of symbol
> name. Alternatively, we could take advantage of Lisp property
> list and use it to attach arbitrary information to symbols.
> In this case we probably would be forced to use uninterned symbols,
> which would require adjustment in other parts of FriCAS.
> Anyway, at some moment we will have a way to represent faces
> and then Texmacs interace will have to convert between Texmacs
> and FriCAS form.

So far I have not tried to do anything with fonts. I think that even
in unicode blackboard bold, Frakur and script symbols are treated as
different symbols - not distinguished by font as such. Of course
there are a lot of other font characteristics like font size, color
and various decorations etc. that might be "nice" to have available
for FriCAS input and output. But for now just Greek and a few more
characters that work would be enough.

>
>>
>> > Producing for example TeX or MathML form inside Texmacs
>> > session should give sensible results.
>>
>> ??? What purpose would TeX and MathML serve inside TeXmacs?
>
> Why not? User may want programtically generate TeX or MathML
> documents and if Texmacs aspires to be preferable interface
> it should allow to do so.
>

I suppose that you need to write it to some file somewhere. The
application of TeXmacs that makes most sense to me is as a
mathematical document system that includes FriCAS computed results.
Using TeXmacs for more general things is much less interesting to me.
There are other better alternative, e.g. the command line interface,
for that.

The minimum kind of iohook that we need is to allow synchronization of
input and output. TeXmacs needs to be able to tell when computer
output starts and ends. This would allow TeXmacs to fold and hide
this output by default. Right now including

)compile xxx

commands in the FriCAS session input results in a loss of
synchonization so that parts of the compiler output seem to appear in
other unrelated FriCAS inputs.