run external command and capture output without temporary files

[Ralf Hemmecke]

Is there a fricas way that one execute a shell command line (represented
as String in FriCAS) and capture the output (String with newlines would
be enough)?

Ralf

[Waldek Hebisch]

This is done by using shell redirection to a file and returning content
of the file.

--
Waldek Hebisch

[Ralf Hemmecke]

> This is done by using shell redirection to a file and returning content
> of the file.

Yes, but as the subject said, is there a way without redirection and
temporary file?

Meanwhile I have found this.

https://github.com/sellout/external-program

Waldek, do you think that would be a way to communicate with external
programs, running a shell script or even starting another CAS process
(Sage or whatever) and communicate with it through streams?

In fact, I am looking for a way to call Sage (in a syncronous or
asynchronous way) from within FriCAS

Ralf

[Waldek Hebisch]

On Tue, Apr 11, 2023 at 09:45:28PM +0200, Ralf Hemmecke wrote:
> > This is done by using shell redirection to a file and returning content
> > of the file.
>
> Yes, but as the subject said, is there a way without redirection and
> temporary file?

Well, without redirection output would to to terminal, so redirection
is a must. And if you want output as a single "piece", then you need
temporary storage and temporary file is classic way to get such
storage.

Of course, all needed redirections and buffering could be hidden
in an utility function. AFAICS need to run external programs and
capture output is rare enough that nobody bothered to write
a special function.

> Meanwhile I have found this.
>
> https://github.com/sellout/external-program
>
> Waldek, do you think that would be a way to communicate with external
> programs, running a shell script or even starting another CAS process (Sage
> or whatever) and communicate with it through streams?
>
> In fact, I am looking for a way to call Sage (in a syncronous or
> asynchronous way) from within FriCAS

You do not say what you want from Sage. IMO "proper" way of using
external programs involves appropriate communitation protocol and
ways to translate data. Several years ago GAP folks wrote an
article about interface of this kind. They based their protocol
on Open Math, which is troublesome because semantics of Open Math
is underspecified (simple things are standarised, anything
"interesting" seem to be vendor specific extention), but as
general direction it looked about right. IIUC Singular implemented
the same protocol. OTOH I am not sure if there are good ways to
get information out of Sage.

Many years ago Martin Rubey tried "Pyton in Lisp" package. The
idea was to run Pytion code from FriCAS and in that way get access
to Sage. IIRC this worked fine for pure Pyton, but Sage is
kind of opposite to "pure Python": most code is in external
programs.

--
Waldek Hebisch

[Ralf Hemmecke]

On 13.04.23 16:18, Waldek Hebisch wrote:
> Of course, all needed redirections and buffering could be hidden in
> an utility function. AFAICS need to run external programs and
> capture output is rare enough that nobody bothered to write a
> special function.

I have written a simple stupid interface that can call any external program.

I can call Sage via

)compile runextcmd.spad
RUN ==> run $ RunExternalCommand
out := RUN("/bin/sh sage.sh", "[x^2 for x in range(40)]")

or

setExecutable("sage", "/bin/sh sage.sh")
out := RUN("sage", "[x^2 for x in range(40)]")

This will give the value:

(5)
["[0,", " 1,", " 4,", " 9,", " 16,", " 25,", " 36,", " 49,", " 64,",
" 81,",
" 100,", " 121,", " 144,", " 169,", " 196,", " 225,", " 256,", "
289,",
" 324,", " 361,", " 400,", " 441,", " 484,", " 529,", " 576,", "
625,",
" 676,", " 729,", " 784,", " 841,", " 900,", " 961,", " 1024,", "
1089,",
" 1156,", " 1225,", " 1296,", " 1369,", " 1444,", " 1521]"]
Type: List(String)

Of course, that needs to be brought back into proper FriCAS objects like

str := concat out

(6)
"[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225,
256, 289,
324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900,
961, 1024,
1089, 1156, 1225, 1296, 1369, 1444, 1521]"

(7) -> inf := parse(str)$InputForm

(7)
(construct 0 1 4 9 16 25 36 49 64 81 100 121 144 169 196
225 256 289 324 361 400 441 484 529 576 625 676 729 784
841
900 961 1024 1089 1156 1225 1296 1369 1444 1521)
Type: InputForm


(8) -> interpret(inf)

(8)
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225,
256, 289,
324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900,
961, 1024,
1089, 1156, 1225, 1296, 1369, 1444, 1521]
Type: List(NonNegativeInteger)


sage.sh is:
==============================
#!/bin/sh
base=$(basename $1 .tmp)
cat $1 | sage -q | sed 's/^sage: //' > $base.out
==============================

The first parameter that script gets is a filename of the form
rxcXXXXX.tmp where the XXXXX is a random number (random(2^32)).

Clearly, this enables me not only to call Sage, but also Maple,
Mathematica, etc. The question is only whether I do the massaging of the
output of the respective programs (sage, mathematica, maple) in the
respective shell script or via string transformations in FriCAS. For
Sage I have chosen the script, because string handling in FriCAS is
rather limited without having an equivalent to regular expression
handling like in sed/awl/perl.

If I could eliminate the need of accessing the file system that would be
great. I guess, it should be diable if I use specific features from
SBCL, but I would like to have something that works for any lisp that
FriCAS supports.

> You do not say what you want from Sage. IMO "proper" way of using
> external programs involves appropriate communitation protocol and
> ways to translate data.

Proper way is more involved. What I want is just to give our users
something at hand that enables them to build on top of a simple
communication protocol (like my RunExternalCommand) and do the remaining
"interpret(parse make_output_fricas_like(external_output)" in an ad hoc
fashion.

This might or might not be the start of a true interface to external CAS
(not really my goal), but I just wanted to call two functions from Sage
and use their output for comparison/testing of my code.

> OTOH I am not sure if there are good ways to get information out of
> Sage.

In general, you are right. But I do not want to write an interface to
everything in Sage. That would be equally difficult as the interface to
FriCAS in Sage. It also only covers a part of FriCAS.

Recently, Kurt told me about his experiments.

https://github.com/nilqed/spadlib/blob/master/pipe/src/pipe.spad

I haven't yet experimented with it, but it looks interesting.
I might probably use that for myself, but this code seems to be bound to
SBCL and not easily doable if the underlying lisp can be any other of
the FriCAS supported lisps.

Ralf

[Dima Pasechnik]

A proper interface to FriCAS in Sage ought to be like the one we have for Maxima - via embedded libECL, where Maxima is loaded as an precompiled Lisp code.

There is no overhead and flakiness associated with files or pipes, all the communication is in memory.

See https://github.com/sagemath/sage/blob/develop/src/sage/interfaces/maxima_lib.py

Dima

--
You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/e3406730-bb70-d19b-e99f-fabea71b8c9e%40hemmecke.org.

[Ralf Hemmecke]

On 15.04.23 13:04, Dima Pasechnik wrote:
> A proper interface to FriCAS in Sage ought to be like the one we have for
> Maxima - via embedded libECL, where Maxima is loaded as an precompiled Lisp
> code.

Not my goal currently. Anyway, it is restricted to ECL and even though
the "temporary file" problem would be solve, one still needs a lot of
machinery to translate the actual objects from one system into the other
if one needs the actual value.

I generic interface is anyway never needed. I guess also Sage just picks
out a couple of useful features from other systems and then optimizes an
interface to such functions. And maybe this means to use a different
machinery to interface two different parts of the other system.

Ralf

[Dima Pasechnik]

On Sat, 15 Apr 2023, 15:39 Ralf Hemmecke, <ra...@hemmecke.de> wrote:

On 15.04.23 13:04, Dima Pasechnik wrote:
> A proper interface to FriCAS in Sage ought to be like the one we have for
> Maxima - via embedded libECL, where Maxima is loaded as an precompiled Lisp
> code.

Not my goal currently. Anyway, it is restricted to ECL

any Lisp with a library wrapper should be easily adapted here, I suppose. 

 and even though
the "temporary file" problem would be solve, one still needs a lot of
machinery to translate the actual objects from one system into the other
if one needs the actual value.

I generic interface is anyway never needed. I guess also Sage just picks
out a couple of useful features from other systems

no, not really. E.g. with libgap (library interface to GAP) you can really write anything you can do in GAP in Python syntax in Sage, 

the interface is not requiring you to write boilerplate interface code.

I don't know much about ECL  Lisp interface in Sage, but I gather you can easily call Lisp functions and get the results back.

I don't know anything about how FriCAS translates its calls to Lisp and back, so I can't say how easily it would be to achieve a similar feat for FriCAS.

Dima

and then optimizes an
interface to such functions. And maybe this means to use a different
machinery to interface two different parts of the other system.

Ralf

--
You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/5e577230-7cb6-d0fc-4a2a-9e45807a600f%40hemmecke.de.

[Ralf Hemmecke]

On 16.04.23 12:18, Dima Pasechnik wrote:
>> I generic interface is anyway never needed. I guess also Sage just picks
>> out a couple of useful features from other systems

> no, not really. E.g. with libgap (library interface to GAP) you can really
> write anything you can do in GAP in Python syntax in Sage,
> the interface is not requiring you to write boilerplate interface code.

OK, maybe I do not know enough of how that works, but we clearly have to
distinguish two types of interfaces.
A) control an external program P by the host program and allow output
from P to be shown in your system
B) same as A) plus allow your system to use the computed values from P
in your system.

Maybe what you describe is interface type (B), but I guess, that still
needs quite some effort to translate the data or at least write some
routines to interpret the ASCII output of P. And what I meant is that in
this part Sage covers only a part of FriCAS.

Just calling P from my system and only showing its output in my system
is not so hard. The problem is to use the data that is computed
externally. I wonder why OpenMath is not yet generally accepted as an
interface communication language.

Ralf

[Dima Pasechnik]

On Sun, Apr 16, 2023 at 12:28 PM Ralf Hemmecke <ra...@hemmecke.de> wrote:
>
> On 16.04.23 12:18, Dima Pasechnik wrote:
> >> I generic interface is anyway never needed. I guess also Sage just picks
> >> out a couple of useful features from other systems
>
> > no, not really. E.g. with libgap (library interface to GAP) you can really
> > write anything you can do in GAP in Python syntax in Sage,
> > the interface is not requiring you to write boilerplate interface code.
>
> OK, maybe I do not know enough of how that works, but we clearly have to
> distinguish two types of interfaces.
> A) control an external program P by the host program and allow output
> from P to be shown in your system
> B) same as A) plus allow your system to use the computed values from P
> in your system.
>
> Maybe what you describe is interface type (B), but I guess, that still
> needs quite some effort to translate the data or at least write some
> routines to interpret the ASCII output of P.

No, there is, ideally, no ASCII involved! E.g. Sage and GAP use the same machine
precision integers (int), and the same long integers (GMP integers).
So they copy this data in memory (or even not copy, just modify in place).

The main issue here becomes memory management of such objects,
especially as GAP has its won garbade collector.
(Same with the Lisp interface).

And what I meant is that in
> this part Sage covers only a part of FriCAS.
>
> Just calling P from my system and only showing its output in my system
> is not so hard. The problem is to use the data that is computed
> externally. I wonder why OpenMath is not yet generally accepted as an
> interface communication language.
>
> Ralf
>

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/1f97d3e5-071f-d315-40dc-64667dd7e2f9%40hemmecke.de.

[Grégory Vanuxem]

Hi folks,

In src/interp/vmlisp.lisp you have:
#+:sbcl
(defun OBEY (S)
#-:win32 (if *OBEY-STDOUT*
(sb-ext::process-exit-code
(sb-ext::run-program "/bin/sh" (list "-c" S) :input t
:output *standard-output* :error *standard-output*))
(sb-ext::process-exit-code
(sb-ext::run-program "/bin/sh"
(list "-c" S) :input t :output t :error t)))
#+:win32 (sb-ext::process-exit-code
(sb-ext::run-program "sh"
(list "-c" S) :input t :output t :error t :search t)))

You will remark I added some of your code for notebook support in my
FriCAS copy, here *OBEY-STDOUT*

The ')system' command uses it, but it is also accessible from SPAD via
the OBEY Lisp function instead.

In your jfricas code (webspad.lisp), you have
(defvar webspad-stream (make-string-output-stream))

and use above instead of
:output *standard-output*
:output webspad-stream

And finally use (get-output-stream-string webspad-stream) to return it.
That way you will even use the FriCAS GC.

I guess you will obtain an SExpression that has a 'string' operation
for coercion.

That can even be done in a '.fricas.input' startup file to rewrite the
OBEY LISP function for example.

Cheers,
__
Greg

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/7d99e560-93bd-2c82-9ba4-2c8c67564fe2%40hemmecke.org.

[Ralf Hemmecke]

Hi Greg,

On 16.04.23 18:25, Grégory Vanuxem wrote:
> In src/interp/vmlisp.lisp you have:
> #+:sbcl
> (defun OBEY (S)
> #-:win32 (if *OBEY-STDOUT*
> (sb-ext::process-exit-code
> (sb-ext::run-program "/bin/sh" (list "-c" S) :input t
> :output *standard-output* :error *standard-output*))
> (sb-ext::process-exit-code
> (sb-ext::run-program "/bin/sh"
> (list "-c" S) :input t :output t :error t)))
> #+:win32 (sb-ext::process-exit-code
> (sb-ext::run-program "sh"
> (list "-c" S) :input t :output t :error t :search t)))

Oh, yes. Thank you. It seems that I forget my own code. However, it only
works currently for sbcl and with my patches for jfricas. So the
restriction to SBCL is what bothers me for the general case. For me
personally it might work, but otherwise... :-(

It somehow looks like Kurt used the same mechanism.

https://github.com/nilqed/spadlib/blob/master/pipe/src/pipe.spad

Ralf

[Grégory Vanuxem]

Oh, I did not see this mail either. You want to use a program which
have an read-eval-print-loop.

[Grégory Vanuxem]

Le dim. 16 avr. 2023 à 19:54, Ralf Hemmecke <ra...@hemmecke.org> a écrit :
>
> Hi Greg,
>
> On 16.04.23 18:25, Grégory Vanuxem wrote:
> > In src/interp/vmlisp.lisp you have:
> > #+:sbcl
> > (defun OBEY (S)
> > #-:win32 (if *OBEY-STDOUT*
> > (sb-ext::process-exit-code
> > (sb-ext::run-program "/bin/sh" (list "-c" S) :input t
> > :output *standard-output* :error *standard-output*))
> > (sb-ext::process-exit-code
> > (sb-ext::run-program "/bin/sh"
> > (list "-c" S) :input t :output t :error t)))
> > #+:win32 (sb-ext::process-exit-code
> > (sb-ext::run-program "sh"
> > (list "-c" S) :input t :output t :error t :search t)))
>
> Oh, yes. Thank you. It seems that I forget my own code. However, it only
> works currently for sbcl and with my patches for jfricas.

I also have some problems like this.
This is what bother me a lot with Common LISP.

> So the
> restriction to SBCL is what bothers me for the general case. For me
> personally it might work, but otherwise... :-(
>
> It somehow looks like Kurt used the same mechanism.
>
> https://github.com/nilqed/spadlib/blob/master/pipe/src/pipe.spad

I think this is what you're looking for. But this is Spad code, and if
you need to support more implementation that will be annoying from my
point of view. Personally I would write it directly in LISP. You can
easily "translate" it to ECL if you need this.

I do not own ECL but beginning by changing (sb-ext::run-program ...)
to (ext:system ...), should do the trick to initialize the pipe
between FriCAS and the other REPL. I would also suggest looking at the
ECL or GCL documentation. More importantly, find the way to create the
two ways stream string with (sb-ext:process-output proc)
(sb-ext:process-input proc) as parameters. I do not know how actually.
Camm should know I think.

I forgot, detach the created process maybe.

Greg

[Grégory Vanuxem]

Hello,

Le sam. 15 avr. 2023 à 11:37, Ralf Hemmecke <ra...@hemmecke.org> a écrit :
> This might or might not be the start of a true interface to external CAS
> (not really my goal), but I just wanted to call two functions from Sage
> and use their output for comparison/testing of my code.

I am curious. Which ones please?

Greg

[Ralf Hemmecke]

ModularForms(Gamma0(level), weight, prec=20).basis()

Ralf

[Grégory Vanuxem]

Good question :)

Nevertheless it's a pity the SAGE documentation is more and more,
apparently, closed. I mean which libraries they use. In the past, they
used to express algorithms and libraries used depending on which case.

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/18a975e0-3813-e9d2-31e7-e882672045e4%40hemmecke.de.

[Grégory Vanuxem]

Aujourd'hui j'ai d'autres chats à fouetter :)

[Grégory Vanuxem]

Hello Ralf, *

Attached should do the job I think.
I added to pipe.spad support of ECL and 'runShellCommand : String -> String'

Have a good day!
__
Greg

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/7d99e560-93bd-2c82-9ba4-2c8c67564fe2%40hemmecke.org.