An idea to greatly simplify socket communication protocol / sman process tree

[Qian Yun]

Currently our socket communication protocol is messy because we have:

sman -- SessionIOServer -- use pty to capture FriCAS standard output
session -- SessionServer -- act as IO multiplex
FRICASsys -- SpadServer -- does calculation
hypertex -- MenuServer -- talks to SpadServer via SessionServer!

Theses are servers, but they also act as clients and connect to
each other.


My idea is that FRICASsys should not print to *standard-output*,
instead to a new stream *spad-output*.

With this new indirection layer, we can control where the output
goes -- it can go to standard-output (for terminal interactive
usage), or it can go through socket for hypertex clients.
(This should also solve the ')spool' problem BTW.)

This way we can eliminate SessionIOServer.

Next, I think we should eliminate SessionServer. I think we can
do IO multiplex in the SpadServer loop.

This way the communication protocol is greatly simplified.

The following patch already captures most algebra output.
We need to find the rest dozens of places that writes to
*standard-ouput*.

I think in theory this could really work!

- Qian

diff --git a/src/interp/int-top.boot b/src/interp/int-top.boot
index 81cddac4..0bd9074c 100644
--- a/src/interp/int-top.boot
+++ b/src/interp/int-top.boot
@@ -212,6 +212,7 @@ intloopReadConsole(b, n)==
b := CONS(a, b)
ncloopEscaped a => "iterate"
n := intloopProcessStrings(nreverse b, n)
+ PRINC GET_-OUTPUT_-STREAM_-STRING _*SPAD_-OUTPUT_*
princPrompt()
b := []

diff --git a/src/interp/vmlisp.lisp b/src/interp/vmlisp.lisp
index e82b4cf1..0e5d337d 100644
--- a/src/interp/vmlisp.lisp
+++ b/src/interp/vmlisp.lisp
@@ -44,6 +44,8 @@

(in-package "BOOT")

+(defvar *spad-output* (make-string-output-stream))
+
;; defuns

(defun define-function (f v)
@@ -621,7 +623,7 @@
(defun |make_append_stream| (filespec)
(CONS T (MAKE_APPENDSTREAM filespec)))

-(defun |mkOutputConsoleStream| () (CONS NIL *standard-output*))
+(defun |mkOutputConsoleStream| () (CONS NIL *spad-output*))

(defun SHUT (st) (if (streamp st) (close st) -1))

[Qian Yun]

On 10/16/24 9:05 PM, Qian Yun wrote:
>
> My idea is that FRICASsys should not print to *standard-output*,
> instead to a new stream *spad-output*.
>
> With this new indirection layer, we can control where the output
> goes -- it can go to standard-output (for terminal interactive
> usage), or it can go through socket for hypertex clients.
> (This should also solve the ')spool' problem BTW.)
>

Hmm, one problem is that if a computation takes a long time,
then we will not see its real-time output, instead we only
see the whole output when the computation is done.

I wonder if there's way to solve this problem...

- Qian

[Waldek Hebisch]

My plan is that _nothing_ which may go to other processes should directly
use Lisp streams like *standard-output*. If you look at 'make_out_stream'
and 'mkOutputConsoleStream' the definitions may look silly, but
the purpose is to make result incompatilbe with Lisp streams, so
we know that corresponding streams use only "approved" functions.

Functions handling those streams need to be changes so that they
perform formatting without using Lisp stream functions. Then we
can redirect result to use sockets.

--
Waldek Hebisch

[Qian Yun]

On 10/16/24 11:49 PM, Waldek Hebisch wrote:
>
> Functions handling those streams need to be changes so that they
> perform formatting without using Lisp stream functions. Then we
> can redirect result to use sockets.
>

The "direct stream to socket" puzzled me.

Now looks like we should use "socket-stream", which is not a part
of CL standard, we may need to use external Lisp library like
usocket. We will not use the sock_send/get_Int/String
FFI C functions on the Lisp side.

Now the architecture looks like this:

FRICASsys opens SpadServer, many spadclients can connect to it,
for each client, FRICASsys creates a socket-stream for it,
for the active client, we assign its socket-stream to *spad-stream*,
so that output goes via socket automatically.

- Qian

[Qian Yun]

On 10/17/24 8:24 AM, Qian Yun wrote:
>
>
> On 10/16/24 11:49 PM, Waldek Hebisch wrote:
>>
>> Functions handling those streams need to be changes so that they
>> perform formatting without using Lisp stream functions.  Then we
>> can redirect result to use sockets.
>>
>
> The "direct stream to socket" puzzled me.
>
> Now looks like we should use "socket-stream", which is not a part
> of CL standard, we may need to use external Lisp library like
> usocket.  We will not use the sock_send/get_Int/String
> FFI C functions on the Lisp side.

To clarify things up:

1. Currently we have stuff like "(PRINC STR *standard-output*)".

2. The above propose is like "(PRINC STR *spad-output*)" where
*spad-output* is bound to a socket-stream and can be switched
at runtime.

3. The following propose uses "(SPRINC STR *spad-output*)",
where SPRINC can do ")spool", and use FFI to print to
*spad-output*, which might be a socket file descriptor.


The usocket Lisp library does not support unix domian socket,
also this higher abstraction differs from our current idiom.

I plan to use the FFI for C functions instead. (Add new
wrappers for "select", "accept", etc.)

Very preliminary tests show that this approach is workable.

Also with this redesigned architecture, it would be very
easy for SageMath to communicate with FriCAS through socket.
This method should be more stable and support more Lisps
than the call-lisp-from-c method.

- Qian

[Ralf Hemmecke]

Qian,

I do not understand all this stuff, but it somehow sounds as if it might
have influence on the connection that I use in jfricas. I remember you
have an eye on this. Can you confirm?

Thank you
Ralf

[Qian Yun]

I've uploaded a very early draft to illustrate my points.
(No error handling, no buffer overflow detection,
interface not refined.)

https://github.com/oldk1331/fricas/tree/newserver

In fact, this is already kind of have enough functionality to support
jfricas! Because jfricas is a single client connected to FriCAS,
and no hypertex or graphics needed, so no IO multiplex needed.

Currently, simply connect to the unix domain socket and you
can do REPL! It should also need only a few lines of python
to connect to this NEW SpadServer. No need for hunchentoot.
Should greatly simplifies the packaging of jfricas,
thus it should attract more users.

You can test my code by:

start fricas and type ")boot $NEWSERVER := true" to enable the
new server.

In another terminal, in build directory, use the following to
connect to FriCAS:

export SPADNUM=`ls /tmp/.s* | cut -d 's' -f 2`; echo $SPADNUM;
export NEWSERVER=1; ./target/x86_64-linux-gnu/lib/spadclient

- Qian

On 10/19/24 2:37 PM, 'Ralf Hemmecke' via FriCAS - computer algebra

[Qian Yun]

It's easier than I originally thought. In fact, I have almost
moved the whole socket communication into Lisp (viewman works,
hyperdoc partly works), making "sman" and "session" redundant.

https://github.com/oldk1331/fricas/commits/newserver/

(Code is in very very early stage, if you really want to
run it, apply the patches to 1.3.11-full release tarball,
and "export NEWSERVER=1".)

This is concrete proof of concept that this idea really works.

- Qian

[Qian Yun]

One disadvantage is that the Lisp output is still to the
terminal of FRICASsys instead of spadclient.
(We can redirect SPAD output but not Lisp output.)

For example the debugger and trace output.

There is a solution at C level, namely "dup2".
We can redirect the 0/1/2 of FRICASsys to socket that
connects to spadclient.

Preliminary test shows that this works really well.

For SBCL, the *debug-io* is not associated with 0/1/2
(stdin/stdout/stderr), instead it is /dev/tty.
So we might need to set this first:

(setq *debug-io* (make-two-way-stream *standard-input*
*standard-output*))

- Qian