[PATCH] proper fix output truncation at beginning

[Qian Yun]

There was this problem when using fricas(sman) in a pipe,
the output may get truncated at beginning or end.

I used a workaround before, but now I have find the root cause
and a proper fix.

First, "expose-bug.patch" in the attachment reverts the workaround
and expose the bug during the book building.

The root cause is that, the first prompt is send from "FRICASsys"
to "session", skipping "sman", causing a race.

If "session" gets "(1) -> \0" from "FRICASsys", then gets
"a very long string" from "sman", then the socket may send them
together, so "spadclient" gets this:

"(1) -> \0a very long string"

When printed by "fputs", those after '\0' is truncated.

Now, with my patch, the initial prompt is sent from "FRICASsys"
to terminal, then to "sman", then to "session", then to "spadclient".

- Qian

[Qian Yun]

Hi Waldek, what's your view on this patch?

- Qian

[Waldek Hebisch]

That is good catch, we should apply it.

However, it looks to me that you eliminated one (significant) problem,
but there is still parallel signalling between processes, which
may lead to races. That is we should change communication protocal
between our processes to be essentially message based.

--
Waldek Hebisch

[Qian Yun]

On 12/25/24 10:07 PM, Waldek Hebisch wrote:
>
> However, it looks to me that you eliminated one (significant) problem,
> but there is still parallel signalling between processes, which
> may lead to races. That is we should change communication protocal
> between our processes to be essentially message based.
>

What's your opinion on my thread titled "An idea to greatly
simplify socket communication protocol / sman process tree"?
(Namely to move the main IO loop from "session" to "FRICASsys",
thus greatly reduce the number of processes and parallel signalling.)

- Qian

[Waldek Hebisch]

I am affraid of potential problems:
- FRICASsys is compute-intensive, computations may block
communication leading to trouble (this is problematic now,
but having more things in FRICASsys may make problem worse).
- duplicating file decriptors without knowledge of Lisp may
lead to troubles. In particular, in the past we had trouble
which was due to buffering between Lisp I/O and OS I/O.

To explain a bit more: I would prefer a protocol where at start
of message we have size. Size allows reliably finding boundaries
between messages. But this looks impossible with current Lisp
I/O.

Let me mention anouther thing. For some period we had trouble
with Control-C handling in sbcl. It turned out that sbcl
developers changed internals and decided to use SIGPIPE for
their Control-C handling. But our C routines intercepted
SIGPIPE, which broke "new" sbcl Control-C handling (I write
"new" in the quotes, as this is old story now).

Note that arguably Unix signals are for applications to use
and I did not find any info about reserving SIGPIPE in sbcl
documentation. File descriptors for Lisp streams IMO are
part of Lisp implementation, they may break our use at any
time.

--
Waldek Hebisch

[Qian Yun]

On 12/27/24 7:32 PM, Waldek Hebisch wrote:
> On Thu, Dec 26, 2024 at 06:14:15PM +0800, Qian Yun wrote:
>> On 12/25/24 10:07 PM, Waldek Hebisch wrote:
>>>
>>> However, it looks to me that you eliminated one (significant) problem,
>>> but there is still parallel signalling between processes, which
>>> may lead to races. That is we should change communication protocal
>>> between our processes to be essentially message based.
>>>
>>
>> What's your opinion on my thread titled "An idea to greatly
>> simplify socket communication protocol / sman process tree"?
>> (Namely to move the main IO loop from "session" to "FRICASsys",
>> thus greatly reduce the number of processes and parallel signalling.)
>
> I am affraid of potential problems:
> - FRICASsys is compute-intensive, computations may block
> communication leading to trouble (this is problematic now,
> but having more things in FRICASsys may make problem worse).

On one hand, we can think "the app" (FriCAS as a whole) as a REPL,
when it is in the "eval" stage, we simply wait for the computation
to finish.

On the other hand, we can think "the app" as a single-threaded
HTTP server, when multiple incoming requests arrive, we handle
each request one by one and block the rest.

Either way, current implementation is unnecessarily complicated.

- Qian

[Qian Yun]

On 12/25/24 10:07 PM, Waldek Hebisch wrote:
>

> That is good catch, we should apply it.
>
> However, it looks to me that you eliminated one (significant) problem,
> but there is still parallel signalling between processes, which
> may lead to races. That is we should change communication protocal
> between our processes to be essentially message based.
>

Sorry for my late reply. I'm taking this as a "yes" to commit
this patch?

- Qian

[Waldek Hebisch]

Yes, the "however" part is about future developement.

--
Waldek Hebisch