[PATCH] improvements to the reference book

[Qian Yun]

I'd like to upstream my improvements to the reference book.

The patches are in this branch:
https://github.com/oldk1331/fricas/commits/new-ps-book/

1. build book with PS images generated by FriCAS
https://github.com/oldk1331/fricas/commit/b1def5000c1fe4db3c68ac6a189dccb6b839d667

2. fix chapter 14 "Browse", remove unused part and fix
some lines to match the behavior of current HyperDoc
https://github.com/oldk1331/fricas/commit/41b56c63c32b7eb30585fb809ffb698100d51888

3. build book with pdflatex, with png images replacing
the old PS images
https://github.com/oldk1331/fricas/commit/b66627d44257d1c85ef9d9668ef95f107e92c3c4

4. fix display issue of "output" in book section 6.19
https://github.com/oldk1331/fricas/commit/4276c76327ab731770506908753162fa7d918de8

====

There has been a few minor improvements in the past days,
and I've built the book many times without failure.

Note that in patch 3, I've changed the Makefile target
from "book.pdf" to "book".

- Qian

[Qian Yun]

Any comments?

I believe the first and second patch are good to go.

As for the third patch, I think the png image can be improved a bit
more, by tweaking the fonts a little.

And for the fourth patch, I'd like to include
)set output length 88
in src/doc/htex2input.awk, because for those verbatim
"algebra stream" output, the default setting is a bit short.

- Qian

[Waldek Hebisch]

On Fri, May 17, 2024 at 08:38:35PM +0800, Qian Yun wrote:
> I'd like to upstream my improvements to the reference book.

I fetched your branch. Book builds fine on new machine, but failed
on my old one. One problem is that on old machine I need
'--outfile=...' option to 'epstopdf' like:

epstopdf tmp/3DOptCf3.VIEW/image.ps --outfile=tmp/3DOptCf3.pdf

After fixing this build failed later with latex error, probably
one of the files is truncated.

--
Waldek Hebisch

[Waldek Hebisch]

In second run 'make book' worked. In third run it failed again.

--
Waldek Hebisch

[Qian Yun]

OK, what's the version of texlive on this old machine?
What's its CPU model and how many threads are you using to build
the book?

- Qian

[Waldek Hebisch]

On Mon, May 27, 2024 at 08:50:31AM +0800, Qian Yun wrote:
> OK, what's the version of texlive on this old machine?

epstopdf has inside comment saying: 2014-06-18

> What's its CPU model and how many threads are you using to build
> the book?

Quad core Core 2 2.4.GHz. I just did 'make book', that is serial
build.

>
> - Qian
>
> On 5/27/24 07:37, Waldek Hebisch wrote:
> > On Fri, May 17, 2024 at 08:38:35PM +0800, Qian Yun wrote:
> > > I'd like to upstream my improvements to the reference book.
> >
> > I fetched your branch. Book builds fine on new machine, but failed
> > on my old one. One problem is that on old machine I need
> > '--outfile=...' option to 'epstopdf' like:
> >
> > epstopdf tmp/3DOptCf3.VIEW/image.ps --outfile=tmp/3DOptCf3.pdf
> >
> > After fixing this build failed later with latex error, probably
> > one of the files is truncated.
> >
>

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/f821d0f1-c669-493e-8e80-c7e2a8001811%40gmail.com.

--
Waldek Hebisch

[Qian Yun]

On 5/27/24 10:00, Waldek Hebisch wrote:
> On Mon, May 27, 2024 at 08:50:31AM +0800, Qian Yun wrote:
>> OK, what's the version of texlive on this old machine?
>
> epstopdf has inside comment saying: 2014-06-18

Changelog shows "2017/01/07 v2.26". I can add "--output="
to make it more backwards compatible.

>> What's its CPU model and how many threads are you using to build
>> the book?
>
> Quad core Core 2 2.4.GHz. I just did 'make book', that is serial
> build.
>

I limit my CPU to 800MHz, but still can't reproduce it.

Can you tweak the sleep value in commit 8009c4f2 and see if
that helps. If file gets truncated at the end, tweak value in
src/sman/sman.c, if file gets truncated at the beginning,
tweak value in src/sman/spadclient.c.

- Qian

[Waldek Hebisch]

On Mon, May 27, 2024 at 06:02:43PM +0800, Qian Yun wrote:
>
> On 5/27/24 10:00, Waldek Hebisch wrote:
> > On Mon, May 27, 2024 at 08:50:31AM +0800, Qian Yun wrote:
> > > OK, what's the version of texlive on this old machine?
> >
> > epstopdf has inside comment saying: 2014-06-18
>
> Changelog shows "2017/01/07 v2.26". I can add "--output="
> to make it more backwards compatible.

It is '--outfile='. Please to it. This is low-cost way to have
compatibility with older machines

> > > What's its CPU model and how many threads are you using to build
> > > the book?
> >
> > Quad core Core 2 2.4.GHz. I just did 'make book', that is serial
> > build.
> >
>
> I limit my CPU to 800MHz, but still can't reproduce it.

I suspect that this may disc drive. I have mirrored pair of
hard discs. After large writes processes writing to disc effectively
stop for few seconds, but CPU bound go forward.

--
Waldek Hebisch

[Qian Yun]

On 5/27/24 22:10, Waldek Hebisch wrote:>
>>>> What's its CPU model and how many threads are you using to build
>>>> the book?
>>>
>>> Quad core Core 2 2.4.GHz. I just did 'make book', that is serial
>>> build.
>>>
>>
>> I limit my CPU to 800MHz, but still can't reproduce it.
>
> I suspect that this may disc drive. I have mirrored pair of
> hard discs. After large writes processes writing to disc effectively
> stop for few seconds, but CPU bound go forward.
>

Can't reproduce it on my mechanical hard drives either.

From your description it looks like it's the missing lines
at the end problem.

Do you think this is a problem that needs to be fixed before commit?

- Qian

[Ralf Hemmecke]

On 5/29/24 11:24, Qian Yun wrote:
> Do you think this is a problem that needs to be fixed before commit?

If I had something to say, I would like to solve that problem once and
for all. Would it be too hard to synchronize the writeout of files by a
temporary file on hard disk (or some cheaper option if possible)?

I could also imagine another solution.
If I am not wrong then these truncated files come from .htex files.
So, the respective .input files that are fed to fricas are generated.
We simply add two lines to these files, one at the beginning, and one at
the end something like

-- %%% BEGIN %%%

-- %%% ENE %%%

and then check the generated file for existence of these lines. If they
are not there, then the .input file has to be sent again through fricas.
That has the chance of looping forever. But is probability of an
infinite loop higher or lower than what we have now?

I especially do not like that the beginning or end of the file can be
truncated at a place that does not lead to a LaTeX error.
(I know that this can also already happen on "master", but we should do
something about it.)

So the book would be incomplete, but that would only be recognized if
someone actually reads the book and figures out that something is
missing. However, the book will most probably only be read by newcomers
that cannot be expected to see that something is (wrongly) missing.

Opinions?

Ralf

[Qian Yun]

I tried more debugging and I hope I solve this problem
in the correct direction.

Everything is connected now... These are all related:

https://github.com/fricas/fricas/issues/171
https://github.com/fricas/fricas/issues/170
https://github.com/fricas/fricas/pull/101


One thing remains unclear to me: is "wait_for_client_read"
really useful?

Basically what this patch does is:
close the process (sman/session/spadbuf/viewman/hypertex)
only after its input source is closed.

This should make sure no data is lost in the socket buffer.

Please test if this patch solves the problem.

- Qian

[Waldek Hebisch]

On Wed, May 29, 2024 at 05:24:44PM +0800, Qian Yun wrote:
>
> On 5/27/24 22:10, Waldek Hebisch wrote:>
> > > > > What's its CPU model and how many threads are you using to build
> > > > > the book?
> > > >
> > > > Quad core Core 2 2.4.GHz. I just did 'make book', that is serial
> > > > build.
> > > >
> > >
> > > I limit my CPU to 800MHz, but still can't reproduce it.
> >
> > I suspect that this may disc drive. I have mirrored pair of
> > hard discs. After large writes processes writing to disc effectively
> > stop for few seconds, but CPU bound go forward.
> >
>
> Can't reproduce it on my mechanical hard drives either.
>
> From your description it looks like it's the missing lines
> at the end problem.

I did more trial. With diff:

diff --git a/src/sman/sman.c b/src/sman/sman.c
index 652e270..5b0c861 100644
--- a/src/sman/sman.c
+++ b/src/sman/sman.c
@@ -773,10 +773,10 @@ monitor_children(void)
}
switch(proc->death_action) {
case Die:
- fricas_sleep(100); /* wait a bit for socket buffer to be processed */
+ fricas_sleep(500); /* wait a bit for socket buffer to be processed */
kill_all_children();
clean_up_sockets();
- fricas_sleep(100);
+ fricas_sleep(500);
exit(0);
case NadaDelShitsky:
break;

first build went fine, but a few later failed. Actually, it looks
worse than previous version where probability of success looked
higher.

Yes, problem is because some .tex files are truncated. In one run it
was 'ug10.tex', in few cases it was 'SEGBIND.tex'. In other cases
I did not check the files but LaTeX error messuge indicated truncation.

> Do you think this is a problem that needs to be fixed before commit?

We should understand what is happening. As I wrote, HyperDoc and
graphics protocal is inherently racy, but in previously it worked
relatively well. Ocasional hangs did happen. But truncated files
is a new thing. AFAICS book build is similar to build of HyperDoc
pages, parts that differ do not involve FRICASsys, HyperDoc or graphics.
On this machine I have build HyperDoc pages thousends of times
and I believe that each time build was correct. There is no 'xvfb'
on this machine and building graphic pages pops images on the screen,
so I build graphic pages less frequently, but I think it was
hundreds of times. I only recently tried to build book on this
machine as book needs extra LaTeX packages. So I do not know if
degradation was already in previous book process or is a new thing.
But clearly there is something fishy with current build process
compared to build of HyperDoc pages.

--
Waldek Hebisch

[Ralf Hemmecke]

On 5/29/24 16:51, Waldek Hebisch wrote:
> We should understand what is happening. As I wrote, HyperDoc and
> graphics protocal is inherently racy, but in previously it worked
> relatively well.

Well, the difference is that the hyperdoc pictures are built with

${GEN_VIEWPORTFILES_PHT}: %.pht: %.ht ${INPUT_EXTRA_INPUT} ${SMAN} \
$(fricas_target_bindir)/hypertex$(EXEEXT) stamp-initial-ht-db
echo $<
unset DAASE; HTPATH=.; export HTPATH; \
FRICAS_INITFILE='' ${XVFB} ${SMAN} -noihere -paste $<

I have no idea what -noihere is actually doing, but maybe it is
connected to the "race problem". Or maybe the -paste option does
something fancy that cause the race not to happen.

Previously, pictures for the book were taken from the "pasted" pictures
of this run and converted by imagemagick. The new version needs a new
fricas run to generate the .ps format.

Anyway, I would like to have some safety check that the generated .tex
files are non-truncated.

Unfortunately, I will be away till Monday.

Ralf

[Qian Yun]

On 5/29/24 22:51, Waldek Hebisch wrote:
>
> first build went fine, but a few later failed. Actually, it looks
> worse than previous version where probability of success looked
> higher.
>
> Yes, problem is because some .tex files are truncated. In one run it
> was 'ug10.tex', in few cases it was 'SEGBIND.tex'. In other cases
> I did not check the files but LaTeX error messuge indicated truncation.

Can you try my patch from yesterday and see if this one helps.

>> Do you think this is a problem that needs to be fixed before commit?
>
> We should understand what is happening. As I wrote, HyperDoc and
> graphics protocal is inherently racy, but in previously it worked
> relatively well. Ocasional hangs did happen. But truncated files
> is a new thing.

I do not think this "truncated file" problem is a new thing.
We simply did not test it before, so it was not noticed until now.

The "sman -paste" invocation of "hypertex" does not have this problem
because it uses only socket IO to FRICASsys, it's purely sequential.

While the "usage of sman/FRICASsys in pipe" is more complex, it involves
both socket IO and stdio, so the race happens there.

- Qian

[Waldek Hebisch]

On Thu, May 30, 2024 at 07:43:30PM +0800, Qian Yun wrote:
>
>
> On 5/29/24 22:51, Waldek Hebisch wrote:
> >
> > first build went fine, but a few later failed. Actually, it looks
> > worse than previous version where probability of success looked
> > higher.
> >
> > Yes, problem is because some .tex files are truncated. In one run it
> > was 'ug10.tex', in few cases it was 'SEGBIND.tex'. In other cases
> > I did not check the files but LaTeX error messuge indicated truncation.
>
> Can you try my patch from yesterday and see if this one helps.

I could, but it takes time. And such tweaking is risky, while it
may solve problem on my machine we risk breaking other. I would
prefer to get closer to reasons so that we can be confident that
book build really works.

> > > Do you think this is a problem that needs to be fixed before commit?
> >
> > We should understand what is happening. As I wrote, HyperDoc and
> > graphics protocal is inherently racy, but in previously it worked
> > relatively well. Ocasional hangs did happen. But truncated files
> > is a new thing.
>
> I do not think this "truncated file" problem is a new thing.
> We simply did not test it before, so it was not noticed until now.

Well, I check HyperDoc pages, it is extremally unlikely that
breakge would be not detected in long time. More precisely
I run diff between versions and most of the time it is trivial
(just some semi-random numbers differ). Since pages did not
change in unexpected ways potential breakge would be _very_
repreateable (and current truncation clearly have random behaviour)
and stay the same for long time. HyperDoc parses pages, so truncation
likely would give errors from HyperDoc when vieved. Since I look at pages
(and I think other folks also look at them) it is extremaly unlikely
that such gross error would be missed.

Concerning book, I did a few trials with version in the trunk, and
it worked fine on each trial. That is too little to be sure,
but is strong indication that trouble is due to recent changes.

> The "sman -paste" invocation of "hypertex" does not have this problem
> because it uses only socket IO to FRICASsys, it's purely sequential.

Yes, socket I/O is free from worst races. Pure use of stdio also
should be good.

> While the "usage of sman/FRICASsys in pipe" is more complex, it involves
> both socket IO and stdio, so the race happens there.

Well, clearly we should try to limit simultaneous use of socket IO
and stdio.

--
Waldek Hebisch

[Qian Yun]

On 5/31/24 00:38, Waldek Hebisch wrote:
> On Thu, May 30, 2024 at 07:43:30PM +0800, Qian Yun wrote:
>>
>>
>> On 5/29/24 22:51, Waldek Hebisch wrote:
>>>
>>> first build went fine, but a few later failed. Actually, it looks
>>> worse than previous version where probability of success looked
>>> higher.
>>>
>>> Yes, problem is because some .tex files are truncated. In one run it
>>> was 'ug10.tex', in few cases it was 'SEGBIND.tex'. In other cases
>>> I did not check the files but LaTeX error messuge indicated truncation.
>>
>> Can you try my patch from yesterday and see if this one helps.
>
> I could, but it takes time. And such tweaking is risky, while it
> may solve problem on my machine we risk breaking other. I would
> prefer to get closer to reasons so that we can be confident that
> book build really works.
>

I'll explain more. The output is truncated because some subprocess
of sman are killed before socket buffer is outputted.

The direction of IO goes like this:

FRICASsys <=> (forked child) sman <=> session <=> spadclient <=> stdio

When FRICASsys quits, the SpadServer socket and pty closes,
sman detects that and quits, causing SessionIOServer to be closed,
session detects that and quits, causing SessionServer to be closed,
spadclient detects that and quits.
(I ignored hypertex in this picture, it should also quit properly.)

Each process quits before processing all of its IO, so the output
will not be truncated.

The core idea is to detect socket shutdown, from "man recv":

When a stream socket peer has performed an orderly shutdown,
the return value will be 0.

I removed my previous workaround and applied this patch,
and I no longer have this truncation issue.

If this patch works on your side as well, I can improve the
details of this patch and upstream it.

>
> Concerning book, I did a few trials with version in the trunk, and
> it worked fine on each trial. That is too little to be sure,
> but is strong indication that trouble is due to recent changes.
>

Trunk version uses "FRICASsys" in pipe, that is fine.
My version uses "sman with FRICASsys" in pipe, which causes problem,
but I think it existed in the past as well.

>> The "sman -paste" invocation of "hypertex" does not have this problem
>> because it uses only socket IO to FRICASsys, it's purely sequential.
>
> Yes, socket I/O is free from worst races. Pure use of stdio also
> should be good.
>
>> While the "usage of sman/FRICASsys in pipe" is more complex, it involves
>> both socket IO and stdio, so the race happens there.
>
> Well, clearly we should try to limit simultaneous use of socket IO
> and stdio.
>

A minor correction: socket IO and stdio are sequential, but the signal
to kill the process is parallel, causing the race problem.
As I explained, this patch makes each process exits normally, instead
of being killed by sman simultaneously.

- Qian

[Waldek Hebisch]

^^^^^^

Do you mean "after"?

> will not be truncated.
>
> The core idea is to detect socket shutdown, from "man recv":
>
> When a stream socket peer has performed an orderly shutdown,
> the return value will be 0.
>
> I removed my previous workaround and applied this patch,
> and I no longer have this truncation issue.
>
> If this patch works on your side as well, I can improve the
> details of this patch and upstream it.

Trying 3 times it worked without failure. Will do more tests,
but seem to solve the truncation problem.

> > Concerning book, I did a few trials with version in the trunk, and
> > it worked fine on each trial. That is too little to be sure,
> > but is strong indication that trouble is due to recent changes.
> >
>
> Trunk version uses "FRICASsys" in pipe, that is fine.
> My version uses "sman with FRICASsys" in pipe, which causes problem,
> but I think it existed in the past as well.

Well, I agree that "sman in the pipe" almost surely was buggy.

> > > The "sman -paste" invocation of "hypertex" does not have this problem
> > > because it uses only socket IO to FRICASsys, it's purely sequential.
> >
> > Yes, socket I/O is free from worst races. Pure use of stdio also
> > should be good.
> >
> > > While the "usage of sman/FRICASsys in pipe" is more complex, it involves
> > > both socket IO and stdio, so the race happens there.
> >
> > Well, clearly we should try to limit simultaneous use of socket IO
> > and stdio.
> >
>
> A minor correction: socket IO and stdio are sequential, but the signal
> to kill the process is parallel, causing the race problem.
> As I explained, this patch makes each process exits normally, instead
> of being killed by sman simultaneously.

Yes, avoiding signal is good.

--
Waldek Hebisch

[Grégory Vanuxem]

Hello Qian,

Can you test this, start fricas with just Git cloned fricas and your
patch applied (ven. 31 mai 2024 21:07:10 CEST):

fricas -rl -noht

After, start HD with )hd, browse dilog operation, you have the
descriptions in a second window. Personally I move the second window a
little on the right.
After, call some routines and close FriCAS with ')q', HD hasn't the focus.

└─$ fricas -rl -noht
Checking for foreign routines
FRICAS="/home/greg/.local/lib/fricas/target/x86_64-linux-gnu"
spad-lib="/home/greg/.local/lib/fricas/target/x86_64-linux-gnu//lib/libspad.so"
foreign routines found
openServer result 0
FriCAS Computer Algebra System
Version: FriCAS 2024-04-15 built with sbcl 2.4.4
Timestamp: ven. 31 mai 2024 20:51:34 CEST
-----------------------------------------------------------------------------
Issue )copyright to view copyright notices.
Issue )summary for a summary of useful system commands.
Issue )quit to leave FriCAS and return to shell.
-----------------------------------------------------------------------------


(1) -> )hd
(1) -> dilog(0.3)

(1) 0.8893776242_8603873859_98408
Type: Complex(Float)
(2) -> )q

FriCAS is not instantly closed.

That happens also with Clozure CL.

Personally, I had to hard kill (- 15) sman or wait a long time. I
already encountered this with a message like this:

XIO: fatal IO error 4 (Interrupted system call) on X server ":0"
after 2715 requests (2715 known processed) with 0 events remaining.

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/150ffce1-80f3-461a-a4b7-984235bd0d8b%40gmail.com.

[Grégory Vanuxem]

Unapplying the src/hyper/event.c patch does not solve this issue for
information. So not relied apparently but no longer works.

[Ralf Hemmecke]

On 5/30/24 13:43, Qian Yun wrote:
> The "sman -paste" invocation of "hypertex" does not have this problem
> because it uses only socket IO to FRICASsys, it's purely sequential.

Hmm... cannot we also have something like this -paste option for the .ps
file generation?

We could also avoid the truncation problem in the following way.

- generate foo.input such that all the graphics commands will not be
executed by fricas. (That's just a little tweaking of a .awk script.)

- generate foo-graphics.input which it basically the same as with
the patch by Qian

- Send foo-graphics.input through fricas (as Qian's patch does, i.e.
with sman controlling everything). The (algebra) output is completely
ignored, but it will generate the .ps files.

- Send foo.input directly through FRICASsys (that is basically the same
as it is in current trunk).

That solves the problem at the cost of maybe running the non-graphics
commands twice. To avoid that "double" execution, we might introduce
a special xtc command that contains the preparation steps for graphics
commands that should appear in the .tex file together with output and
only execute those but ignore all the other xtc commands.
I have not actually checked whether there are such commands at all, but
maybe there are.)

Ralf

[Waldek Hebisch]

On Fri, May 31, 2024 at 09:23:32PM +0200, Grégory Vanuxem wrote:
> Hello Qian,
>
> Can you test this, start fricas with just Git cloned fricas and your
> patch applied (ven. 31 mai 2024 21:07:10 CEST):
>
> fricas -rl -noht
>
> After, start HD with )hd, browse dilog operation, you have the
> descriptions in a second window. Personally I move the second window a
> little on the right.
> After, call some routines and close FriCAS with ')q', HD hasn't the focus.
>

<snip>

> (1) -> )hd
> (1) -> dilog(0.3)
>
> (1) 0.8893776242_8603873859_98408
> Type: Complex(Float)
> (2) -> )q
>
> FriCAS is not instantly closed.
>
> That happens also with Clozure CL.

I see the following using clef and 'sin(x)' instead of 'dilog(0.3)':

- using version from Jan 23 both HyperDoc and FriCAS itself quit
- using versions between March 1 an my current developement version
FriCAS itself quits, but HyperDoc stays and I needed to click
Exit button to close HyperDoc
- using my developement version with patches to book build, including
latest patch FriCAS does not exit and I had to close main HyperDoc
window. Browse window vanished when I moved cursor into it.

It seems that most details does not matter, but it is important to
activate Browse.

--
Waldek Hebisch

[Waldek Hebisch]

On Fri, May 31, 2024 at 09:23:32PM +0200, Grégory Vanuxem wrote:

> Hello Qian,
>
> Can you test this, start fricas with just Git cloned fricas and your
> patch applied (ven. 31 mai 2024 21:07:10 CEST):
>
> fricas -rl -noht
>
> After, start HD with )hd, browse dilog operation, you have the
> descriptions in a second window. Personally I move the second window a
> little on the right.
> After, call some routines and close FriCAS with ')q', HD hasn't the focus.

<snip>

> (1) -> )hd
> (1) -> dilog(0.3)
>
> (1) 0.8893776242_8603873859_98408
> Type: Complex(Float)
> (2) -> )q
>
> FriCAS is not instantly closed.

AFAICS first regression is 1936e8639a992a5e46127bc5f4eb27662ddbbd47,
here HyperDoc started by ')hd' stays after FriCAS proper has quit,
07b4d99eb6c41443711654c65f0e8f66e7bf6c00 works OK.


--
Waldek Hebisch

[oldk1331]

>> Each process quits before processing all of its IO, so the output
> ^^^^^^
>
> Do you mean "after"?

To Waldek: yes.

> Hmm... cannot we also have something like this -paste option
> for the .ps file generation?

To Ralf: we can, but we also decided to decouple book generation
with hypertex pages generation.

About the other workaround, yes, we can workaround the problem
by running the input file twice, once for the graphics, once
for text output. But I believe my patch can handle the IO
correctly now, so workaround is not needed.

> After, call some routines and close FriCAS with ')q',
> HD hasn't the focus.

To Greg, thanks for reporting this.

This is a not a new issue, but it is newly exposed after my patch.
The problem is that in hyper/event.c, it uses "select" instead of
"sselect", which means when HyperDoc gets interrupted, the "select"
system call is not automatically retried, instead it goes to the
X11 event loop and blocked by it.


To Greg and Waldek, I'll open a new thread, posting a patch
to make HyperDoc exit properly.

- Qian

[Ralf Hemmecke]

On 6/1/24 06:56, oldk1331 wrote:
>> Hmm... cannot we also have something like this -paste option
>> for the .ps file generation?
>
> To Ralf: we can, but we also decided to decouple book generation
> with hypertex pages generation.

I somehow meant that there is another option that is passed to the
fricas call that works similar to -paste in terms of serializing IO as
you described for the -paste option, but works for generating .ps files.
Of course, the overall goal is decoupling the book and hyperdoc generation.

But if you have solved the problem in another way, also fine. I can only
test from Monday on. Which patch do you mean?

Ralf

[Qian Yun]

The patch was sent on May 29. Anyway, I've rebased
https://github.com/oldk1331/fricas/tree/new-ps-book

And the patch is updated a little at
https://github.com/oldk1331/fricas/commit/233020620ac6d83b5a8c497541c4d3e38148ceec

- Qian

[Waldek Hebisch]

On Sat, Jun 01, 2024 at 12:56:26PM +0800, oldk1331 wrote:
>
> > Hmm... cannot we also have something like this -paste option
> > for the .ps file generation?
>
> To Ralf: we can, but we also decided to decouple book generation
> with hypertex pages generation.

Well, I am not sure why Ralf wants decoupling, but for me it
is no-goal. In fact, since at core process is similar it is
natural to reuse as much code as possible. And reuse already
done computations. Which means doing what code in FriCAS
wiki is doing: produce both text and LaTeX output in the same
run.

> > After, call some routines and close FriCAS with ')q',
> > HD hasn't the focus.
>
> To Greg, thanks for reporting this.
>
> This is a not a new issue, but it is newly exposed after my patch.
> The problem is that in hyper/event.c, it uses "select" instead of
> "sselect", which means when HyperDoc gets interrupted, the "select"
> system call is not automatically retried, instead it goes to the
> X11 event loop and blocked by it.
>
>
> To Greg and Waldek, I'll open a new thread, posting a patch
> to make HyperDoc exit properly.

It seems that all discovered issues are resolved now. It is not
clear to me it is worth splitting disscusion into separate threads.
Concerning splitting patches, I think it would be natural to
commit chages to HyperDoc program and graphics as one piece (assuming
they are compatible with current way of building book) and changes
to FriCAS book and src/doc Makefile as another piece.

I have some small comment to changes that you did in .htex files.
Namely on page 876 subsection 'abbrs' you changed 'limited to 7
characters' into 'limited to 8 characters'. The exact statement
is that _category_ abbreviations are limited to 7 characters
and domain/package abbreviations can use 8 characters.
I would probably skip 'limited to' part here.

Abbreviations are mentioned in other places and we probably should
put precise statement somewhere, but description of 'Browse' does
not look like good place. And such change is independent of other
changes that you are making.

--
Waldek Hebisch

[Qian Yun]

On 6/2/24 02:24, Waldek Hebisch wrote:
> On Sat, Jun 01, 2024 at 12:56:26PM +0800, oldk1331 wrote:
>>
>>> Hmm... cannot we also have something like this -paste option
>>> for the .ps file generation?
>>
>> To Ralf: we can, but we also decided to decouple book generation
>> with hypertex pages generation.
>
> Well, I am not sure why Ralf wants decoupling, but for me it
> is no-goal. In fact, since at core process is similar it is
> natural to reuse as much code as possible. And reuse already
> done computations. Which means doing what code in FriCAS
> wiki is doing: produce both text and LaTeX output in the same
> run.

There's \htonly and \texonly, so it has to run twice.

>
> It seems that all discovered issues are resolved now. It is not
> clear to me it is worth splitting disscusion into separate threads.
> Concerning splitting patches, I think it would be natural to
> commit chages to HyperDoc program and graphics as one piece (assuming
> they are compatible with current way of building book) and changes
> to FriCAS book and src/doc Makefile as another piece.

I think the HyperDoc patch is relatively independent and should
commit separately, after all it solves Greg's problem.

As for the "file truncation problem", or let's rephrase it as
"sman in pipe problem". (I just noticed it is mentioned in the
"doc/fricas.1" man page.) One thing I'm still not very sure:
what's the purpose of "wait_for_client_read" in "sread"?
In which context is it useful? I'm inclined to remove it.

Another thing that I'm more certain: since in session.c
we use "session_socket_mask" instead of "socket_mask",
"sread" ("get_int" etc) can have unexpected side effects,
I need to take a deeper look to see if there's a better solution.

A final problem is, should "sman" restart "viewman" if it exits?
(https://github.com/fricas/fricas/pull/101)
If "viewman" exits normally, "sman" should not restart it,
otherwise "sman" (and the whole session) will not quit to terminal.

> I have some small comment to changes that you did in .htex files.
> Namely on page 876 subsection 'abbrs' you changed 'limited to 7
> characters' into 'limited to 8 characters'. The exact statement
> is that _category_ abbreviations are limited to 7 characters
> and domain/package abbreviations can use 8 characters.
> I would probably skip 'limited to' part here.
>
> Abbreviations are mentioned in other places and we probably should
> put precise statement somewhere, but description of 'Browse' does
> not look like good place. And such change is independent of other
> changes that you are making.
>

Branch rebased and updated.

- Qian

[Qian Yun]

On 6/2/24 08:29, Qian Yun wrote:
>
>
> On 6/2/24 02:24, Waldek Hebisch wrote:
>
>>
>> It seems that all discovered issues are resolved now.  It is not
>> clear to me it is worth splitting disscusion into separate threads.
>> Concerning splitting patches, I think it would be natural to
>> commit chages to HyperDoc program and graphics as one piece (assuming
>> they are compatible with current way of building book) and changes
>> to FriCAS book and src/doc Makefile as another piece.
>
> I think the HyperDoc patch is relatively independent and should
> commit separately, after all it solves Greg's problem.
>
> As for the "file truncation problem", or let's rephrase it as
> "sman in pipe problem". (I just noticed it is mentioned in the
> "doc/fricas.1" man page.)  One thing I'm still not very sure:
> what's the purpose of "wait_for_client_read" in "sread"?
> In which context is it useful?  I'm inclined to remove it.

To answer my own question: we should remove "wait_for_client_read".
This function is intended for the scenario "client closes connection
and tries to connect again." But for Unix domain socket,
they can't be closed unexpectedly, unlike TCP connection over Internet.
And since the reconnection needs to exchange PID and 'purpose',
but there's no code to handle this logic on the client side.

So when "fricas_read" returns 0, it means the other end of the
unix domain socket is closed, and we should prepare for proper shutdown.

> Another thing that I'm more certain: since in session.c
> we use "session_socket_mask" instead of "socket_mask",
> "sread" ("get_int" etc) can have unexpected side effects,
> I need to take a deeper look to see if there's a better solution.

In session.c, we should pay extra attention when the socket peer
closes, I think these cases are already considered.

> A final problem is, should "sman" restart "viewman" if it exits?
> (https://github.com/fricas/fricas/pull/101)
> If "viewman" exits normally, "sman" should not restart it,
> otherwise "sman" (and the whole session) will not quit to terminal.

I modify "sman" to check for the return status of its subprocesses,
and do not restart a child process if it exits successfully.

Branch rebased and updated. Please review and test.
https://github.com/oldk1331/fricas/tree/new-ps-book

- Qian

[Qian Yun]

Any remaining issues and questions?

Can we move forward to include these patches?

- Qian

[Waldek Hebisch]

On Sun, Jun 09, 2024 at 01:55:36PM +0800, Qian Yun wrote:
> Any remaining issues and questions?
>
> Can we move forward to include these patches?

Personally I would keep 'wait_for_client_read' just now.
Our processes can exit and at least in principle restart
and reconnect. So some way of handling reconnection is
needed. The code indeed looks like it does not work,
but keeping it probably is safer.

In longer term we should create diagram of expected interactions
between processes and verify that communication between
processes works as desired. But this is larger effort
and will take time. For now I prefer "most conservative"
solution. As I wrote in another mail, I would like to do
new release in June and would like to include the changes.

--
Waldek Hebisch

[Ralf Hemmecke]

Sorry, but I am still unhappy with Qian's stuff.

I just compiled with -j1 and it hangs here.

$tail -f /dev/shm/hemmecke/fricas/b/src/doc/tmp/ug07.spool
-- \end{TeXOutput}

-- \begin{MessageOutput}
Type:
TwoDimensionalViewport
-- \end{MessageOutput}
-- \end{xtcnooutput}
-- \includespadgraph{2DppcB}
-- \begin{xtcnooutput}
write(%, "tmp/2DppcB", "postscript"); close(%)

^C

and leaving the following processes (see below).

If this becomes the release, then we will ship a broken version that
only finishes compiling when you are lucky.

I try to investigate the patches now. Some seem acceptable, but I think
piping the generated (from .htex) .input files through fricas (i.e.
sman) instead of directly to FRICASsys causes all the problems with
hanging compilation or truncation of output.

Ralf

hemmecke 817003 0.0 0.0 10324 3712 pts/4 S+ 21:54 0:00
/bin/bash -c if grep "^write.*postscript" tmp/ug07.input; then \ ?(unset
DAASE; FRICAS=/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu; export
FRICAS; \ ??echo ')read tmp/ug07.input' | FRICAS_INITFILE='' xvfb-run -a
-s '-screen 0 1024x768x24' \
??/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/bin/fricas -noht
-noclef > tmp/ug07.spool); \ else \ ?(unset DAASE;
FRICAS=/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu; export
FRICAS; \ ??echo ')read tmp/ug07.input' | FRICAS_INITFILE='' \
??/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/bin/fricas -nosman
> tmp/ug07.spool); \ fi
hemmecke 817013 0.0 0.0 10324 1828 pts/4 S+ 21:54 0:00
/bin/bash -c if grep "^write.*postscript" tmp/ug07.input; then \ ?(unset
DAASE; FRICAS=/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu; export
FRICAS; \ ??echo ')read tmp/ug07.input' | FRICAS_INITFILE='' xvfb-run -a
-s '-screen 0 1024x768x24' \
??/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/bin/fricas -noht
-noclef > tmp/ug07.spool); \ else \ ?(unset DAASE;
FRICAS=/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu; export
FRICAS; \ ??echo ')read tmp/ug07.input' | FRICAS_INITFILE='' \
??/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/bin/fricas -nosman
> tmp/ug07.spool); \ fi
hemmecke 817018 0.0 0.0 2896 1792 pts/4 S+ 21:54 0:00
/bin/sh /usr/bin/xvfb-run -a -s -screen 0 1024x768x24
/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/bin/fricas -noht -noclef
hemmecke 817152 0.0 0.0 2828 1664 pts/4 S+ 21:54 0:00
/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/bin/sman -noht
-noclef -ws /dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/bin/FRICASsys
hemmecke 817160 0.0 0.2 24587916 75648 pts/10 Ssl+ 21:54 0:00
/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/bin/FRICASsys --
/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/bin/
hemmecke 817161 0.0 0.0 2820 1280 pts/4 S+ 21:54 0:00
/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/lib/session
hemmecke 817162 99.7 0.0 2812 1408 pts/4 R+ 21:54 14:55
/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/lib/spadclient
hemmecke 817163 0.0 0.0 4492 1920 pts/4 S+ 21:54 0:00
/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/lib/viewman
hemmecke 817209 0.0 0.0 2828 644 pts/4 S+ 21:54 0:00
/dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/bin/sman -noht
-noclef -ws /dev/shm/hemmecke/fricas/b/target/x86_64-linux-gnu/bin/FRICASsys

[Waldek Hebisch]

On Mon, Jun 10, 2024 at 10:23:18PM +0200, Ralf Hemmecke wrote:
> Sorry, but I am still unhappy with Qian's stuff.
>
> I just compiled with -j1 and it hangs here.
>
> $tail -f /dev/shm/hemmecke/fricas/b/src/doc/tmp/ug07.spool
> -- \end{TeXOutput}
>
> -- \begin{MessageOutput}
> Type:
> TwoDimensionalViewport
> -- \end{MessageOutput}
> -- \end{xtcnooutput}
> -- \includespadgraph{2DppcB}
> -- \begin{xtcnooutput}
> write(%, "tmp/2DppcB", "postscript"); close(%)
>
> ^C
>
> and leaving the following processes (see below).
>
> If this becomes the release, then we will ship a broken version that only
> finishes compiling when you are lucky.
>
> I try to investigate the patches now. Some seem acceptable, but I think
> piping the generated (from .htex) .input files through fricas (i.e. sman)
> instead of directly to FRICASsys causes all the problems with hanging
> compilation or truncation of output.

Quite likely.

--
Waldek Hebisch

[Ralf Hemmecke]

Dear Qian,

I studied your commit

"fix display issue of "output" in book section 6.19"

https://github.com/fricas/fricas/commit
/0ab9749f0faff760bc998235436584553b67b3f3

That looks quite good.

I would still like that the change from \xtc to \noOutputXtc in
REGSET.htex is not included.

https://github.com/fricas/fricas/commit/0ab9749f0faff760bc998235436584553b67b3f3#diff-36d4a3eafe860f579e9d7b4a2c0e732dd439947d53137c78aa5bae8d5a4330f5L381

The reason is that I would still like to see the type included in the
book even though the value is hidden.

I've not yet checked whether

)set output length 88

gives good output for all cases. I'll check that later.

What I do not quite understand is the need for tex2tex.awk (no
documentation in that file). The code seems to tell me that you would
like to combine lines like

\begin{AlgebraOutput}
blah
\end{algebraOutput}
\begin{AlgebraOutput}
blah2
\end{AlgebraOutput}

into

\begin{AlgebraOutput}
blah
blah2
\end{AlgebraOutput}

so that there will be no white lines in the book.

I am not sure that this is the best solution, but I have nothing better
myself at the moment.

Ralf

[Qian Yun]

On 6/11/24 02:43, Waldek Hebisch wrote:
> On Sun, Jun 09, 2024 at 01:55:36PM +0800, Qian Yun wrote:
>> Any remaining issues and questions?
>>
>> Can we move forward to include these patches?
>
> Personally I would keep 'wait_for_client_read' just now.
> Our processes can exit and at least in principle restart
> and reconnect. So some way of handling reconnection is
> needed. The code indeed looks like it does not work,
> but keeping it probably is safer.

Our code already can handle client reconnect.
The server socket is put into "select" main loop and can
handle new connections.

This 'wait_for_client_read' happens at "read", it may even
happen on client side!

On the other hand, it's hard to imagine that a unix domain
socket connection get disconnected unexpectedly. In this
case, we should consider shutting down process, instead of
trying to recover from it.

Off topic a little on Internet connection: there's code
suggest we can use Internet connection instead of unix domain
socket connection. But I've find in many places the code
relies on unix domain socket specifically, such as EPIPE
and detection socket peer shutdown. So shall we cleanup
those Internet connection parts?

> In longer term we should create diagram of expected interactions
> between processes and verify that communication between
> processes works as desired. But this is larger effort
> and will take time. For now I prefer "most conservative"
> solution. As I wrote in another mail, I would like to do
> new release in June and would like to include the changes.
>

As a compromise, I can keep 'wait_for_client_read' in 'sread'
and only replace the problematic 'sread' calls.

- Qian

[Qian Yun]

Can you reprodce this?

We can see that 'spadclient' is taking 100% CPU.
Which is stange.

I'd like to see a "strace" output of this process.

- Qian

[Qian Yun]

On 6/11/24 05:49, Ralf Hemmecke wrote:
> Dear Qian,
>
> I studied your commit
>
> "fix display issue of "output" in book section 6.19"
>
> https://github.com/fricas/fricas/commit
> /0ab9749f0faff760bc998235436584553b67b3f3
>
> That looks quite good.
>
> I would still like that the change from \xtc to \noOutputXtc in
> REGSET.htex is not included.
>
> https://github.com/fricas/fricas/commit/0ab9749f0faff760bc998235436584553b67b3f3#diff-36d4a3eafe860f579e9d7b4a2c0e732dd439947d53137c78aa5bae8d5a4330f5L381
>
> The reason is that I would still like to see the type included in the
> book even though the value is hidden.

The reason to discard its output is, as the text bellow described,
it will output a large portion of debugging output.

> I've not yet checked whether
>
> )set output length 88
>
> gives good output for all cases. I'll check that later.
>
> What I do not quite understand is the need for tex2tex.awk (no
> documentation in that file). The code seems to tell me that you would
> like to combine lines like
>
> \begin{AlgebraOutput}
> blah
> \end{algebraOutput}
> \begin{AlgebraOutput}
> blah2
> \end{AlgebraOutput}
>
> into
>
> \begin{AlgebraOutput}
> blah
> blah2
> \end{AlgebraOutput}
>
> so that there will be no white lines in the book.
>
> I am not sure that this is the best solution, but I have nothing better
> myself at the moment.
>
> Ralf
>

Yes, its intention is to combine those lines to discard white lines
in between. I'll add some comments.

Due to the workflow of awk, it seems that I can not do this in a single
awk script, and have to add this new script.

- Qian

[Ralf Hemmecke]

>> I would still like that the change from \xtc to \noOutputXtc in
>> REGSET.htex is not included.
>>
>> https://github.com/fricas/fricas/commit/0ab9749f0faff760bc998235436584553b67b3f3#diff-36d4a3eafe860f579e9d7b4a2c0e732dd439947d53137c78aa5bae8d5a4330f5L381
>>
>> The reason is that I would still like to see the type included in the
>> book even though the value is hidden.
>
> The reason to discard its output is, as the text bellow described,
> it will output a large portion of debugging output.

Oh, maybe I understand now. That debugging output is a problem after
your modification. Otherwise the semicolon at the end at least removes
the output value for that computation. Why suddenly the debugging output
is a problem, I do not yet understand.

> Yes, its intention is to combine those lines to discard white lines
> in between.  I'll add some comments.

> Due to the workflow of awk, it seems that I can not do this in a single
> awk script, and have to add this new script.

Yes, it maybe difficult to do it in one pass. I still don't like
tex2tex.awk. I'll think about it.

Ralf

[Qian Yun]

On 6/11/24 16:24, Ralf Hemmecke wrote:
>>> I would still like that the change from \xtc to \noOutputXtc in
>>> REGSET.htex is not included.
>>>
>>> https://github.com/fricas/fricas/commit/0ab9749f0faff760bc998235436584553b67b3f3#diff-36d4a3eafe860f579e9d7b4a2c0e732dd439947d53137c78aa5bae8d5a4330f5L381
>>>
>>> The reason is that I would still like to see the type included in the
>>> book even though the value is hidden.
>>
>> The reason to discard its output is, as the text bellow described,
>> it will output a large portion of debugging output.
>
> Oh, maybe I understand now. That debugging output is a problem after
> your modification. Otherwise the semicolon at the end at least removes
> the output value for that computation. Why suddenly the debugging output
> is a problem, I do not yet understand.
>

The problem is that the debugging output does not have a newline
at the end, causing "-- \begin{AlgebraOutput}" to not start at
beginning of a line.

Also the debugging output is one very long line, it would not
fit well on the book.

- Qian

[Ralf Hemmecke]

> The problem is that the debugging output does not have a newline
> at the end, causing "-- \begin{AlgebraOutput}" to not start at
> beginning of a line.

Oh. I must check what actually happens without tex2tex.awk. Perhaps can
do so tonight.

> Also the debugging output is one very long line, it would not
> fit well on the book.

Of course, debugging output should not be in the book. Before your
modification that would have been automatically removed and only the
type infomation survived.

Ralf

[Qian Yun]

This can be considered as a sub-issue of the "sman in pipe"
issue.

When running in terminal, Ctrl-D is interpreted as EOF,
causing "read" to return 0, this does not mean the input
is finished.

When running in pipe, EOF ("read" returns 0) means the
end of input, we should remove fd 0 from "select",
otherwise we get this problem of 100% CPU in "spadclient".

- Qian

[Tim Daly]

If you're interested in either expanding the graphics examples from the book

or testing the graphics intereface you might find this useful:

http://axiom-developer.org/axiom-website/bookvol8.1.pdf

It contains code and example graphics from

CRC Standard Curves and Surfaces

https://www.amazon.com/Standard-Surfaces-Advances-Applied-Mathematics/dp/0849301963

and Pasta By Design

https://www.amazon.com/Pasta-Design-George-L-Legendre/dp/0500515808