GCL and ECL build status regarding to recent patches

[Qian Yun]

There are patches for GCL and ECL recently, and I wonder
if there are more coming?

I didn't take a deeper look into the failures, but this is
build status from my side:

GCL-2.6.13_pre fails to build on Linux with git HEAD.

ECL fails to build on macOS with git HEAD.

- Qian

[Dima Pasechnik]

We use 2 patches for ECL/fricas:
https://github.com/sagemath/sage/tree/develop/build/pkgs/fricas/patches



> - Qian
>
> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/3e7ef019-1507-c015-b061-eaaa6116ea24%40gmail.com.

[Waldek Hebisch]

On Sat, Jul 23, 2022 at 12:51:41PM +0800, Qian Yun wrote:
> There are patches for GCL and ECL recently, and I wonder
> if there are more coming?

Currently no. The patches fix things that I know about
and know how to fix.

> I didn't take a deeper look into the failures, but this is
> build status from my side:
>
> GCL-2.6.13_pre fails to build on Linux with git HEAD.

I commited GCL patches should make porting to newere GCL-s
easier, but does not solve main problems. Also, I commited
only parts compatible with older versions.

> ECL fails to build on macOS with git HEAD.

Could you say more about this? Main change is that
FriCAS now generates extern declarations for C
functions that we call via FFI. That is supposed
to replace Sage patch (which IMO is problematic).

Anyway, with the patches FriCAS build for me
using gcl-2.6.12, ecl-16.1.2 and ecl-21.2.1.
So with ECL any problem most likely is OS or
C compiler. Of course, C compiler can emit
warnings whenever it wants and if one inserts
'-Werror' to default C options it may easily
lead to spurious failures.

BTW: ATM I find Github CI reports kind of useless,
they does not tell me more than what I already
know due to Murphy: there is failure somewhere.
Up to now I was unable to see build logs to get
more info. One answer to stackoverflow question
claimed that one has to authenticate as project
admin to see logs but I saw no docs how this is
supposed to work.

--
Waldek Hebisch

[Dima Pasechnik]

well, yes, one has to log in to GitHub to see full logs of the Actions (CI) runs.

although with a bit of extra effort one can upload them (from Actions runs) somewhere outside of GitHub.

--
                              Waldek Hebisch

--
You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/20220723124649.GA30366%40fricas.math.uni.wroc.pl.

[Qian Yun]

On 7/23/22 20:46, Waldek Hebisch wrote:
>> GCL-2.6.13_pre fails to build on Linux with git HEAD.
>
> I commited GCL patches should make porting to newere GCL-s
> easier, but does not solve main problems. Also, I commited
> only parts compatible with older versions.

OK, not high priority for GCL-2.6.13_pre, but the error
happens when first trying to generate database:

======= it stops somewhere randomly around "JGB". no other error message
)compile JGB.spad

Compiling FriCAS source code from file
/tmp/fricas/src/algebra/JGB.spad using old system compiler.
JGB abbreviates package JetGroebner
------------------------------------------------------------------------
initializing NRLIB JGB for JetGroebner
compiling into NRLIB JGB
****** Domain: JB already in scope
****** Domain: P already in scope

>> System error:


(1) -> mv: cannot stat 'category.daase': No such file or directory
mv: cannot stat 'compress.daase': No such file or directory
mv: cannot stat 'interp.daase': No such file or directory
mv: cannot stat 'operation.daase': No such file or directory
make[3]: *** [Makefile:620: stamp-db] Error 1
make[3]: Leaving directory '/tmp/fricas/src/algebra'

>> ECL fails to build on macOS with git HEAD.
>
> Could you say more about this? Main change is that
> FriCAS now generates extern declarations for C
> functions that we call via FFI. That is supposed
> to replace Sage patch (which IMO is problematic).

Glad to know you are trying to fix this issue.
I'm attaching the failure log and generated
fricas-lisp.c.

> Anyway, with the patches FriCAS build for me
> using gcl-2.6.12, ecl-16.1.2 and ecl-21.2.1.
> So with ECL any problem most likely is OS or
> C compiler. Of course, C compiler can emit
> warnings whenever it wants and if one inserts
> '-Werror' to default C options it may easily
> lead to spurious failures.
>
> BTW: ATM I find Github CI reports kind of useless,
> they does not tell me more than what I already
> know due to Murphy: there is failure somewhere.
> Up to now I was unable to see build logs to get
> more info. One answer to stackoverflow question
> claimed that one has to authenticate as project
> admin to see logs but I saw no docs how this is
> supposed to work.
>

For CI logs, if you are logged into GitHub, you will be able
to see the logs. But recent CI status for macOS build
is "canceled", which is strange. I'm restarting them
and one has passed (they are testing sbcl on macOS after all).

- Qian

[Waldek Hebisch]

On Sat, Jul 23, 2022 at 09:16:11PM +0800, Qian Yun wrote:
>
> >>ECL fails to build on macOS with git HEAD.
> >
> >Could you say more about this? Main change is that
> >FriCAS now generates extern declarations for C
> >functions that we call via FFI. That is supposed
> >to replace Sage patch (which IMO is problematic).
>
> Glad to know you are trying to fix this issue.
> I'm attaching the failure log and generated
> fricas-lisp.c.

Actually, here relevant file is fricas-lisp.eclh

AFAICS we need slight change to fricas-lisp.lisp:

--- ../trunk.cur/src/lisp/fricas-lisp.lisp 2022-07-21 15:46:46.797054021 +0000
+++ fricas-lisp.lisp 2022-07-23 15:51:19.552480068 +0000
@@ -571,8 +572,8 @@
(let ((l-ret (c-type-to-ffi return-type))
wrapper)
`(progn
- ,(ext:with-backend :c/c++
- `(FFI:clines ,(make_extern return-type c-name arguments)))
+ (ext:with-backend :c/c++
+ (FFI:clines ,(make_extern return-type c-name arguments)))
,(if strs
(let* ((sym (gensym))
(wargs (mapcar #'car fargs))

> For CI logs, if you are logged into GitHub, you will be able
> to see the logs. But recent CI status for macOS build
> is "canceled", which is strange. I'm restarting them
> and one has passed (they are testing sbcl on macOS after all).

Hmm, I saw in Github message that they no longer want to support
macos-10.15. They turned of support for a day when I made 3 commits,
maybe that is the reason.

--
Waldek Hebisch

[Waldek Hebisch]

On Sat, Jul 23, 2022 at 09:16:11PM +0800, Qian Yun wrote:
>

> >>ECL fails to build on macOS with git HEAD.
> >
> >Could you say more about this? Main change is that
> >FriCAS now generates extern declarations for C
> >functions that we call via FFI. That is supposed
> >to replace Sage patch (which IMO is problematic).
>
> Glad to know you are trying to fix this issue.
> I'm attaching the failure log and generated
> fricas-lisp.c.

AFAICS to trigger error it is enought to set ECL
C compiler options like:

(setf c:*user-cc-flags* "-Werror -Wimplicit-function-declaration")

With such setting I got error also on Linux. I have now
commited fix for the problem. However, IMO '-Werror' is
inapropriate for portable programs. C compilers generate
warnings for a lot of reasons, some serious (like the
implict declaration problem), some more of sort 'look
at this code, maybe you want to rewrite it'. If program
is tied to specific compiler version, then one can tune
warning options to disable unwanted ones. But that
is inpractical if code is supposed to compile with
wide range of compilers...

--
Waldek Hebisch

[Qian Yun]

On 7/24/22 00:09, Waldek Hebisch wrote:
> On Sat, Jul 23, 2022 at 09:16:11PM +0800, Qian Yun wrote:
>>
>>>> ECL fails to build on macOS with git HEAD.
>>>
>>> Could you say more about this? Main change is that
>>> FriCAS now generates extern declarations for C
>>> functions that we call via FFI. That is supposed
>>> to replace Sage patch (which IMO is problematic).
>>
>> Glad to know you are trying to fix this issue.
>> I'm attaching the failure log and generated
>> fricas-lisp.c.
>
> Actually, here relevant file is fricas-lisp.eclh
>
> AFAICS we need slight change to fricas-lisp.lisp:
>

Thanks, now git HEAD builds with ECL on macOS on my side.
I can close https://github.com/fricas/fricas/issues/59.

>
>> For CI logs, if you are logged into GitHub, you will be able
>> to see the logs. But recent CI status for macOS build
>> is "canceled", which is strange. I'm restarting them
>> and one has passed (they are testing sbcl on macOS after all).
>
>
> Hmm, I saw in Github message that they no longer want to support
> macos-10.15. They turned of support for a day when I made 3 commits,
> maybe that is the reason.
>

Thanks for the info, I have updated the macOS version in CI script.

- Qian

[Waldek Hebisch]

On Sat, Jul 23, 2022 at 09:16:11PM +0800, Qian Yun wrote:
>
>

> On 7/23/22 20:46, Waldek Hebisch wrote:
> >>GCL-2.6.13_pre fails to build on Linux with git HEAD.
> >
> >I commited GCL patches should make porting to newere GCL-s
> >easier, but does not solve main problems. Also, I commited
> >only parts compatible with older versions.
>
> OK, not high priority for GCL-2.6.13_pre, but the error
> happens when first trying to generate database:
>
> ======= it stops somewhere randomly around "JGB". no other error message
> )compile JGB.spad
>
> Compiling FriCAS source code from file
> /tmp/fricas/src/algebra/JGB.spad using old system compiler.
> JGB abbreviates package JetGroebner
> ------------------------------------------------------------------------
> initializing NRLIB JGB for JetGroebner
> compiling into NRLIB JGB
> ****** Domain: JB already in scope
> ****** Domain: P already in scope
>
> >> System error:
>

I have re-tried GCL 2.6.13_pre on Linux. I see similar error.
Running offending part by hand with

)lisp (si::use-fast-links nil)
)set break break

I got more sensible error message. Apparrently GCL is running
out of memory to load code. But this is strange: GCL has allocated
44G heap and there is 2.9G resident, so there should be plenty
of memory. And at that stage there is about 4.5 M od code
to load. There is also previously compiled code in the same image,
but this should not be very big.
in
--
Waldek Hebisch

[Qian Yun]

On 7/24/22 10:41, Waldek Hebisch wrote:
>
> I have re-tried GCL 2.6.13_pre on Linux. I see similar error.
> Running offending part by hand with
>
> )lisp (si::use-fast-links nil)
> )set break break
>
> I got more sensible error message. Apparrently GCL is running
> out of memory to load code. But this is strange: GCL has allocated
> 44G heap and there is 2.9G resident, so there should be plenty
> of memory. And at that stage there is about 4.5 M od code
> to load. There is also previously compiled code in the same image,
> but this should not be very big.
> in

Err, did you finish this email?

I did some lookup in this direction, and found Environment Variable
GCL_MEM_MULTIPLE.

It means how many memory GCL will use. However, a smaller
value (0.1) can make compile go further, while bigger value (0.5) can't.

(Of course, the effect of this variable is dependent on system memory,
so YMMV.)

With GCL_MEM_MULTIPLE=0.1, now build fails at:

======
U64Int is already explicitly exposed in frame initial
U64Int will be automatically loaded when needed from
/tmp/fricas/src/algebra/U64INT.NRLIB/U64INT

)lisp (make-databases nil nil)


"building operation.daase"
"building category.daase"

>> System error:


(1) -> mv: cannot stat 'category.daase': No such file or directory
mv: cannot stat 'compress.daase': No such file or directory

======

- Qian

[Waldek Hebisch]

I had to go down to GCL_MEM_MULTIPLE=0.07 (and GCL_MEM_MULTIPLE=0.02
works the same) to get this. AFAICS the error is becuse interpsys
is unable to evaluate BasicType. The trouble is that DATABASE property
on BasicType is NIL. But it should be set by compilation. Apparently,
running LOCALDATABASE by hand sets DATABASE property and compilation
calls LOCALDATABASE, so it should be set. But it is not...

--
Waldek Hebisch

[Waldek Hebisch]

ATM only one patch is needed to build using 2.6.13pre and FriCAS
from distribution tarball:

--- ../trunk/src/lisp/fricas-lisp.lisp 2022-07-23 20:46:47.697097794 +0200
+++ ../dist/src/lisp/fricas-lisp.lisp 2022-07-25 17:25:07.328035488 +0200
@@ -986,7 +986,7 @@
(let ((ns (namestring name)))
(if (and (consp (pathname-directory name))
(eq (car (pathname-directory name))
- #-:GCL :absolute #+:GCL :root))
+ :absolute ))
ns
(concatenate 'string (get-current-directory) "/" ns))))
#+:cmu

This patch is incompatible with earlier versions of GCL, but I think
I have now way to do the same in backwards compatible way.

OTOH testsuite has several failures.

--
Waldek Hebisch

[Waldek Hebisch]

On Mon, Jul 25, 2022 at 04:21:09PM +0000, Waldek Hebisch wrote:
>
> ATM only one patch is needed to build using 2.6.13pre and FriCAS
> from distribution tarball:
>
> --- ../trunk/src/lisp/fricas-lisp.lisp 2022-07-23 20:46:47.697097794 +0200
> +++ ../dist/src/lisp/fricas-lisp.lisp 2022-07-25 17:25:07.328035488 +0200
> @@ -986,7 +986,7 @@
> (let ((ns (namestring name)))
> (if (and (consp (pathname-directory name))
> (eq (car (pathname-directory name))
> - #-:GCL :absolute #+:GCL :root))
> + :absolute ))
> ns
> (concatenate 'string (get-current-directory) "/" ns))))
> #+:cmu
>
> This patch is incompatible with earlier versions of GCL, but I think
> I have now way to do the same in backwards compatible way.
>
> OTOH testsuite has several failures.

I also tried to bump safety in interp-proclaims.lisp:

diff -ru ../trunk/src/interp/interp-proclaims.lisp ../dist/src/interp/interp-proclaims.lisp
--- ../trunk/src/interp/interp-proclaims.lisp 2021-08-01 14:26:15.421744317 +0200
+++ ../dist/src/interp/interp-proclaims.lisp 2022-07-25 19:18:31.141460429 +0200
@@ -1,4 +1,4 @@
#+:GCL
(progn
(eval-when (:execute :compile-toplevel :load-toplevel)
- (proclaim '(optimize (safety 1) (debug 3)))))
+ (proclaim '(optimize (safety 3) (debug 3)))))

With this setting I am again getting _huge_ memory use (one of test
processes had 19G resident when I killed the test run). With
GCL_MEM_MULTIPLE=0.03 tests run OK. This run (at safety 3)
produced only one error (in errortrap.input), which happens also
with older versions. Unfortunatly, at safety 3 GCL is quite
slow (more than 22 times CPU time than sbcl, more than 26 times
real time on parallel testsuite run).

--
Waldek Hebisch