pure lisp build (build without C)

[Qian Yun]

Actually it is not that hard to build FriCAS using only Lisp compiler
(without C compiler).

Of course, this means there will be no hyperdoc, graphics, clef, sman.
Just FriCASsys, just algebra.

This can be useful, if you are using the emacs/texmacs/jupyter/sagemath
interface.

This can also be useful when porting FriCAS to different Lisp. No need
to worry about FFI when doing pure Lisp build.

The following diff is just a quick hack to achieve this.
You use this to test the build: "make -jN CC=not-exist all-fricassys"

Waldek, if you think such "--without-c" build option is useful, I can
develop it into a full patch.

- Qian

============

diff --git a/src/Makefile.in b/src/Makefile.in
index cf3eb2e1..9ec5fe40 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -35,7 +35,7 @@ all-doc: all-hyper all-fricassys
cd doc && ${MAKE}
all-lib:
@cd lib && ${MAKE}
-all-lisp: all-lib
+all-lisp:
cd lisp && $(MAKE) all-lisp
all-boot: all-lisp
@cd boot && ${MAKE}
@@ -45,7 +45,7 @@ all-interpsys: all-boot

all-fricassys: all-asq
cd interp && $(MAKE) all-fricassys
-all-algebra: all-interpsys util-ht
+all-algebra: all-interpsys
@ echo 30b running find-algebra-files
(cd $(srcdir)/algebra; \
. ../scripts/find-algebra-files) \
diff --git a/src/etc/Makefile.in b/src/etc/Makefile.in
index 4d077a01..54ed193a 100644
--- a/src/etc/Makefile.in
+++ b/src/etc/Makefile.in
@@ -56,7 +56,6 @@ bin_PROGRAMS = asq$(EXEEXT)

$(fricas_target_bindir)/asq$(EXEEXT): $(srcdir)/asq.c $(fricas_c_macros)
@echo 4 making $@ from $<
- ${CC} ${CCF} $(fricas_includes) -o $@ $<

$(fricas_target_libdir)/summary: $(srcdir)/summary
cp $< $@
diff --git a/src/lisp/Makefile.in b/src/lisp/Makefile.in
index 30f61509..1196d9a9 100644
--- a/src/lisp/Makefile.in
+++ b/src/lisp/Makefile.in
@@ -102,7 +102,7 @@ fricas-config.lisp:
if [ x$(fricas_has_directory_operations) = xyes ] ; then \
echo '(if (not (member :fricas_has_remove_directory' \
' *features*)) ' \
- '(push :fricas_has_remove_directory *features*))' \
+ '(push :fricas_has_no_remove_directory *features*))' \
>> $@ ; \
fi

[Ralf Hemmecke]

Hi Qian,

> Waldek, if you think such "--without-c" build option is useful, I can
> develop it into a full patch.

You don't speak to me, but I think it would be quite interesting to
separate the lisp sfuff from the C-stuff.

Graphics is an issue, but some people might be content with the other
parts of the algebra. In the long run in makes anyway sense to come up
with better graphics. FriCAS should simply be able to connect to a
number of external graphics viewers.

I would not immediately throw C away, but having a configure option to
just build FRICASsys would be quite useful.

Ralf

[Waldek Hebisch]

On Thu, Jun 30, 2022 at 07:29:14PM +0800, Qian Yun wrote:
> Actually it is not that hard to build FriCAS using only Lisp compiler
> (without C compiler).
>
> Of course, this means there will be no hyperdoc, graphics, clef, sman.
> Just FriCASsys, just algebra.
>
> This can be useful, if you are using the emacs/texmacs/jupyter/sagemath
> interface.
>
> This can also be useful when porting FriCAS to different Lisp. No need
> to worry about FFI when doing pure Lisp build.
>
> The following diff is just a quick hack to achieve this.
> You use this to test the build: "make -jN CC=not-exist all-fricassys"
>
> Waldek, if you think such "--without-c" build option is useful, I can
> develop it into a full patch.

I personally see no need for such option, but it is hard to
know what other people want.

Concerning porting: current Makefile structure worked quite well
for porting to variuous Lisps even if at first port did not
handle FFI. For porting to Poplog I used 'contrib/load-fricas.lisp'
(had to edit it by hand but at some moment it allowed loading
FriCAS even though normal algebra bootstrap would fail in Poplog).

Also, without FFI we would have no gmp for sbcl. With ECL
no C compiler means no compiler at all.

Let me add that I consider making FFI mandatory and using
foreign code to speed up algebra.

--
Waldek Hebisch

[Qian Yun]

On 7/1/22 04:32, Waldek Hebisch wrote:
>
> I personally see no need for such option, but it is hard to
> know what other people want.

So no objection here. I think it is a good thing to properly
layer a project -- it is good abstraction. In our case,
it is the Lisp layer and C layer, and OS layer (shell stuff).

>
> Also, without FFI we would have no gmp for sbcl. With ECL
> no C compiler means no compiler at all.

It is just an option for sbcl.

For ECL I think it will call C compiler regardless of our CC setting.

> Let me add that I consider making FFI mandatory and using
> foreign code to speed up algebra.
>

The "speed up" is GMP right? Currently only available for
sbcl and clozurecl.

On the same subject, what's your opinion on providing GMP
for all Lisps via CFFI?

- Qian

[Waldek Hebisch]

On Fri, Jul 01, 2022 at 08:30:46AM +0800, Qian Yun wrote:
>
>
> On 7/1/22 04:32, Waldek Hebisch wrote:
> >
> >I personally see no need for such option, but it is hard to
> >know what other people want.
>
> So no objection here. I think it is a good thing to properly
> layer a project -- it is good abstraction. In our case,
> it is the Lisp layer and C layer, and OS layer (shell stuff).
>
> >
> >Also, without FFI we would have no gmp for sbcl. With ECL
> >no C compiler means no compiler at all.
>
> It is just an option for sbcl.
>
> For ECL I think it will call C compiler regardless of our CC setting.
>
> >Let me add that I consider making FFI mandatory and using
> >foreign code to speed up algebra.
> >
>
> The "speed up" is GMP right?

Not only. We could make more use of GMP than we make now,
but there are also other possibilities. One is to have
fast routines for polynomials mod p. We could use BLAS
to speed up linear algebra.

> Currently only available for
> sbcl and clozurecl.
>
> On the same subject, what's your opinion on providing GMP
> for all Lisps via CFFI?

ECL and GCL use GMP to implement their fixnums. This is
actually a big point of our GMP support: for sbcl and Clozure
CL we speed up _native_ bignums. So avoiding _explict_
use of FFI is a feature. We could could make more use
of GMP and in such case FFI binding would be useful. But
it is likely that any such use would go via C wrapper.

I must admit that for clisp I decided that GMP binding
is too much work for the effect: clisp is slow anyway.

--
Waldek Hebisch

[Qian Yun]

On 7/1/22 04:32, Waldek Hebisch wrote:
>

> Concerning porting: current Makefile structure worked quite well
> for porting to variuous Lisps even if at first port did not
> handle FFI. For porting to Poplog I used 'contrib/load-fricas.lisp'
> (had to edit it by hand but at some moment it allowed loading
> FriCAS even though normal algebra bootstrap would fail in Poplog).

I'm working on Kurt's fricas0, and I'd like to add ABCL support in
similar ways.

- Qian

[Qian Yun]

Just experimented with something interesting recently.

CFFI: is a portable library that abstracts different lisp's
FFI interface.

We have our own abstraction "fricas-foreign-call" defined for
each lisp, but with CFFI, we can have this for all lisp:

=========
(eval-when (:compile-toplevel :load-toplevel :execute)

(setf *c-type-to-ffi*
'((int :int)
(c-string :string)
(double :double)))

(defun c-args-to-cffi (arguments)
(mapcar (lambda (x) (list (nth 0 x) (c-type-to-ffi (nth 1 x))))
arguments))

(defun cffi-foreign-call (name c-name return-type arguments)
(let ((cffi-args (c-args-to-cffi arguments))
(cffi-ret (c-type-to-ffi return-type)))
`(cffi:defcfun (,c-name ,name) ,cffi-ret ,@cffi-args)))

(defmacro fricas-foreign-call (name c-name return-type &rest arguments)
(cffi-foreign-call name c-name return-type arguments)
))
===========

C2FFI: this is Clang-based FFI wrapper generator. It can automatically
generate FFI bindings for every function.

AUTOWRAP: this is based on C2FFI, makes it easier to use.

Following few lines generates full bindings to mpfr:

====
(ql:quickload "cl-autowrap") ;; use quicklisp to install and load
(autowrap:c-include "/usr/include/mpfr.h" :spec-path
'(cl-autowrap-test)) ;; this single line generates full bindings
(cffi-sys:%load-foreign-library "mpfr" "libmpfr.so") ;; load the library
(mpfr-get-version) ;; calls the generated function and returns "4.1.0-p7"
====

[Waldek Hebisch]

CFFI was available when I implemented "fricas-foreign-call".
At that time my conclusion was that CFFI handles easy parts, but
we also had to deal with nasty issues. Main thing is that
we want foreign call to do nontrivial amount of work, which
usually means some data aggregates. Clozure CL folks claimed
that only way to handle aggregates is to copy data between Lisp
and foreign memory. sbcl has 'pinned objects' which allows
safe passing Lisp data to foreign routines. IIUC ECL claims
that passing Lisp data to foreign routines is always safe
(without any special precautions).

For routines like matrix-vector multiplication cost of
copy is comparable to cost of operation and probably in
implementation needing copy using foreign call for
matrix-vector multiplication has no advantage (similarly,
we do _not_ use GMP add, only more expensive operations).

What I would like to call:
- matrix-matrix multiplication from BLAS
- routines like solve, SVD, eigenvalue decomposition from Lapack
- multiplication for polys over Z_p

That is rather small number of routines (of order of 20). Generating
signatures for them is not a problem. What remains to solve:
- detecting if there is BLAS/Lapack available and if yes in
which library (there are few competing BLAS libraries and
Lapack may be configured to use any of them). To make
things more "interesting" original BLAS and Lapack interface
was in Fortran 77, there are now C wrappers
- as I mentioned we need to solve problem of transfering data
to foreign routines

As I mentioned for sbcl we can use pinned objects, for Clozure CL
and Clisp we need copy, for GCL ATM we just pass pointer to C
side, but this is not guaranteed to work (basically we bet that
GCL will not run its garbage collector during foreign call, which
seem to work in current use. So the remaing part is avalability
of BLAS/Lapack. In other words, problem is not FFI itself but
managing dependencies. This affects binary distributions
because BLAS/Lapack tend to use machine specific instructions
(BLAS using say AVX will fail on machine without AVX instructions).

There is related, but different issue of making FFI easy to
use from Spad program. IMH here main issue is mapping
between Spad types and C types. In OpenAxion Gaby implemented
something, description is not clear but it seems that Gaby
decided that bunch of Spad types will map to C types (that
is user gives Spad types and interface uses corresponding
C types for call). AFAICS this has serious limitations
and for FriCAS I would like something better.

--
Waldek Hebisch

[Kurt Pagani]

On 02.07.2022 19:08, Waldek Hebisch wrote:
> On Sat, Jul 02, 2022 at 10:10:11PM +0800, Qian Yun wrote:

...

>
> What I would like to call:
> - matrix-matrix multiplication from BLAS
> - routines like solve, SVD, eigenvalue decomposition from Lapack
> - multiplication for polys over Z_p
>
> That is rather small number of routines (of order of 20). Generating
> signatures for them is not a problem. What remains to solve:
> - detecting if there is BLAS/Lapack available and if yes in
> which library (there are few competing BLAS libraries and
> Lapack may be configured to use any of them). To make
> things more "interesting" original BLAS and Lapack interface
> was in Fortran 77, there are now C wrappers
> - as I mentioned we need to solve problem of transfering data
> to foreign routines
>
> As I mentioned for sbcl we can use pinned objects, for Clozure CL
> and Clisp we need copy, for GCL ATM we just pass pointer to C
> side, but this is not guaranteed to work (basically we bet that
> GCL will not run its garbage collector during foreign call, which
> seem to work in current use. So the remaing part is avalability
> of BLAS/Lapack. In other words, problem is not FFI itself but
> managing dependencies. This affects binary distributions
> because BLAS/Lapack tend to use machine specific instructions
> (BLAS using say AVX will fail on machine without AVX instructions).
>

That sounds interesting.

There is also a very useful fortran to lisp translator (f2cl) available:

https://trac.common-lisp.net/f2cl/
https://trac.common-lisp.net/f2cl/wiki/GettingF2cl
http://quickdocs.org/f2cl/

"""
The easiest way to get F2CL is to use ​Quicklisp. You can just say
(ql:quickload :f2cl)
to get f2cl.
"""

I translated some examples (*)
(https://github.com/nilqed/f2cl/tree/master/packages/) some years ago (e.g.
BLAS) and it worked well, i.e. it seems easier to me to do lisp->spad than
c->spad.

(*) fortran sources partly from netlib.org

[Waldek Hebisch]

On Sun, Jul 03, 2022 at 03:41:50AM +0200, Kurt Pagani wrote:
>
> There is also a very useful fortran to lisp translator (f2cl) available:
>
> https://trac.common-lisp.net/f2cl/
> https://trac.common-lisp.net/f2cl/wiki/GettingF2cl
> http://quickdocs.org/f2cl/
>
> """
> The easiest way to get F2CL is to use Quicklisp. You can just say
> (ql:quickload :f2cl)
> to get f2cl.
> """
>
> I translated some examples (*)
> (https://github.com/nilqed/f2cl/tree/master/packages/) some years ago (e.g.
> BLAS) and it worked well, i.e. it seems easier to me to do lisp->spad than
> c->spad.
>
> (*) fortran sources partly from netlib.org

Well, from my point of view main point of BLAS is speed. Currently
speed that we can get from Spad code is limited by code generator
in underlying Lisp. It is not clear for me if f2cl gives us any
gain for real (double computations). There is probably gain for
complex due to incompatiblity of Spad Complex with Lisp complex
types. But I expect system BLAS to have much higher speed
(IIRC in the past optimized BLAS had 4-10 time better performance
than "reference BLAS" which is probably base for translation to
Lisp). On BLAS type code I would expect sbcl to get 2-4 times
slower code that Fortran compiler. And since optimized BLAS
uses special machine instructions and newer machines introduced
more such instructions I expect that gap between optimized BLAS
and f2cl result will be bigger.

--
Waldek Hebisch