Handle more C types in FriCAS FFI macros (fricas-lisp.lisp)

[Grégory Vanuxem]

Hello,

I hesitated to add this as an issue at GitHub.com (feature request), I
put it here first.

FriCAS supports what I would call preliminary support of FFI stuff via
a set of CL macros. I would like to suggest adding more types in this
regard. For example, it could be possible to add double-float arrays,
boolean and the like. I modified, and well tested, the following for
example. For information, only for SBCL and Clozure CL, but adding
those additional types for other CL implementations should be easy I
think:

===================================================
diff --git a/home/greg/Tmp/fricas/src/lisp/fricas-lisp.lisp
b/src/lisp/fricas-lisp.lisp
index 357aa063..8de10db7 100644
--- a/home/greg/Tmp/fricas/src/lisp/fricas-lisp.lisp
+++ b/src/lisp/fricas-lisp.lisp
@@ -363,10 +363,13 @@ with this hack and will try to convince the GCL
crowd to fix this.
)

(defvar *c_type_as_string* '(
+ (void "void")
+ (bool "bool")
(int "int")
(c-string "char *")
(double "double")
(char-* "char *")
+ (double-* "double *")
))

(defun c_type_as_string(c_type) (nth 1 (assoc c_type *c_type_as_string*)))
@@ -465,10 +468,13 @@ with this hack and will try to convince the GCL
crowd to fix this.
(eval-when (:compile-toplevel :load-toplevel :execute)

(setf *c-type-to-ffi* '(
- (int SB-ALIEN::int)
- (c-string SB-ALIEN::c-string)
- (double SB-ALIEN::double)
- (char-* (sb-alien:* sb-alien:char))
+ (void sb-alien::void)
+ (bool (sb-alien::boolean 8))
+ (int sb-alien::int)
+ (c-string (sb-alien::c-string))
+ (double sb-alien::double-float)
+ (char-* (sb-alien:* sb-alien:char))
+ (double-* (sb-alien:* sb-alien:double-float))
))

(defun c-args-to-sbcl (arguments)
@@ -490,10 +496,13 @@ with this hack and will try to convince the GCL
crowd to fix this.
(eval-when (:compile-toplevel :load-toplevel :execute)

(setf *c-type-to-ffi* '(
+ (void :void)
+ (bool :signed-byte)
(int :int)
(c-string :address)
(double :double-float)
(char-* :address)
+ (double-* :address)
))

(defun c-args-to-openmcl (arguments)
@@ -518,7 +527,7 @@ with this hack and will try to convince the GCL
crowd to fix this.
`(ccl::external-call ,c-name ,@fargs ,l-ret))
(fun-body
(if strs
- `(ccl::with-cstrs ,strs ,call-body)
+ `(ccl::with-encoded-cstrs :utf-8 ,strs ,call-body)
call-body)))
`(defun ,name ,largs ,fun-body))))

@@ -532,7 +541,7 @@ with this hack and will try to convince the GCL
crowd to fix this.

(setf *c-type-to-ffi* '(
(int :int)
- (c-string :cstring )
+ (c-string :cstring)
(double :double)
(char-* :pointer-void)
===================================================

The patch file is attached. One remark, I added UTF-8 to Clozure CL
encoded-cstrs in such a way some non-ascii characters can be used in
the Spad source code, SBCL has as default external format UTF-8. It
can be modified via sb-ext:*default-external-format* and
sb-ext:*default-c-string-external-format*. This is not an intent to
fully support UTF-8 in Spad code, it just allows, for example, to use:

x quo y == ibinfunc("÷", x, y)
instead of (commented here)
-- x quo y == ibinfunc("div", x, y)
x rem y == ibinfunc("%", x, y)

where ibinfunc is a macro that expands binary functions operating on integers.

Regards,

- Greg

PS: I am adding the content of this email to GitHub for record.

[Waldek Hebisch]

On Thu, Feb 29, 2024 at 01:01:45PM +0100, Grégory Vanuxem wrote:
> Hello,
>
> I hesitated to add this as an issue at GitHub.com (feature request), I
> put it here first.
>
> FriCAS supports what I would call preliminary support of FFI stuff via
> a set of CL macros. I would like to suggest adding more types in this
> regard. For example, it could be possible to add double-float arrays,
> boolean and the like. I modified, and well tested, the following for
> example. For information, only for SBCL and Clozure CL, but adding
> those additional types for other CL implementations should be easy I
> think:

Well, there are some things which are easy, but a limited use, like
adding 'void' or 'boolean' (but even boolean is not entirely
strightforward, as one needd to know corresponding C type, which
in principle could be 32-bit integer (I did not check what it is
in various implementations, but AFAICS C compiler can choose size
that it finds most convenient)).

IMO before going forward with implementation we should first
define what FFI should do. One basic question is "who owns
the data". sbcl FFI slightly prefers variant when C code
owns data. This avoid memory management by Lisp implementation.
But it also means that memory management is user responsibility.
This may be acceptable when you want to use existing C library
which requires manual memory management anyway. But for me
_main_ use case is using C code as fast "primitives" that
operate on Lisp data.

Concerning Lisp arrays, some years ago Clozure CL folks claimed
that the only correct way to pass Lisp arrays is to create temporary
copy in specially allocated memory and pass that to C code.
Basically, Clozure CL can move normal Lisp objects at any time, so
C code is not allowed direct access to Lisp data. In sbcl there is
'with-pinned-objects' which solves this problem. ECL does not moves
Lisp object (for ease of C interface) so no trouble here. IIUC GCL
has special "non-movable" allocation for things that are passed via
FFI, but with if we want freely pass Lisp arrays to FFI we would have
to allocate all of them in non-movable way and it is not clear to me
if this would fly if we allocate _all_ arrays in non-movable way. For
Clisp it seems that we need to copy. Poplog has similar approach to
GCL: there is non-movable pool of allocations for FFI.

So, one issue is how we properly pass Lisp objects to C. Note:
passing test do not really tell you if the code is correct. AFAIK
Lisp moves object during garbage collection, so to get bad behaviour
there are several conditions. Murphy says that such conditions will
never happen during test but only when what you compute is important
enough.

Another issue is how to specify foreign routines and foreign types
in Spad. One possiblity is modified 'import' statement. Minimally
one need here ablity to specify type. But frequently when using
FFI foreign name is different that the name we would like to use
in Spad, so probably we should add optional renaming. Instead
of 'import' statement we could try to add "foreign packages",
that is use normal package heading to specify types and add
some magic token like "Foreign('C)" as implementation so that
Spad compiler knows that is should create foreing calls for
all functions.

--
Waldek Hebisch

[Ralf Hemmecke]

On 3/1/24 01:33, Waldek Hebisch wrote:
> IMO before going forward with implementation we should first
> define what FFI should do. One basic question is "who owns
> the data". sbcl FFI slightly prefers variant when C code
> owns data. This avoid memory management by Lisp implementation.
> But it also means that memory management is user responsibility.
> This may be acceptable when you want to use existing C library
> which requires manual memory management anyway. But for me
> _main_ use case is using C code as fast "primitives" that
> operate on Lisp data.

I would like to be able to call functions from FLINT or NTL.
AFAICS one has to provide memory for FLINT or NTL to work on, even for
the result (correct me, if I am wrong).

Ralf

[Waldek Hebisch]

I am not sure about this. But in general calling FLINT or NTL
in official way is rather inconvenient.

--
Waldek Hebisch

[Ralf Hemmecke]

Yes, inconvenience maybe. But the alternative is to not being able to
call it at all. With inconvenience the situation would still improve.
Well, I currently do not need it desparately, but it would be a
nice-to-have, even if it were inconvenient.

Ralf

[Grégory Vanuxem]

Hello Waldek, *,

Le ven. 1 mars 2024 à 01:33, Waldek Hebisch <de...@fricas.org> a écrit :
>
> On Thu, Feb 29, 2024 at 01:01:45PM +0100, Grégory Vanuxem wrote:
> > Hello,
> >
> > I hesitated to add this as an issue at GitHub.com (feature request), I
> > put it here first.
> >
> > FriCAS supports what I would call preliminary support of FFI stuff via
> > a set of CL macros. I would like to suggest adding more types in this
> > regard. For example, it could be possible to add double-float arrays,
> > boolean and the like. I modified, and well tested, the following for
> > example. For information, only for SBCL and Clozure CL, but adding
> > those additional types for other CL implementations should be easy I
> > think:
>
>
> Well, there are some things which are easy, but a limited use, like
> adding 'void' or 'boolean' (but even boolean is not entirely
> strightforward, as one needd to know corresponding C type, which
> in principle could be 32-bit integer (I did not check what it is
> in various implementations, but AFAICS C compiler can choose size
> that it finds most convenient)).

Yes, I agree, it's not totally satisfactory, but it already allows you
to easily and succinctly code an interface to a C library, for
example. In general, booleans require special handling. With Clozure
CL, for example, a wrapper function is required, and this also applies
to strings. This is what I'm using for the moment:

(defmacro boot::|jl_bool_function_dbl_dbl| (func arg1 arg2)
`(if (eq (jl_bool_function_dbl_dbl ,func ,arg1 ,arg2) 0) nil t))))
(defmacro boot::|jl_string_eval_string| (str)
`(ccl::%get-cstring (jl_string_eval_string ,str))

It's very practical, by the way. Using this set of macros has been on
my TODO list for some time and I don't like modifying FriCAS
internals. What I'd like in interfacing with Julia is to be able to
produce a very simple patch to apply to vanilla FriCAS. And then be
able to add different spad files if/when necessary.

> IMO before going forward with implementation we should first
> define what FFI should do. One basic question is "who owns
> the data". sbcl FFI slightly prefers variant when C code
> owns data. This avoid memory management by Lisp implementation.
> But it also means that memory management is user responsibility.
> This may be acceptable when you want to use existing C library
> which requires manual memory management anyway. But for me
> _main_ use case is using C code as fast "primitives" that
> operate on Lisp data.

Very interesting. I know I do not take into account, right now,
stack/heap allocation differences between the different CL
implementations. I have to look deeper into this.
Presently I first need to manage the interaction
between two GCs, that is not really funny, but necessary.

I also prefer using "fast "primitives" that operate on Lisp data", I
use this scheme for parts of BLAS/LAPACK interface, but since I use a
high level interface this is not always possible to use "inplace"
operations. BTW there is also a partial BLAS/LAPACK interface in Julia that
uses the historical Fortran scheme, and you know that, with Fortran
you manage yourself memory areas (arrays) and just give them as
function/subroutine arguments.

> Concerning Lisp arrays, some years ago Clozure CL folks claimed
> that the only correct way to pass Lisp arrays is to create temporary
> copy in specially allocated memory and pass that to C code.

I did not know that, personally, with Clozure CL I use
'ccl::with-pointer-to-ivector' before calling C functions.

> Basically, Clozure CL can move normal Lisp objects at any time, so
> C code is not allowed direct access to Lisp data. In sbcl there is
> 'with-pinned-objects' which solves this problem.

Yes for SBCL. I was not aware of that for Clozure CL. To speak frankly
I am struggling with the Clozure CL GC, it
"never" recalls unreferenced memory, I have to write an email to
openmcl-devel to know how to _really_ do a GC reclaim. Maybe my code
is wrong, I do not "see" it reclaiming memory (with C printf help).
BTW it is always possible to temporarily stop the GC job but of course
this is not a solution that's too hard (and awful).

> ECL does not moves
> Lisp object (for ease of C interface) so no trouble here. IIUC GCL
> has special "non-movable" allocation for things that are passed via
> FFI, but with if we want freely pass Lisp arrays to FFI we would have
> to allocate all of them in non-movable way and it is not clear to me
> if this would fly if we allocate _all_ arrays in non-movable way. For
> Clisp it seems that we need to copy. Poplog has similar approach to
> GCL: there is non-movable pool of allocations for FFI.

I note. Supporting other CL implementations is on my road for what I'm doing.

>
> So, one issue is how we properly pass Lisp objects to C. Note:
> passing test do not really tell you if the code is correct. AFAIK
> Lisp moves object during garbage collection, so to get bad behaviour
> there are several conditions. Murphy says that such conditions will
> never happen during test but only when what you compute is important
> enough.

:)

I like his "law". Funnily expressed in french in fact, but when you
know what his job was you understand it: that will arrive...

>
> Another issue is how to specify foreign routines and foreign types
> in Spad. One possiblity is modified 'import' statement. Minimally
> one need here ablity to specify type. But frequently when using
> FFI foreign name is different that the name we would like to use
> in Spad, so probably we should add optional renaming. Instead
> of 'import' statement we could try to add "foreign packages",
> that is use normal package heading to specify types and add
> some magic token like "Foreign('C)" as implementation so that
> Spad compiler knows that is should create foreing calls for
> all functions.

Very ambitious, but that would be marvelous. I totally agree with Ralf
concerning $Lisp call vs. spad calls, I have a lot of things to clean
up in this regard. Calling routines using a specific FFI at the spad
level would be very clean. Using Lisp/C wrapper etc. is really not
satisfactory.

- Greg

[Camm Maguire]

Greetings! Grégory, will reply to your post in separate email. But in
general there are several places where fricas creates calls to 'list
which exceed GCL's call-arguments-limit when translating boot files to
.clisp. 2.6.x was not rigorous about enforcing this, but 2.7.0 is.
Where should I look in the boot parser source to suggest a patch
breaking this into an nconc of several compliant list calls, or better
yet just writing a quoted list (since apparently all contents are
strings) which the reader can handle just fine?

Take care,
--
Camm Maguire ca...@maguirefamily.org
==========================================================================
"The earth is but one country, and mankind its citizens." -- Baha'u'llah

[Waldek Hebisch]

On Sat, Mar 02, 2024 at 08:56:14AM -0500, Camm Maguire wrote:
> Greetings! Grégory, will reply to your post in separate email. But in
> general there are several places where fricas creates calls to 'list
> which exceed GCL's call-arguments-limit when translating boot files to
> .clisp. 2.6.x was not rigorous about enforcing this, but 2.7.0 is.
> Where should I look in the boot parser source to suggest a patch
> breaking this into an nconc of several compliant list calls, or better
> yet just writing a quoted list (since apparently all contents are
> strings) which the reader can handle just fine?

AFAICS this is handled in 'src/boot/tytree1.boot'. Note that in
FriCAS lists are mutable and Boot code makes significant use of
of mutation. So translating _all_ list construction to quoted
lists would be wrong. Boot really does not look at control/data
flow so finding out possibilties to use quoted lists is probably
not practical.

The simplest solution may be shadowing Lisp LIST and using a macro
instead.

--
Waldek Hebisch

[Camm Maguire]

Greetings, and thanks so much for the suggestion!

This works. I now have a 2.7.0 and fricas tree with minor modifications
ready for commit but for (I think) one remaining problem. 2.6.14
compiles my fricas tree just fine, but 2.7.0 gives two identical errors
compiling the algebra:

)compile GUESSINT.spad

Compiling FriCAS source code from file
/mnt/sda4/debian/fricas/src/algebra/GUESSINT.spad using old
system compiler.
GUESSINT abbreviates package GuessInteger
****** comp fails at level 2 with expression: ******
(|Sel| (|Expression| (|Integer|)) |retract|)
****** level 2 ******
$x:= ((Sel (Expression (Integer)) retract) G1)
$m:= (Fraction (Integer))
$f:=
((((#:G1 #) (* #) (+ #) (- #) ...)))

>> Apparent user error:
not known that (Expression (Integer)) has (OR (AND (has (Integer) (IntegralDomain)) (has (Integer) (RetractableTo (Integer)))) (has (Integer) (RetractableTo (Fraction (Integer)))))

I'm digging into this and will find it eventually, but if you have any
debugging suggestions I'm sure that would speed things up.

Take are,

Waldek Hebisch <heb...@fricas.org> writes:

> On Sat, Mar 02, 2024 at 08:56:14AM -0500, Camm Maguire wrote:

>> Greetings! Grégory, will reply to your post in separate email. But in
>> general there are several places where fricas creates calls to 'list
>> which exceed GCL's call-arguments-limit when translating boot files to
>> .clisp. 2.6.x was not rigorous about enforcing this, but 2.7.0 is.
>> Where should I look in the boot parser source to suggest a patch
>> breaking this into an nconc of several compliant list calls, or better
>> yet just writing a quoted list (since apparently all contents are
>> strings) which the reader can handle just fine?
>

> AFAICS this is handled in 'src/boot/tytree1.boot'. Note that in
> FriCAS lists are mutable and Boot code makes significant use of
> of mutation. So translating _all_ list construction to quoted
> lists would be wrong. Boot really does not look at control/data
> flow so finding out possibilties to use quoted lists is probably
> not practical.
>
> The simplest solution may be shadowing Lisp LIST and using a macro
> instead.

[Camm Maguire]

Greetings! Has fricas removed axiom's )fin (restart) pair? How does
one escape to/from lisp at the fricas prompt?

Take care,

[Qian Yun]

(1) -> )help fin
====================================================================
A.10. )fin
====================================================================

User Level Required: development

Command Syntax:

- )fin

Command Description:

This command is used by FriCAS developers to leave the FriCAS system and
return to the underlying Lisp system. To return to FriCAS, issue the
``(|spad|)'' function call to Lisp.




This works for SBCL. Not sure about GCL.

- Qian

[Camm Maguire]

Greetings! Thanks so much -- it was the replacement of (|restart|) with
(|spad|) that confused me.

Any other insights or suggestions on how to quickly pin down the error I
report below?

Take care,

[Waldek Hebisch]

On Wed, Mar 06, 2024 at 10:11:01AM -0500, Camm Maguire wrote:
> Greetings! Thanks so much -- it was the replacement of (|restart|) with
> (|spad|) that confused me.
>
> Any other insights or suggestions on how to quickly pin down the error I
> report below?

This could be bug in Spad compiler, triggered by different form of
S-expression (parts of expressions are sorted in implementation-dependent
way). I will try to look deeper into this, but ATM I am traveling, and
can not do this.

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/87sf132y62.fsf%40maguirefamily.org.

--
Waldek Hebisch

[Camm Maguire]

Greetings, and thanks so much for your feedback!

I've gotten it down to these gensyms in the |catlist| variable in
|knownInfo1|:

(((|Hashable|) T) ((|canonicalsClosed|) T) ((|Canonical|) T)
((|OpenMath|) T) ((|ConvertibleTo| (|String|)) T)
((|PolynomialFactorizationExplicit|) T)
((|LinearlyExplicitOver| (|Integer|)) T)
((|multiplicativeValuation|) T) ((|canonicalUnitNormal|) T)
((|StepThrough|) T) ((|ConvertibleTo| #:G1) T) ((|RealConstant|) T)
((|CombinatorialFunctionCategory|) T) ((|PatternMatchable| #:G16) T)
((|ConvertibleTo| #:G15) T) ((|ConvertibleTo| #:G14) T)
((|RetractableTo| #:G13) T) ((|ConvertibleTo| #:G12) T)
((|DifferentialRing|) T) ((|CharacteristicZero|) T)
((|PartialOrder|) T) ((|OrderedRing|) T) ((|OrderedIntegralDomain|) T)
((|EuclideanDomain|) T) ((|LeftOreRing|) T) ((|noZeroDivisors|) T)
((|CommutativeStar|) T) ((|Module| %) T) ((|Algebra| %) T)
((|unitsKnown|) T) ((|NonAssociativeRing|) T) ((|Monoid|) T)
((|MagmaWithUnit|) T) ((|SemiRing|) T) ((|RightModule| %) T)
((|SemiRng|) T) ((|AbelianGroup|) T) ((|CoercibleTo| #:G0) T))

Should be:

(((|Hashable|) T) ((|canonicalsClosed|) T) ((|Canonical|) T)
((|OpenMath|) T) ((|ConvertibleTo| (|String|)) T)
((|PolynomialFactorizationExplicit|) T)
((|LinearlyExplicitOver| (|Integer|)) T)
((|multiplicativeValuation|) T) ((|canonicalUnitNormal|) T)
((|StepThrough|) T) ((|ConvertibleTo| (|Float|)) T)
((|RealConstant|) T) ((|CombinatorialFunctionCategory|) T)
((|PatternMatchable| (|Integer|)) T)
((|ConvertibleTo| (|Pattern| (|Integer|))) T)
((|ConvertibleTo| (|InputForm|)) T) ((|RetractableTo| (|Integer|)) T)
((|ConvertibleTo| (|Integer|)) T) ((|DifferentialRing|) T)
((|CharacteristicZero|) T) ((|PartialOrder|) T) ((|OrderedRing|) T)
((|OrderedIntegralDomain|) T) ((|EuclideanDomain|) T)
((|LeftOreRing|) T) ((|noZeroDivisors|) T) ((|CommutativeStar|) T)
((|Module| %) T) ((|Algebra| %) T) ((|unitsKnown|) T)
((|NonAssociativeRing|) T) ((|Monoid|) T) ((|MagmaWithUnit|) T)
((|SemiRing|) T) ((|RightModule| %) T) ((|SemiRng|) T)
((|AbelianGroup|) T) ((|CoercibleTo| (|OutputForm|)) T))

Narrowing.....

Take care,