Problems with GMP on Mac OS

[Piet van Oostrum]

While I was exploring Computer Algebra Systems I came along Fricas, and got interested.
I decided to install it on my MacBook Pro with Mac OS 10.10 (Yosemite).
I could have installed the binary, but as this is about two years old, I decided to install from source. I downloaded https://sourceforge.net/projects/fricas/files/fricas/1.3.2/fricas-1.3.2-full.tar.bz2 and started from there. I made a command line installation, not an Application bundle.

I installed in a separate ax-build directory as recommended.
I used sbcl 1.3.19 (installed with macports).

There were, however, problems with GMP support.
Here are a couple of observations:

1. First I installed without GMP support, and later I decided to add GMP support. I did a 'make clean' in between. However, it appeared that GMP was not used (this is hard to find out, by the way).
I found that ax-build/src/lisp/num_gmp.lisp was still the old one from the non-GMP build. After deleting that file and re-running 'make' I was one step further towards using GMP, but some other hurdles still had to be taken.

Solution: 'make clean' in src/lisp/Makefile.in should also remove num_gmp.lisp.

2. I have GMP installed by macports, hence it is in a non-standard location (/opt/local). This makes that configure doesn't find it. This can easily be solved by using

CPPFLAGS=-I/opt/local/include LDFLAGS=-L/opt/local/lib ../configure ...

or setting these environment variables before starting configure.

(But see also point 3)

3. That is, however, not sufficient to compile ax-build/src/lib/gmp_wrap.so.
It needs this in the Makefile:
$(GMP_WRAP_SO): $(srcdir)/gmp_wrap.c
$(CC) -I/opt/local/include -L/opt/local/lib -g -fPIC $(SOLIB_FLAGS) $< $(GMP_LIB) -o $@

This is a bit harder, as then configure must patch the Makefile. But configure doesn't know the location of GMP.

Another option is:

$(GMP_WRAP_SO): $(srcdir)/gmp_wrap.c
$(CC) %(CPPFLAGS) $(LDFLAGS) -g -fPIC $(SOLIB_FLAGS) $< $(GMP_LIB) -o $@

and running 'make' with these environment variables set.
I think the best solution would be to add options -gmp-includes and -gmp-libraries to configure, similar to -x-includes and -x-libraries. Then configure can put this information in the Makefile.
This would also be useful for point 4.

4. In src/lisp/num_gmp.lisp, in init-gmp the statement

(setf *gmp-multiplication-initialized* t)

is the 'else' branch of the last 'if'. I think it should be part of the 'then' branch, like

(if (ignore-errors
(|quiet_load_alien| wrapper-lib) t)
(progn
(install-gmp-multiplication)
(setf *gmp-multiplication-initialized* t))))))

Alternatively, 'when' could be used instead of 'if'.

5. On Mac OS, the gmp dynamic library, is called "libgmp.dylib", not "libgmp.so".
So, this must be patched in init-gmp. Moreover, as in my system it is in a non-standard location, either the full path name must be given, or the directory /opt/local/lib must be given in the environment variable LD_LIBRARY_PATH.

I have tried both: use in init-gmp:
(if (ignore-errors (|quiet_load_alien| "/opt/local/lib/libgmp.dylib") t)

or add

LD_LIBRARY_PATH=/opt/local/lib
export LD_LIBRARY_PATH

to the 'fricas' shell script.

In either case, configure must know the location, so this asks for the -gmp-libraries option mentioned in point 3 above.

With all changes mentioned above, fricas runs on Mac OS, with GMP support. The speedup with GMP is not as dramatic as mentioned in some previous messages, however. For example, on my MacBook, calculating 10^6 digits of 'e' goes from 354 sec. to 135 sec. I don't know if this means that the GMP is not working completely?

I am planning to make a macports port for fricas. As macports has a provision to provide patches for the build process, the problems mentioned above can be solved. However, for stand-alone builds, some changes should be made to the original sources.

--
Piet van Oostrum <pi...@vanoostrum.org>
WWW: http://piet.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

[oldk1331]

Hello Piet, I'm glad that you are interested in FriCAS and solved this
"build bug" of FriCAS with gmp on Mac OS.

I'll help you to get related patches merged and I'm looking forward to
see FriCAS on macports.

> For example, on my MacBook, calculating 10^6 digits of 'e' goes
> from 354 sec. to 135 sec.

How did you do that? Here is what I do:
(1) -> digits(10^6);

Type: PositiveInteger
(2) -> )time on
(2) -> exp1()$Float;

Type: Float
Time: 0.22 (EV) + 0.03 (OT) = 0.24 sec

[Piet van Oostrum]

If I do it this way, I get a speedup from 8.3 sec to 0.38 sec.

What I did was
(1) -> )time on
(2) -> numeric(%e, 1000000);

which, apparently, takes much longer, and doesn't give that much of a speedup.
--
Piet van Oostrum <pie...@vanoostrum.org>

[oldk1331]

Hmm, if you do this, 'numeric' will be fast:

)time on
numeric(%e, 1)
numeric(%e, 10^6);

I tried to debug this, but haven't found anything yet.

====

You can format some patches to the mailing list:

1. about Makefile,

There is already file num_gmp.lisp, so you should not remove it
by "make clean", you should rename the Makefile target to a different one.

2. about *gmp-multiplication-initialized*,

I think that variable is better modified by 'install-gmp-multiplication'
and 'unistall-gmp-multiplication'.

3. about gmp headers/library path,

Do you familar with autoconf? A patch would be great...

[Piet van Oostrum]

oldk1331 <oldk...@gmail.com> writes:

> Hmm, if you do this, 'numeric' will be fast:
>
> )time on
> numeric(%e, 1)
> numeric(%e, 10^6);
>
> I tried to debug this, but haven't found anything yet.
>

Yes, I had noticed that, when I first did 'numeric(%e, 10^4);' and then
'numeric(%e, 10^6);' the total time was shorter than only doing the
last one. Very strange.

I did some timing experiments with different lisp flavours, both the
split calculaion and the single calculation. Here are the results: (By
the way, it would be nice if Fricas would not only display its own
version, but also the Lisp version).
As you can see, there are considerable time differences.

-----------------------------------------------------------------------------
*** CCL ***
-----------------------------------------------------------------------------
(1) -> )time on
(1) -> numeric(%e)

(1) 2.7182818284_590452354
Type: Float
Time: 0.01 (IN) + 0.03 (EV) + 0.28 (OT) = 0.31 sec
(2) -> numeric(%e, 1000000);

Type: Float
Time: 0.00 (IN) + 138.44 (EV) + 0.00 (OT) = 138.44 sec
(3) ->
-----------------------------------------------------------------------------
(1) -> )time on
(1) -> numeric(%e, 1000000);

Type: Float
Time: 0.01 (IN) + 3413.76 (EV) + 0.25 (OT) = 3414.02 sec
(2) ->
-----------------------------------------------------------------------------
*** ECL ***
-----------------------------------------------------------------------------
(1) -> )time on
(1) -> numeric(%e)

(1) 2.7182818284_590452354
Type: Float
Time: 0.00 (IN) + 0.01 (EV) + 0.18 (OT) = 0.20 sec
(2) -> numeric(%e, 1000000);

Type: Float
Time: 0.83 (EV) = 0.83 sec
(3) ->
-----------------------------------------------------------------------------
(1) -> )time on
(1) -> numeric(%e, 1000000);

Type: Float
Time: 0.00 (IN) + 326.82 (EV) + 0.16 (OT) = 326.99 sec
(2) ->
-----------------------------------------------------------------------------
*** SBCL - GMP ***
-----------------------------------------------------------------------------
(1) -> )time on
(1) -> numeric(%e)

(1) 2.7182818284_590452354
Type: Float
Time: 0.01 (IN) + 0.03 (EV) + 0.78 (OT) = 0.81 sec
(2) -> numeric(%e, 1000000);

Type: Float
Time: 0.00 (IN) + 7.71 (EV) + 0.00 (OT) = 7.71 sec
(3) ->
-----------------------------------------------------------------------------
(1) -> )time on
(1) -> numeric(%e, 1000000);

Type: Float
Time: 0.01 (IN) + 343.34 (EV) + 0.70 (OT) = 344.05 sec
(2) ->
-----------------------------------------------------------------------------
*** SBCL + GMP ***
-----------------------------------------------------------------------------
(1) -> )time on
(1) -> numeric(%e)

(1) 2.7182818284_590452354
Type: Float
Time: 0.02 (IN) + 0.04 (EV) + 0.88 (OT) = 0.93 sec
(2) -> numeric(%e, 1000000);

Type: Float
Time: 0.35 (EV) = 0.35 sec
(3) ->
-----------------------------------------------------------------------------
(1) -> )time on
(1) -> numeric(%e, 1000000);

Type: Float
Time: 0.01 (IN) + 132.94 (EV) + 0.82 (OT) = 133.77 sec
(2) ->
-----------------------------------------------------------------------------

[Piet van Oostrum]

oldk1331 <oldk...@gmail.com> wrote:

> You can format some patches to the mailing list:
>
> 1. about Makefile,
>
> There is already file num_gmp.lisp, so you should not remove it
> by "make clean", you should rename the Makefile target to a different one.

OK, I have done this. I still need to test it.

> 2. about *gmp-multiplication-initialized*,
>
> I think that variable is better modified by 'install-gmp-multiplication'
> and 'unistall-gmp-multiplication'.

I think that is a good idea. I will test and supply a patch.

> 3. about gmp headers/library path,
>
> Do you familar with autoconf? A patch would be great...
>

I know it a bit, probably enough. I have now started from the tar.bz2
distribution, so I did not use autoconf. I would have to change to the
svn or git sources to test that.

I think the best solution would be to change the '--enable-gmp'
configure option to '--with-gmp=/dir/of/gmp', e.g.
'--with-gmp-/opt/local' for macports (default). People can put gmp
anywhere in their filesystem, so autoconf cannot find it on its own.
The old '--enable-gmp' could be kept as an alias for 'with-gmp' (without
directory), if so desired.

[oldk1331]

Hi Piet, I think I solved the slow exp computation.

During the initialization of Expression(Float) (to be more
precise, ElementaryFunction(Float, Expression(Float)) ),
it tries to compute and cache %pi, sqrt(2), sqrt(3) first, compute
them to 10^6 digits takes a lot of time. This bug shows
that it's useless to cache %pi/sqrt(2)/sqrt(3) for Expression Float,
also note that Float already caches %e, so the second compute is fast.

I propose the following patch:

--- a/src/algebra/elemntry.spad
+++ b/src/algebra/elemntry.spad
@@ -208,7 +208,7 @@
-- following code is conditionalized on arbitraryPrecision to recompute in
-- case user changes the precision

- if R has TranscendentalFunctionCategory then
+ if R has TranscendentalFunctionCategory and not R has
arbitraryPrecision then
Pie := pi()$R :: F
else
Pie := kernel(oppi, []$List(F))
@@ -222,7 +222,7 @@
isqrt1 := imaginary()$R :: F
else isqrt1 := sqrt(-1::F)

- if R has RadicalCategory then
+ if R has RadicalCategory and not R has arbitraryPrecision then
isqrt2 := sqrt(2::R)::F
isqrt3 := sqrt(3::R)::F
else

All tests passed:
https://travis-ci.org/oldk1331/fricas/builds/254377613

[oldk1331]

My previous patch isn't quite right (the conditions are wrong),
this is the fixed patch:

https://github.com/oldk1331/fricas/commit/38f9c9ec01046a7d8dab7abc8ce39c3e05b21ffe.patch

All tests passed: https://travis-ci.org/oldk1331/fricas/builds/254671682

[Piet van Oostrum]

I have applied this patch, but, unfortunately, it doesn't solve the timing problem.

numeric(%e, 1000000); takes 0.3 sec when preceded by numeric(%e, 1)
but 130 sec if given as the first expression.

[Ralf Hemmecke]

On 07/18/2017 06:46 PM, Piet van Oostrum wrote:
> I have applied this patch, but, unfortunately, it doesn't solve the timing problem.
>
> numeric(%e, 1000000); takes 0.3 sec when preceded by numeric(%e, 1)
> but 130 sec if given as the first expression.

I don't know for certain, but I fear that there is another phenomenon
that causes that time delay.

When FriCAS instantiates a domain, the function vector is not initialize
for all functions by the actual code of the function, but rather with a
function that (on the first call) sets the function vector entry to the
actual value. However, the first call might still use the old vector
entry, i.e., additionally to the actual computation, the function vector
entry is set again and again.

Well, that's only a guess of mine, but I remember some time ago, I had a
discussion with Waldek (or was it about Aldor?) with exactly that problem.

Ralf

[oldk1331]

On a slow VPS without gmp, I test with 'numeric(%e, 10^5)', without this patch
it takes 9s; after patch it takes 0.4s.

May I ask how did you applied the patch?

[Piet van Oostrum]

oldk1331 <oldk...@gmail.com> writes:

> On a slow VPS without gmp, I test with 'numeric(%e, 10^5)', without this patch
> it takes 9s; after patch it takes 0.4s.
>
> May I ask how did you applied the patch?
>

I just patched the source (elemntry.spad) and then ran 'make'. However,
I found the problem: the pre-generated lisp files were still used. So I
had to re-generate these and now it works: I get 0.3 sec. EV time for
numeric(%e, 10^6), and 0.81 OT.

Is there a simple trick to regenerate only those lisp files where the
.spad file is changed?

[oldk1331]

Well, we (developers) usually work on SVN trunk version (or git master branch),
so there's no "pre-generated" problem. If you modify a file without changing
function signatures, usually "make" will successfully generate a newer AXIOMsys
binary that contains the changes. (Note: you can use
AXIOM=./target/x86_64-linux-gnu/ ./target/x86_64-linux-gnu/bin/AXIOMsys
to invoke development version binary.)

Glad to know that this patch works.

[Waldek Hebisch]

>
> While I was exploring Computer Algebra Systems I came along Fricas, and got interested.
> I decided to install it on my MacBook Pro with Mac OS 10.10 (Yosemite).
> I could have installed the binary, but as this is about two years old, I decided to install from source. I downloaded https://sourceforge.net/projects/fricas/files/fricas/1.3.2/fricas-1.3.2-full.tar.bz2 and started from there. I made a command line installation, not an Application bundle.
>
> I installed in a separate ax-build directory as recommended.
> I used sbcl 1.3.19 (installed with macports).
>
> There were, however, problems with GMP support.
> Here are a couple of observations:
>
> 1. First I installed without GMP support, and later I decided to add GMP support. I did a 'make clean' in between. However, it appeared that GMP was not used (this is hard to find out, by the way).
> I found that ax-build/src/lisp/num_gmp.lisp was still the old one from the non-GMP build. After deleting that file and re-running 'make' I was one step further towards using GMP, but some other hurdles still had to be taken.
>
> Solution: 'make clean' in src/lisp/Makefile.in should also remove num_gmp.lisp.
>

This is red herring: ax-build/src/lisp/num_gmp.lisp is just copy of
'num_gmp.lisp' to avoid Lisp insanity with pathnames. Changing configure
options is tricky thing, my impression was that one should do
at least 'make distclean' for that. Anyway, FriCAS makes no
effort to supprt such use.

> 2. I have GMP installed by macports, hence it is in a non-standard location (/opt/local). This makes that configure doesn't find it. This can easily be solved by using
>
> CPPFLAGS=-I/opt/local/include LDFLAGS=-L/opt/local/lib ../configure ...
>
> or setting these environment variables before starting configure.
>
> (But see also point 3)
>
> 3. That is, however, not sufficient to compile ax-build/src/lib/gmp_wrap.so.
> It needs this in the Makefile:
> $(GMP_WRAP_SO): $(srcdir)/gmp_wrap.c
> $(CC) -I/opt/local/include -L/opt/local/lib -g -fPIC $(SOLIB_FLAGS) $< $(GMP_LIB) -o $@
>
> This is a bit harder, as then configure must patch the Makefile. But configure doesn't know the location of GMP.
>
> Another option is:
>
> $(GMP_WRAP_SO): $(srcdir)/gmp_wrap.c
> $(CC) %(CPPFLAGS) $(LDFLAGS) -g -fPIC $(SOLIB_FLAGS) $< $(GMP_LIB) -o $@
>
> and running 'make' with these environment variables set.
> I think the best solution would be to add options -gmp-includes and -gmp-libraries to configure, similar to -x-includes and -x-libraries. Then configure can put this information in the Makefile.
> This would also be useful for point 4.

Yes.

> 4. In src/lisp/num_gmp.lisp, in init-gmp the statement
>
> (setf *gmp-multiplication-initialized* t)
>
> is the 'else' branch of the last 'if'. I think it should be part of the 'then' branch, like
>
> (if (ignore-errors
> (|quiet_load_alien| wrapper-lib) t)
> (progn
> (install-gmp-multiplication)
> (setf *gmp-multiplication-initialized* t))))))

Yes, that was the intent.

> Alternatively, 'when' could be used instead of 'if'.
>
> 5. On Mac OS, the gmp dynamic library, is called "libgmp.dylib", not "libgmp.so".
> So, this must be patched in init-gmp. Moreover, as in my system it is in a non-standard location, either the full path name must be given, or the directory /opt/local/lib must be given in the environment variable LD_LIBRARY_PATH.
>
> I have tried both: use in init-gmp:
> (if (ignore-errors (|quiet_load_alien| "/opt/local/lib/libgmp.dylib") t)
>
> or add
>
> LD_LIBRARY_PATH=/opt/local/lib
> export LD_LIBRARY_PATH
>
> to the 'fricas' shell script.
>
> In either case, configure must know the location, so this asks for the -gmp-libraries option mentioned in point 3 above.
>

Yes.

> With all changes mentioned above, fricas runs on Mac OS, with GMP support. The speedup with GMP is not as dramatic as mentioned in some previous messages, however. For example, on my MacBook, calculating 10^6 digits of 'e' goes from 354 sec. to 135 sec. I don't know if this means that the GMP is not working completely?
>

That was mostly explained in other messages. Let me just add that
GMP is not used for "small" numbers (below hundreds of digits) and
not for addition and subtraction. So you get large speedup for
multiplication, division, square roots and gcd of large numbers.
How much that matter depends very much on specific computation.

> I am planning to make a macports port for fricas. As macports has a provision to provide patches for the build process, the problems mentioned above can be solved. However, for stand-alone builds, some changes should be made to the original sources.
>

--

Waldek Hebisch

[Waldek Hebisch]

oldk1331 wrote:
> 2. about *gmp-multiplication-initialized*,
>
> I think that variable is better modified by 'install-gmp-multiplication'
> and 'unistall-gmp-multiplication'.

No. The intent of '*gmp-multiplication-initialized*' was to
avoid loading libraries second time. 'install-gmp-multiplication'
and 'unistall-gmp-multiplication' may be called several times
without re-loading libraries.

--
Waldek Hebisch

[Waldek Hebisch]

Piet van Oostrum wrote:
>
> oldk1331 <oldk...@gmail.com> writes:
>
> > On a slow VPS without gmp, I test with 'numeric(%e, 10^5)', without this patch
> > it takes 9s; after patch it takes 0.4s.
> >
> > May I ask how did you applied the patch?
> >
> I just patched the source (elemntry.spad) and then ran 'make'. However,
> I found the problem: the pre-generated lisp files were still used. So I
> had to re-generate these and now it works: I get 0.3 sec. EV time for
> numeric(%e, 10^6), and 0.81 OT.
>
> Is there a simple trick to regenerate only those lisp files where the
> .spad file is changed?

Well, in general .lsp file depends on several (hundreds) .spad
files. In typical cases those dependencies may be ignored
and

)compile file.spad

on FriCAS command line will do right thing and allow you to
test the file. In fact, this is how I do most of my
developement: edit, compile, try, repeat.

Concerning Makefile-s, when recompiling they make shortcuts
trying to do minimal work and ignore some dependencies (but
some dependencies needed for build subvert this intent).
However, change from pregenerated files to compiling
from .spad sources requires full rebuild. Simply, there is
different set of dependencies and trying to avoid rebuild
would enormously complicate Makefile-s (OK, I do not know if
it is possible to express needed logic as a Makefile)
for little gain.

--
Waldek Hebisch

[Piet van Oostrum]

Waldek Hebisch wrote:

> >
> > While I was exploring Computer Algebra Systems I came along Fricas, and got interested.
> > I decided to install it on my MacBook Pro with Mac OS 10.10 (Yosemite).
> > I could have installed the binary, but as this is about two years old, I decided to install from source. I downloaded https://sourceforge.net/projects/fricas/files/fricas/1.3.2/fricas-1.3.2-full.tar.bz2 and started from there. I made a command line installation, not an Application bundle.
> >
> > I installed in a separate ax-build directory as recommended.
> > I used sbcl 1.3.19 (installed with macports).
> >
> > There were, however, problems with GMP support.
> > Here are a couple of observations:
> >
> > 1. First I installed without GMP support, and later I decided to add GMP support. I did a 'make clean' in between. However, it appeared that GMP was not used (this is hard to find out, by the way).
> > I found that ax-build/src/lisp/num_gmp.lisp was still the old one from the non-GMP build. After deleting that file and re-running 'make' I was one step further towards using GMP, but some other hurdles still had to be taken.
> >
> > Solution: 'make clean' in src/lisp/Makefile.in should also remove num_gmp.lisp.
> >
>
> This is red herring: ax-build/src/lisp/num_gmp.lisp is just copy of
> 'num_gmp.lisp' to avoid Lisp insanity with pathnames. Changing configure
> options is tricky thing, my impression was that one should do
> at least 'make distclean' for that. Anyway, FriCAS makes no
> effort to supprt such use.

ax-build/src/lisp/num_gmp.lisp is not always a copy of 'num_gmp.lisp'. It can also be a stub generated by the makefile, if no gmp support is selected. In fact, if you would do an in-source build (which doesn't work on Mac OS), then that would destroy the original num_gmp.lisp. So it makes sense to give the generated/copied file a new name.
Also on systems where libgmp is on a non-standard location, or on Mac OS where the extension is .dylib rather than .so it makes sense to give the file to be compiled a different name.

I have a patch for this.

[Piet van Oostrum]

Waldek Hebisch wrote:

> > 2. I have GMP installed by macports, hence it is in a non-standard location (/opt/local). This makes that configure doesn't find it. This can easily be solved by using
> >
> > CPPFLAGS=-I/opt/local/include LDFLAGS=-L/opt/local/lib ../configure ...
> >
> > or setting these environment variables before starting configure.
> >
> > (But see also point 3)
> >
> > 3. That is, however, not sufficient to compile ax-build/src/lib/gmp_wrap.so.
> > It needs this in the Makefile:
> > $(GMP_WRAP_SO): $(srcdir)/gmp_wrap.c
> > $(CC) -I/opt/local/include -L/opt/local/lib -g -fPIC $(SOLIB_FLAGS) $< $(GMP_LIB) -o $@
> >
> > This is a bit harder, as then configure must patch the Makefile. But configure doesn't know the location of GMP.
> >
> > Another option is:
> >
> > $(GMP_WRAP_SO): $(srcdir)/gmp_wrap.c
> > $(CC) %(CPPFLAGS) $(LDFLAGS) -g -fPIC $(SOLIB_FLAGS) $< $(GMP_LIB) -o $@
> >
> > and running 'make' with these environment variables set.
> > I think the best solution would be to add options -gmp-includes and -gmp-libraries to configure, similar to -x-includes and -x-libraries. Then configure can put this information in the Makefile.
> > This would also be useful for point 4.
>
> Yes.

I have a working solution now with configure options --with-gmp=/dir/where/gmp, or alternatively, --with-gmp-include=/dir/where/gmp/include --with-gmp-lib=/dir/where/gmp/lib
I think this is the cleanest solution and is well supported by autoconf.

I also found a tricky problem with the current configure.ac, when libgmp is in a non-standard location. Then '-lgmp' could be settled permanently in the variable 'LIBS' and that may cause other tests to fail. I have also solved this.

[Piet van Oostrum]

Waldek Hebisch wrote:

> > 4. In src/lisp/num_gmp.lisp, in init-gmp the statement
> >
> > (setf *gmp-multiplication-initialized* t)
> >
> > is the 'else' branch of the last 'if'. I think it should be part of the 'then' branch, like
> >
> > (if (ignore-errors
> > (|quiet_load_alien| wrapper-lib) t)
> > (progn
> > (install-gmp-multiplication)
> > (setf *gmp-multiplication-initialized* t))))))
>
> Yes, that was the intent.

I can send a patch for this. Also, I think there is a misplaced parenthesis:
The second unistall-gmp-multiplication:

(defun unistall-gmp-multiplication()
(sb-ext:unlock-package "SB-BIGNUM")
(setf (symbol-function 'sb-bignum::multiply-bignums)
(symbol-function 'orig-multiply-bignums))
(setf (symbol-function 'sb-bignum::bignum-truncate)
(symbol-function 'orig-bignum-truncate))
(setf (symbol-function 'sb-bignum::bignum-gcd)
(symbol-function 'orig-bignum-gcd))
(sb-ext:lock-package "SB-BIGNUM"))
(sb-ext:unlock-package "COMMON-LISP")
(setf (symbol-function 'common-lisp:isqrt)
(symbol-function 'orig-isqrt))
(sb-ext:lock-package "COMMON-LISP")

The last four lines are now outside the function. I think they should be inside:

(defun unistall-gmp-multiplication()
(sb-ext:unlock-package "SB-BIGNUM")
(setf (symbol-function 'sb-bignum::multiply-bignums)
(symbol-function 'orig-multiply-bignums))
(setf (symbol-function 'sb-bignum::bignum-truncate)
(symbol-function 'orig-bignum-truncate))
(setf (symbol-function 'sb-bignum::bignum-gcd)
(symbol-function 'orig-bignum-gcd))
(sb-ext:lock-package "SB-BIGNUM")
(sb-ext:unlock-package "COMMON-LISP")
(setf (symbol-function 'common-lisp:isqrt)
(symbol-function 'orig-isqrt))
(sb-ext:lock-package "COMMON-LISP"))

What do you think?

[Piet van Oostrum]

OK, clear.
By the way, is 'unistall-gmp-multiplication' a typo for 'uninstall-gmp-multiplication'?

[Waldek Hebisch]

Piet van Oostrum wrote:
>
> I have a working solution now with configure options --with-gmp=/dir/where/gmp, or alternatively, --with-gmp-include=/dir/where/gmp/include --with-gmp-lib=/dir/where/gmp/lib
> I think this is the cleanest solution and is well supported by autoconf.

Just a little remark: IIUC Gnu theology, '--with-...' options are
supposed to give locations of things but do not change functionality.
For change in functionality they want '--enable-...' options.
That is why FriCAS currently has '--enable-gmp'. So Gnu theology
seem to require separate '--enable-gmp' so that GMP gets used
at all and separte '--with-...' options to specify location.
Giving location but prefering GMP to be _not_ used looks odd,
but apparently that is what GNU standards want. I have no
strong opinion on this, just want FriCAS behave in way expected
by users. But if users expect that build system follows GNU
standards then separate '--enable-gmp' and '--with' options
make sense.

> I also found a tricky problem with the current configure.ac, when libgmp is in a non-standard location. Then '-lgmp' could be settled permanently in the variable 'LIBS' and that may cause other tests to fail. I have also solved this.

IIUC LIBS is supposed to contanin everything needed to use library.
Extra data for GMP should either fail GMP test or be harmless
for other tests. In particular, if there is same magic in
environment needed for GMP it should work everwhere.

--
Waldek Hebisch

[Waldek Hebisch]

Piet van Oostrum wrote:

Of course the last four lines should be inside.

--
Waldek Hebisch

[Waldek Hebisch]

Piet van Oostrum wrote:
>
> Waldek Hebisch wrote:
>
> > oldk1331 wrote:
> > > 2. about *gmp-multiplication-initialized*,
> > >
> > > I think that variable is better modified by 'install-gmp-multiplication'
> > > and 'unistall-gmp-multiplication'.
> >
> > No. The intent of '*gmp-multiplication-initialized*' was to
> > avoid loading libraries second time. 'install-gmp-multiplication'
> > and 'unistall-gmp-multiplication' may be called several times
> > without re-loading libraries.
>
> OK, clear.
> By the way, is 'unistall-gmp-multiplication' a typo for 'uninstall-gmp-multiplication'?

Yes.

--
Waldek Hebisch

[Piet van Oostrum]

So then, here is a patch for num_gmp.lisp

[Waldek Hebisch]

Piet van Oostrum wrote:
>
> Waldek Hebisch wrote:
>
> > >
> > > While I was exploring Computer Algebra Systems I came along Fricas, and got interested.
> > > I decided to install it on my MacBook Pro with Mac OS 10.10 (Yosemite).
> > > I could have installed the binary, but as this is about two years old, I decided to install from source. I downloaded https://sourceforge.net/projects/fricas/files/fricas/1.3.2/fricas-1.3.2-full.tar.bz2 and started from there. I made a command line installation, not an Application bundle.
> > >
> > > I installed in a separate ax-build directory as recommended.
> > > I used sbcl 1.3.19 (installed with macports).
> > >
> > > There were, however, problems with GMP support.
> > > Here are a couple of observations:
> > >
> > > 1. First I installed without GMP support, and later I decided to add GMP support. I did a 'make clean' in between. However, it appeared that GMP was not used (this is hard to find out, by the way).
> > > I found that ax-build/src/lisp/num_gmp.lisp was still the old one from the non-GMP build. After deleting that file and re-running 'make' I was one step further towards using GMP, but some other hurdles still had to be taken.
> > >
> > > Solution: 'make clean' in src/lisp/Makefile.in should also remove num_gmp.lisp.
> > >
> >
> > This is red herring: ax-build/src/lisp/num_gmp.lisp is just copy of
> > 'num_gmp.lisp' to avoid Lisp insanity with pathnames. Changing configure
> > options is tricky thing, my impression was that one should do
> > at least 'make distclean' for that. Anyway, FriCAS makes no
> > effort to supprt such use.
>
> ax-build/src/lisp/num_gmp.lisp is not always a copy of 'num_gmp.lisp'. It can also be a stub generated by the makefile, if no gmp support is selected. In fact, if you would do an in-source build (which doesn't work on Mac OS), then that would destroy the original num_gmp.lisp.

IIUC in such case GNU 'make' will skip the rule as "cyclic dependency"
so source version will be in use. Not sure what other 'make'-s
would do. Anyway, using the same name is a bug. And 'make clean'
should remove generated file.

> So it makes sense to give the generated/copied file a new name.
> Also on systems where libgmp is on a non-standard location, or on Mac OS where the extension is .dylib rather than .so it makes sense to give the file to be compiled a different name.

Well, there are separate issues. 'num_gmp.lisp' is a Lisp file
which in principle can be loaded even if there is no GMP in the
system. 'gmp_wrap.c' is C file used to produce wrapper library.
'gmp_wrap.c' needs GMP headers and whatever system needs to
produce shared library. After installation corresponding
library is located in FriCAS installation tree. I see
no reason against using the same name 'gmp_wrap.so' on all
systems. Actual GMP library comes from system and we need to
use name chosen by the system.

--
Waldek Hebisch

[Ralf Hemmecke]

On 07/19/2017 11:39 PM, Waldek Hebisch wrote:
> Piet van Oostrum wrote:
>>
>> I have a working solution now with configure options --with-gmp=/dir/where/gmp, or alternatively, --with-gmp-include=/dir/where/gmp/include --with-gmp-lib=/dir/where/gmp/lib
>> I think this is the cleanest solution and is well supported by autoconf.
>
> Just a little remark: IIUC Gnu theology, '--with-...' options are
> supposed to give locations of things but do not change functionality.
> For change in functionality they want '--enable-...' options.
> That is why FriCAS currently has '--enable-gmp'. So Gnu theology
> seem to require separate '--enable-gmp' so that GMP gets used
> at all and separte '--with-...' options to specify location.

--enable-gmp=/path/to/gmp

would also be an option. But if we really need to specify include and
lib dir of gmp separately, I'd be in favour of --enable-gmp and then
--with-gmp-include=... and --with-gmp-lib=...

In case --enable-gmp is given without argument, configure should test if
it finds the respective library/include files so that --with-gmp-lib and
--with-gmp-include would only be needed if that test fails.

> Giving location but prefering GMP to be _not_ used looks odd,

Yes, maybe, but --with-foo only provides paths similar to environment
variables. "configure" may or may not use these.

Ralf

[Piet van Oostrum]

Well, GCC uses --with-gmp. I don't think you can be more 'GNUish' than GCC.

From https://gcc.gnu.org/install/prerequisites.html:

GNU Multiple Precision Library (GMP) version 4.3.2 (or later)

Necessary to build GCC. If a GMP source distribution is found in a subdirectory of your GCC sources named gmp, it will be built together with GCC. Alternatively, if GMP is already installed but it is not in your library search path, you will have to configure with the --with-gmp configure option. See also --with-gmp-lib and --with-gmp-include. The in-tree build is only supported with the GMP version that download_prerequisites installs.

~/src/gcc/gcc/configure --help | fgrep gmp
--with-gmp-dir=PATH this option has been REMOVED
--with-gmp=PATH specify prefix directory for the installed GMP
--with-gmp-include=PATH/include plus
--with-gmp-lib=PATH/lib
--with-gmp-include=PATH specify directory for installed GMP include files
--with-gmp-lib=PATH specify directory for the installed GMP library

I would suggest to use the same system, and keep --enable-gmp for backwards compatibility.

[Ralf Hemmecke]

On 07/20/2017 09:49 AM, Piet van Oostrum wrote:
> Well, GCC uses --with-gmp. I don't think you can be more 'GNUish' than GCC.

...

> I would suggest to use the same system, and keep --enable-gmp for backwards compatibility.

Yes.

* configure should test for availability of gmp libraries and build with
gmp by default. (--enable-gmp is not necessary)
* If libraries are in an uncommon path then --with-gmp (and friends)
should give hints.
* --enable-gmp=no would not build without GMP.
* --enable-gmp and no availablility of gmp header/library is a fatal
error at configure time.

Is this GNUish enough?

Ralf

[Piet van Oostrum]

Ralf Hemmecke wrote:

> * --enable-gmp=no would not build without GMP.

There are three negations in the above line.
Do you perhaps mean:
* --enable-gmp=no would build without GMP.

The interesting question remains: Should building with GMP be the default (if at all possible) or should it be explicitly enabled?

[Ralf Hemmecke]

On 07/20/2017 11:23 AM, Piet van Oostrum wrote:
> Ralf Hemmecke wrote:
>
>> * --enable-gmp=no would not build without GMP.
>
> There are three negations in the above line. Do you perhaps mean: *
> --enable-gmp=no would build without GMP.

Of course.

> The interesting question remains: Should building with GMP be the
> default (if at all possible) or should it be explicitly enabled?

Yes, an interesting question, given that there are forks of GMP.
http://www.mpir.org/ (no idea how API compatible MPIR is to GMP.

Other than that I don't see a good argument against making GMP the
default. I only faintly remember that Waldek included GMP support
initially only for sbcl more or less as an experiment. Promising users
that GMP works on all platforms an with any FriCAS supported LISP
variant, is probably too much effort at the moment.

Ralf

[Piet van Oostrum]

Waldek Hebisch wrote:

> Piet van Oostrum wrote:

> [...]

> > I also found a tricky problem with the current configure.ac, when libgmp is in a non-standard location. Then '-lgmp' could be settled permanently in the variable 'LIBS' and that may cause other tests to fail. I have also solved this.
>
> IIUC LIBS is supposed to contanin everything needed to use library.
> Extra data for GMP should either fail GMP test or be harmless
> for other tests. In particular, if there is same magic in
> environment needed for GMP it should work everwhere.

GMP is only used for the compilation of src/lib/gmp_wrap.c and there it is taken care of in the Makefile, which contains $(GMP_LIB) in the compilation command. Configure sets this to '-lgmp'. So there is no need to also put it in $LIBS. In fact in some cases putting it in $LIBS makes it to fail other tests. For example in my case, where libgmp is in /opt/local/lib, '-lgmp' makes only sense if preceded by '-L/opt/local/lib'.

Now when $LIBS contained '-lgmp', the test for ptyopen failed because then it will compiled with -lgmp, which isn't found as the '-L/opt/local/lib' is not provided. So I suggest to keep the '-lgmp' out of $LIBS.

The Makefile for gmp_wrap.c must still be enhanced to add -I and -L options on the compile command, in case libgmp is in a non-standard location. I have a patch for that.

[Waldek Hebisch]

Piet van Oostrum wrote:
>
> Waldek Hebisch wrote:
>
> > Piet van Oostrum wrote:
> > [...]
> > > I also found a tricky problem with the current configure.ac, when libgmp is in a non-standard location. Then '-lgmp' could be settled permanently in the variable 'LIBS' and that may cause other tests to fail. I have also solved this.
> >
> > IIUC LIBS is supposed to contanin everything needed to use library.
> > Extra data for GMP should either fail GMP test or be harmless
> > for other tests. In particular, if there is same magic in
> > environment needed for GMP it should work everwhere.
>
> GMP is only used for the compilation of src/lib/gmp_wrap.c and there it is taken care of in the Makefile, which contains $(GMP_LIB) in the compilation command. Configure sets this to '-lgmp'. So there is no need to also put it in $LIBS. In fact in some cases putting it in $LIBS makes it to fail other tests. For example in my case, where libgmp is in /opt/local/lib, '-lgmp' makes only sense if preceded by '-L/opt/local/lib'.

Well, _we_ put '-lgmp' in GMP_LIB (in configure.ac which gets propagated
further). We do not touch LIBS -- handling of LIBS is entirely done
by autoconf macros. Now, my point is that at point where '-lgmp'
gets into flags variable it should get there together with appropriate
'-L...' options (if any such option is needed). So, first we need
to fix handling of GMP_LIB. AFACS we can avoid '-lgmp' in LIBS
by giving dummy success action to AC_CHECK_LIB.

>
> Now when $LIBS contained '-lgmp', the test for ptyopen failed because then it will compiled with -lgmp, which isn't found as the '-L/opt/local/lib' is not provided. So I suggest to keep the '-lgmp' out of $LIBS.

As I wrote, LIBS is set by autoconf. '-lgmp' gets into LIBS
as byproduct of test. AFACS we can avoid '-lgmp' in LIBS
by giving dummy success action to AC_CHECK_LIB.

--
Waldek Hebisch

[Waldek Hebisch]

Well, the tricky question is what to do when there is no GMP
in sight? We could fall back to non-GMP build, but than
effect of build depends in non-obvoius way on build environment.

So we probably should fail by default if GMP is not available.
But this is not nice if system has no GMP and user is
satisfied by non-GMP build.

Extra remark: GMP support is only for sbcl and Closure CL.
ECL and GCL use GMP, so there is no need for extra support.
Supporting GMP with Clisp would take a lot of effort and
given that Clisp in general is slow is probably not worth
the effort.

--
Waldek Hebisch

[Waldek Hebisch]

Piet van Oostrum wrote:
>
> Well, GCC uses --with-gmp. I don't think you can be more 'GNUish' than GCC.
>
> >From https://gcc.gnu.org/install/prerequisites.html:
>
> GNU Multiple Precision Library (GMP) version 4.3.2 (or later)
>
> Necessary to build GCC.

^^^^^^^^^^^^^^^^^^^^^^^

You _must_ have GMP to build GCC. So GCC does not need '--enable'
because the feature is not optional -- it is always on. The
'--with' argument just specify location.

--
Waldek Hebisch

[Piet van Oostrum]

You're right. So what about using both, but letting --with-gmp implicitely also setting --enable-gmp. I think that would be the easiest.

[Piet van Oostrum]

Yes, that makes it tricky to have it enabled by default. I think the cleanest is to disable GMP by default, and to fail if it is enabled but not possible because of the wrong Lisp, or because it is not available.

[Piet van Oostrum]

Or by saving LIBS before the test and restoring it after the test. This is also done sometimes for other variables. For example LDFLAGS in the current test for GMP availability.

[Piet van Oostrum]

Waldek Hebisch wrote:

> Well, _we_ put '-lgmp' in GMP_LIB (in configure.ac which gets propagated
> further). We do not touch LIBS -- handling of LIBS is entirely done
> by autoconf macros. Now, my point is that at point where '-lgmp'
> gets into flags variable it should get there together with appropriate
> '-L...' options (if any such option is needed). So, first we need
> to fix handling of GMP_LIB. AFACS we can avoid '-lgmp' in LIBS
> by giving dummy success action to AC_CHECK_LIB.

I have fixed that (adding the proper -I and -L options) on my local machine, in order to be able to compile it on my system.

[Piet van Oostrum]

Waldek Hebisch wrote:

> Piet van Oostrum wrote:
[...]

> > So it makes sense to give the generated/copied file a new name.
> > Also on systems where libgmp is on a non-standard location, or on Mac OS where the extension is .dylib rather than .so it makes sense to give the file to be compiled a different name.
>
> Well, there are separate issues. 'num_gmp.lisp' is a Lisp file
> which in principle can be loaded even if there is no GMP in the
> system. 'gmp_wrap.c' is C file used to produce wrapper library.
> 'gmp_wrap.c' needs GMP headers and whatever system needs to
> produce shared library. After installation corresponding
> library is located in FriCAS installation tree. I see
> no reason against using the same name 'gmp_wrap.so' on all
> systems. Actual GMP library comes from system and we need to
> use name chosen by the system.

Yes, it is OK to use 'gmp_wrap.so' also on Mac. No problem, it just needs to be compiled with the proper -I and -L flags.

But on the Mac, num_gmp.lsp must be changed.

The line
(if (ignore-errors (|quiet_load_alien| "libgmp.so") t)
on a Mac must be changed to
(if (ignore-errors (|quiet_load_alien| "libgmp.dylib") t)
or if libgmp is in a non-standard location, like /opt/local/lib, then to

(if (ignore-errors (|quiet_load_alien| "/opt/local/lib/libgmp.dylib") t)

I will supply my patches to this mailing list.
I have everything now compiling correctly on my Mac.

[Piet van Oostrum]

Here are my patches to get Fricas + GMP working on Mac OS.

[Waldek Hebisch]

Piet van Oostrum wrote:
>
> Here are my patches to get Fricas + GMP working on Mac OS.

Thanks for the patches. Some comments:

- patch to num_gmp.lisp is already applied
- I think that '--enable-gmp=no' should disable GMP regardless
of other options
- I think that giving dummy action to avoid change to LIBS is
cleaner than save-restore
- I think that GMP_DYLIB is a bit misleading

Attached is a version which I modified to take those issues into
account, in particular I replace GMP_DYLIB by GMP_EXT. I also
shortend long lines and changed order of paragraphs in INSTALL
to make it more consistent.

I hope this version works on Macs (I only tested on Linux).

BTW. I will be for some days without net access.

--
Waldek Hebisch

[Piet van Oostrum]

Maybe it would be advisable to give a warning if --enable-gmp=no is given with --with-gmp?

I tested these patches on the Mac and it works.

I compiled without pre-generated lisp files (with 'make PREGENERATED=') because of the previous patch to elemntry.spad. The compile took some 10 hours on my MacBook Pro.

I noticed a lot of warnings of the type:
Unexpected HT command: \spad
and a lot of similar ones. I suppose these are the TeX commands in the documentation and they can be ignored. I checked one of the generated Lisp files and it looked OK.
However, it would be nice if these messages could be suppressed.
--
Piet van Oostrum <pie...@vanoostrum.org>

[Ralf Hemmecke]

> I compiled without pre-generated lisp files (with 'make PREGENERATED=') because of the previous patch to elemntry.spad. The compile took some 10 hours on my MacBook Pro.

10 hours? I run the build with sbcl and make -j8 (on i7 2720QM. It takes
(including the build of the fricas-aldor interface less than 30 min real
time.

> I noticed a lot of warnings of the type:
> Unexpected HT command: \spad
> and a lot of similar ones. I suppose these are the TeX commands in the documentation and they can be ignored. I checked one of the generated Lisp files and it looked OK.
> However, it would be nice if these messages could be suppressed.

Probably.

I must have eleminated most of those messages in my mathjax branch.
Probably, because in some of the commits I've cleaned up all those
different \spad... commands.

Now I only have about 27 problems left. Most of them complain about \url.

https://github.com/hemmecke/fricas/commits/mathjax
(But take care, that's nothing to build upon, since I take the freedom
to rebase it on top of SVN trunk.)

Ralf