Spurious bugs when interfacing FriCAS - Julia

[Grégory Vanuxem]

Hello Waldek, hello all,

First of all Julia is essentially aimed at scientific computing (numerical) so not a real competitor.

I continue to interface it and FriCAS with SBCL but I encounter an annoying bug. Maybe you will be able to help me here. At least explain to me how Domains and Packages are loaded and used.

The base is done for me, but... The Julia library requires an initialisation and I can't figure out how to initialize its environment at startup, I always encounter a fatal error: ReadOnlyMemoryError

See :

============================================================

└─$ fricas -nosman -rl
Checking for foreign routines
FRICAS="/usr/local/lib/fricas/target/x86_64-linux-gnu"
spad-lib="/usr/local/lib/fricas/target/x86_64-linux-gnu/lib/libspad.so"
gmp-lib="/usr/local/lib/fricas/target/x86_64-linux-gnu/lib/gmp_wrap.so"
julia-lib="/usr/local/lib/fricas/target/x86_64-linux-gnu/lib/julia_wrap.so"
foreign routines found
openServer result -2
                       FriCAS Computer Algebra System
                         Version: FriCAS 2022-07-16
                  Timestamp: mer. 21 déc. 2022 10:35:13 CET

(1) -> )lisp (init-julia-env)

Value = T
(1) -> )lisp (jl-dbl-function "exp" 1.0)

Value = 2.718281828459045

(1) -> exp(1.0)

fatal: error thrown and no exception handler available.
ReadOnlyMemoryError()
unknown function (ip: 0x52f7062f)
unknown function (ip: 0x52ca710e)
unknown function (ip: 0x52a47557)
unknown function (ip: 0x530ec65c)
unknown function (ip: 0x52e32622)
unknown function (ip: 0x52a146dd)
unknown function (ip: 0x52a1ed73)
unknown function (ip: 0x52c021fc)
unknown function (ip: 0x52af5b24)
unknown function (ip: 0x52a4a816)
unknown function (ip: 0x52c301b9)
unknown function (ip: 0x52aa254b)
unknown function (ip: 0x52a0e138)
unknown function (ip: 0x52b0ddf3)
unknown function (ip: 0x52a15b9a)
unknown function (ip: 0x52a49e58)
unknown function (ip: 0x52c38e76)
unknown function (ip: 0x52a1b335)
unknown function (ip: 0x52aa4ba8)
unknown function (ip: 0x52b0adb7)
unknown function (ip: 0x52c24c68)
unknown function (ip: 0x52a11f60)
unknown function (ip: 0x52ab0634)
unknown function (ip: 0x52b410cb)
unknown function (ip: 0x52b40de2)
unknown function (ip: 0x52c97778)
unknown function (ip: 0x52a4f136)
unknown function (ip: 0x52ca51db)
unknown function (ip: 0x52ca5059)
unknown function (ip: 0x52ca6283)
unknown function (ip: 0x52ca4ed8)
unknown function (ip: 0x52ca4afd)
unknown function (ip: 0x52ab1645)
unknown function (ip: 0x52ca3359)
unknown function (ip: 0x52b4498d)
unknown function (ip: 0x52ca2ee2)
unknown function (ip: 0x52ca2c04)
unknown function (ip: 0x52ca2a10)
unknown function (ip: 0x52ca2869)
unknown function (ip: 0x52aeaf1c)
unknown function (ip: 0x52a8914b)
unknown function (ip: 0x53366b14)
unknown function (ip: 0x53366919)
unknown function (ip: 0x53366741)
call_into_lisp_ at /usr/local/lib/fricas/target/x86_64-linux-gnu/bin/FRICASsys (unknown line)
initialize_lisp at /usr/local/lib/fricas/target/x86_64-linux-gnu/bin/FRICASsys (unknown line)
main at /usr/local/lib/fricas/target/x86_64-linux-gnu/bin/FRICASsys (unknown line)
unknown function (ip: 0x7fc6f30cb189)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at /usr/local/lib/fricas/target/x86_64-linux-gnu/bin/FRICASsys (unknown line)

===============================================================

This is because above I begin by initializing Julia's env. So I can't even build FriCAS if I automatize at startup the Julia's env.

But, apparently, and I don't even know if I am in the right thinking direction, if I "play" with different FriCAS's domains this does not happen. Sometimes in fact.

See :

==============================================================

$ fricas -nosman -rl
Checking for foreign routines
FRICAS="/usr/local/lib/fricas/target/x86_64-linux-gnu"
spad-lib="/usr/local/lib/fricas/target/x86_64-linux-gnu/lib/libspad.so"
gmp-lib="/usr/local/lib/fricas/target/x86_64-linux-gnu/lib/gmp_wrap.so"
julia-lib="/usr/local/lib/fricas/target/x86_64-linux-gnu/lib/julia_wrap.so"
foreign routines found
openServer result -2
                       FriCAS Computer Algebra System
                         Version: FriCAS 2022-07-16
                  Timestamp: mer. 21 déc. 2022 10:35:13 CET
(1) -> sqrt(2)

         +-+
   (1)  \|2
                                                        Type: AlgebraicNumber
(2) -> 2*x+7

   (2)  2 x + 7
                                                    Type: Polynomial(Integer)
(3) -> sqrt %

         +-------+
   (3)  \|2 x + 7
                                                    Type: Expression(Integer)
(4) -> sqrt(2.0)

   (4)  1.4142135623_730950488
                                                                  Type: Float
(5) -> )lisp (init-julia-env)

Value = T
(5) -> )lisp (jl-dbl-function "exp" 1.0)

Value = 2.718281828459045
(5) -> )lisp (jl-eval-string "using SpecialFunctions")

Value = NIL
(5) -> )lisp (jl-eval-string "print(erf(0.7))")
0.6778011938374184
Value = NIL
(5) ->
(5) -> 2*x+7

   (5)  2 x + 7
                                                    Type: Polynomial(Integer)
(6) -> sqrt(2.0)

   (6)  1.4142135623_730950488
                                                                  Type: Float
(7) -> sqrt(2.0::SF)

   (7)  1.4142135623730951
                                                            Type: DoubleFloat

========================================================

========================================================

As you can see I hope, it appears that it works in this case. And frankly I can't figure out why.

I was thinking that some times ago, Waldek, you explained that during the load of a FriCAS library (domain, package or whatever) a vector (array) was created or populated with addresses of its different functions among other things.

Am I wrong? And maybe here, someone can tell me in which direction I have to look?

__

Greg

PS1: Julia: https://julialang.org/

PS2: If you want to give it a try, I insist it's just laying the foundations right now and I want good bases: https://github.com/gvanuxem/fricas (jfricas branche). You'll also need Julia. My advice is to use the julialang.org version not the Debian one, it's really a big project, difficult to maintain I think.

After, personally I do a:

$ ./configure --without-x --enable-gmp --enable-julia --with-julia-config=/usr/local/share/julia/julia-config.jl && make -j4 && sudo make install

Julia can also be run in a directory for info., your home directory for example.

[Grégory Vanuxem]

Quick reminder just in case. Compiling a file before or after using Julia interface : ReadOnlyMemoryError

[Waldek Hebisch]

On Wed, Dec 21, 2022 at 12:38:15PM +0100, Grégory Vanuxem wrote:
> Hello Waldek, hello all,
>
> First of all Julia is essentially aimed at scientific computing (numerical)
> so not a real competitor.
> I continue to interface it and FriCAS with SBCL but I encounter an annoying
> bug. Maybe you will be able to help me here. At least explain to me how
> Domains and Packages are loaded and used.
>
> The base is done for me, but... The Julia library requires an
> initialisation and I can't figure out how to initialize its environment at
> startup, I always encounter a fatal error: ReadOnlyMemoryError

I have never seen such a message.

>
> See :
> ============================================================
> └─$ fricas -nosman -rl
> Checking for foreign routines
> FRICAS="/usr/local/lib/fricas/target/x86_64-linux-gnu"
> spad-lib="/usr/local/lib/fricas/target/x86_64-linux-gnu/lib/libspad.so"
> gmp-lib="/usr/local/lib/fricas/target/x86_64-linux-gnu/lib/gmp_wrap.so"
> julia-lib="/usr/local/lib/fricas/target/x86_64-linux-gnu/lib/julia_wrap.so"
> foreign routines found
> openServer result -2
> FriCAS Computer Algebra System
> Version: FriCAS 2022-07-16
> Timestamp: mer. 21 déc. 2022 10:35:13 CET
>
> (1) -> )lisp (init-julia-env)

Big question is what 'init-julia-env' is doing.

> Value = T
> (1) -> )lisp (jl-dbl-function "exp" 1.0)
>
> Value = 2.718281828459045
> (1) -> exp(1.0)
> fatal: error thrown and no exception handler available.
> ReadOnlyMemoryError()
> unknown function (ip: 0x52f7062f)

<snip>

> This is because above I begin by initializing Julia's env. So I can't even
> build FriCAS if I automatize at startup the Julia's env.
>
> But, apparently, and I don't even know if I am in the right thinking
> direction, if I "play" with different FriCAS's domains this does not
> happen. Sometimes in fact.

If you think that problem is related. would suggest trying:

)set messages autoload on

--
Waldek Hebisch

[Grégory Vanuxem]

Hello Waldek,

Le mar. 27 déc. 2022 à 02:43, Waldek Hebisch
<de...@fricas.math.uni.wroc.pl> a écrit :

>
> On Wed, Dec 21, 2022 at 12:38:15PM +0100, Grégory Vanuxem wrote:
> > Hello Waldek, hello all,

[SNIPPET]

>> The base is done for me, but... The Julia library requires an
>> initialisation and I can't figure out how to initialize its environment at
>> startup, I always encounter a fatal error: ReadOnlyMemoryError

> I have never seen such a message.

Me too, and, difficult to debug in fact.

> >
> > See :
> > ============================================================
> > └─$ fricas -nosman -rl
> > Checking for foreign routines
> > FRICAS="/usr/local/lib/fricas/target/x86_64-linux-gnu"
> > spad-lib="/usr/local/lib/fricas/target/x86_64-linux-gnu/lib/libspad.so"
> > gmp-lib="/usr/local/lib/fricas/target/x86_64-linux-gnu/lib/gmp_wrap.so"
> > julia-lib="/usr/local/lib/fricas/target/x86_64-linux-gnu/lib/julia_wrap.so"
> > foreign routines found
> > openServer result -2
> > FriCAS Computer Algebra System
> > Version: FriCAS 2022-07-16
> > Timestamp: mer. 21 déc. 2022 10:35:13 CET
> >
> > (1) -> )lisp (init-julia-env)
>
> Big question is what 'init-julia-env' is doing.

It's a call to the julia 'jl_init' routine in libjulia.so. Julia has
its own interpreter, stack, GC etc. so its environment needs to be initialized.
In fact I wonder if SBCL and Julia are compatible and how to
circumvent this. I'll see.

See: https://docs.julialang.org/en/v1/manual/embedding/

(defun init-julia-env()
(if (not *julia-initialized*)
(progn
(sb-alien:alien-funcall
(sb-alien:extern-alien "jl_init" (sb-alien:function
sb-alien:void)))
(setf *julia-initialized* t))))

> > Value = T
> > (1) -> )lisp (jl-dbl-function "exp" 1.0)
> >
> > Value = 2.718281828459045
> > (1) -> exp(1.0)
> > fatal: error thrown and no exception handler available.
> > ReadOnlyMemoryError()
> > unknown function (ip: 0x52f7062f)
> <snip>
> > This is because above I begin by initializing Julia's env. So I can't even
> > build FriCAS if I automatize at startup the Julia's env.
> >
> > But, apparently, and I don't even know if I am in the right thinking
> > direction, if I "play" with different FriCAS's domains this does not
> > happen. Sometimes in fact.
>
> If you think that problem is related. would suggest trying:
>
> )set messages autoload on

Good idea, I didn't think about that. I'll give it a try. This is for
sure related but not only I think.
That's relatively difficult because of possible memory corruption
behind the scene.

For now to test my spad code FriCAS is initialized via .fricasrc with:
)set mess type off
sqrt 2;sqrt 2.0;x^3=7;x^3+7;
)set mess type on
--)lisp (init-julia-env)

But for example it's totally impossible to compile a .spad file if
Julia is initialized so I have to work around that to compile my spad
code i.e. use FriCAS with uninitialized Julia.
__
Greg


>
> --
> Waldek Hebisch
>
> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/20221227022657.ypgx275453f43ai5%40fricas.math.uni.wroc.pl.

[Waldek Hebisch]

On Tue, Dec 27, 2022 at 10:52:04AM +0100, Grégory Vanuxem wrote:
> Hello Waldek,
>
> Le mar. 27 déc. 2022 à 02:43, Waldek Hebisch
> <de...@fricas.math.uni.wroc.pl> a écrit :
> >
> > On Wed, Dec 21, 2022 at 12:38:15PM +0100, Grégory Vanuxem wrote:
> > > Hello Waldek, hello all,
> [SNIPPET]
>
> >> The base is done for me, but... The Julia library requires an
> >> initialisation and I can't figure out how to initialize its environment at
> >> startup, I always encounter a fatal error: ReadOnlyMemoryError
>
> > I have never seen such a message.
>
> Me too, and, difficult to debug in fact.

<snip>

> > > fatal: error thrown and no exception handler available.
> > > ReadOnlyMemoryError()
> > > unknown function (ip: 0x52f7062f)
> > <snip>

Just a little remark: Google says that "error thrown and no
exception handler available" is Julia error message. Since
you are _not_ calling Julia this must be exception handler
installed by Julia. But sbcl depends on exceptions and
its execetions should go to sbcl handlers. If Julia and
sbcl wants to take over the same signal (likely SIGSEGV),
then you are in trouble.

--
Waldek Hebisch

[Tim Daly]

The system is just Lisp code so you can use a lisp break to debug.

I'm not sure if it still exists but you can get to the heart of the

problem by using the lisp debugger. At startup type:

)set break break

any low-level error will throw you into the Lisp REPL and tell you everthing

you might want to know.

-- Tim

[Grégory Vanuxem]

Le mar. 27 déc. 2022 à 14:41, Waldek Hebisch

You're in touch with God I believe*, thanks. I believe something is right here.
__
Greg
* not yo be taken literally (SIGTRAP for info)

[Tim Daly]

Try starting Julia with:

julia --lisp

This isn't documented because it's not meant to be an official part of Julia.

At present, Julia's parser is written in Scheme. That command lets you use

the Scheme implementation that Julia uses internally.

Then look at

(trace <procedure-name>)

to get closer to the failure.

[Tim Daly]

If you want to find a SIGSEGV trap run the whole thing under GDB.

This link might help.

http://www.unknownroad.com/rtfm/gdbtut/gdbsegfault.html

Set the trap on the line issuing the error message. It will let you

see the stack.

Good programming practice for setting things like SIGSEGV is to

remember the current setting, install your handler, and then call

the saved setting. This creates a chain of handlers.

I believe Julia is open source. If Julia isn't doing that, then fix it

and post a patch.

An easier path is to just comment out the SIGSEGV setting code

and see if the fault goes away.

Tim

On Thursday, December 29, 2022 at 11:01:26 AM UTC-5 g.va...@gmail.com wrote:

[Grégory Vanuxem]

Hi,

In fact I think this is the well known SBCL message: « heap is exhausted, augment dynamic space size » or something like that. But Julia catchs the signal and stops all apparently. I have rebuilt the SBCL deb file with a modified debian/rules (8 Gb of dynamic space size) and now I can compile, use polynomials and even no longer have to use a modified .fricasrc file. All seems right.

By the way, many thanks for all advices I'll use them, I need to look deeper in signals handler in Julia. Yes, Julia us open  source (MIT licence).

__

Greg

--

You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/955b588c-3404-4969-be57-f15bb94fdcc3n%40googlegroups.com.

[Waldek Hebisch]

On Sat, Dec 31, 2022 at 04:29:48PM +0100, Grégory Vanuxem wrote:
> Hi,
>
> In fact I think this is the well known SBCL message: « heap is exhausted,
> augment dynamic space size » or something like that.

Hmm, you should not see this on so small computation. SBCL uses
SIGSEGV to determine if pages are modified or not. If you run
sbcl via gdb you should see a lot of SIGSEGV-s which in C program
would mean crash, but are normal in sbcl.

You write that you can run when you give 8G to sbcl. I would be
_very_ suspicious of such a solution. Simply, I saw too many
programs that were horribly broken, yet under light tests
appeared to work correctly. One possiblity is that having 8G
you did not manage to trigger sbcl garbage collection (or maybe
major collection since sbcl collector is generational).

--
Waldek Hebisch

[Grégory Vanuxem]

Oh, thanks for the information. I have to look deeper in this for sure.
Moreover, I'll soon add arrays support, so probably malloc, free,
Julia's garbage collector manipulations and so on so I can not let
remain issues like this one.

Le sam. 31 déc. 2022 à 22:29, Waldek Hebisch
<de...@fricas.math.uni.wroc.pl> a écrit :
>

> --
> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/20221231221319.fdm4yfoakbt5vbnc%40fricas.math.uni.wroc.pl.

[Tim Daly]

If you run Julia in a separate process and use a socket connection

then the SIGSEGV use by SBCL won't matter as the handlers would

be swapped by the system at process swap time.

If you are using sb-ext:run-program the sub-program, in this case,

Julia would have to replace-and-chain SIGSEGV.

Since the failure is being caught in Julia there could be a couple

reasons. One I already mentioned (replace and chain failure).

Another might be how sb-ext:run-program handles process memory.

It could be that there really isn't enough process memory to run Julia

as a sub-process. For size-limited computations enlarging memory

as you did would work fine. Julia seems to use 200MB of memory

https://discourse.julialang.org/t/julia-memory-overhead/1861

Julia provides "handle-signals" to enable or disable default signal

handlers: julis --handle-signals=no

A more subtle error might be that SBCL moved a memory reference

from the embedded process interface. See

https://docs.julialang.org/en/v1/manual/embedding/

A known approach is to use swank (this is how emacs embed certain programs):

https://github.com/brown/swank-client

If you really want to play hardball then allocate sbcl memory with

the allocated object is larger than BYTES-CONSED-BETWEEN-GCS.

This it tends to get promoted to gen 1 right after allocation, while the reference

is still live. Allocate all Julia memory from addresses within that block.

Or you could memory-map a "file" and do what appears to be file-io

but use memory references into the pseudo-file.

When all else fails you could hack Julia source code with an SBCL

friendly function to call along with some SBCL friendly routines to

copy Julia/SBCL objects back and forth.

There are many ways to skin a cat. Be sure to read the side of the box:

This is a quantum entanglement test. Please be sure that you

observe the cat before choosing to skin it.

Tim

--
You received this message because you are subscribed to a topic in the Google Groups "FriCAS - computer algebra system" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fricas-devel/9WNtYSSJp0w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fricas-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/20221231221319.fdm4yfoakbt5vbnc%40fricas.math.uni.wroc.pl.

[Grégory Vanuxem]

Le dim. 1 janv. 2023 à 11:10, Tim Daly <axio...@gmail.com> a écrit :
>
> If you run Julia in a separate process and use a socket connection
> then the SIGSEGV use by SBCL won't matter as the handlers would
> be swapped by the system at process swap time.
>
> If you are using sb-ext:run-program the sub-program, in this case,
> Julia would have to replace-and-chain SIGSEGV.

No, it's a shared library (a wrapper) built in the fricas/src/lib
directory and loaded at startup with libjulia.so.
(defun init-julia(wrapper-lib)
(if (ignore-errors (|quiet_load_alien| wrapper-lib) t)
(|quiet_load_alien| "libjulia.so")))

> Since the failure is being caught in Julia there could be a couple
> reasons. One I already mentioned (replace and chain failure).

Yes, my first thought is to temporarily bypass the Julia signals
handler and see what happens at SBCL level. When I encountered the bug
mentioned at the beginning of this thread what came in my mind was a
bad memory access to arrays or something like that and I modified
src/lisp/interp-proclaims to (debug 3) (safety 3) but, yes, Julia
catches the signal and crash as it has no way to handle this error:

fatal: error thrown and no exception handler available.
ReadOnlyMemoryError()

So this modification was without effect.

> Another might be how sb-ext:run-program handles process memory.
> It could be that there really isn't enough process memory to run Julia
> as a sub-process. For size-limited computations enlarging memory
> as you did would work fine. Julia seems to use 200MB of memory
> https://discourse.julialang.org/t/julia-memory-overhead/1861
>
> Julia provides "handle-signals" to enable or disable default signal
> handlers: julis --handle-signals=no

Yes!!! Good! I'll test that. Since this is implemented there must be a
way to bypass Julia signals handler via the Julia library.
Otherwise, I think I'll have to first look at Julia source code and
eventually do a temporary dirty hack to Julia.

__
Greg

PS: I'll take a look at the second part of your mail later as it
contains tons of things.

> You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/CAJn5L%3DKKtUB2RvaPJ2vWODk%2Bm1D7xSSVvF9fQd%3D28pvvLuTYdg%40mail.gmail.com.