Please help: wierd bug with library loading

[Ralf Hemmecke]

Hello (in particular Waldek),

while trying to run my code I ran into a terrible bug. Obviously,
loading a library into a session is NOT the same as compiling it in that
session.

Well, the particular problem here is perhaps that
etaQuotientIdealGenerators is defined in the category FooCat from which
the package FooPkg inherits. When I copy the etaQuotientIdealGenerators
code into FooPkg, i.e., define the function inside the package,
everything is find. However, such a code duplication is what I wanted to
avoid by introducing the category.

Instructions to reproduce:

echo ")read foo.input" |fricas -nosman
echo ")read fool.input" |fricas -nosman

The first run compiles and then ends in

etaQuotientIdealGenerators(6, [[1,-2]])$FooPkg(Integer, Integer)


(1) []
Type:
List(Polynomial(Integer))

while the second command yields:

trex:~/scratch/spad>echo ")read fool.input" |fricas -nosman
Checking for foreign routines
AXIOM="/home/hemmecke/g/fricas-bisect/install/lib/fricas/target/x86_64-linux-gnu"
spad-lib="/home/hemmecke/g/fricas-bisect/install/lib/fricas/target/x86_64-linux-gnu/lib/libspad.so"
foreign routines found
openServer result -2
FriCAS Computer Algebra System
Version: FriCAS e407a5a0040d7de504f0d8db31d5cf2ce6678d66
Timestamp: Tue Jul 18 21:51:24 CEST 2017
-----------------------------------------------------------------------------
Issue )copyright to view copyright notices.
Issue )summary for a summary of useful system commands.
Issue )quit to leave FriCAS and return to shell.
-----------------------------------------------------------------------------

(1) -> )lib FOOCAT

FooCat is now explicitly exposed in frame initial
FooCat will be automatically loaded when needed from
/home/hemmecke/scratch/spad/FOOCAT.NRLIB/FOOCAT
)lib FOOPKG

FooPkg is now explicitly exposed in frame initial
FooPkg will be automatically loaded when needed from
/home/hemmecke/scratch/spad/FOOPKG.NRLIB/FOOPKG
etaQuotientIdealGenerators(6, [[1,-2]])$FooPkg(Integer, Integer)


>> System error:
The index 1 is too large.

Is there a way to load the libraries without recompiling the code?
Or perhaps an easy way to fix the problem.

Ralf

[oldk1331]

I might find something:

The FooCat category comes with a default implementation
of function etaQuotientIdealGenerators, so the compiler
generates a "dummy domain" for that, "FOOCAT-".

If you add ")lib FOOCAT-", then there's no problem.

So, if my analysis is correct, we should automatically
load XXXCAT- when ')lib XXXCAT' is issued.

[Ralf Hemmecke]

On 09/08/2017 04:52 AM, oldk1331 wrote:
> I might find something:
>
> The FooCat category comes with a default implementation
> of function etaQuotientIdealGenerators, so the compiler
> generates a "dummy domain" for that, "FOOCAT-".
>
> If you add ")lib FOOCAT-", then there's no problem.

THANK YOU. That's a pretty solution for now.

> So, if my analysis is correct, we should automatically
> load XXXCAT- when ')lib XXXCAT' is issued.

Let's see what Waldek says. But I would appreciate that the default
package is automatically loaded if the category is loaded via )lib.

Ralf

[Waldek Hebisch]

I can not say if not loading category defaults is a bug or
missing feature. But certainly I would like to get rid of
crash with no hint to the reason...

--
Waldek Hebisch

[Ralf Hemmecke]

On 09/11/2017 07:18 PM, Waldek Hebisch wrote:
>>> So, if my analysis is correct, we should automatically
>>> load XXXCAT- when ')lib XXXCAT' is issued.
>>
>> Let's see what Waldek says. But I would appreciate that the default
>> package is automatically loaded if the category is loaded via )lib.
>
> I can not say if not loading category defaults is a bug or
> missing feature.

Your statement sounds a bit strange. If I interpret you correctly then
"not loading" is a bug and "loading a default package when loading the
category" is a missing feature.

In this interpretation, I don't care about a proper error message.
Better would be to load the default package of a category automatically
with the category. I don't see any good reason why "not loading" is a
"good" behaviour.

Asking the user to load the default package explicitly, is bad. User A
provides version 1 of a library with a category. User B compiles this
(external code) and then uses )lib to load the code. In release 2 User I
simply moves code from some domain(s) into a category default without
changing any functionality. Then user B would have to compile and add a
")lib FOOCAT-" to his load-libraries script.

Even if there were a proper error message, I find that additional )lib
command unnecessary.

I opt for fixing this bug by automatically loading category defaults
with the category unless one of the original Axiom developers tell me
why "not loading" is a "good thing".

Ralf

[oldk1331]

Strictly speaking, the argument passed to ')library' is a directory,
it often happens to be an abbreviation of Domain/Category/Package.

So strictly speaking, when you pass ")lib FOOCAT", it doesn't
know that FOOCAT is a Category, it just sees a directory.

We can make it a feature that when ")lib XXX" is issued,
execute ")lib XXX-" automatically if such directory exists.
It would be hard to know if the XXX and XXX- are created
at the same time and compatible, but I don't think that
would be a problem in 99% cases.

This patch is a prototype, it can't handle the case when
')dir' option is also passed. If this idea is accepted, I
will rewrite it in daase.lisp and write documentation for ")help lib".

diff --git a/src/interp/i-syscmd.boot b/src/interp/i-syscmd.boot
index 6b099395..07b4c569 100644
--- a/src/interp/i-syscmd.boot
+++ b/src/interp/i-syscmd.boot
@@ -2026,6 +2026,10 @@
library(args) ==
$newConlist : local := []
original_directory := GET_-CURRENT_-DIRECTORY()
+ for dir in args repeat
+ defaultcat := STRCONC(dir, '"-")
+ if PROBE_-FILE(STRCONC(defaultcat, '".NRLIB/"))
+ then args := cons(defaultcat, args)
LOCALDATABASE(args, $options)
extendLocalLibdb($newConlist)
CHDIR(original_directory)

[Ralf Hemmecke]

Hi Waldek,

I don't see this patch applied. Can Qian commit it?

EMail thread is here
https://groups.google.com/forum/#!searchin/fricas-devel/Please$20help$3A$20wierd$20bug$20with$20library$20loading%7Csort:date/fricas-devel/_XPWYc4RACA/5no2Qv5eDAAJ

or here...

https://www.mail-archive.com/fricas...@googlegroups.com/msg11471.html

Ralf

[Waldek Hebisch]

Ralf Hemmecke wrote:
>
> I don't see this patch applied. Can Qian commit it?
>
> EMail thread is here
> https://groups.google.com/forum/#!searchin/fricas-devel/Please$20help$3A$20wierd$20bug$20with$20library$20loading%7Csort:date/fricas-devel/_XPWYc4RACA/5no2Qv5eDAAJ
>
> or here...
>
> https://www.mail-archive.com/fricas...@googlegroups.com/msg11471.html
>
> Ralf
>

I find the patch problematic. It makes assumptions that may be
valid or not. IIRC we store information about default package
somewhere, so proper patch would dig out this information and
act on it. Possibly we need to fix propagation of this information.

--
Waldek Hebisch

[Ralf Hemmecke]

Hi and happy new year to everyone!

On 1/2/19 4:08 PM, Waldek Hebisch wrote:
>> https://www.mail-archive.com/fricas...@googlegroups.com/msg11471.html

> I find the patch problematic. It makes assumptions that may be
> valid or not. IIRC we store information about default package
> somewhere, so proper patch would dig out this information and
> act on it. Possibly we need to fix propagation of this information.

OK, I agree.

If I read the original function

library(args) ==
$newConlist : local := []
original_directory := GET_-CURRENT_-DIRECTORY()

LOCALDATABASE(args, $options)
extendLocalLibdb($newConlist)
CHDIR(original_directory)
terminateSystemCommand()

correctly, then it gets the a list of names of constructors (short name,
i.e., what makes up for the directory name like FOO.NRLIB).

IIUC you are saying that instead of checking whether

PROBE_-FILE(STRCONC(defaultcat, '".NRLIB/"))

gives true, one should simply ask the FriCAS database whether

(a) the constructor is a category constructor,
(b) if yes, whether there is a corresponding default package,
(c) what the abbrev of the default package is.

To amand the patch would probably be simple to do if you could tell me
whether (a-c) is in the database at all (probably true) and how are the
calls to ask for this information.

Ralf

[Waldek Hebisch]

Ralf Hemmecke wrote:
>
> On 1/2/19 4:08 PM, Waldek Hebisch wrote:
> >> https://www.mail-archive.com/fricas...@googlegroups.com/msg11471.html
>
> > I find the patch problematic. It makes assumptions that may be
> > valid or not. IIRC we store information about default package
> > somewhere, so proper patch would dig out this information and
> > act on it. Possibly we need to fix propagation of this information.
>
> OK, I agree.
>
> If I read the original function
>
> library(args) ==
> $newConlist : local := []
> original_directory := GET_-CURRENT_-DIRECTORY()
> LOCALDATABASE(args, $options)
> extendLocalLibdb($newConlist)
> CHDIR(original_directory)
> terminateSystemCommand()
>
> correctly, then it gets the a list of names of constructors (short name,
> i.e., what makes up for the directory name like FOO.NRLIB).

Most FriCAS functions can handle directory paths in names, I would
have to check if this is allowed here.

> IIUC you are saying that instead of checking whether
>
> PROBE_-FILE(STRCONC(defaultcat, '".NRLIB/"))
>
> gives true, one should simply ask the FriCAS database whether
>
> (a) the constructor is a category constructor,
> (b) if yes, whether there is a corresponding default package,
> (c) what the abbrev of the default package is.
>

Silightly different:

(a) load constructor (needed to update database etc.)
(b) check if there is default to load and do it if needed.

> To amand the patch would probably be simple to do if you could tell me
> whether (a-c) is in the database at all (probably true) and how are the
> calls to ask for this information.

Again some searching of source is needed to find out how
info about default is handled. Note that proper logic may be
somewhat tricky due to Aldor support and current organization
of code.


--
Waldek Hebisch

[oldk1331]

Let's use the example from the origin post:

FOOCAT is the category, FOOCAT- is the default domain the implements
part of FOOCAT, FOOPKG is the package that uses FOOCAT-.

Now, the information about FOOCAT- is not mentioned in FOOCAT.NRLIB
at all.  It is inserted in FOOPKG during compilation.

So if we issue ")library FOOCAT" to a fresh system, then we have
no information about FOOCAT- in the database.

[Ralf Hemmecke]

> Let's use the example from the origin post:
>
> FOOCAT is the category, FOOCAT- is the default domain the implements
> part of FOOCAT, FOOPKG is the package that uses FOOCAT-.
>
> Now, the information about FOOCAT- is not mentioned in FOOCAT.NRLIB
> at all. It is inserted in FOOPKG during compilation.
>
> So if we issue ")library FOOCAT" to a fresh system, then we have
> no information about FOOCAT- in the database.

Yes, that's seemingly true. But while I was looking a bit deeper I saw
showdatabase.
https://github.com/fricas/fricas/blob/master/src/interp/daase.lisp#L585

I used it to show me what information I can get. Indeed for

)lisp (showdatabase '|FooCat|)

I get ... (see end of mail). Most interestingly, the entry DEFAULTDOMAIN
shows NIL. I don't know whether this entry will be ever set, because in
daase.lisp one finds

=======================================
; there are only a small number of categories that have default domains.
; rather than keep this slot in every domain we maintain a list here.

(defvar |$defaultdomain_list| '(
(|MultisetAggregate| |Multiset|)
(|FunctionSpace| |Expression|)
(|AlgebraicallyClosedFunctionSpace| |Expression|)
(|ThreeSpaceCategory| |ThreeSpace|)
(|DequeueAggregate| |Dequeue|)
(|ComplexCategory| |Complex|)
(|LazyStreamAggregate| |Stream|)
(|AssociationListAggregate| |AssociationList|)
(|QuaternionCategory| |Quaternion|)
(|PriorityQueueAggregate| |Heap|)
(|PointCategory| |Point|)
(|PlottableSpaceCurveCategory| |Plot3D|)
(|PermutationCategory| |Permutation|)
(|StringCategory| |String|)
(|FileNameCategory| |FileName|)
(|OctonionCategory| |Octonion|)))
=========================================

Maybe I misinterpret the semantics of "defaultdomain", but this list has
certainly nothing to do with the default/add domain of a category,
otherwise I would see KeyedDictionary there.

Anyway, apart from this problem with defaultdomain, a strategy to find
where )lib should look for the a (possible) default package, could be to
simply check whether in ")lib FooCat" FooCat is a category

(15) -> )lisp (getdatabase '|FooCat| 'constructorkind)

Value = |category|

and if it is, to probe FooCat& (i.e. name with "&" appended).

(15) -> )lisp (getdatabase '|FooCat&| 'object)

Value = "/home/hemmecke/scratch/FOOCAT-.NRLIB/FOOCAT-"


Concerning a fix for Aldor...
There does not seem to be any problem at all as the attached bar*.*
files show. Everything works fine. The default package seems to be
loaded when ")lib barcat" is issued.

Running foocomp.input works fine while foolib.input (in a fresh session)
gives

(2) -> foozwei(1)$F

>> System error:
The index 1 is too large.

To me it just makes sense to fix FriCAS so that it automatically load a
default package of a category if the category is loaded via the )lib
command.

Waldek, do you think, it makes sense to follow the above strategy to
prepare a patch for such a fix?

Ralf




===================================================================
)lisp (showdatabase '|FooCat|)

CONSTRUCTORKIND: category
COSIG: (NIL T T)
OPERATION: NIL
CONSTRUCTORMODEMAP:
(((|FooCat| |#1| |#2|) (|Category|) (|EuclideanDomain|) (|Ring|)) (T
|FooCat|))
CONSTRUCTORCATEGORY:
(|Join|
(CATEGORY |package|
(SIGNATURE |relationsIdealGenerators|
((|List| (|Polynomial| |t#1|)) (|List| |t#2|)))
(SIGNATURE |etaQuotientIdealGenerators|
((|List| (|Polynomial| |t#1|)) (|NonNegativeInteger|)
(|List| (|List| (|Integer|)))))))
OPERATIONALIST:
((|relationsIdealGenerators| (((|List| (|Polynomial| |#1|)) (|List|
|#2|)) 6))
(|etaQuotientIdealGenerators|
(((|List| (|Polynomial| |#1|)) (|NonNegativeInteger|)
(|List| (|List| (|Integer|))))
6)))
MODEMAPS:
((|etaQuotientIdealGenerators| (*1 *2 *3 *4)
(AND (|isDomain| *3 (|NonNegativeInteger|))
(|isDomain| *4 (|List| (|List| (|Integer|))))
(|ofCategory| *1 (|FooCat| *5 *6)) (|ofCategory| *5
(|EuclideanDomain|))
(|ofCategory| *6 (|Ring|)) (|isDomain| *2 (|List| (|Polynomial|
*5)))))
(|relationsIdealGenerators| (*1 *2 *3)
(AND (|isDomain| *3 (|List| *5)) (|ofCategory| *1 (|FooCat| *4 *5))
(|ofCategory| *4 (|EuclideanDomain|)) (|ofCategory| *5 (|Ring|))
(|isDomain| *2 (|List| (|Polynomial| *4))))))
HASCATEGORY: NIL
OBJECT: /home/hemmecke/scratch/FOOCAT.NRLIB/FOOCAT
NILADIC: NIL
ABBREVIATION: FOOCAT
CONSTRUCTOR?: T
CONSTRUCTOR: NIL
DEFAULTDOMAIN: NIL
ANCESTORS: NIL
SOURCEFILE: /home/hemmecke/scratch/foo.spad
CONSTRUCTORFORM: (FooCat C L)
CONSTRUCTORARGS: (C L)
PREDICATES:
NIL
DOCUMENTATION: NIL
PARENTS: NIL
Value = NIL

[oldk1331]

But in a fresh session, after you issue ")lib FOOCAT", you can't get
information about FooCat&:

(1) -> )lib FOOCAT
FooCat is now explicitly exposed in frame frame1

FooCat will be automatically loaded when needed from

/home/oldk/tmp/FOOCAT.NRLIB/FOOCAT
(1) -> )lisp (getdatabase '|FooCat| 'constructorkind)

Value = |category|
(1) -> )lisp (getdatabase '|FooCat&| 'object)

Value = NIL


After you load FOOCAT- or in a session that ")compile foo",
only then you can get this information.

(1) -> )lib FOOCAT-
FooCat& is now explicitly exposed in frame frame1
FooCat& will be automatically loaded when needed from
/home/oldk/tmp/FOOCAT-.NRLIB/FOOCAT-
(1) -> )lisp (getdatabase '|FooCat&| 'object)

Value = "/home/oldk/tmp/FOOCAT-.NRLIB/FOOCAT-"

[Ralf Hemmecke]

On 1/9/19 8:13 AM, oldk1331 wrote:
> But in a fresh session, after you issue ")lib FOOCAT", you can't get
> information about FooCat&:
>
> (1) -> )lib FOOCAT
> FooCat is now explicitly exposed in frame frame1
> FooCat will be automatically loaded when needed from
> /home/oldk/tmp/FOOCAT.NRLIB/FOOCAT
> (1) -> )lisp (getdatabase '|FooCat| 'constructorkind)
>
> Value = |category|
> (1) -> )lisp (getdatabase '|FooCat&| 'object)
>
> Value = NIL

Arrrrhhhh... Thank you, very much!

Qian, now I am on your side, meaning the following.

If ")lib FOOCAT" happens, then
(1) load FOOCAT as is done now.
(2) find the location of FOOCAT.NRLIB
(2.1) find the long name |FooCat| corresponding to abbrev FOOCAT.
(How?)
(2.2) Issue GETDATABASE('|FooCat|, 'object') to get the full path
of FOOCAT.NRLIB

(3) check existence of FOOCAT-.NRLIB in the path of (2.2)
(4) If possible, then load FOOCAT-.NRLIB.

One problem I see with this approach is in (2.1). I currently don't know
how to extract the long name.

The second problem is that a developer might have compiled a category
FooCat with a default package at some point in the development of a
bigger software package. Later he decides to delete the default package
in foocat.spad. If FOOCAT-.NRLIB then still lies in his filesystem then
the new ")lib FOOCAT" will find it and load it. In order to prevent this
from happening, it must be checked that FOOCAT-.NRLIB is not older than
FOOCAT.NRLIB.

We can ignore the Aldor part as shown in my last mail.

Ralf

[oldk1331]

On 1/9/19 3:42 PM, Ralf Hemmecke wrote:
> One problem I see with this approach is in (2.1). I currently don't know
> how to extract the long name.

Hmmm, about the path... As shown in ")help lib", when issuing
")lib XXX", the XXX part is already the filename of XXX.NRLIB,
and an explicit directory name can be passed by argument ")dir".
So there is no need to consult the database, just load XXX- in
the same directory as XXX.

> it must be checked that FOOCAT-.NRLIB is not older than
> FOOCAT.NRLIB.

This is the concern I'm having too. But I'm afraid FOOCAT-
is created earlier than FOOCAT ("stat" show that it's 8ms).

But there should be no harm if we load an outdated FOOCAT-
which is no longer being used elsewhere?

[Ralf Hemmecke]

> So there is no need to consult the database, just load XXX- in
> the same directory as XXX.

OK. Then I also agree.

>> it must be checked that FOOCAT-.NRLIB is not older than
>> FOOCAT.NRLIB.
>
> This is the concern I'm having too. But I'm afraid FOOCAT-
> is created earlier than FOOCAT ("stat" show that it's 8ms).

Hmmm... I guess the default package is created when FooCat is compiled,
i.e., roughly FOOCAT.NRLIB and FOOCAT-.NRLIB must have the same time.
So checking that the file dates are not too different (say about 10 sec)
the default package is still considered up-to-date.

In fact, I considere it a bug, if I compile a file foocat.spad with a
category FooCat (abbrev FOOCAT), that has *no* default package, then the
compiler *must* update FOOCAT-.NRLIB (in this case delete it).

Waldek, can this be easily realised?

> But there should be no harm if we load an outdated FOOCAT-
> which is no longer being used elsewhere?

Really? The FriCAS spad compiler is not as strict as Aldor. You can
compile a domain/package successfully even though you have not given a
function implementation for every export of your domain. Now suppose, I
have not implemented foozwei and also no longer a default package. Then
everything works locally since the foozwei implementation is taken from
the (outdated) FOOCAT-.NRLIB, if that (falsely) gets loaded.
No, no, the correct way IMHO is that the compiler always updates
FOOCAT-.NRLIB. FOOCAT and FOOCAT- should be considered as *one*
unseparable entity. Created together and loaded together. Even better,
they should just live in the same FOOCAT.NRLIB folder.

Ralf

[oldk1331]

I agree that this should be the best solution.