Serial part of build

[Waldek Hebisch]

I did some measurement on serial part of build, that is execution
of 'boo_db.input'. On my machine times (in seconds) are:

parsing algebra 1.024178
first stage of database computation 0.197644
bootstrap mode compilation of whole
algebra 5.792356
normal compilation of categories an
core (inlinable) domains 10.547418
making databases 1.691597

Sum 19.253193

Parsing time is not great, as parsers can do more than 2 million
lines per second (actually, compiler like Tiny C can do whole
compilation at speed close to 2 million lines per second),
so probably about 10 to 20 times slower than possible. Still,
it is one of fastest stages, so currently not problematic.

First stage of database computation computes ABBREVIATION, COSIG,
NILADIC, CONSTRUCTORKIND, CONSTRUCTORMODEMAP and CONSTRUCTORCATEGORY.
I think that time for this stage is reasonable.

We compile whole algebra in bootstrap mode mainly to populate
rest of database, that is OPERATIONALIST, MODEMAPS, OBJECT,
ANCESTORS, PREDICATES. More precisely, except for categories
and and core domains compilation does not need to load code,
so strictly speaking object code for normal domains is not
needed at that stage. So actually, for normal domains OBJECT
(which holds location of loadable object file) is not needed.
For domains ANCESTORS field is empty (Lisp nil), so no need
to compute it. So what we need is MODEMAPS, OPERATIONALIST
and PREDICATES. Actually, we need PREDICATES only to simplify
conditions when computing MODEMAPS and OPERATIONALIST (and
to understand runtime code).

I think that we currently need to compile categories in
bootstrap mode because normal compilation loads categories.

Normal compilation of core (inlinable) domains is needed
because to inline we need to load compiled code. We also
need normal compilation for categories with defauls, as
they affect which function will be used at runtime, so
also are needed to decide if a call actually calls an
inlinable function. I am not sure if we need to compile
other categories in normal mode.

Thinking loudly, MODEMAPS are not hard to compute from
CONSTRUCTORCATEGORY. Our HyperDoc code computes lists of
operations based on modemaps and category ANCESTORS in
about 1.5 seconds. Currently category ANCESTORS and
OPERATIONALIST are computed by JoinInner, which needs
category vectors which are produced by executable code
of category constructor. More precisely, category
which only has 'with' part is handled directly to
produce thing acceptable to JoinInner, but this part
is then passed to JoinInner to finish the job.

So, there seem to be substantial potential to speed up
serial part. Minimally we should be able to skip
bootstrap mode compilation of normal domains, replacing
it by computation of appropriate database fields.
But after that stage we have computed all important
database fields, so in principle we should be able
to dump databases. And since all data is in memory
we should be able to skip loading info from 'index.KAF'
so dumping databases should be faster than normal
'make_datebases' call.

Once we created databases compilation should be able to go
in parallel. That is we may be forced to ensure that some
dependencies are respected, but there should be potential
for parallel build. Actually, thinking more, even without
deeper work we should be able to dump databases after
bootstrap mode compilation. That probably would somewhat
increase total compile time, but allow more parallelism,
so give net gain in real time.

Better would be to decouple creating info about categories
from use of executable code. That would allow much earlier
start of normal compilation and would completely skip some
compilations. Optimistically, after say 4 second bootstrap
we should be able to go for normal parallel compilation.

--
Waldek Hebisch

[Qian Yun]

On 8/11/26 9:43 AM, Waldek Hebisch wrote:
> I did some measurement on serial part of build, that is execution
> of 'boo_db.input'. On my machine times (in seconds) are:
>
> parsing algebra 1.024178
> first stage of database computation 0.197644
> bootstrap mode compilation of whole
> algebra 5.792356

> normal compilation of categories an
> core (inlinable) domains 10.547418

^^ This part means boot_doms.input, right?

It is especially long for ECL (due to gcc).

For the purpose of making databases, compile spad to lisp
is enough, and compile lisp to binary can be paralleled
in the next stage (with dependencies sorted out). Right?

- Qian

[Waldek Hebisch]

On Tue, Aug 11, 2026 at 09:58:04AM +0800, Qian Yun wrote:
> On 8/11/26 9:43 AM, Waldek Hebisch wrote:
> > I did some measurement on serial part of build, that is execution
> > of 'boo_db.input'. On my machine times (in seconds) are:
> >
> > parsing algebra 1.024178
> > first stage of database computation 0.197644
> > bootstrap mode compilation of whole
> > algebra 5.792356
>
> > normal compilation of categories an
> > core (inlinable) domains 10.547418
>
> ^^ This part means boot_doms.input, right?

Yes.

> It is especially long for ECL (due to gcc).
>
> For the purpose of making databases, compile spad to lisp
> is enough, and compile lisp to binary can be paralleled
> in the next stage (with dependencies sorted out). Right?

Looks so. Actually, only 'index.KAF' is used by 'make_databases'.

When compiling Aldor files there is ')nolispcompile' option
which skips compilation of generated Lisp file. But ATM
there is no such option for Spad compiler, we probably
should add it. Currently, when we pass .NRLIB to ')compile'
is should just do Lisp compilation. AFAICS this .NRLIB
compilation should have no additional dependencies (so
only .NRLIB and Lisp + index.KAF inside).

> > making databases 1.691597
> >
> > Sum 19.253193
> >

--
Waldek Hebisch

[Waldek Hebisch]

I have now a test implementation of ')nolispcompile' for Spad.
When I modified algebra Makefile so that it uses ')nolispcompile'
in 'komp_all.input' build using sbcl fails because sbcl does not
load files with extention '.lsp'. Trying build with ECL and gcl
I get nontrivial savings:

gcl trunk gcl lisp ECL trunk ECL lisp
real 3m41.112s 2m51.479s 6m46.102s 4m44.896s
user 20m60.617s 19m18.201s 35m70.348s 33m45.919s
sys 3m35.873s 3m26.645s 4m50.418s 3m18.549s

Below the same stages as with sbcl:

parsing 1.84 7.420393
proc1 0.14 1.294984
komp_all 5.20 35.380560
boot_doms 10.19 44.528851
database 1.12 10.035884

gcl also compiles via C, but for some reason compilation is much
faster than ECL. Since komp_all is doing only Lisp compilation,
difference of time is due to speed of Spad compiler. So it seems
that in case of Spad compiler ECL object code is much slower than
gcl object code. Two steps (final making of database and first
stage of database computation) seem to be faster than with sbcl,
OTOH sbcl is significantly faster on parsing.

--
Waldek Hebisch

[Waldek Hebisch]

A little update. After change to get_database2 I can also pass
')nolispcompile' in 'boot_doms.input' after that real time for
build with GCL decreased by about 50 seconds to 2m1.182s. There
is smaller decrease (about 30 seconds) in CPU time. Real time for
build with ECL decreased by about 68 seconds to 3m36.846s. There
is also decrease in CPU time by about 15 seconds.

I also tried to skip second compilation of categories without
default, but this seem to have negligible effect (less than
measurement noise). Basically, 185 files in boot_doms.input
seem to need almost the same time as 319 files.

When I changed extention of Lisp files to '.lisp' build with
sbcl works. Real time is about the same, but there is increase
in paralellism, that is CPU time is bigger. Apparently, what
we save on Lisp compilation we loose due to more expensive
loading. Even skipping Lisp cmpilation in 'komp_all.input',
which should be pure gain seem to have no positive effect.

In case of sbcl there may be also question of speed of interpreter.
We evaluate categories in somewhat trivial way, but since we do
this a lot of time compiled code may win.

With sbcl doing at start of 'boo_db.input'

(proclaim '(optimize ((speed 0) (safety 0) (compilation-speed 3))))

reduced build time by something like 2-3 seconds. Rather small
improvement, but seem to be real.

BTW: We could cut cost of C compilation in ECL and SBCL build by
using lower optimization settings. And possibly we could try
to use Tiny C for compilations in bostrap phase (Tiny C compiles
sevaral times faster than GCC).

--
Waldek Hebisch

[Qian Yun]

On 8/12/26 10:04 AM, Waldek Hebisch wrote:
> A little update. After change to get_database2 I can also pass
> ')nolispcompile' in 'boot_doms.input' after that real time for
> build with GCL decreased by about 50 seconds to 2m1.182s. There
> is smaller decrease (about 30 seconds) in CPU time. Real time for
> build with ECL decreased by about 68 seconds to 3m36.846s. There
> is also decrease in CPU time by about 15 seconds.

How many CPU cores are you using?

As for SBCL, can you try with :sb-fastevl?

>
> BTW: We could cut cost of C compilation in ECL and SBCL build by
> using lower optimization settings. And possibly we could try
> to use Tiny C for compilations in bostrap phase (Tiny C compiles
> sevaral times faster than GCC).
>

I tried that before when I was trying to reduce the 3 hour build time
for ECL on Cygwin. Maybe I did something wrong, but "-O0", "-pipe",
does not have noticeable speedup. As for TCC, it can't be used
with ECL because of missing _complex support IIRC.

- Qian

[Tim Daly]

Why retain NRLIBs at all? There is enough memory these

days to keep everything in memory. NRLIBs existed because

there wasn't enough main memory. NRLIBs do a double

lookup, one to find the index and one to seek to that index.

That structure was created to optimize disk access times.

Back in the IBM days disk drives were slow and small.

Indeed, one of the databases was created by me (compress)

to wedge the system into a laptop with almost no memory.

A minor bit of hacking on GETDATABASE could restructure

the NRLIBs to be flat in-memory structures or just hash table

entries. The in-memory structures can be dumped/restored.

The GETDATABASE code could hide all of the changes. All

that would be needed is a one-time pass to map GETDATABASE 

calls over every NRLIB entry to write an in-memory image. Once

created the NRLIBs could be removed forever.

Hacking the databases to be in-memory data structures

allows creating lots of additional information attached to

each category and domain such as the data needed by

hyperdoc. A minor bit of creativity would be to attach the

test files to domains as user-available examples.

Eliminating I/O would likely reduce build times.

Tim

[Waldek Hebisch]

On Wed, Aug 12, 2026 at 10:17:01AM +0800, Qian Yun wrote:
> On 8/12/26 10:04 AM, Waldek Hebisch wrote:
> > A little update. After change to get_database2 I can also pass
> > ')nolispcompile' in 'boot_doms.input' after that real time for
> > build with GCL decreased by about 50 seconds to 2m1.182s. There
> > is smaller decrease (about 30 seconds) in CPU time. Real time for
> > build with ECL decreased by about 68 seconds to 3m36.846s. There
> > is also decrease in CPU time by about 15 seconds.
>
> How many CPU cores are you using?

I have 12 cores with hyperthreading, so 24 logical cores. I used
'make -j 20'.

> As for SBCL, can you try with :sb-fastevl?

I did not look how much effort it is. IIRC with past sbcl
versions it was incompatible with our code.

> > BTW: We could cut cost of C compilation in ECL and SBCL build by
> > using lower optimization settings. And possibly we could try
> > to use Tiny C for compilations in bostrap phase (Tiny C compiles
> > sevaral times faster than GCC).
> >
>
> I tried that before when I was trying to reduce the 3 hour build time
> for ECL on Cygwin. Maybe I did something wrong, but "-O0", "-pipe",
> does not have noticeable speedup.

I think that "-pipe" should have limited effect. "-O0" rather
reliably leads to shorter compile time. OTOH if you try "add to"
ECL options instead of replacing them, then ECL option may be still
in effect. I some other cases I have found that using fake 'gcc'
which _removes_ unwanted options and only leaves desired ones worked,
while trying to add options had no effect (I had tried to compile
older code using '-Werror' in Makefiles with new compiler having more
warnings).

On Windows/Cygwin things like process startup time and/or antivirus
may dominate runtime. When using Clisp it is making a lot of calls
to 'fsync'. Many years ago Tim Daly tried to build early version
of FriCAS compiling algebra from Lisp files using Clisp in VM running
on top of Windows. He reported long build time (few hours). I was
puzzled, as on Linux it was few seconds. Later I realized that
on native Linux at that time 'fsync' was ignored, once 'fsync'
was properly implemented the same long build time happened on Linux.
Presumably 'fsync' was vorking in VM running on top of Windows...

To put it differently, you need to have some performance data to
know what is slowing down build.

> As for TCC, it can't be used
> with ECL because of missing _complex support IIRC.

Currently 'sfsfun.boot' uses Lisp complex numbers. Some elementary
function may be using Lisp complex numbers but I am not aware of
other uses in Spad code. I would hope that ECL puts _Complex in
generated C code only if source uses complex numbers. And I meant
using Tiny C only for bootstrap part, from 'boo_db.input'.

--
Waldek Hebisch

[Waldek Hebisch]

On Wed, Aug 12, 2026 at 10:17:01AM +0800, Qian Yun wrote:

> On 8/12/26 10:04 AM, Waldek Hebisch wrote:
> > A little update. After change to get_database2 I can also pass
> > ')nolispcompile' in 'boot_doms.input' after that real time for
> > build with GCL decreased by about 50 seconds to 2m1.182s. There
> > is smaller decrease (about 30 seconds) in CPU time. Real time for
> > build with ECL decreased by about 68 seconds to 3m36.846s. There
> > is also decrease in CPU time by about 15 seconds.
>
> How many CPU cores are you using?

Attached is relevant diff:
- implementation of ')nolispcompile' (compiler.boot and lisplib.bpot)
- changes (probably incomplete) to use .lisp extention (gdata.boot,
m_utils.boot, partialy algebra Makefile.in)
- change to algebra Makefile.in to use ')nolispcompile' and cater
to different file extention

Just change 2 variables in the Makefile.in and you get different
build variant. If desirable we could choose variant at configure
time.

--
Waldek Hebisch

[Waldek Hebisch]

On Wed, Aug 12, 2026 at 01:13:03AM -0700, Tim Daly wrote:
> Why retain NRLIBs at all? There is enough memory these
> days to keep everything in memory. NRLIBs existed because
> there wasn't enough main memory. NRLIBs do a double
> lookup, one to find the index and one to seek to that index.
> That structure was created to optimize disk access times.
>
> Back in the IBM days disk drives were slow and small.
> Indeed, one of the databases was created by me (compress)
> to wedge the system into a laptop with almost no memory.
>
> A minor bit of hacking on GETDATABASE could restructure
> the NRLIBs to be flat in-memory structures or just hash table
> entries. The in-memory structures can be dumped/restored.
> The GETDATABASE code could hide all of the changes. All
> that would be needed is a one-time pass to map GETDATABASE
> calls over every NRLIB entry to write an in-memory image. Once
> created the NRLIBs could be removed forever.
>
> Hacking the databases to be in-memory data structures
> allows creating lots of additional information attached to
> each category and domain such as the data needed by
> hyperdoc. A minor bit of creativity would be to attach the
> test files to domains as user-available examples.
>
> Eliminating I/O would likely reduce build times.

NRLIB-s help with interactive developement. Textual form (.lsp
and index.KAF) helps with debugging compiler problems. Dumping
data structures with pointers requires some effort. IIUC ECL
has no support for dumping, usually other Lisp-s can dump whole
process, but not specific data structure. So the best we have
is printing which we use now. We could gain some speed by
using defult printer mode, but to ease debugging FriCAS
insists on pretty-printing. NRLIB-s also allow parallel
compilation which makes big difference to build time.

In first bootstrap pass we compile categories to memory, without
disc trace. I will try to compile more files in this way, but
at the end we need databases that other instances of FriCAS can
load and we need compiled code for categories (Lisp source or
fasl). As I wrote, Spad compiler need data from categories and
currently uses runtime support to get this data. As long as
this is the case besides databases we need loadable code for
categories. Spad inlining optimization load .NRLIBS and uses
runtime system to determine which function will be called.
If we don not know which function will be called we can not
apply this optimization and this makes quite a difference to
runtime speed.

Generating databases entirely in memory (without using data from
.NRLIB-s) look feasible and just now I am trying to do this.
I am thinking about eliminating need to load categories, but
this is bigger task, probably not for next release. And reworking
optimization in Spad compiler probably will need other changes,
so is even more in the future.

So, with current state of Spad compiler we get dependencies
similar to lattice structure that you created at early period
of open source Axiom developement. FriCAS avoids need for
explicit information of similar kind by compiling enough things
during bootstrap stage so that all dependencies are satisfied
and rest of compilation is in arbitrary order. The main part
of bootstrap works in single Lisp image, with benefit of sharing
things in memory. But this is by neccesity serial, so on
multicore machines any increase of CPU time for this stage
directly increases real time for build. So, we want to avoid
doing unneccessary work in bootstrap stage, but we need to
produce enough files (currently databases and NRLIB-s) so that
later compilation can go smoothly.

--
Waldek Hebisch

[Tim Daly]

> IIUC ECL
> has no support for dumping, usually other Lisp-s can dump whole
> process, but not specific data structure. So the best we have
> is printing which we use now. We could gain some speed by
> using defult printer mode, but to ease debugging FriCAS
> insists on pretty-printing.

Any common lisp can print and read a structure, even a circular one.
Set *print-circle* to t.

All of the parallel compiles could load the data structure.

The GETDATABASE code handles all of the NRLIB behavior.
Converting, optimizing, and combining those data structures
into a lisp data structure should be transparent to everything.

As for getting debugging information it would be easy to extend
GETDATABASE with a set of lisp (or Spad) functions to examine
the databases. They could all live in a GetDatabase domain.

The key advantage is that everything about NRLIBs is hidden
behind and accessed from GETDATABASE.

With a bit of clever code it would be possible to create various
visual graph representations of the categories and domains.
These could be viewed in a browser and arranged so it could
be "walked". I used to have those graphs on my website but
they were static. Using a single data structure would allow the
ability to add/modify the structure to do things like tracing or
modifying dependencies.

A slightly more ambitious goal would be to allow "triggers" in
the data structure so one could add behaviour, that is, code
that runs when a function is invoked. This would make it easy
to do things like show a visual trace of domain functions called
during a computation in an active browser window. A lisper
would create "around methods" without changing code.

Anyway, the point is that hacking NRLIBs is a well-contained
transparent project that anyone could do. It is a good and useful
side project.

Tim

> --
> You received this message because you are subscribed to a topic in the Google Groups "FriCAS - computer algebra system" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/fricas-devel/Dl11d9o_E5U/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to fricas-devel...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/fricas-devel/anyQU9wgPlfSnnoR%40fricas.org.

[Qian Yun]

Some test results from github actions (4 cores):

1. CCL takes longer. Because CCL does not have interpreter mode,
(load "xx.lisp") file will result in compilation first.
So this optimization does not work for CCL.

2. CLISP time is almost the same.

3. For ECL, around 25% faster.

4. For GCL, around 10~20% faster.

5. For SBCL, it takes longer.

6. For CMUCL, seems faster (only 1 data point).


Obviously, with more cores, the speed up is more effective.

- Qian

[Waldek Hebisch]

On Thu, Aug 13, 2026 at 06:55:04AM +0800, Qian Yun wrote:
> On 8/12/26 10:37 PM, Waldek Hebisch wrote:
> > On Wed, Aug 12, 2026 at 10:17:01AM +0800, Qian Yun wrote:
> >> On 8/12/26 10:04 AM, Waldek Hebisch wrote:
> >>> A little update. After change to get_database2 I can also pass
> >>> ')nolispcompile' in 'boot_doms.input' after that real time for
> >>> build with GCL decreased by about 50 seconds to 2m1.182s. There
> >>> is smaller decrease (about 30 seconds) in CPU time. Real time for
> >>> build with ECL decreased by about 68 seconds to 3m36.846s. There
> >>> is also decrease in CPU time by about 15 seconds.
> >>
> >> How many CPU cores are you using?
> >
> > Attached is relevant diff:
> > - implementation of ')nolispcompile' (compiler.boot and lisplib.bpot)
> > - changes (probably incomplete) to use .lisp extention (gdata.boot,
> > m_utils.boot, partialy algebra Makefile.in)
> > - change to algebra Makefile.in to use ')nolispcompile' and cater
> > to different file extention
> >
> > Just change 2 variables in the Makefile.in and you get different
> > build variant. If desirable we could choose variant at configure
> > time.
> >
>
> Some test results from github actions (4 cores):
>
> 1. CCL takes longer. Because CCL does not have interpreter mode,
> (load "xx.lisp") file will result in compilation first.
> So this optimization does not work for CCL.

Yes, that is expected result.

> 2. CLISP time is almost the same.

No surprise here. Clisp compiles to bytcode which should be
relatively fast. Loading of bytecode probably is faster.
But when using Clisp other things take a lot of time.

> 3. For ECL, around 25% faster.
>
> 4. For GCL, around 10~20% faster.
>
> 5. For SBCL, it takes longer.
>
> 6. For CMUCL, seems faster (only 1 data point).
>
>
> Obviously, with more cores, the speed up is more effective.

AFAICS when using ECL or GCL we should get some speedup even on
single core (because CPU time is lower). For others balance is
more tricky.

The above is just a preliminary trial. I am looking at deeper
change that keeps more things in memory, that hopefully will
speed up build for any Lisp that we use.

--
Waldek Hebisch

[Waldek Hebisch]

On Thu, Aug 13, 2026 at 01:40:42AM +0200, Waldek Hebisch wrote:
>
> The above is just a preliminary trial. I am looking at deeper
> change that keeps more things in memory, that hopefully will
> speed up build for any Lisp that we use.

A little update. I have now changed 'processGlobals()' to
do compilation of the whole algebra using saved parse tree.
In it I also skip compilation of generated Lisp file for
domains. Using 'sbcl' this gives me about 2 seconds gain
in speed. Small compared to whole build, but AFACS the
two stages that I have now replace previous 3 stages and
produce equivalent result. That is I skip generation of
things fasls that are not used at all. So this is pure gain
regardless of serial or parallel setup.

The 2 stages that I have now take together about 4.5 seconds.
At this point we have almost all data needed to generate
databases. I write "almost" because we lack information
about default packages (default packages are skipped by
bootstrap mode compilation that we use). I am looking
at possibility of including default packages in bootstrap
mode compilation. Currently this fails, but I hope that
this I can solve this.

Profile info about this run shows that about 0.8s is spent
evaluating categories, about 0.8s is spent in JoinInner,
abot 0.7s in Lisp printer, about 0.37s is spent reading
info from index.KAF-s. About 1s is parsing.

Doing things in memory we should be able to save on reading
index.KAF-s and skip most printing (we need Lisp and
index.KAF-s for categories). Currently this 2 stage
process compiles categories twice, I am not sure if it
is enough to compile them once. We should be able to
skip loading category code from NRLIB-s (as we already
have this code in memory). So there are some possible
savings.

But the main thing is generating databases for default
packages. If that is solved we should be able to generate
databases after 4 to 7 seconds of serial time, compared
to 19 seconds in trunk and 17 seconds with the improvement
above.

--
Waldek Hebisch

[Qian Yun]

On 8/12/26 10:37 PM, Waldek Hebisch wrote:

> make_compiler_output_name(dir_name, name) ==
> - CONCAT(dir_name, '"/", name, '".lsp")
> + CONCAT(dir_name, '"/", name, '".lisp")

For the lsp->lisp change, it should also be done in
src/interp/i-syscmd.boot :

compileSpadLispCmd args ==
-- Assume we entered from the "compile" function, so args ~= nil
-- and is a file with file extension .NRLIB

libname := first args
basename := file_basename(libname)
path := make_fname(libname, basename, '"lsp")
file_must_exit(path)

- Qian

[Waldek Hebisch]

If we go with this change, then we probably also need to coordinate
with Aldor compiler (this is actually related to Aldor compilation).
I wrote if, because "lsp" seem to work with Lisps that benefit
from loading Lisp files, while we need '.lisp' for 'sbcl' and
maybe Cmucl and Clozure CL.

--
Waldek Hebisch

[Waldek Hebisch]

On Thu, Aug 13, 2026 at 01:40:42AM +0200, Waldek Hebisch wrote:
>

> The above is just a preliminary trial. I am looking at deeper
> change that keeps more things in memory, that hopefully will
> speed up build for any Lisp that we use.

I have now a better patch (attached). Using it with sbcl serial
part of algebra build on my machine takes about 3.2s, the rest
of algebra build is parallel. This is still proof of concept,
tuning may change a bit (in particular this patch compiles files
that we are going to load, so favours sbcl and Clozure CL). There
are two new options to Spad compiler: ')nolispcompile' which
skips compilation of generated Lisp, and ')noopt' which skips
inlining optimizations (needed for bootstrap). Part changing
extention from '.lsp' to '.lisp' is half done, if we want to
do this there is more work. There is some degradation of output,
it seems that is some cases that worked previously inlining
no longer works (that needs to be investigated).

Some extra comments: this patch uses option to compile .spad
files without creating an NRLIB. This is used for domains,
because generated code is (and was) not needed and patched
compiler uses database info in memory instead of loading it
from index .KAF. More precisely, previously after generating
NRLIB Spad compiler erased database info it had about freshly
compiled constructor and load new info from index.KAF. With
the patch info that would be written to index.KAF is used
to update in memory database. In case of categories we
later need executable code and due to the way Spad loading
works we need index.KAF. So patched compiler writes and
compiles Lisp file and writes index.KAF, but update database
in memory. Path also causes compiler to generate database
info for default packages, which allows generation of
database just after 'processGlobals'. Database is generated
using data present in memory. In effect, we avoid reading
data that we have previously written and skip much of
writing activity. Since for compilation we use already
parsed code (needed for previous 'processGlobals' that did
less work) we save parsing algebra files with is about 1
second. There is slightly more compiling going on, but for
sbcl balance is rather positive, that is we save about 5
seconds of CPU time. The second chande is at Makefile
level, since we have database and category NRLIB-s we
can compile category files in normal mode in parallel.
I also changed Makefile to compile inlinable domains in
parallel. The last step affect what is inlinable.
If we compile inlinable domains serially in a single
image, then there inlining between those domains which
probably leads to more inlining possibilities later.
And AFAICS when inlinable domains are compiled serially,
then exactly the same things are inlined as before.

Compiling in parallel seem to need more CPU time, but the
effect is not big and with '-j 20' there is clear gain
in real time. It is not clear how much increase is
real: with hypertreading 2 logical cores running on a
single physical core compete for resources so are slower
than 2 independent core. Also, on sigle threaded work
CPU inreases clock frequency, so one get more cycles in
given time unit. But there are also reasons for real
increase: each compilation process must separately read
its database info instead of reusing already present
info. Similarly each compilation process must load
code that is needs.

Anyway, with that patch as is and using sbcl I can compile
FriCAS on my machine in 45-46 seconds (compared to 61-62
when using current trunk). After changing variables in
Makeifile.in to load Lisp instead of fasl-s, using ECL I get

real 3m13.548s
user 34m28.368s
sys 3m1.747s

where real time is way better than the trunk and slightly
better than the best ECL time that I had previously. CPU
time is larger, so with ECL on machines with lower core
count it may be better to do more work serially.

To put it differenly, building database in memory seem to
be pure win for any Lisp that we use (we basically skip some
work that was done previously). OTOH parallel build
apparently may increase build time, depending on core count
and Lisp that we use.

--
Waldek Hebisch

[Qian Yun]

I think that for SBCL, (load "f"), it looks for f.lisp, never f.lsp.

Maybe we can tweak "load_quietly" to avoid coordinated change
with Aldor, by passing "f.lsp" or "f.fasl" explicitly.

- Qian