[windows] TMFORM and "spaces in PATH"

[Qian Yun]

Another tricky bug on Windows is that, we can't compile TMFORM
with default heap size, so we need to pass "--dynamic-space-size 4096".

That is not a problem on Linux, but on Windows, it's complicated.

First, by default, SBCL installation path contains spaces.
("C:\Program Files\Steel Bank Common Lisp\2.0.0")

Our configure script and Makefile are not happy about it.

Is it a bug that we can't compile FriCAS with SBCL installed in a
directory which contains spaces (on either linux or windows).

Second, I can tweak build script to make previous problem go away,
but to combine "SBCL path contains space" and "pass additional
parameters to SBCL", it seems impossible? I can't work out the
correct combination between quote chars and escape chars.

Anyway, we can simply install SBCL into "C:\sbcl" and call it a day,
but still, can we improve the current situation?

Third, is it a bug that compile TMFORM requires so much RAM?
We should look into it to see if RAM usage can be reduced.

- Qian

[Waldek Hebisch]

On Thu, Mar 04, 2021 at 06:46:14PM +0800, Qian Yun wrote:
> Another tricky bug on Windows is that, we can't compile TMFORM
> with default heap size, so we need to pass "--dynamic-space-size 4096".
>
> That is not a problem on Linux, but on Windows, it's complicated.
>
> First, by default, SBCL installation path contains spaces.
> ("C:\Program Files\Steel Bank Common Lisp\2.0.0")
>
> Our configure script and Makefile are not happy about it.
>
> Is it a bug that we can't compile FriCAS with SBCL installed in a
> directory which contains spaces (on either linux or windows).
>
> Second, I can tweak build script to make previous problem go away,
> but to combine "SBCL path contains space" and "pass additional
> parameters to SBCL", it seems impossible? I can't work out the
> correct combination between quote chars and escape chars.

IIUC Windows has "shortcuts" which are supposed to act like
Unix symbolic links. Maybe it is enough to create a
shortcut?

> Anyway, we can simply install SBCL into "C:\sbcl" and call it a day,
> but still, can we improve the current situation?
>
> Third, is it a bug that compile TMFORM requires so much RAM?
> We should look into it to see if RAM usage can be reduced.

AFAICS RAM usage during compilation of TMFORM is really sbcl
bug. It seem that the problem is the same as AWAIC.
Namely, we need to initialise a table and at Lisp level
it is longish segment of code with calls but no jump.
Apparently sbcl thinks that this is excellent opportunity
to optimize and collects insane amounts of information,
consequently running out of resources.

For the record, ECL has no trouble to compile (tranlate to C)
those files. On AWAIC gcc exibits high memory usage and
long compile time but after 10 minutes it finished.

We could try to tweak Lisp output by Spad compiler, but it is
not clear how much we can gain in this way.

--
Waldek Hebisch

[Qian Yun]

> IIUC Windows has "shortcuts" which are supposed to act like
> Unix symbolic links. Maybe it is enough to create a
> shortcut?
>

You mean create "shortcuts"/symbolic links of SBCL installation
directory? That's still a workaround, isn't it?

Anyway, this problem is not that important, we can make people
install SBCL into a directory without spaces, for now.

[Waldek Hebisch]

On Thu, Mar 04, 2021 at 11:59:23PM +0800, Qian Yun wrote:
>
> > IIUC Windows has "shortcuts" which are supposed to act like
> > Unix symbolic links. Maybe it is enough to create a
> > shortcut?
> >
>
> You mean create "shortcuts"/symbolic links of SBCL installation
> directory? That's still a workaround, isn't it?

Yes.

--
Waldek Hebisch

[Ralf Hemmecke]

>> Third, is it a bug that compile TMFORM requires so much RAM?
>> We should look into it to see if RAM usage can be reduced.
>
> AFAICS RAM usage during compilation of TMFORM is really sbcl
> bug. It seem that the problem is the same as AWAIC.
> Namely, we need to initialise a table and at Lisp level
> it is longish segment of code with calls but no jump.

Are we talking about TexmacsFormat?

The only long list is for CorkFunc. But that is a list of records. What
can be difficult for that to be compiled in Lisp? Maybe it helps to list
the data as a list of lists and do the construction a bit differently.

Maybe I should write a new "FormatTexmacs" in my Formatter framework.

Ralf

[Waldek Hebisch]

On Thu, Mar 04, 2021 at 05:02:41PM +0100, Ralf Hemmecke wrote:
> >> Third, is it a bug that compile TMFORM requires so much RAM?
> >> We should look into it to see if RAM usage can be reduced.
> >
> > AFAICS RAM usage during compilation of TMFORM is really sbcl
> > bug. It seem that the problem is the same as AWAIC.
> > Namely, we need to initialise a table and at Lisp level
> > it is longish segment of code with calls but no jump.
>
> Are we talking about TexmacsFormat?
>
> The only long list is for CorkFunc. But that is a list of records. What
> can be difficult for that to be compiled in Lisp?

Just one explanation: Spad data is mutable. Lisp constant
data is immutable, so Spad _have to_ translate list for CorkFunc
as a sequence of calls to 'construct'. This is 794 calls,
not very large number and _should_ not be problem for Lisp
compiler. But apparently it is. AWAIC produces more than
3000 calls and calls are more complicated than in TMFORM.
Still, with capacities of modern machine there should be
no problem. But sbcl behaviour in AWAIC is really much
worse then on TMFORM...

> Maybe it helps to list
> the data as a list of lists and do the construction a bit differently.

AFAICS there is no reasonable workaround at Spad level (I do not
consider splitting table between several packages as reasonable).
I principle Spad compiler could recognize initialization pattern,
produce list which is Lisp constant (so Lisp compiler presumably
would handle it resonably) and then use iteration over constant
list to initialize Spad list. But that would be rather significant
addition to Spad compiler.

--
Waldek Hebisch

[Ralf Hemmecke]

On 05.03.21 14:25, Waldek Hebisch wrote:
> On Thu, Mar 04, 2021 at 05:02:41PM +0100, Ralf Hemmecke wrote:
>>>> Third, is it a bug that compile TMFORM requires so much RAM?
>>>> We should look into it to see if RAM usage can be reduced.
>>>
>>> AFAICS RAM usage during compilation of TMFORM is really sbcl
>>> bug. It seem that the problem is the same as AWAIC.
>>> Namely, we need to initialise a table and at Lisp level
>>> it is longish segment of code with calls but no jump.
>>
>> Are we talking about TexmacsFormat?
>>
>> The only long list is for CorkFunc. But that is a list of records. What
>> can be difficult for that to be compiled in Lisp?
>
> Just one explanation: Spad data is mutable. Lisp constant
> data is immutable, so Spad _have to_ translate list for CorkFunc
> as a sequence of calls to 'construct'. This is 794 calls,
> not very large number and _should_ not be problem for Lisp
> compiler. But apparently it is.

First of all, I actually do not understand why this Cork stuff is in
texmacs.spad at all. FriCAS does not generate any Unicode data.
The only way unicode can appear is inside a String. This is what
texmacs.spad takes care of. Obviously, the Texmacs people seem to use
mathematical symbols in strings when working with FriCAS. What I do not
quite understand is why TexmacsFormat has to do that this translation
from unicode encoding to Texmacs encoding. I would have guessed that
Texmacs itself is able to do this conversion.

Anyway, attached you find a modified texmacs.spad, where I simply
translated the List(Record(key:SI, entry:String)) into a simple
List(String) and constructed the respective HashTable from that list.
On my laptop, the compilation if that texmacs.spad is more or less
instant, at least it takes a few seconds less than the texmacs.spad that
is currently in the repository. Presumably, it takes also less RAM, but
I don't know how to check that.

> AWAIC produces more than

Sorry, what is AWAIC?

>> Maybe it helps to list
>> the data as a list of lists and do the construction a bit differently.
>

> AFAICS there is no reasonable workaround at Spad level...

I wonder whether my attempt might work.

Ralf

[Waldek Hebisch]

We have this wonderful thing called ChangeLog and corresponding
entry is:

2014-10-01 Bill Page <bill...@newsynthesis.org>,

Texmacs interface encodes scheme Cork symbols as unicode characters
....

IIRC around that time Bill also wrote to the list about
it.

As I wrote currently due to Guile dependencies Texmacs does
not build on my machine. Unfortunately, I was unable to
find in-depth documentation of communication format used
by Texmacs. Few things I know are from experimentaion...

> Anyway, attached you find a modified texmacs.spad, where I simply
> translated the List(Record(key:SI, entry:String)) into a simple
> List(String) and constructed the respective HashTable from that list.
> On my laptop, the compilation if that texmacs.spad is more or less
> instant, at least it takes a few seconds less than the texmacs.spad that
> is currently in the repository. Presumably, it takes also less RAM, but
> I don't know how to check that.

Well, writing integers as strings is not really satisfactory
solution. OTOH, given how problematic TMFORM is we may have
no better way...

> > AWAIC produces more than
>
> Sorry, what is AWAIC?

Table of atomic data. It is (maybe earlier version) in the
FriCAS wiki. Kurt Pagani (I am not sure, but probably he produced
Spad translation of the table) some time ago wrote about it.
While not a top priority item it would be nice to include it,
or at least have it as working "optional package". Unfortunatly,
ATM compilation with sbcl fails (clisp and ECL can compile it).

To say the truth, precomputed tables are quite useful to
speed up computations. For example, standard trick to
speed up work with towers of finite fields is to use
precomputed table of embeddings. So better (well,
reasonable) handling of intialized tables would be
quite useful.

> >> Maybe it helps to list
> >> the data as a list of lists and do the construction a bit differently.
> >
> > AFAICS there is no reasonable workaround at Spad level...
>
> I wonder whether my attempt might work.

I think it should work.


--
Waldek Hebisch

[Ralf Hemmecke]

> Well, writing integers as strings is not really satisfactory
> solution. OTOH, given how problematic TMFORM is we may have
> no better way...

OK, we don't need integers as strings, we can also use the utf8
character itself. See attachment for a second attempt.

The texmacs.input file is attached to show how I computed and checked
the new table.

Hope you like it.
Can I commit?

Ralf

[Waldek Hebisch]

This has problem, in C locale (which is reasonable setup for
build machine) sbcl reject anything non-ASCII. So ATM this
is not acceptable.

I will look if I can quickly change Spad compiler to produce
something that causes no trouble to sbcl. If chaging Spad
compiler requires too much work, I would go with string
version.

BTW: This is yet another reason to rewrite our scanners.
After rewrite we could use binary mode for reading files,
so could handle UTF-8 independent of locale settings.

--
Waldek Hebisch

[Qian Yun]

I'm glad that Ralf is tackling the third TMFORM issue.

Meanwhile I found a simple solution/workaround for the first two:

If we put SBCL path (with spaces) into PATH, then there's no
problem to use --with-lisp="sbcl --dynamic-space-size 4096".

To truly solve this issue, I guess we will need a separate
"--with-lisp-args" option. But I don't see the need for support this,
after all, we do not support building FriCAS when srcdir or builddir
contains spaces. I'm satisfied with current situation now.

- Qian

[Qian Yun]

On 3/8/21 12:48 AM, Qian Yun wrote:
> I'm glad that Ralf is tackling the third TMFORM issue.
>
> Meanwhile I found a simple solution/workaround for the first two:
>
> If we put SBCL path (with spaces) into PATH, then there's no
> problem to use --with-lisp="sbcl --dynamic-space-size 4096".

Seems --with-lisp="sbcl --dynamic-space-size 4096" is not working,
even on Linux? You can check it by

)lisp (/ (- sb-vm:dynamic-space-end sb-vm:dynamic-space-start) (expt
1024 2))

Maybe related with :SAVE-RUNTIME-OPTIONS keyword of save-lisp-and-die?
Will investigate later.

- Qian

[Qian Yun]

On 3/8/21 1:42 AM, Qian Yun wrote:
>
>
> On 3/8/21 12:48 AM, Qian Yun wrote:
>> I'm glad that Ralf is tackling the third TMFORM issue.
>>
>> Meanwhile I found a simple solution/workaround for the first two:
>>
>> If we put SBCL path (with spaces) into PATH, then there's no
>> problem to use --with-lisp="sbcl --dynamic-space-size 4096".
>
> Seems --with-lisp="sbcl --dynamic-space-size 4096" is not working,
> even on Linux?  You can check it by
>
>   )lisp (/ (- sb-vm:dynamic-space-end sb-vm:dynamic-space-start) (expt
> 1024 2))
>
> Maybe related with :SAVE-RUNTIME-OPTIONS keyword of save-lisp-and-die?
> Will investigate later.
>
> - Qian
>

Oops, seems I mistype it as '--with-lisp="sbcl --dynamic-space-size=4096"'.

There's no real problem here.

- Qian

[Waldek Hebisch]

On Sun, Mar 07, 2021 at 05:32:11PM +0100, Waldek Hebisch wrote:
>
> I will look if I can quickly change Spad compiler to produce
> something that causes no trouble to sbcl. If chaging Spad
> compiler requires too much work, I would go with string
> version.

I have now a patch that seem to work well. I am still testing
it, but on TMFORM sbcl reports 64M memory consed, which means
we should have no memory use prolems compiling TMFORM.

The patch "optimizes" constant coercion, if integer constant
is safely in SingleInteger ranges this constant is used as-is
(saving call to "coerce"). Similar thing is done for
coercion from Float to DoubleFloat: if Float is constant
it is converted to DoubleFloat at compile time. There
is extra twist: in context expecting DoubleFloat floating
point constant should now produce DoubleFloat (up to now
this was an error).

Anyway, I am attaching it if somebody wants to comment or
test it.

Note1: Size of SingleInteger is known only at Lisp compile
time, which may be later than Spad compilation and may
use different Lisp than Spad compilation, so we know if
integer fits into SingleInteger only at Lisp compile time
and in principle Lisp compiler may produce nonsense for out-of
range constants. That is why coercion was postponed to
runtime.

BTW: For TMFORM there is workaround at Spad level, namely
instead of code like

60::SI

one could use

qconvert(60)@SI

That skip call to 'coerce' (and corresponding size check).

--
Waldek Hebisch

[Qian Yun]

This patch tested OK, and I only have one small question:

Why define the macro 'mk_DF' instead of using 'make_DF' function?

- Qian

[Waldek Hebisch]

On Mon, Mar 08, 2021 at 11:00:22PM +0800, Qian Yun wrote:
> This patch tested OK, and I only have one small question:
>
> Why define the macro 'mk_DF' instead of using 'make_DF' function?

Macro will always run at compile time, even in dumb compiler.
Function may be delayed to runtime, at least partially defeating
optimization.

--
Waldek Hebisch

[oldk1331]

But this macro only expands to this function call, it does not do computation at compile time.

--
You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/20210308162033.GA1217%40math.uni.wroc.pl.

[Waldek Hebisch]

On Tue, Mar 09, 2021 at 12:31:34AM +0800, oldk1331 wrote:
> But this macro only expands to this function call, it does not do
> computation at compile time.
>

No. Macro is executed at compile time and result used as expansion.
Most of our macros use "`" (backtick) to delay evaliation and
just insert some variable pieces inside, like:

(defmacro QAREF1(v i)
`(aref (the (simple-array T (*)) ,v) ,i))

Here backtick means that main part will be taken literaly and
only 'v' and 'i' replaced by parameters. But in

(defmacro |mk_DF|(x e) (|make_DF| x e))

there is no backtick, so it runs at compile time. See:

(1) -> )lisp (macroexpand '(|mk_DF| 17 -3))

Value = 0.017
(1) -> )lisp (macroexpand '(QAREF1 17 -3))

Value = (AREF (THE (SIMPLE-ARRAY T (*)) 17) -3)

Of course, in the second case code is wrong because 17 is not a
vector and -3 is not valid index (negative). But 'macroexpand'
does not execute code (and in some weird context this code
could be valid, so 'macroexpand' just dully performs expansion).

--
Waldek Hebisch

[Qian Yun]

Thanks, I applied the patch and compiled FriCAS to see its result.

You are right, it is working.

The thing confused me is that, this is double compilation.

First, the SPAD compiler complies *.spad to *.lsp with this macro.
Then, when *.lsp are read and compiled, this macro works at compile time.

I have no further questions.

- Qian