Deprecated operations in scene.spad

[Waldek Hebisch]

scene.spad contains several operators marked as deprecated. In
about 8 years there were no siginficant changes to scene.spad.
I think that there is time to either remove the depreciated
operations or decide that thay should stay indefinitely and
remove depreciacation notice.

--
Waldek Hebisch

[Martin Baker]

I don't mind, I have given up with this software now.

When I wrote the code I was hoping that this approach would replace old
boot and C code with something much more maintainable and flexible. Also
the scenegraph architecture makes it much easier to export to 2D and 3D
graphics files like SVG and X3D and being written in SPAD it would have
been easy to add other graphics formats.

I'm not critisising because I don't see the big picture but it seems to
me that going from boot code to SPAD in lots of small steps rather than
one big step is impossibly slow and and risks replacing messy boot code
with messy SPAD code.

Martin

[Ralf Hemmecke]

Unfortunately, I have not much experience with graphics code and not yet
much need of using some, however, I would really like to be able to
create nice looking graphics easily with FriCAS, be it via SPAD code or
via external libraries. In particular in connection with jFriCAS, the
old graphics code does not seem appropriate.

I now looked a bit on the scene.spad code. It seems that it is quite
safe to remove the deprecated code. And in the spirit of not keeping old
unused code, I am in favour or removing that code.

On 4/12/24 09:41, Martin Baker wrote:
> I don't mind, I have given up with this software now.

That's sad.

> When I wrote the code I was hoping that this approach would replace
> old boot and C code with something much more maintainable and
> flexible.

I do not understand why you think that not both graphics systems can
live besides each other (as they do now)?

It requires lot of efforts to maintain code and to promote it.
Now, since you seem to be gone, the your code must be studied and
understood by some other person. I would be happier if you maintained
that part of the codebase in FriCAS by your own efforts.

Yes, from my own experience I know that it is not always easy to get my
ideas for new code against the conservativeness of Waldek. However,
seeing it from a maintainer's point of view, it is quite clear that a
maintainer cannot lightly accept new code since that code not only
brings new features, but maybe a burden of lurking bugs. And of the
original developer is gone, who will take over that burden?

> Also the scenegraph architecture makes it much easier to export to 2D
> and 3D graphics files like SVG and X3D and being written in SPAD it
> would have been easy to add other graphics formats.

So? Can you show it? I'd be happy if FriCAS were able to show graphics
in a Jupyter notebook. Any ideas? I have no clue how I would do it. And
honestly, I do not have time and interest in doing it myself. So I would
rather like that other (more knowledgable) people step in and make that
happen.

It's open source development so people do what they can and what they
are interested in. Of course, in that sense it is fine if you do
something that interests you more. Open source development is mostly
spare time fun. One cannot force someone to "voluntarily" do something
for someone else. One has to respect different people's interests and
wishes. Thus, development can be slow or there are more developers that
commit themselves for a longer time of being maintainers for different
parts of the software.

Ralf

[Martin Baker]

On 12/04/2024 09:23, Ralf Hemmecke wrote:
> I do not understand why you think that not both graphics systems can
> live besides each other (as they do now)?

Well, I think that the interactive graphics should work seamlessly with
the things that scene.spad can do such as outputting to various file
formats and having much more control over the output. For all this to
work together without duplication it all needs to be written in SPAD.

The problem is interactive graphics require multi-threading and I can't
see anyone agreeing to a limitation that SPAD can only work on top of
some specialised lisp with multi-threading. I suspect that is the reason
for the C code to allow multi-threading.

People usually suggest using their favorite 3rd party graphical front
end but I can't see Waldek making FriCAS totally reliant on some 3rd
party software.

Another problem is that graphic interfaces, hardware, file formats
change rapidly and these interfaces need more maintainable code. For
example will FriCAS support X11 to wayland transition?

Martin

[Ralf Hemmecke]

On 4/12/24 16:20, Martin Baker wrote:
> On 12/04/2024 09:23, Ralf Hemmecke wrote:
>> I do not understand why you think that not both graphics systems
>> can live besides each other (as they do now)?
>
> Well, I think that the interactive graphics should work seamlessly
> with the things that scene.spad can do such as outputting to various
> file formats and having much more control over the output. For all
> this to work together without duplication it all needs to be written
> in SPAD.

So, yes. If I were you, I would simply ignore the existing graphics
system and provide all the features that *you* think would be great and
help you in your work. That would inclulde connecting (or writing from
scratch) an interactive graphics system in SPAD. Why would you care what
Waldek or anybody else who has write access to the FriCAS repo says? You
can put everything into your own repo and try to promote it to users.
If your system is superior, it will at some point be seen and more users
will ask to properly include it into FriCAS.
Clearly, nobody can assume that his/her code will be included into the
"official" code base. But with open source it is terribly easy to create
your own fork and show that your stuff is better. Yes, with forking
comes maintenance and other costs, so one must decide whether to fork,
fight with the maintainers for inclusion or simply abandon the project.

It's not an easy taskt to make the world a better place.

> The problem is interactive graphics require multi-threading and I
> can't see anyone agreeing to a limitation that SPAD can only work on
> top of some specialised lisp with multi-threading. I suspect that is
> the reason for the C code to allow multi-threading.

Hmmm... strangely enough we have jFriCAS, it was easier if FriCAS is
compiled with a lisp that has a webserver built-in. And, in fact,
jFriCAS is only properly working with SBCL (at least I haven't tested
with other lisps).
What I want to say is that your interactive graphics system can work on
a certain type of lisp. It just means that by default FriCAS provides
the old graphics system and your features would require the users to
compile with a specific lisp. Why should that be a problem. Your
graphics system would be optional and whether to compile it or not is
only a configure option away. That would reduce maintenance cost for
anybody else except you. You then must show that you are interested and
prove it by maintaining your code and fixing bugs.
Maybe Waldek would still be against, but having a better interactive
graphics system would get support from me for its includion as an
(optional) add-on.

> People usually suggest using their favorite 3rd party graphical
> front end but I can't see Waldek making FriCAS totally reliant on
> some 3rd party software.

Yes, that is dangerous. But allowing users to decide, is another.

> Another problem is that graphic interfaces, hardware, file formats
> change rapidly and these interfaces need more maintainable code.

Yes, of course, some people like old code more since they are used to it
and know how it works, the younger generation probably want newer system
and newer code. And surely, knowing all the newer graphics systems I
would be unwilling to support any weird old stuff and fight for
inclusion of newer technology. That's not a bad thing.

> For example will FriCAS support X11 to wayland transition?

Well, there are only two options. 1) It will. 2) FriCAS will lose it's
graphics capabilities. Which one would you choose if you were a
maintainer of FriCAS?

Ralf

[Hill Strong]

There is nothing to stop anyone from breaking the requirement of using any Lisp as the destination for Boot code. I translated one section of Boot code directly into Icon. Of course there is much in Boot than assumes Lisp lists. But that is not an impossible task to change.

You could then use the iconc compiler to transform the icon code to C and compile to machine code.

I have not continued that development as I am much more interested in changing the Spad language to be a more consistent block structured language, including adding in static  environmental structures ala ALGOL and it's descendents. 

--
You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fricas-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/4df02ef4-1b24-42f1-a68c-3cb122a162f8%40hemmecke.org.

[Waldek Hebisch]

On Fri, Apr 12, 2024 at 08:41:18AM +0100, Martin Baker wrote:
> I don't mind, I have given up with this software now.
>
> When I wrote the code I was hoping that this approach would replace old
> boot and C code with something much more maintainable and flexible. Also
> the scenegraph architecture makes it much easier to export to 2D and 3D
> graphics files like SVG and X3D and being written in SPAD it would have
> been easy to add other graphics formats.

Hmm, I am not sure what _exactly_ you expected. I have a book about
"object oriented program analysis". It stress that one should take
into account "critical factors for success". For me personally, and
I suspect that for most FriCAS users "critical factors for success"
is ability to use higher level interactive commands and pop up
windows.

I considerd hooking 'draw' to your framework. However, architecturaly
nice way would direct all graphics trough your framework. But at the
end _I_ want image in a window as one of options. Which means appropriate
"output format". Writing convertor from scengraph data structures to
say X11 drawing commands is a bit more than I am willing to do.
To say the truth I do know how much work is needed, I would first
go trough your code and analize what it needs.

Anyway, I hope to use your framework to produce SVG files, but as
a user most of my personal needs are covered by old framework. And
without implementing new featurs your framework can not do all
things that I need.

> I'm not critisising because I don't see the big picture but it seems to
> me that going from boot code to SPAD in lots of small steps rather than
> one big step is impossibly slow and and risks replacing messy boot code
> with messy SPAD code.

'src/interp' subdirectory contains about 62000 lines of code. I expect
replacement to have similar size. If you want to include in rewrite
C code, then considering that we have about 50000 lines of C code
we need 20000-30000 lines of additional code. Considering that FriCAS
is a volunteer part time project writing and debugging that amount of
code will take several years (say 5 years). If you go via big shot
rewrite FriCAS would probably remain broken during rewrite period.
And that is relatively optimistic estimate. Debugging tricky problems
can sink a lot of time. How many developers are going to stay in
a "broken" project? How many users are going to stay with project
that makes no useful release?

Krystian Baclawski attempted rewrite of Spad compiler as new code.
It took him few years and still his compiler is not usable on
algebra code, to finish it would require substantial work.

--
Waldek Hebisch

[Waldek Hebisch]

On Sat, Apr 13, 2024 at 10:38:34AM +1000, Hill Strong wrote:
> There is nothing to stop anyone from breaking the requirement of using any
> Lisp as the destination for Boot code. I translated one section of Boot
> code directly into Icon. Of course there is much in Boot than assumes Lisp
> lists. But that is not an impossible task to change.

If one want to test independence of Lisp, then IMO Poplog is an
interesting possiblity. Poplog has execution engine that is in
many aspects similar to Lisp, but different in some other aspects.
For example in Poplog "words" have single value, which may be
a piece of data or a function. Poplog have data types needed by
Spad: booleans, big integers, lists, vectors, arrays. Poplog have
"words" that could be a basis for Spad symbols. Or alternatively,
in Poplog one can define new type for Spad symbols.

Poplog execution engine is general enough to support several
languages. Core language is Pop11, which has Pascal like
syntax, but higher level semantics. Name Poplog came because
Poplog execution engine supports Prolog (so Poplog is an
abbreviation of Pop11+Prolog). There are 2 other languages
implemented in Poplog: SML and Common Lisp. Currently FriCAS
can run using Common Lisp. But it would be interesting to
target lower level of Poplog directly, bypassing Common Lisp.

--
Waldek Hebisch

[Dima Pasechnik]

On 13 April 2024 02:38:34 CEST, Hill Strong <elders...@gmail.com> wrote:
>There is nothing to stop anyone from breaking the requirement of using any
>Lisp as the destination for Boot code. I translated one section of Boot
>code directly into Icon. Of course there is much in Boot than assumes Lisp
>lists. But that is not an impossible task to change.
>
>You could then use the iconc compiler to transform the icon code to C and
>compile to machine code.

But that's what ECL is doing, compiling to C and then calling a C compiler? Would it be more useful to figure out why the resulting FriCAS runs considerably slower than the one built using SBCL ?

Dima

[Hill Strong]

One of the reasons that started me looking at a non-Lisp base was the fundamental use of lists in the Boot implementation. Within Icon (or in my case using the Unicon system), lists are implemented in such a way that accessing any element was much faster than via a cons cell. You could treat a list in much the same manner as an array, even though it is not. The other thing that I saw as a poor fit was the use of lists (cons cells) for what was really a record structure. Some people are quite happy to use lists as their general purpose structure of choice, others prefer to use more appropriate structures. This is simply a matter of choice for the individual.

As far as ECL is concerned, I don't have any insight to offer here as I don't as a matter of course use any Common Lisp implementation for any programming that I do.Most of my programming is based on using Unicon/Icon for my interest today. Being effectively retired, I don't have to use whatever is in use by others.

Though I was at one stage considering looking at the practicality of using Scheme as the implementation language for Boot. There are just too many fundamental differences in philosophy between Common Lisp and Scheme and I put that project to one side.

Hill

To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/AA56A632-E864-49D0-BA9C-31AE14C76F91%40gmail.com.

[Waldek Hebisch]

On Sat, Apr 13, 2024 at 12:54:22PM +0200, Dima Pasechnik wrote:
>
> But that's what ECL is doing, compiling to C and then calling a C compiler? Would it be more useful to figure out why the resulting FriCAS runs considerably slower than the one built using SBCL ?

On social level it is clear: ECL has small developement resources
and speed is not a top priority. sbcl descends from CMUCL which
was designed for performance and a lot of effort went into
improving performance.

On more technical level some things are known for long time.
First, sbcl has well working profiler which is very helpful
in identifying bottlenecks. ECL folks basically say: use
C tools. But IME C profilers work well for identifying
low-level troubles. But in machine generated code slowness
is frequently spread out over large area and C tools do not
work well. Second, there is semantic mismatch between high-level
untyped operations offerd by default in Lisp and typed low-level
code needed for high performance. Spad code is typed and Spad
complier makes some effort to preserve types in its translation
to Lisp. sbcl makes quite good job extracting low level types
from Lisp code and when such types are available generates
resonably good code (gcc can typically generate code that runs
2 times faster, but what sbcl code generator is doing is not bad).

ECL is much worse, in many cases it generates calls to general
routines instead of directly doing the job. My benchmarking
indicated that function calls in ECL are much slower than
in sbcl. IIUC this is because Lisp semantics allows fancy
argument processing and merely checking that no such
processing is need takes time and is relatively inefficient
in C. sbcl is doing equvalent thing using machine code idioms
which are more efficient for this purpose. Today, experiment
with parsing suggest that ECL garbage collector may be
seriously mistuned for FriCAS use: in sbcl parsing took 55 seconds.
In ECL FriCAS process run out of memory after burning 38
hours of CPU time. During that apparently it fully used
9 cores, so real time was about 4 hours. Since FriCAS code is
single-threaded the only semi-reasonable use of multiple cores
is for parallel garbage collection. It is possible that
ECL was heroically trying to fit data within its memory limit
(for large part of time process size was of order of ECL
heap limit). But even it that were the case, it looks like
mistuning: when program is at the edge of available memory
it is likely to fail in the future.

At least theoretically bad code generated by ECL could be
mitigated by directly generating C code from Spad. But
mistuned/slow garbage collector means that we probably
should avoid ECL for heavier work.

Long ago I informed Juan Jose Garcia-Ripoll who was developing
ECL about various problems and he was able to mitigate some
of them increasing ECL speed about 4 times. But core issues
remain and are harder to solve. And current ECL group actually
decreased performance, at least in terms of CPU efficiency,
which leads to longer build/test times.

--
Waldek Hebisch

[Waldek Hebisch]

On Fri, Apr 19, 2024 at 01:18:13PM +0200, Waldek Hebisch wrote:
> On Sat, Apr 13, 2024 at 12:54:22PM +0200, Dima Pasechnik wrote:
> >
> > But that's what ECL is doing, compiling to C and then calling a C compiler? Would it be more useful to figure out why the resulting FriCAS runs considerably slower than the one built using SBCL ?

<snip>

> ECL is much worse, in many cases it generates calls to general
> routines instead of directly doing the job. My benchmarking
> indicated that function calls in ECL are much slower than
> in sbcl.

A little uptate on this: recent ECL actually generates reasonable
code for function calls. I had a litte benchmark which measures
speed of calls and retried it. It turns out that main slowdown
is due to the way ECL implement its threading support. Benchamrk
is doing 10000000 and chain of 3 calls per iteration. Innermost
call increments a counter. Results are:

0.120 sec sbcl-2.2.9
0.239 sec ecl-23.9.9 with threads disabled
0.270 sec gcl-2.6.14
0.646 sec ecl-23.9.9 default build

So with threading disabled on this benchmark ECL is slightly
faster than GCL and gives about half of sbcl speed. Threading
slows down ECL to about 5.4 times slower than sbcl. More
precisely, with threading enabled each ECL compiled function
is doing call to 'pthread_getspecific'. Essentially only
difference between enabling threading and disabling it is
presence of this call.

Concerning the factor 2 between "good" ECL and sbcl:
- ECL generates extra stores to global variables (probably
as part of debugging support)
- ECL generates extra checks
- ECL vectors use double indirection, sbcl uses simple indirection,
so ECL generated code needs more leads than sbcl code
- sbcl uses tail call optimization, gcc compiles ECL code to
normal calls

At machine code level sbcl code does not look great, but it
is performig much less work than ECL code.

I also tried on a bit larger scale and compiled FriCAS using
single threaded ECL. Results are
ecl-23.9.9 default cl-23.9.9 no threads
Build: real 12m27.006s 9m34.639s
user 46m56.687s 38m55.354s
sys 3m47.180s 2m29.636s
Tests: real 2m33.963s 1m33.818s
user 14m23.708s 7m34.652s
sys 0m37.619s 0m20.587s

As you can see there is substantial reduction in CPU time, and
slightly smaller reduction in real time. Some reduction in
CPU time is likely because mutithreaded ECL is running someting
(probably garbage collection) in extra threads. Testsuite runs
in parallel, but there are some long running tests that increase
real time, relation between real time and CPU time is a bit
complicated.

--
Waldek Hebisch