vector image (graphviz?) representing domain relationships?

[Qian Yun]

Hi Ralf,

I remembered there was discussion about using
vector image (graphviz?) to represent domain relationships.

Any progress/pointers?

- Qian

[Tim Daly]

An SVG full name graph

http://axiom-developer.org/axiom-website/bookvol10.2full.html

An SVG abbreviation graph

http://axiom-developer.org/axiom-website/bookvol10.2full.html

[Qian Yun]

Sadly it is encoded by hand.

I was looking for an automated process to generate such graph
from the SPAD DATABASE. If there's no such program, I might
try to write one.

One reason I want such a visualization graph is that I want
to see the domains that are not used by other domains,
(and visualize how certain domains are grouped/related together)
thus by striping those domains, I can get a "core" of SPAD
domains. I have no idea how big/small this "core" will be.

- Qian

[Ralf Hemmecke]

Hi Qian,

long time ago I had such a program for Aldor and its library,
but I have not yet found it. Still looking ...

However, fricas.github.io/api probably comes close to what you want.
However that would only lead to a dependency graph among the categories.

It is much harder to dig out whether some domain/package *uses* another
domain/package. And such dependencies are probably also interesting.

So for the category part, I suggest to look into src/doc/api.spad and
tweak it a bit so that it outputs in a .dot format instead of .rst.

Maybe it is also possible to use the tikz latex package, but honestly, I
would be more happy if we had a dependency graph as a visualization of
the APL, i.e. in some web-presentable format.

Ralf

[Tim Daly]

The "core" domains are those in the minimal bootstrap set.

This took a while to develop. The NAG Axiom required a running

Axiom to recompile but I was not allowed to distribute the NAG

version. My first task was to find the "core" set of domains

so Axiom could build from scratch.

You'll notice in the graph that there are a few "subsets" of nodes.

This is a side-effect of expertise from people who worked on Axiom.

One goal was to find "domain experts" to work on new domain clusters.

I had developed a whole subset related to robotics, my other area of

expertise, involving coordinate systems, transforms, Jacobians,

Lyapunov stability matrices, etc.

( https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant )

I never released it. The only domain that made it into the distribution is

the Denavit-Hartenberg (DHMATRIX) domain.

https://en.wikipedia.org/wiki/Denavit%E2%80%93Hartenberg_parameters

This is used to create the forward solution of the robot end effector.

[Tim Daly]

Ralf,

I chose the SVG format because it is web-scalable. The total graph is too

complex to fit on a web page without scaling. Useful scales are on the

fractions at the top of the page.

[Qian Yun]

On 9/21/24 6:50 PM, Ralf Hemmecke wrote:
> Hi Qian,
>
> long time ago I had such a program for Aldor and its library,
> but I have not yet found it. Still looking ...
>
> However, fricas.github.io/api probably comes close to what you want.
> However that would only lead to a dependency graph among the categories.
>
> It is much harder to dig out whether some domain/package *uses* another
> domain/package. And such dependencies are probably also interesting.

It is easy to verify that if a domain is not used by any other domains.
Then we remove it from the graph and do the trimming again.

> So for the category part, I suggest to look into src/doc/api.spad and
> tweak it a bit so that it outputs in a .dot format instead of .rst.

I'll look into it.

> Maybe it is also possible to use the tikz latex package, but honestly, I
> would be more happy if we had a dependency graph as a visualization of
> the APL, i.e. in some web-presentable format.
>
> Ralf
>

It seems that Graphivz (.dot format) already supports WEB format and
even interactive visualization.

- Qian

[Qian Yun]

On 9/21/24 6:55 PM, Tim Daly wrote:
> The "core" domains are those in the minimal bootstrap set.
> This took a while to develop. The NAG Axiom required a running
> Axiom to recompile but I was not allowed to distribute the NAG
> version. My first task was to find the "core" set of domains
> so Axiom could build from scratch.

Thanks for this information. I'll check the relevant part in Axiom.

Still, I wonder if an automated process can do this.

- Qian

[Tim Daly]

Quan,

One suggestion for finding dependencies is to turn on autoload messages.

Starting from a clean image you can find the list of domains that need to

be autoloaded. Choosing one of the listed domains you can apply the same

technique to see what it requires.

I suppose you could capture the code used for autoloading and use it to

develop a standalone tool. Or you could use the full image, capture the

messages, and automate a depth-first domain walk mentioned above.

You'll discover that trying to write code to parse the algebra source code

to be "a challenge" to put it mildly. Only Bill Burge managed to do that.

It is VERY hard to decide where that "+" sign comes from by inspecting

source code.

Be aware that it took a whole summer to find that minimal boostrap set

and a long time to create the SVG dependency graph.

There is no such thing as a simple job. (Daly's law of programming)

[Qian Yun]

On 9/21/24 7:17 PM, Tim Daly wrote:
> Quan,
>
> One suggestion for finding dependencies is to turn on autoload messages.
> Starting from a clean image you can find the list of domains that need to
> be autoloaded. Choosing one of the listed domains you can apply the same
> technique to see what it requires.
>

I wonder if it is possible to get those dependencies from the static
*.daase files (aka the database).

- Qian

[Tim Daly]

I think the information you want is only available at runtime.

Look at the GETDATABASE function. It would be the best place to

capture dynamic lookups.

Looking up a function and finding the implementation either at the

domain or category level is a complex process. At any time in the

future someone can add / change a domain / category which 

will change the graph. You'll probably have to generate a new graph

with every system build or, at least, every algebra change.

Trying to discover this information from static *.daase files is basically

writing a new compiler, which would naturally require implementing

large portions of the interpreter. The Aldor effort was supposed to create

a new compiler before it went off the rails into a new programming language.

This is the reason my new version is pure Common Lisp. Creating the

tool you want is probably a one-day coding effort.

[Waldek Hebisch]

On Sat, Sep 21, 2024 at 06:13:05PM +0800, Qian Yun wrote:
>
> I was looking for an automated process to generate such graph
> from the SPAD DATABASE. If there's no such program, I might
> try to write one.
>
> One reason I want such a visualization graph is that I want
> to see the domains that are not used by other domains,
> (and visualize how certain domains are grouped/related together)
> thus by striping those domains, I can get a "core" of SPAD
> domains. I have no idea how big/small this "core" will be.

There are different kinds of dependencies. One is category inheritance,
domain "depends" on its category which in turn inherit from other
categories. In particular this process gives you list of exported
operations. Another kind of dependency is via types of arguments
or return values, that is types appearing in siganture. Third
kind of dependency is when domain calls functions in other constructors.
The third kind may be somewhat obscured because of inlining.
Then you have practical runtime dependency "domain A needs to
load constructor B".

Inheritace information is contained in databases and used by browser.
In principle extracting inheritance information is should be easy.
But drawing it looks tricky. More precisely, inheritance graph
for a single domain is likely to be sizeable, say 50 to 150 nodes.
Automatiacally drawing such graph so that it is readible requires
some effort.

If you add dependencies via argument types, then graph will be
larger.

Concerning calls to other constructors, currently this is discovered
by looking at compiled code of a domain, more precisely domain
vector. Due to this it misses inlined functions.

Concerning "core" of FriCAS, in my early experiments with bootstap
of algebra (that was in Axiom era) I tried to compile a domain,
when it failed I added domains and categories needed to compile it
and tried again. IIRC I got to few hundreds constructors.

If you look at algebra you will see reasons: any domain is likely
to use someting from really fundamental domains like Boolean or
Integer. Boolean inherits from Finite which brings Integer into
play. Integer has complicated category structure which will bring
into play probably about 100 categories. In particular we get
PolynomialFactorizationExplicit which pulls in polynomials.

As Tim wrote, you can do

)set messages autoload on

and see messages about loading various constructors. Note that
loading is lazy, so this gives only messages about actually
used domains and still simple operations lead to loading of
many domains.

--
Waldek Hebisch