Modularization project: V. The blocs

[Matthias Koeppe]

(index of previous posts: https://github.com/sagemath/sage/issues/29705)

A common wisdom applies: Do your combinatorial explosions at home, not in public.

In other words, we should not attempt to define a distribution package for every possible community or subfield of mathematics that Sage supports.

The proposed design instead introduces 3 types of distribution packages:

Packages that are named after a basic mathematical structure but may cover a wide range of generalizations/applications of this structure. People who work in a specialized research area will of course recognize what structures they need. But the down-to-earth naming also creates discoverability for a broader audience. Not many more distribution packages than these 7 are needed:

- sagemath-combinat: everything combinatorial except for graphs.

- sagemath-graphs: also provides posets, designs, abstract complexes, quivers, etc.

- sagemath-groups

- sagemath-modules: also has matrices, tensors, homology, coding theory, etc.

- sagemath-polyhedra: also provides fans, hyperplane arrangements, etc.

- sagemath-schemes

- sagemath-symbolics

Packages that are named after an upstream library. This makes technical sense because the dependencies will be localized to this distribution package, but it also helps give attribution and visibility to these libraries and projects that Sage depends on. (These narrowly focused packages could also invigorate collaboration with the upstream projects.) Examples:

- sagemath-flint

- sagemath-gap

- sagemath-giac

- sagemath-ntl

Packages that are named after a technical functionality. 

- sagemath-plot

- sagemath-repl: The top-level interactive environment with the preparser.

- sagemath-standard: Everything as provided by a standard installation of the Sage distribution.

Each of the distribution packages can define a list of extras (nicknames for sets of other distribution packages that provide additional advertised features). When using pip to install a distribution, users can use square bracket notation to add (adjoin?) these extras.

Need fast linear algebra over GF(pⁿ) and over cyclotomic fields?

Use pip install "sagemath-modules[GFpn,CyclotomicField]".

So you need Berkovich spaces and elliptic curves over the p-adics?

That's pip install "sagemath-schemes[padics]".

So you want to do algebraic topology?

Just do pip install "sagemath-graphs" "sagemath-modules[FiniteField]".

Algebraic combinatorics?

That may mean pip install sagemath-combinat sagemath-modules for you.

Representation theory?

That may be pip install sagemath-groups sagemath-combinat or, shorter, pip install "sagemath-groups[representations]".

Graph theory and everything related that the Sage distribution provides?

Use pip install "sagemath-graphs[standard]".

All of this is added in https://github.com/sagemath/sage/pull/35095 (WIP).

To try it out, use ./bootstrap && make pypi-wheels

[David Roe]

I assume that the MANIFEST.in.m4 file contains a list of what's in each package, right?  There's also sagemath-objects and sagemath-categories (are there any others?); can you send analogous links for those so that we can understand how the library is being broken up into pieces?  Is every file part of exactly one package (aside from sagemath-standard, which is the union of all of them?).  From the quick glance I've taken so far, the division looks pretty reasonable.

David

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/edd7c02a-74d5-4d04-b598-2a923c0410cen%40googlegroups.com.

[Matthias Koeppe]

On Friday, June 16, 2023 at 11:07:36 AM UTC-7 David Roe wrote:

I assume that the MANIFEST.in.m4 file contains a list of what's in each package, right? 

That's right, this is the current mechanism.

I have plans for switching this to something more maintainable, which will also enforce that the set of modules is exactly partitioned into the various didstributions.

 

There's also sagemath-objects and sagemath-categories (are there any others?);

Yes, these are two of the most fundamental distributions. See this diagram for the runtime dependency relations. 

sagemath-objects has everything that makes the coercion system work (including functorial constructions).

 On top of that, sagemath-categories provides the full set of categories, basic rings, functions, without introducing non-Python library dependencies.

can you send analogous links for those so that we can understand how the library is being broken up into pieces?  Is every file part of exactly one package (aside from sagemath-standard, which is the union of all of them?).

It depends on whether we're talking about the modules directly included in (shipped by) a distribution package, or also the modules shipped by the declared runtime dependencies of the distribution package. 

Each module is shipped by exactly one distribution package. 

sagemath-standard will ship very few modules directly, but will have lots of other distributions as declared runtime dependencies, which will be installed automatically.

[David Roe]

How do you handle files that depend on multiple libraries (like both FLINT and NTL for example)?  Are these only included in sagemath-standard?
David

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/da02f9b5-a33c-4848-8fbf-ba6ece6a3580n%40googlegroups.com.

[Matthias Koeppe]

On Friday, June 16, 2023 at 12:52:16 PM UTC-7 David Roe wrote:

How do you handle files that depend on multiple libraries (like both FLINT and NTL for example)?  Are these only included in sagemath-standard?

In many cases, such multiple dependencies are gratuitous and can be resolved by splitting a Cython module into several parts. 

For example: sage.matrix.misc: Split by library dependency #35758

Some of the basic ring element implementations are highly encumbered with such multi-library dependencies.

As an extreme example, NumberFieldElement_quadratic cimports everything under the sun. This can be fixed by moving methods that use conversion out to separate new Cython modules, which will be imported rather than cimported. I have already eliminated many compile-time dependencies on PARI from basic ring element implementations such as the Integers in this way.

FLINT and NTL is an interesting example. FLINT, in the configuration that we use, actually has a dependency on NTL -- just to provide a conversion facility between the libraries. See https://github.com/sagemath/sage/issues/32908; I have inquired with upstream.

When dependencies have been reduced to, say, 2 simultaneous libraries per module, then we make can ship the module as part of a distribution that has only 1 required runtime dependency but 2 build-time dependencies that are used for the affected modules. To be able to import the affected module, another distribution has to be present. (That's Rules 3./4.)

For example, in sagemath-modules I use a build-time dependency on numpy, so that I can build matrix implementations such as Matrix_real_double_dense (which cimports numpy). But numpy is not a declared runtime dependency of sagemath-modules; it is only an extra that can be activated by using pip install "sagemath-modules[RDF]" (or, equivalently, pip install sagemath-modules numpy).

[Matthias Koeppe]

... 5 months later...

On Friday, June 16, 2023 at 11:19:23 AM UTC-7 Matthias Koeppe wrote:

On Friday, June 16, 2023 at 11:07:36 AM UTC-7 David Roe wrote:

I assume that the MANIFEST.in.m4 file contains a list of what's in each package, right? 

That's right, this is the current mechanism.

I have plans for switching this to something more maintainable, which will also enforce that the set of modules is exactly partitioned into the various distributions.

The next step here is a new tool "sage -fixdistributions", under review at https://github.com/sagemath/sage/pull/36135 

 

 There's also sagemath-objects and sagemath-categories (are there any others?);

Yes, these are two of the most fundamental distributions. See this diagram for the runtime dependency relations. 

sagemath-objects has everything that makes the coercion system work (including functorial constructions).

 On top of that, sagemath-categories provides the full set of categories, basic rings, functions, without introducing non-Python library dependencies.

can you send analogous links for those so that we can understand how the library is being broken up into pieces?  Is every file part of exactly one package (aside from sagemath-standard, which is the union of all of them?).

It depends on whether we're talking about the modules directly included in (shipped by) a distribution package, or also the modules shipped by the declared runtime dependencies of the distribution package. 

Each module is shipped by exactly one distribution package. 

sagemath-standard will ship very few modules directly, but will have lots of other distributions as declared runtime dependencies, which will be installed automatically.

An updated diagram of ↖↑↗ run-time dependencies ("install-requires"), ⇠⇡⇢ build-time dependencies

Ready for review: A restructuring of our "all.py" files along these dependencies in https://github.com/sagemath/sage/pull/36676

This is an opportunity to review the contents of the proposed distributions implemented in Mega-PR https://github.com/sagemath/sage/pull/35095 (~50 kLOC changes, not open for review).

As https://github.com/sagemath/sage/pull/36676 rewrites all "all.py" files, it is also an opportunity for a deliberate coding style decision for these files. I welcome all constructive discussions in the PR.