Proposal for Sage 9.3: Modularization of sagelib; in particular, splitting out a sage_objects package

[Matthias Koeppe]

I propose the following goal for Sage 9.3.

Modularize sagelib so that individual parts of it can be built and run with a much smaller set of libraries.

In particular, create a package "sage_objects" that makes Sage objects, the element/parent framework, categories, the coercion system and the related metaclasses available. This package would, for example, not provide or know about integers, polynomials, or matrices (thus avoiding all the dependencies on the various libraries implementing these). Ideally it would only have a build dependency on Cython and no runtime dependencies. 

(Doctesting sage_objects would still need more of Sage, but some it could also be done by mocking instead of with the real classes.)

a) I think these facilities could see a wider use in the mathematical Python community, and in this way we could hope to benefit from a larger developer base. For example, I would hope that sage_objects could be ported for use with PyPy instead of CPython.

b) Building and deploying a user package would be easier if it could depend on a small package such as "sage_objects" only instead of the whole multigigabyteness of Sage.

I would hope that this can be done without changing the overall structure of the source tree, using the mechanism of native namespace packages introduced in Python 3.3 (see https://trac.sagemath.org/ticket/28925).

[Michael Orlitzky]

On 5/3/20 8:09 PM, Matthias Koeppe wrote:
> I propose the following goal for Sage 9.3.
>
> Modularize sagelib so that individual parts of it can be built and run
> with a much smaller set of libraries.
>

A noble goal, but it's going to be difficult given the widespread use of
lazy_import to create circular references. We're probably looking at a
lot of refactoring before any useful modules can be excised. But that
doesn't mean it's not worth trying.

A good first step might be to ban new circular imports, but that means
that e.g. parent structures and their element classes would need to be
defined in the same files for things like

class FooElement(...):
pass

class Foo(...):
Element = FooElement

to work. And I'm sure that's not the only widespread antipattern that
will be uncovered.

[David Roe]

On Sun, May 3, 2020 at 8:27 PM Michael Orlitzky <mic...@orlitzky.com> wrote:

A good first step might be to ban new circular imports, but that means
that e.g. parent structures and their element classes would need to be
defined in the same files for things like

  class FooElement(...):
      pass

  class Foo(...):
      Element = FooElement

to work. And I'm sure that's not the only widespread antipattern that
will be uncovered.

If the element is written in Cython and the parent in Python they can't be in the same file.

In general I worry about rules that prevent us from breaking up large source files into pieces.

David

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/7c3e1653-b279-9f43-03e8-3f03c682fdf8%40orlitzky.com.

[Vincent Delecroix]

The category framework can not really be separated from the
Parent/Element/CategoryObject (e.g. coercion is involved in
comparisons and binary operations). And the former can not
live without the integers (e.g. the output of .cardinality()
expects an integer). More dramatically, in sage.categories

bimodules imports QQ, RR
classical_crystals imports SR
finite_coxeter_groups imports QQbar

One can try to not populate the whole category system.

Given the current status of the library, I don't think this
is reasonable to state it as a milestone goal. But still, it
is desirable to introduce modularity. Starting with a
sage-core / sage-rest would be nice.

Best
Vincent

[Sébastien Labbé]

On Monday, May 4, 2020 at 2:09:28 AM UTC+2, Matthias Koeppe wrote:

I propose the following goal for Sage 9.3.

Do you mean the next cycle 9.2 or really the one after?

[Matthias Koeppe]

Yes, the one after 9.2 for modularization.

I think 9.2 will have to be focused on major package upgrades that are enabled by dropping Python 2 support.

[Matthias Koeppe]

On Sunday, May 3, 2020 at 11:18:12 PM UTC-7, vdelecroix wrote:

Le 04/05/2020 à 02:27, Michael Orlitzky a écrit :
> On 5/3/20 8:09 PM, Matthias Koeppe wrote:
>> I propose the following goal for Sage 9.3.
>>
>> Modularize sagelib so that individual parts of it can be built and run
>> with a much smaller set of libraries.
>

> it's going to be difficult given the widespread use of
> lazy_import to create circular references. We're probably looking at a
> lot of refactoring before any useful modules can be excised.
>

> A good first step might be to ban new circular imports, but that means
> that e.g. parent structures and their element classes would need to be
> defined in the same files

My proposal does NOT say that every Python module should become a separate Python package. It is not a prerequisite to fight against all circular imports.

A parent class and its corresponding element class would certainly not be separated into separate packages. 

But of course they can stay in separate modules (source files) as they are now. 

 

The category framework can not really be separated from the
Parent/Element/CategoryObject (e.g. coercion is involved in
comparisons and binary operations).

Thanks, yes, I am aware of this. That's why in my proposal all of these would be in the same package sage_objects.

 

And the former can not
live without the integers (e.g. the output of .cardinality()
expects an integer).

Noted. This will need a solution. 

 

More dramatically, in sage.categories

  bimodules imports QQ, RR
  classical_crystals imports SR
  finite_coxeter_groups imports QQbar

One can try to not populate the whole category system.

Good points. 

Given the current status of the library, I don't think this
is reasonable to state it as a milestone goal. But still, it
is desirable to introduce modularity. Starting with a
sage-core / sage-rest would be nice.

Well, the technical design of this modularization, even if not completely implemented within 9.3, will inform work on https://trac.sagemath.org/ticket/21507 (Meta-ticket: Make sagelib a pip-installable Python source package, listed on PyPI).

[Matthias Koeppe]

On Sunday, May 3, 2020 at 11:08:10 PM UTC-7, David Roe wrote:

In general I worry about rules that prevent us from breaking up large source files into pieces.

No worries, certainly not part of what I have in mind - see other posts.

 

[Matthias Koeppe]

On Sunday, May 3, 2020 at 5:09:28 PM UTC-7, Matthias Koeppe wrote:

I propose the following goal for Sage 9.3.

Modularize sagelib so that individual parts of it can be built and run with a much smaller set of libraries.

In particular, create a package "sage_objects" that makes Sage objects, the element/parent framework, categories, the coercion system and the related metaclasses available. 

I have created https://trac.sagemath.org/ticket/29705 for this.

[Matthias Koeppe]

On Sunday, May 3, 2020 at 5:09:28 PM UTC-7 Matthias Koeppe wrote:

I propose the following goal for Sage 9.3.

Modularize sagelib so that individual parts of it can be built and run with a much smaller set of libraries.

In particular, create a package "sage_objects" that makes Sage objects, the element/parent framework, categories, the coercion system and the related metaclasses available. This package would, for example, not provide or know about integers, polynomials, or matrices (thus avoiding all the dependencies on the various libraries implementing these). Ideally it would only have a build dependency on Cython and no runtime dependencies. 

(Doctesting sage_objects would still need more of Sage, but some it could also be done by mocking instead of with the real classes.)

A quick update on this.

1) There is now a branch on https://trac.sagemath.org/ticket/29865  that can build a pip-installable source distribution "sage-objects", providing a small core of sagelib. It has only minimal dependencies (Cython, cysignals, gmpy2, pkgconfig; and I hope to reduce the dependencies further) and can load sage.categories.all. I could use help from developers familiar with the lazy_import facility to solve some remaining problems.

2) Work has also progressed on sagelib's build system to support modularization. Tickets leading to https://trac.sagemath.org/ticket/29701 move most code from setup.py to sage_setup, remove the use of module_list and replace the OptionalExtension facility.