Modularization project: I. The goals

[Matthias Koeppe]

Dear all,

I proposed the modularization project three years ago, in May 2020, in the post https://groups.google.com/g/sage-devel/c/M9QTWtln6zU/m/UHwkrmTKBQAJ

The most recent substantial discussions on sage-devel on this topic took place in Oct/Nov 2021; and I gave a presentation on it in June 2022 at the virtual Sage Days. (See https://github.com/sagemath/sage/issues/29705 for links to all of these and the hundreds of tickets that went into the project; many thanks to all individuals who collaborated on and/or reviewed these tickets over the years.)

So it is timely to give you an update on the modularization project. I'll start here with a short post that explains the motivation and the goals.

A. The project is about the modularization of the Sage library (contents of SAGE_ROOT/src/sage); it is unrelated to work on the Sage distribution (the externally maintained packages tracked in SAGE_ROOT/build/pkgs).

B. The Python and Cython modules of the Sage library have an extensive web of interdependencies ("everything depends on everything"). As the developer's guide section on modularization  explains, these dependencies range from build-time to module-level to method-level to "doctest-only" dependencies.

C. The Python and Cython modules of the Sage library also have numerous dependencies on externally maintained non-Python packages: To just build the (Cython modules of) the Sage library, installations of at least the following are required: arb boost brial ecl eclib ecm fflas_ffpack flint libgd gap giac givaro glpk gmp gsl iml lcalc libbraiding libhomfly libpng linbox m4ri m4rie mpc mpfi mpfr ntl pari rw singular symmetrica, as well as a number of Python packages. Then, to be able to start Sage, more Python packages need to be installed.

D. As a consequence of B and C, it was impossible to build or run parts of the Sage library. And it is impossible to install the whole Sage library using Python infrastructure (pip). (Yes, I know that conda exists.)

E. This is the key mechanism that isolates the Sage developer community from the (vastly larger) Python community. As is, no portion of Sage is viable to be used as the dependency of any Python packages. Potential developers of features face the difficult decision: "Are the facilities that a portion of Sage offers really important enough for my project --- that it makes sense to accept being isolated from the rest of the Python world?" Outside of a narrow context of research mathematics, the decision will often be "no".

This is a severe bug in the setup of the Sage community. The modularization project fixes this bug by creating viable distributions of portions of the Sage library. Viable means: Separately buildable with minimal dependencies, separately runnable, separately testable -- and thus making it possible to attract many more developers. And it can be done without sacrificing the rich integration with the rest of the Sage library.

More later.

Matthias

[Michael Orlitzky]

On Thu, 2023-06-08 at 14:09 -0700, Matthias Koeppe wrote:
>
> *D. *As a consequence of B and C, it was *impossible to build or run parts
> of the Sage library.* And it is *impossible to install the whole Sage
> library using Python infrastructure* (pip). (Yes, I know that conda exists.)
>

Of itself, modularization is a noble goal. But in the implementation,
please be wary of focusing on users we don't have at the expense of the
users we do have. The amount of additional complexity and maintenance
this entails is daunting, and your tentative plans for pip packaging
sometimes conflict with common-sense improvements for existing users
and developers, as I've been trying to argue (for the second time) on
issue #32242.

Pip isn't a real build system or package manager. Most egregiously, it
can't do anything with the non-python software on which sage subsists.
To make sage-via-pip work, we'll have to maintain a new pseudo-
distribution on pypi that either ships people pre-built wheels or wraps
autotools/cmake/etc in python. As was made clear in recent threads,
many developers don't want to be maintaining the *first* sage
distribution, much less a second one.

I think we're all happy to modularize so long as it doesn't create work
elsewhere. (At the very least, modularization improves build times.)
But given that you may be the only person who has pip-installablity as
a goal, I think you have to be more sensitive to the complaints when
that goal conflicts with others. Complexity is our biggest problem
right now, and uncompromising modularization is only adding to it.

[Matthias Koeppe]

On Thursday, June 8, 2023 at 4:40:06 PM UTC-7 Michael Orlitzky wrote:

Pip [...] can't do anything with the non-python software on which sage subsists.

To make sage-via-pip work, we'll have to maintain a new pseudo-
distribution on pypi that either ships people pre-built wheels or wraps
autotools/cmake/etc in python. As was made clear in recent threads,
many developers don't want to be maintaining the *first* sage
distribution, much less a second one.

That's right, building binary wheels to be distributed on PyPI (in addition to the source distributions) will be one of the key steps in order to make it very user-friendly -- i.e., on par with doing "pip install scipy" on standard platforms.

It is normal practice of Python projects to create binary wheels and make them available on PyPI. There is sufficient mainstream infrastructure (such as https://github.com/pypa/cibuildwheel) that makes it easy. 

And it is, in fact, even easier for us (almost free) because we can reuse the Sage distribution for this; we do NOT have to write new ad-hoc installation scripts for the non-Python dependencies. For example, I have recently provided a wheel-building workflow for an upstream project, PySCIPOpt, that does exactly this.

https://github.com/scipopt/PySCIPOpt/pull/652/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711R16

[Dima Pasechnik]

some other examples of such pip-installable spin-offs of parts of
Sage, some with binary wheels, some without, are pplpy, cysignals,
primecountpy
(more of this should come)

At some point we should consider fully switching to gmpy2 to provide
interfaces for gmp, mpfr, etc.
Are there any performance advantages of Sage's gmp (ZZ type) over gmpy2 ?

I'd also like Sage the distribution becoming a more well-defined
downstream of Sage the library, as a part of modularisation.


>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/efdc3b85-5690-49f7-814b-f5602dd89ee0n%40googlegroups.com.

[Matthias Koeppe]

On Friday, June 9, 2023 at 12:40:52 AM UTC-7 Dima Pasechnik wrote:

some other examples of such pip-installable spin-offs of parts of
Sage, some with binary wheels, some without, are pplpy, cysignals,
primecountpy
(more of this should come)

Yes! Making sure that all of these have (automatically built) binary wheels would be very important work.

 

At some point we should consider fully switching to gmpy2 to provide
interfaces for gmp, mpfr, etc.

Indeed. This idea is tracked in https://github.com/sagemath/sage/issues/35559 and if this can be done without sacrificing performance, this would be a welcome simplification.

 

[Travis Scrimshaw]

I strongly disagree with your conclusion that this is a bug, much less a severe one. My understanding of William's goal (please correct me if I am wrong) was to put everything together so nobody was trying to build a better wheel. To me, by splitting everything up into these small pieces, it seems contrary to that goal as how is someone suppose to know the piece they want already exists? For instance, the backend libraries all have existed on their own, and they aren't going to change.

What you seem to be proposing is a large maintenance burden as we have to

1) Write code that keeps the modularization both "upstream" and "downstream" within Sage's ecosystem.

1b) Write code in nonstandard ways to break certain dependencies that are there in practice.

2) Write doctests with tags (at various levels, e.g., individual, block, module) that will need to be updated.

2) Test the code with all of the different dependencies so we don't break (1) and (2).

My reading is that your main goal is to make Sage pip-installable, but that seems independent of breaking Sage up into smaller parts. Even breaking up Sage into smaller portions for more 'precise' dependencies for these supposed downstream Python developers won't really fix B and C. I agree with Dima that splitting off some of the smaller pieces into pip-installable projects that Sage then depends on is a good thing to do, but I think this is better down with a bottom-up approach by taking it away piece by piece.

I also would like to see an increase in the number of developers and users. Yet, we are still primarily math software with an eye towards research, which means we will always have a somewhat limited developer base. I am hoping you could answer the following questions in detail with your next post.

I) How do you expect the amount of work the average developer will have to put in to add a new component (e.g., a Python module on a new linear optimization routine) to Sage? To fix a small bug? A larger bug (say, requiring changing 3 or 5 files)?

II) Related to (I), but what automated tools do you expect to produce and when will they be available? What extend will they reduce the developer overhead? How accessible with their output be to read?

III) How do you expect this to bring new developers in? Do you have any evidence (including anecdotal) that this would increase the number of developers? (Since switching to GH, I have not noticed much of a change in the number of developers or the amount of contributions. I recall this being one of the claimed benefits to the switch. This feels like it should have had a larger impact than the proposed modularization.) Why would these larger-Python-ecosystem developers even contribute their code to Sage rather than just do their own off-shoot/one-off projects (that we wouldn't know about)?

IV) How would a user know what components are out there? Why don't we need to first have a better package-manager/distribution-platform/etc. first (which includes all of the downstream Sage pip-packages) before we try to split Sage up? I would expect this to be a requirement for modularization. Or are you still expecting people just to install one "Sage" thing and never have to worry about anything else?

V) How are we going to make sure code doesn't bitrot? In particular, who is responsible for fixing code that breaks "downstream"? Right now, that is clearly the work of the author(s), but could the following situation happen? Suppose we have a base Sage module A, then Bob writes his own separate library B that depends on A that we then include as part of the main "Sage" distribution. I make a change to A that breaks B. Who should fix it? What if Bob is no longer maintaining B?

There are many things I would like to see with your proposal included into Sage, and thank you for undertaking such a huge project. I have some reservations about the benefits over the costs, and I think it would be beneficial to have a more detailed plan before proceeding further.

Best,

Travis

[Matthias Koeppe]

On Sunday, June 11, 2023 at 6:20:03 PM UTC-7 Travis Scrimshaw wrote:

My understanding of William's goal (please correct me if I am wrong) was to put everything together so nobody was trying to build a better wheel. To me, by splitting everything up into these small pieces, it seems contrary to that goal as how is someone suppose to know the piece they want already exists?

The Sage distribution will continue to exist. There will be no user-visible change coming from the modularization project for the users of the Sage distribution.

My reading is that your main goal is to make Sage pip-installable, but that seems independent of breaking Sage up into smaller parts. Even breaking up Sage into smaller portions for more 'precise' dependencies for these supposed downstream Python developers won't really fix B and C.

It works the other way around. B and C (applied to individual Cython/Python modules) were/are _obstacles_ to making modularized pip-installable distributions.

I have been fixing B and C (in numerous tickets) so that I can make modularized distributions. It didn't come for free; but I have already done most of the necessary work... and clear plans for the other work required.

 

[...]. I am hoping you could answer the following questions in detail with your next post. [...]

Thanks a lot for your questions; I'll address them in post II.

[William Stein]

On Sun, Jun 11, 2023 at 6:53 PM Matthias Koeppe <matthia...@gmail.com> wrote:

On Sunday, June 11, 2023 at 6:20:03 PM UTC-7 Travis Scrimshaw wrote:

My understanding of William's goal (please correct me if I am wrong) was to put everything together so nobody was trying to build a better wheel. To me, by splitting everything up into these small pieces, it seems contrary to that goal as how is someone suppose to know the piece they want already exists?

I didn't split things up into small pieces only because it is very, very difficult, takes a lot of time, and I just didn't have the resources to do it.   Other priorities like increasing doctest coverage were higher.    From a software engineering perspective modularization is very valuable, and if we had more resources we definitely would have divided Sage into more modular blocks long ago.   I'm extremely glad that Matthias is working on this difficult problem, and I appreciate that you (Travis) are putting so much effort into clarifying what the heck is going on.

Sage is about building the car instead of reinventing the wheel.  However, cars are built out of a large number of modular components (e.g., wheels, carburetor, etc.), and it's good engineering to put a lot of effort into properly building those components.   I'm absolutely blown away and thrilled Sage is still around after almost 20 years, and that you guys are actually doing that sort of work!

--

William (http://wstein.org)

[Nathan Dunfield]

For the SnapPy project [1], modularization of Sage would very useful.  Currently, we provide SnapPy as a stand-alone GUI application on Windows and macOS and as a pip-install Python module on those platforms plus Linux.  When used in Sage, the SnapPy module gains extra functionality by leveraging various parts of sage.all, including features that are of high interest to our users.  It would be fantastic if we could include those in the stand-alone applications (including on Windows, which accounts for 2/3 of the downloads of the full application).

Best,

Nathan

[1] https://snappy.computop.org

[Matthias Koeppe]

Hi Nathan,

What parts of Sage does SnapPy use? (I was starting to look at https://github.com/search?q=repo%3A3-manifolds%2FSnapPy+%2F%28import+sage%7Csage.*import%29%2F&type=code&p=1 but this would take a while...)

Matthias

[Matthias Koeppe]

On Monday, June 12, 2023 at 11:18:05 AM UTC-7 Nathan Dunfield wrote:

[...] various parts of sage.all, including features that are of high interest to our users.  It would be fantastic if we could include those in the stand-alone applications (including on Windows, which accounts for 2/3 of the downloads of the full application).

Indeed. The modularized distributions also make it much easier (or even feasible) to port Sage features to new platforms. A native Windows build would be one important new platform; Web Assembly (wasm, pyodide) would be another one, see #34539.

One would proceed as follows:

1. make sagemath-objects build on the new platform

2. make sagemath-categories build on the new platform

3. make sagemath-categories pass its tests on the new platform 

4.–n. continue with other distributions that build upon sagemath-categories 

[Nathan Dunfield]

On Monday, June 12, 2023 at 2:58:18 PM UTC-4 Matthias Koeppe wrote:

What parts of Sage does SnapPy use? 

Primarily the various rings/fields, including matrices over them and basic linear algebra.  Specifically, interval arithmetic (Real/ComplexIntervalField), polynomial rings (in several variables, including Laurent polynomials), number fields, and the basic rings ZZ, QQ, GF, RealField, ComplexField, etc.  Also, the ptolemy submodule uses Sage for groebner_basis calculations.  I suspect most of what we use would be part of any initial pip-installable version of the Sage library.  Right now, in the stand-alone apps we backstop some of Sage's functionality (e.g. arbitrary precision floats) with CyPari, which is itself an example of part of Sage being modularized. 

Best,

Nathan

[William Stein]

Hi Sage Devs,

As further motivation for this discussion of "Modularization project:
I. The goals", here is a quote from a discussion just now (with
permission) from Eric Deeds, who is the vice chair for Life Sciences
Math Courses and Professor of Integrative Biology and Physiology at
UCLA:

"I think we may also switch to using Python with libraries, rather
than SageMath since Python is much more widely used."

At UCLA they currently use SageMath (the full environment and
language) with about 3K students per year, and are considering
switching from Sage to normal Python, since sage is its own custom
language and environment, which isn't exactly the Python language,
etc. This is motivation for modularization of sage, since there is
substantial functionality in Sage, e.g., "plotting vector fields",
which they would like to use still. Of course, I did explain how it
is possible to use normal Python and import sage via "from sage.all
import ...". However, properly supporting using Sage from Python like
any other Python citizen would be even better, and that's something
that would result from the project Matthias is spearheading.

-- William

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/4b14371d-b528-4b43-b350-5920259c88e1n%40googlegroups.com.



--
William (http://wstein.org)

[Travis Scrimshaw]

On Monday, June 12, 2023 at 10:53:38 AM UTC+9 Matthias Koeppe wrote:

On Sunday, June 11, 2023 at 6:20:03 PM UTC-7 Travis Scrimshaw wrote:

My understanding of William's goal (please correct me if I am wrong) was to put everything together so nobody was trying to build a better wheel. To me, by splitting everything up into these small pieces, it seems contrary to that goal as how is someone suppose to know the piece they want already exists?

The Sage distribution will continue to exist. There will be no user-visible change coming from the modularization project for the users of the Sage distribution.

That is simply not true right now. The # optional sage.* doctests as a user-visible change.

My reading is that your main goal is to make Sage pip-installable, but that seems independent of breaking Sage up into smaller parts. Even breaking up Sage into smaller portions for more 'precise' dependencies for these supposed downstream Python developers won't really fix B and C.

It works the other way around. B and C (applied to individual Cython/Python modules) were/are _obstacles_ to making modularized pip-installable distributions.

I don't follow. Separating the components of Sage won't fix B/C, nor will fixing B/C split Sage into smaller components (which will actually make the dependency web worse).

Best,
Travis

[Matthias Koeppe]

I think you cut away the part of my response; reading that part may be contain what you're missing.

[Travis Scrimshaw]

Hi William,

   That is interesting. Although my take on that is following Matthias's proposal, they will just use one (or more) part of Sage as a Python library. So if they switch, in effect they will still be dropping Sage. I don't see Sage as having its own custom language other than some small syntactic sugar items (although I guess the ^ for ** is a big one for people switching to Python), but it does sit within its own environment (which has its pros and cons IMO). It is slightly annoying to bring everything into that environment, but I feel that we can get a bit of a smoother user experience this way.

I guess some of it comes down to what our target audience is. Even if we were just a normal Python package and just do "from sage import *", I think Sage would have a hard time competing against things like NumPy and matplotlib for those first year courses. They are currently too broadly used in many fields/industries, so universities will want to emphasize those. However, Sage is better equipped to do the more specialized math courses and research based computations. I don't think we have spent nearly enough time making our own set of teaching tools to compete; mostly that has been split off/overtaken by the Jupyter notebooks or in the libraries we build off of. I think we would need to have more of an influence in industry jobs to bring more universities around to saying "start Python, then do 'from sage.all import *'."

Perhaps this is the question that I am looking for an answer for, "What is Sage (after modularization)?"

Best,

Travis

[Kwankyu Lee]

The Sage distribution will continue to exist. There will be no user-visible change coming from the modularization project for the users of the Sage distribution.

That is simply not true right now. The # optional sage.* doctests as a user-visible change.

Matthias means that all pieces (the sage library and external packages that the sage library depends on) are all there and available to a sage user after the modularization project. That is why you (and most of us) didn't notice and care much about the modularization project up to now, until those massive `# optional ...` tags start to appear in the code you care about.

I don't think we can dismiss `# optional ...` tags from the sage library if we embrace the modularization efforts of the sage library, as Matthias is trying to convince us. It is more productive to find ways to do it in more pleasing way.

 

[Travis Scrimshaw]

On Friday, June 16, 2023 at 10:49:49 AM UTC+9 Kwankyu Lee wrote:

The Sage distribution will continue to exist. There will be no user-visible change coming from the modularization project for the users of the Sage distribution.

That is simply not true right now. The # optional sage.* doctests as a user-visible change.

Matthias means that all pieces (the sage library and external packages that the sage library depends on) are all there and available to a sage user after the modularization project.

It depends on what you mean by "there" because it is not been explained I think how a user should install and work with Sage afterwards. I agree it is clearly all available, but depending on how we tell a user to install, there could very well be changes.

 

That is why you (and most of us) didn't notice and care much about the modularization project up to now, until those massive `# optional ...` tags start to appear in the code you care about.

I've been following along with different pieces to see how it's going. You are right, in part I am bringing this up now because it is is affecting code I care about (and regularly use to promote Sage), but it is also starting to come with policy implications due to its scale that are not easy to revert.

I don't think we can dismiss `# optional ...` tags from the sage library if we embrace the modularization efforts of the sage library, as Matthias is trying to convince us. It is more productive to find ways to do it in more pleasing way.

I concur.

Best,

Travis

[Kwankyu Lee]

On Friday, June 16, 2023 at 10:58:36 AM UTC+9 Travis Scrimshaw wrote:

... but it is also starting to come with policy implications due to its scale that are not easy to revert.

Right. So I welcome all these discussions. We (the developers and even those focusing on research mathematics) need to be well aware of the modularization efforts: the modularized structure of sage and the developer tools to support it.  

 

[Matthias Koeppe]

On Thursday, June 15, 2023 at 6:58:36 PM UTC-7 Travis Scrimshaw wrote:

I've been following along with different pieces to see how it's going. You are right, in part I am bringing this up now because it is is affecting code I care about (and regularly use to promote Sage)

Well, when you promote Sage to new developers, there is a simple and positive way to explain the new annotations that I would recommend:

"Some people in our developer community are working hard so that when you are ready to make your contributions to the project, you can have the hope that your contribution can be accessed by the Python community, rather than being hidden away in a walled garden (that chronically has too few gardeners). Right now, these tags are only decoration, so don't worry about them."

[William Stein]

On Thu, Jun 15, 2023 at 6:49 PM 'Travis Scrimshaw' via sage-devel
<sage-...@googlegroups.com> wrote:
>
> Hi William,
> That is interesting. Although my take on that is following Matthias's proposal, they will just use one (or more) part of Sage as a Python library. So if they switch, in effect they will still be dropping Sage.

If you really do see "using or more parts of Sage as a Python library"
as "dropping Sage", then I can see why you're so worried by the
modularization proposal.

> I don't see Sage as having its own custom language other than some small syntactic sugar items (although I guess the ^ for ** is a big one for people switching to Python),

There is a list of changes that the preparser introduces, and it's not
trivial, including the fact that all numerical literals are different
Overall, I think the preparser has a significant negative impact on
usability to Sage, because it cuts sage users off from a rich
ecosystem of tooling, e.g., automatic formatters like yapf and black,
the amazing LSP tooling of Python like
https://github.com/python-lsp/python-lsp-server, and it makes the
stack traces that one sees when using Sage confusing. I created the
preparser in the first place, but I can certainly appreciate that it
has a lot of negatives. When I created it, I had no idea how much the
general Python ecosystem would grow, or how much value there would be
in participating in it.

> but it does sit within its own environment (which has its pros and cons IMO). It is slightly annoying to bring everything into that environment, but I feel that we can get a bit of a smoother user experience this way.
>
> I guess some of it comes down to what our target audience is. Even if we were just a normal Python package and just do "from sage import *", I think Sage would have a hard time competing against things like NumPy and matplotlib for those first year courses.

matplotlib plotting api (at least pylab) is like matlab. Sage has a
huge amount to offer related to plotting that is complementary to
matplotlib, due to the mathematica like plotting api, and also rich
support for symbolic functions with fast evaluation.

> They are currently too broadly used in many fields/industries, so universities will want to emphasize those. However, Sage is better equipped to do the more specialized math courses and research based computations. I don't think we have spent nearly enough time making our own set of teaching tools to compete; mostly that has been split off/overtaken by the Jupyter notebooks or in the libraries we build off of. I think we would need to have more of an influence in industry jobs to bring more universities around to saying "start Python, then do 'from sage.all import *'."
>

The point of making Sage part of the broader ecosystem is definitely
not to compete with numpy. It is to provide value to more people, and
to make sage better benefit from a much larger ecosystem. Also,
sympy is very popular and the sort of people that are interested in
installing sympy might also be interested in sage. Sympy was
downloaded over 2 millions times in the last week from pypi:
https://pypistats.org/packages/sympy

One thing is that the GPL license might be a significant non-technical
issue blocking interest in Sage by the broader python ecosystem, where
everything else is under much more commercial friendly licenses. That
ship has sailed (Sage will always be GPL'd), but it is something we
have to be realistic about.

> Perhaps this is the question that I am looking for an answer for, "What is Sage (after modularization)?"

Sage would be part of the Python ecosystem.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/91a2c3b6-a018-4931-890f-ce3ce5ced8b8n%40googlegroups.com.



--
William (http://wstein.org)

[Dima Pasechnik]

On Fri, 16 Jun 2023, 02:08 'Travis Scrimshaw' via sage-devel, <sage-...@googlegroups.com> wrote:

On Monday, June 12, 2023 at 10:53:38 AM UTC+9 Matthias Koeppe wrote:

On Sunday, June 11, 2023 at 6:20:03 PM UTC-7 Travis Scrimshaw wrote:

My understanding of William's goal (please correct me if I am wrong) was to put everything together so nobody was trying to build a better wheel. To me, by splitting everything up into these small pieces, it seems contrary to that goal as how is someone suppose to know the piece they want already exists?

The Sage distribution will continue to exist. There will be no user-visible change coming from the modularization project for the users of the Sage distribution.

That is simply not true right now. The # optional sage.* doctests as a user-visible change.

It's a metadata getting visible, not code change. It is a useful change, anyway.

Perhaps one can introduce a tag #needsmodule

instead of, admittedly over-used, #optional tag,

for these cases.

My reading is that your main goal is to make Sage pip-installable, but that seems independent of breaking Sage up into smaller parts. Even breaking up Sage into smaller portions for more 'precise' dependencies for these supposed downstream Python developers won't really fix B and C.

It works the other way around. B and C (applied to individual Cython/Python modules) were/are _obstacles_ to making modularized pip-installable distributions.

I don't follow. Separating the components of Sage won't fix B/C, nor will fixing B/C split Sage into smaller components (which will actually make the dependency web worse).

Best,
Travis

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/19987597-9875-4306-927c-4a287c70eb5fn%40googlegroups.com.

[Kwankyu Lee]

Perhaps one can introduce a tag #needsmodule

Yours is the latest: https://github.com/sagemath/sage/issues/35750#issuecomment-1594072121

 

[Michael Orlitzky]

>
On 2023-06-15 18:08:35, 'Travis Scrimshaw' via sage-devel wrote:
>
> That is simply not true right now. The # optional sage.* doctests as a
> user-visible change.
>

These tags aren't essential to the modularization itself. They're an
artifact of bad tests:

* doctests are in general just a bad way to test functionality (I
mean TESTS here, not EXAMPLES). Among their many other problems,
they force us to set up a full interactive sage environment, using
more parts of the sage library than would otherwise be necessary
for the test in question.

* For 20 years nobody has cared about writing tests that only test
the thing they're supposed to test. If the tests in (for example)
sage.categories tested only the code in sage.categories, then the
module could be split without any #optional tags. But since those
tests actually use code from all over the place, we have two options:

1) Stop and rewrite all of those tests
2) Declare them optional

We should still rewrite the bad tests eventually, even with the
#optional tags. For that reason, I read them as "#TODO: rewrite
this bad test", which makes me hate them less.

Regardless, I am also sympathetic that this is yet another massive
complicated project that we've started without resolving any of the
other ongoing massive complicated projects. One definitely gets the
feeling that nothing ever gets easier around here. If we rewrote the
bad tests BEFORE modularization, that would ensure that our situation
was monotonically improving, without the local regression from the
introduction of the #optional tags.

[Matthias Koeppe]

On Friday, June 16, 2023 at 12:08:29 AM UTC-7 Michael Orlitzky wrote:

[...] without resolving any of the

other ongoing massive complicated projects. 

One such project that has been ongoing (or rather started and then forgotten) since at least 2012 is the deprecation and removal of is_SomeClass functions, in favor of using isinstance.

It would be great if we could finish this; see https://github.com/sagemath/sage/issues/32414, which also explains why this helps the modularization project.

[Francesco Biscani]

On Fri, 9 Jun 2023 at 02:02, Matthias Koeppe <matthia...@gmail.com> wrote:

On Thursday, June 8, 2023 at 4:40:06 PM UTC-7 Michael Orlitzky wrote:

Pip [...] can't do anything with the non-python software on which sage subsists.
To make sage-via-pip work, we'll have to maintain a new pseudo-
distribution on pypi that either ships people pre-built wheels or wraps
autotools/cmake/etc in python. As was made clear in recent threads,
many developers don't want to be maintaining the *first* sage
distribution, much less a second one.

That's right, building binary wheels to be distributed on PyPI (in addition to the source distributions) will be one of the key steps in order to make it very user-friendly -- i.e., on par with doing "pip install scipy" on standard platforms.

It is normal practice of Python projects to create binary wheels and make them available on PyPI. There is sufficient mainstream infrastructure (such as https://github.com/pypa/cibuildwheel) that makes it easy.

What is your plan for dealing with sage's non-Python dependencies?

Binary wheels which bundle compiled non-Python dependencies using tools such as auditwheel are extremely fragile, as there is nothing preventing the user from installing a package which might bundle a different version of the same library, resulting in hard-to-debug erratic runtime errors due to ABI inconsistencies, symbol collisions, etc.

The scipy situation is not the same, as, to my knowledge, scipy's depends only on widely available libraries with simple and stable ABIs (e.g., BLAS, lapack).

I do not think it's realistic to expect to be able to sanely manage a multi-language software distribution with language-specific package managers.

Cheers,

  Francesco.

[Matthias Koeppe]

On Thursday, June 22, 2023 at 12:49:29 PM UTC-7 Francesco Biscani wrote:

On Fri, 9 Jun 2023 at 02:02, Matthias Koeppe <matthia...@gmail.com> wrote:

building binary wheels to be distributed on PyPI (in addition to the source distributions) will be one of the key steps in order to make it very user-friendly -- i.e., on par with doing "pip install scipy" on standard platforms.

It is normal practice of Python projects to create binary wheels and make them available on PyPI. There is sufficient mainstream infrastructure (such as https://github.com/pypa/cibuildwheel) that makes it easy.

What is your plan for dealing with sage's non-Python dependencies?

Binary wheels which bundle compiled non-Python dependencies using tools such as auditwheel are extremely fragile, as there is nothing preventing the user from installing a package which might bundle a different version of the same library, resulting in hard-to-debug erratic runtime errors due to ABI inconsistencies, symbol collisions, etc.

That's of course a valid concern and a well known weakness of Python packaging. In general, there is no mechanism to prevent users from shooting themselves in the foot. We can, however:

(1) keep the non-Python library dependencies of each of our distribution packages minimal (that's one of the reasons why the plan includes distribution packages that are keyed to a single library or set of libraries, like sagemath-flint); 

(2) test to ensure consistency with the published binary wheels of upstream packages (for example, gmpy2 for the gmp, mpfr, mpc libraries);

(3) work with upstream projects so that they have a consistent process for building wheel packages;

(4) avoid/remove sage-specific library glue when there are already dominant packages in the Python ecosystem that do this job (for example, https://github.com/sagemath/sage/pull/35368 replaces a custom Cython interface to the cbc library by the upstream cylp library).

[William Stein]

(5) provide a WebAssembly option

WebAssembly is typically about half the speed as native code (at best), but it is highly cross platform and self contained.   WebAssembly is difficult mainly when you have to deal with the OS somehow (e.g., filesystem, networking, etc.), and fortunately, a lot of the code in Sage is math libraries that support a non-threaded mode, so are particularly easy to port to WebAssembly.  A good example is Pari, which is one of "sage's non-Python dependencies".

The word "Web" in the name WebAssembly doesn't imply that it is only for use in a web browser. It's used widely outside of browsers.    One could compile something like pari to webassembly, then make use of a popular Python library like https://github.com/wasmerio/wasmer-python.  This wasmer-python itself is an extension module to Python (written in Rust), and the platforms it supports are listed here:

https://github.com/wasmerio/wasmer-python#supported-platforms

Incidentally, building Pari to WebAssembly is officially supported and regularly tested by Bill Allombert.  Here's his demo: 

https://pari.math.u-bordeaux.fr/gpexpwasm.html

 -- William

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/dab4c686-8791-4280-903c-5a982181a352n%40googlegroups.com.



--
William (http://wstein.org)

[Michael Orlitzky]

On Thu, 2023-06-22 at 13:56 -0700, William Stein wrote:
>
> (5) provide a WebAssembly option
>
> WebAssembly is typically about half the speed as native code (at best), but
> it is highly cross platform and self contained. WebAssembly is difficult
> mainly when you have to deal with the OS somehow (e.g., filesystem,
> networking, etc.), and fortunately, a lot of the code in Sage is math
> libraries that support a non-threaded mode, so are particularly easy to
> port to WebAssembly. A good example is Pari, which is one of "sage's
> non-Python dependencies".
>

We always wind up back here. Are we building mathematics software, or
signing on to run the world's most experimental linux distribution?

[Matthias Koeppe]

Well, concerns of software integration and deployment have always been part of the Sage project. And several leading projects in scientific python have already invested in WebAssembly deployment, at least for its obvious uses in providing interactive documentation. I, for one, welcome our new edge computing overlords. The modularization project makes such porting efforts much easier.

[William Stein]

 WebAssembly is not an experimental linux distribution, and it has very little overlap with linux distributions.  The WebAssembly ecosystem is built from the ground up, primarily on the LLVM (and Rust) toolchain, and an ecosystem of free software that is much more liberally licensed (and smaller) than what is typically in Linux distributions.    WebAssembly is neither better nor worse than Linux distributions; instead it is a different thing that solves different problems.   

To pre-empt another potential misconception, WebAssembly is more helpful for the needs of Sage than Java VM's because C/C++ can directly target WebAssembly.

William

[parisse]

I believe that wasm is the future, because you don't have to install anything and computations are done in the browser client, they do not require ressources from a server (except for the initial download of the wasm file). Giac/Xcas does that since many years now, (initially it was a request for Geogebra), for example here is the integral from another topic in a single URL (you can also embed it in an email from Xcas web interface, and it's certainly possible to do the same for other communication channels)

[url=https://www-fourier.univ-grenoble-alpes.fr/%7eparisse/xcasfr.html#exec&filename=%40session&python=1&radian=1&cas=0,0,f%3A%3Dintegrate(x%2F(x%5E5%2Ba%5E5))&cas=0,200,simplify(diff(f))&]session Xcas[/url]

BTW, I presented that to Bill Allombert a few years ago at a PARI meeting in Grenoble, and he decided to support wasm target for PARI.

[Michael Orlitzky]

On Thu, 2023-06-22 at 14:41 -0700, William Stein wrote:
>
> WebAssembly is not an experimental linux distribution, and it has very
> little overlap with linux distributions. The WebAssembly ecosystem is
> built from the ground up, primarily on the LLVM (and Rust) toolchain, and
> an ecosystem of free software that is much more liberally licensed (and
> smaller) than what is typically in Linux distributions. WebAssembly
> is neither better nor worse than Linux distributions; instead it is a
> different thing that solves different problems.

Yes, I know what WebAssembly is. The phrase "pip installability" is a
essentially euphemism for running a binary linux distribution on pypi
using pip as the package manager. On top of the many architecture and
libc-specific binaries that we'll have to build and ship and secure
forever, you're also suggesting that we maintain LLVM/Rust
infrastructure that allows us to ship pre-built WebAssembly targets for
all of those packages as well.

In addition to the usual sage distribution, and now the pypi
distribution, that would mean also maintaining our own LLVM/Rust-based
webassembly distribution. It's recreating exactly what Conda, Nix,
Guix, etc. all do, except with a more experimental toolchain. That's
the main problem: we don't have time to maintain our one existing
distribution, and the installation process is already too complicated
and confusing. With three, it's going to get more complicated, more
confusing, and more bitrotten as we all start to drown under the
increased maintenance burden.

The second issue is that WebAssembly doesn't actually solve the
problems we have, it only pushes them one layer of abstraction up. You
can run an entire OS in WebAssembly in a browser sandbox in docker in a
virtual machine. So what? Now you have two computers to maintain. To
avoid duplicating work, you start to move more and more things into
that browser. Eventually, you're right back where you started; things
conflict, etc. Except now everything doesn't work at half the speed
that it originally didn't work at. There's no way to sweep these
problems under the rug, they're fundamental. And instead of focusing on
them we keep getting distracted by squirrels.

[Dima Pasechnik]

It's strange that nobody said this already, but a lot of this is caused by Sage the distribution intertwined with Sage the library too much.

The right way forward is to decouple these two.

I tried to propose something like this years ago, but didn't get far.

In a nutshell, shift the non-Python things, as well as things like vendored Python, into Sage the distribution project, and use it (or the needed pieces of it) to build Sage the library.

This way Sage the library can still use more or less the same build system (but much simplified, e.g. the broken C++ compiler should just trigger an error, rather than the quest to build gcc/g++), and also Sage the distribution won't need too much build system modification. 

The latter will consume wheels built by Sage the library.

As well, Sage the distribution, ideally, could be replaced by e.g. Conda.

How precisely the partition should look like remains to be worked out, of course.

Dima

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/281e65d5ebeb2884b7638a340233db5855d43285.camel%40orlitzky.com.

[William Stein]

On Fri, Jun 23, 2023 at 1:02 PM Michael Orlitzky <mic...@orlitzky.com> wrote:

The second issue is that WebAssembly doesn't actually solve the
problems we have,

By "the problem we have" I guess maybe you mean "Sage is a lot of work to maintain".    The fundamental 

and massive problem that I think SageMath has is that it is not part of the  Python ecosystem,

by which I mean that there is no good way to do "pip install sagemath-[foo]", in sufficient generality.

PROBLEM: SageMath is not part of the Python ecosystem.

DEFINITION: A piece of software is part of the Python ecosystem, if you can do "pip install <software-name>" on

basically the same platforms as the intersection of where you can install scipy/numpy/matplotlib/pandas,

and with somewhat comparable resource usage (i.e., installing Sagemath can't use 100x of the time/space of

the above, as that would be unfair).

   

There are many tools to help solve this problem, and WebAssembly is definitely one of those tools, and at least

scipy/numpy/matplotlib/pandas are all well supported via WebAssembly these days.

If this problem were solved, I think SageMath would likely have much more usage, which would have 

benefits related to other problems such as "Sage is a lot of work to maintain".

William

 

--

William (http://wstein.org)

[Matthias Koeppe]

On Friday, June 23, 2023 at 1:24:42 PM UTC-7 Dima Pasechnik wrote:

a lot of this is caused by Sage the distribution intertwined with Sage the library too much.

It isn't. That was a problem years ago, but it has been fixed with only a few odd ends (such as sage.misc.package) remaining.

 

[Matthias Koeppe]

On Friday, June 23, 2023 at 1:02:34 PM UTC-7 Michael Orlitzky wrote:

[...] And instead of focusing on

them we keep getting distracted by squirrels.

Around here, hummingbirds are the bigger problem in this regard

[Dima Pasechnik]

it is a problem. One obvious thing I didn't mention is that releases of Sage the library and Sage the distribution could be independent of each other.

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/07d46292-49e8-4201-bdc2-cd74c6c164c6n%40googlegroups.com.

[Matthias Koeppe]

On Monday, June 12, 2023 at 4:54:21 PM UTC-7 Nathan Dunfield wrote:

On Monday, June 12, 2023 at 2:58:18 PM UTC-4 Matthias Koeppe wrote:

What parts of Sage does SnapPy use? 

Primarily the various rings/fields, including matrices over them and basic linear algebra.

In the present design (reflected in the currently "disputed" PRs https://github.com/sagemath/sage/pull/36676 and https://github.com/sagemath/sage/pull/36964, as well as the Mega-PR https://github.com/sagemath/sage/pull/35095):

Linear algebra over basic rings is provided by a distribution sagemath-modules (https://github.com/mkoeppe/sage/tree/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules)

 

  Specifically, interval arithmetic (Real/ComplexIntervalField), polynomial rings (in several variables, including Laurent polynomials), number fields, and the basic rings ZZ, QQ, GF, RealField, ComplexField, etc.

For GF beyond small prime fields, you would additionally need sagemath-pari. https://github.com/mkoeppe/sage/tree/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-pari

For the interval fields, you would need sagemath-flint. https://github.com/mkoeppe/sage/tree/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-flint

For number fields, you may additionally need sagemath-linbox currently because the class NumberFieldElement_quadratic is still encumbered with lots of gratuitous compile-time dependencies ( I have removed dependencies of this kind from other modules such as the Integers already). Rewriting this class using FLINT would be an easy and high-impact improvement of the modularity. https://github.com/mkoeppe/sage/tree/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-linbox

Instead of referring to these libraries explicitly, SnapPy would also be able to declare these dependencies in a high-level way using the "extras" (declared optional dependencies) of sagemath-modules, for example as "sagemath-modules[RIF,CIF,GF,NumberField]". See https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules/pyproject.toml.m4#L58

  Also, the ptolemy submodule uses Sage for groebner_basis calculations.

For any Groebner basis functionality, you would need sagemath-singular. It sounds like this would be declared as an optional dependency of SnapPy even if you decide to make the dependencies on the other Sage distribution packages required dependencies. https://github.com/mkoeppe/sage/tree/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-singular