935 views

Skip to first unread message

Apr 18, 2024, 5:24:35 PMApr 18

to sage-devel

Dear all:

**What is the modularization project?** The Sage developer community has long been aware of the severe problems that the monolithic design of Sage has brought. See in particular the lively 2016 sage-devel thread "How we develop Sage" (https://groups.google.com/g/ sage-devel/c/29ndCD8z94k) initiated by William. In its current incarnation, "modularization project" refers to my proposal from May 2020,

**Has the Sage community been informed and consulted regarding the modularization project? **Yes, in addition to the normal review that all tickets/PRs underwent:

**What does the PR https://github.com/sagemath/sage/pull/36676 do? **Per its title, "Restructure sage.*.all for modularization, replace relative by absolute imports". The PR is "mostly harmless": There are no user-visible changes; it's just a bunch of imports that are moved around. It includes no policy change of any kind; it only executes a design that was previously reviewed and carefully documented in separate PRs. Nothing permanent or irreversible is done here. The new files provide the top-level namespaces needed for doctesting modularized installations of Sage.

**Has it been reviewed?** Yes, David Coudert and John Palmieri did a detailed review. This was completed on November 15, 2023 --- over 5 months ago.

**How did this PR become "disputed"?** Back in November, one commenter floated an (untested) alternative design (https://github.com/sagemath/sage/pull/36676#pullrequestreview-1726079717); I explained in https://github.com/sagemath/sage/pull/36676#issuecomment-1806873154 why it's not suitable. Commenter demanded that the previously reviewed and documented design is reopened for discussion, https://github.com/sagemath/sage/pull/36676#issuecomment-1863667919.

**What are the concerns that have been made known during the voting process for this PR (March/April 2024)?** I will not attempt to paraphrase, but here are links to some posts so that you can find the discussion.

As an alternative to the proposal to back out the PR https://github.com/sagemath/sage/pull/36964 whose **disputed dependency PR https://github.com/sagemath/sage/pull/36676 which had not reached the required 2:1 supermajority **of the dispute-resolution process **(it currently only has a simple majority of 7 votes in favor, 5 votes against)** --- I am asking for your votes on that dependency PR https://github.com/sagemath/sage/pull/36676 to heal the process. This will avoid further delays and disruptions.

- to use modern Python packaging ("PEP 517/518/660") and Python 3's "implicit namespace packages" to

- break the Sage library into separately buildable and installable "distribution packages"

- while keeping the structure of the source tree mostly unchanged, monolithic, for the convenience of the Sage developer community.

For the project, hundreds of tickets/PRs have been prepared and merged over the past 4 years, see the Meta-ticket https://github.com/sagemath/sage/issues/29705 for a list.

- I have given detailed presentations about the project in SageDays 110 (2020), 112.358 (2022), 120 (2023), see https://github.com/sagemath/sage/issues/29705 for links.

- A chapter of the Sage Developer Guide, https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#packaging-the-sage-library, provides a detailed description of the design

- I have posted numerous times to sage-devel, most recently the series "SageMath modularization project: The five by five" (2023-06). See https://github.com/sagemath/sage/issues/29705 for links to all of these.

- Specifically, in the post "Modularization project: V. The blocs" (https://groups.google.com/g/sage-devel/c/kiB32zP3xD4/m/GJ0qF7TTAAAJ, 2023-06), I outlined the design of the pip-installable packages such as sagemath-combinat, sagemath-graphs, sagemath-flint, sagemath-plot etc.

- And in my 2023-11 post https://groups.google.com/g/sage-devel/c/kiB32zP3xD4/m/GJ0qF7TTAAAJ in the same thread, I asked:

> Ready for review: A restructuring of our "all.py" files along these dependencies in https://github.com/sagemath/sage/pull/36676. This is an opportunity to review the contents of the proposed distributions implemented in Mega-PR https://github.com/sagemath/sage/pull/35095 (~50 kLOC changes, not open for review). As https://github.com/sagemath/sage/pull/36676 rewrites all "all.py" files, it is also an opportunity for a deliberate coding style decision for these files. I welcome all constructive discussions in the PR.

- Gonzalo Tornaria https://github.com/sagemath/sage/pull/36676#issuecomment-2048350399, https://github.com/sagemath/sage/pull/36676#issuecomment-2048848093

- Michael Orlitzky https://github.com/sagemath/sage/pull/36676#issuecomment-2048600337

Message has been deleted

Apr 19, 2024, 6:47:38 AMApr 19

to sage-...@googlegroups.com

On 2024-04-18 14:18:37, Matthias Koeppe wrote:

> Dear all:

>

> As an alternative to the proposal to back out the

> PR https://github.com/sagemath/sage/pull/36964 whose *disputed dependency
> Dear all:

>

> As an alternative to the proposal to back out the

> PR https://github.com/sagemath/sage/pull/36676 which had not reached the

> required 2:1 supermajority *of the dispute-resolution process *(it
> currently only has a simple majority of 7 votes in favor, 5 votes against)*

> --- I am asking for your votes on that dependency PR

> https://github.com/sagemath/sage/pull/36676 to heal the process. This will

> avoid further delays and disruptions.

This doesn't circumvent the issue. Voting on PRs with disputed
> https://github.com/sagemath/sage/pull/36676 to heal the process. This will

> avoid further delays and disruptions.

dependencies is nonsensical and counterproductive. We were told they

wouldn't be merged. I have no idea what 36964 is about and if I'm for

or against it because it's a waste of time until its dependencies are

accepted/rejected. Only then (after another two weeks or whatever) can

the vote on 36964 be considered meaningful.

Apr 19, 2024, 8:08:02 AMApr 19

to sage-devel

Dear Matthias!

**> What is the modularization project?** The Sage developer community has long been aware of the severe problems that the monolithic design of Sage has brought. See in particular the lively 2016 sage-devel thread "How we develop Sage" (https://groups.google.com/g/sage-devel/c/29ndCD8z94k) initiated by William.

I looked at a few of the messages in this thread (there are something like hundred messages, I feel unable to read them all). I would like to make two remarks

* 26 persons participated in this thread, roughly 10 of these are still active

* many of the messages were quite critical regarding modularization of the math library in sage

* it seems to me that at least a big part of the thread is about modularization of the *infrastructure* of sage, not the math library.

In any case, I do not see consensus in this thread. Of course, there may be other threads that I am not aware of.

I was not there at the time, and only very recently realized some potential consequences of the direction the modularization might take as indicated in https://github.com/sagemath/sage/pull/36964, and formulated some questions there.

To my surprise, at least Volker issued similar concerns in the thread above. I could not find satisfying answers to his questions either. Also, I must say, that I find "Don't fear. This has been thought through carefully, and you can read about it in the Sage developer's guide." in place of an answer on the edge of abuse. It is at least condescending.

For convenience of discussion, I am reproducing my questions below.

Martin

1.) Is there an example for someone who did not want to use sage because of some dependency of the math library? Or at least a possible reason? @kwankyu's comment above suggests that having something in the "wrong" distribution wouldn't be a big deal. But this begs the question: who profits from cutting the math library into pieces (which look very arbitrary to me and have a curious emphasis on discrete math topics)?

My fear would be that at some point there is a request not to use symbolics in some module, because Lisp is hard to install on some system.

(I don't think this fear is unjustified: in the section of the developer guide you pointed to, I find

> The imports of the symbolic functions ceil() and floor() can likely be replaced by the artithmetic functions [integer_floor()](https://doc.sagemath.org/html/en/reference/rings_standard/sage/arith/misc.html#sage.arith.misc.integer_floor) and [integer_ceil()](https://doc.sagemath.org/html/en/reference/rings_standard/sage/arith/misc.html#sage.arith.misc.integer_ceil).

OK, so some user of that module happily replaces the two functions. Now, I come along and would like to replace some other implementation by a call to something defined in symbolics. But that would be breaking a promise to the user, so it would be really hard to justify.

In fact, this happened to me already, in some sense. I noticed a function definition in `sage.modular.multiple_zeta` with misleading documentation, which could be replaced by a call to code in `sage.combinat`. However, this is *already* hard to do, because it might affect performance (which is a very valid point in my opinion). I think it would be extremely bad to make it even harder.

2.) If this is about dependencies on other software, why aren't the distributions named after these dependencies? (Of course, at some point dependencies might change, for example, there might be a switch from glpk to scip.)

Before I read - by chance - `distribution = sagemath-graphs` somewhere, I thought one would "modularize" things like the repl, user interfaces, and perhaps some low level stuff. But it seems to me now that this is really about the modularization of the mathematics.

Also, I find it hard to believe that it is about dependencies, because the stuff in `abstract_tree.py` and friends has no dependencies on external software (unless you want to LaTeX them, perhaps).

My fear would be that at some point there is a request not to use symbolics in some module, because Lisp is hard to install on some system.

(I don't think this fear is unjustified: in the section of the developer guide you pointed to, I find

> The imports of the symbolic functions ceil() and floor() can likely be replaced by the artithmetic functions [integer_floor()](https://doc.sagemath.org/html/en/reference/rings_standard/sage/arith/misc.html#sage.arith.misc.integer_floor) and [integer_ceil()](https://doc.sagemath.org/html/en/reference/rings_standard/sage/arith/misc.html#sage.arith.misc.integer_ceil).

OK, so some user of that module happily replaces the two functions. Now, I come along and would like to replace some other implementation by a call to something defined in symbolics. But that would be breaking a promise to the user, so it would be really hard to justify.

In fact, this happened to me already, in some sense. I noticed a function definition in `sage.modular.multiple_zeta` with misleading documentation, which could be replaced by a call to code in `sage.combinat`. However, this is *already* hard to do, because it might affect performance (which is a very valid point in my opinion). I think it would be extremely bad to make it even harder.

2.) If this is about dependencies on other software, why aren't the distributions named after these dependencies? (Of course, at some point dependencies might change, for example, there might be a switch from glpk to scip.)

Before I read - by chance - `distribution = sagemath-graphs` somewhere, I thought one would "modularize" things like the repl, user interfaces, and perhaps some low level stuff. But it seems to me now that this is really about the modularization of the mathematics.

Also, I find it hard to believe that it is about dependencies, because the stuff in `abstract_tree.py` and friends has no dependencies on external software (unless you want to LaTeX them, perhaps).

Apr 19, 2024, 1:02:24 PMApr 19

to sage-devel

Michael, note that in my message I asked for a vote on that dependency https://github.com/sagemath/sage/pull/36676.

Apr 19, 2024, 1:03:13 PMApr 19

to sage-devel

On Friday, April 19, 2024 at 5:08:02 AM UTC-7 Martin R wrote:

> What is the modularization project?The Sage developer community has long been aware of the severe problems that the monolithic design of Sage has brought. See in particular the lively 2016 sage-devel thread "How we develop Sage" (https://groups.google.com/g/sage-devel/c/29ndCD8z94k) initiated by William.I looked at a few of the messages in this thread (there are something like hundred messages, I feel unable to read them all). I would like to make two remarks* 26 persons participated in this thread, roughly 10 of these are still active* many of the messages were quite critical regarding modularization of the math library in sage* it seems to me that at least a big part of the thread is about modularization of the *infrastructure* of sage, not the math library.In any case, I do not see consensus in this thread.

Of course there was not a "consensus"'; at that time there was not even a plan.

I was citing the thread, as I indicated, to support my sentence, "The Sage developer community has long been aware of the severe problems that the monolithic design of Sage has brought."

Apr 19, 2024, 1:18:03 PMApr 19

to sage-...@googlegroups.com

On Fri, 2024-04-19 at 09:46 -0700, Matthias Koeppe wrote:

>

> Michael, note that in my message I asked for a vote on that dependency

> https://github.com/sagemath/sage/pull/36676.

>

Even if 36676 gets approval, 36964 must be reverted. It was not
>

> Michael, note that in my message I asked for a vote on that dependency

> https://github.com/sagemath/sage/pull/36676.

>

meaningfully voted upon.

Apr 19, 2024, 2:08:51 PMApr 19

to sage-devel

On Friday, April 19, 2024 at 5:08:02 AM UTC-7 Martin R wrote:

2.) If this is about dependencies on other software, why aren't the distributions named after these dependencies?

Martin, I have answered this already when you asked it in the PR: Some are.

Note that the description of the PR where you asked this question contains the list of the distributions: https://github.com/sagemath/sage/pull/36676

"We restructure the all.py files for modularization. Guided by the technical constraints outlined in https://groups.google.com/g/sage-devel/c/ozh-7ZZ848s, https://github.com/sagemath/sage/pull/35095 defines distribution packages (pip-installable packages) sagemath-{brial, combinat, eclib, flint, gap, giac, glpk, graphs, groups, homfly, lcalc, libbraiding, libecm, linbox, modules, mpmath, ntl, pari, plot, polyhedra, schemes, singular, standard-no-symbolics, symbolics}."

My June 2023 sage-devel post https://groups.google.com/g/sage-devel/c/kiB32zP3xD4 explained that the design use "... 3 types of distribution packages:

- Packages that are named after a basic mathematical structure but may cover a wide range of generalizations/applications of this structure.- Packages that are named after an upstream library.

- Packages that are named after a technical functionality."

- Packages that are named after a technical functionality."

Apr 19, 2024, 3:34:06 PMApr 19

to sage-devel

On Friday 19 April 2024 at 20:08:51 UTC+2 Matthias Koeppe wrote:

On Friday, April 19, 2024 at 5:08:02 AM UTC-7 Martin R wrote:2.) If this is about dependencies on other software, why aren't the distributions named after these dependencies?Martin, I have answered this already when you asked it in the PR: Some are.

Matthias, this does not answer my question. Maybe it is clearer if I ask: why do you introduce distributions sage-graphs, sage-combinat, sage-categories etc.

Martin

Apr 20, 2024, 3:56:30 AMApr 20

to sage-devel

A follow-up question: do I understand correctly that common lisp (via maxima) is the main dependency that prevents sagemath from being pip-installable?

All the best,

Martin

Apr 20, 2024, 5:22:25 AMApr 20

to sage-devel

Yes in a perfect world, but then you don't get a gold star for satisfying some purity test. We should just do the minimal amount of work to get us where we want to be. Lets focus on the direction to go and not too much on the process.

Apr 20, 2024, 11:27:20 AMApr 20

to sage-...@googlegroups.com

On Sat, 2024-04-20 at 02:22 -0700, Volker Braun wrote:

> Yes in a perfect world, but then you don't get a gold star for satisfying

> some purity test. We should just do the minimal amount of work to get us

> where we want to be. Lets focus on the direction to go and not too much on

> the process.

We don't know where we want to be. To figure that out, we should hold
> Yes in a perfect world, but then you don't get a gold star for satisfying

> some purity test. We should just do the minimal amount of work to get us

> where we want to be. Lets focus on the direction to go and not too much on

> the process.

some kind of vote or something.

Apr 20, 2024, 11:55:52 AMApr 20

to sage-...@googlegroups.com

On 20 April 2024 08:56:30 BST, 'Martin R' via sage-devel <sage-...@googlegroups.com> wrote:

>A follow-up question: do I understand correctly that common lisp (via

>maxima) is the main dependency that prevents sagemath from being

>pip-installable?

already works in a venv on a box with enough deps

installed. Takes lot of time. To fix this,

anyhow, someone has to make their hands dirty and do to Lisp interface what's done to Pari/GP (cypari)

PPL (pplpy), primecount (primecountpy), etc.

Apart from Lisp, there is GAP (with the corresponding effort stalled).

That's what is much more urgent than attempting to slice up the maths functionality of sagelib.

Dima

Apr 20, 2024, 11:56:14 AMApr 20

to sage-...@googlegroups.com, vbrau...@gmail.com

Hi Volker,

On Sat, Apr 20, 2024 at 10:22 AM Volker Braun <vbrau...@gmail.com> wrote:

Yes in a perfect world, but then you don't get a gold star for satisfying some purity test. We should just do the minimal amount of work to get us where we want to be. Lets focus on the direction to go and not too much on the process.

the keywords are "where we want to be" and "direction".

The revert should happen as there is no agreement on this.

It would help if we first sort out and agree on these, then we can resume the normal operations.

IMHO it would help if you stopped touching these disputed PRs until this these are done, or at least for a set period of time,

say a month or two.

Dima

On Friday, April 19, 2024 at 7:18:03 PM UTC+2 Michael Orlitzky wrote:On Fri, 2024-04-19 at 09:46 -0700, Matthias Koeppe wrote:

>

> Michael, note that in my message I asked for a vote on that dependency

> https://github.com/sagemath/sage/pull/36676.

>

Even if 36676 gets approval, 36964 must be reverted. It was not

meaningfully voted upon.

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/b9d41da8-57ec-4542-a5d8-7f2690849a49n%40googlegroups.com.

Apr 20, 2024, 11:57:28 AMApr 20

to sage-devel

On Friday, April 19, 2024 at 12:34:06 PM UTC-7 Martin R wrote:

why do you introduce distributions sage-graphs, sage-combinat, sage-categories etc.

Let's follow the link included in my previous message to my June 2023 sage-devel post https://groups.google.com/g/sage-devel/c/kiB32zP3xD4 and see what's written there.

"**Packages that are named after a basic mathematical structure** but may cover a wide range of generalizations/applications of this structure. People who work in a specialized research area will of course recognize what structures they need. But the down-to-earth naming also creates discoverability for a broader audience. Not many more distribution packages than these 7 are needed:

- sagemath-combinat: everything combinatorial except for graphs.

- sagemath-graphs: also provides posets, designs, abstract complexes, quivers, etc.

- sagemath-groups

- sagemath-modules: also has matrices, tensors, homology, coding theory, etc.

- sagemath-polyhedra: also provides fans, hyperplane arrangements, etc.

- sagemath-schemes

- sagemath-symbolics"

- sagemath-combinat: everything combinatorial except for graphs.

- sagemath-graphs: also provides posets, designs, abstract complexes, quivers, etc.

- sagemath-groups

- sagemath-modules: also has matrices, tensors, homology, coding theory, etc.

- sagemath-polyhedra: also provides fans, hyperplane arrangements, etc.

- sagemath-schemes

- sagemath-symbolics"

In short, I introduce these packages to create **discoverability** for potential consumers of portions of the Sage library. (pip-installable packages are often named after the functionality that they provide.)

Apr 20, 2024, 12:16:01 PMApr 20

to sage-...@googlegroups.com

On Sat, 2024-04-20 at 10:07 +0100, Dima Pasechnik wrote:

>

> Apart from Lisp, there is GAP (with the corresponding effort stalled).

>

> That's what is much more urgent than attempting to slice up the maths functionality of sagelib.

>

Also the ancient copy of ginac/pynac we bundle.
>

> Apart from Lisp, there is GAP (with the corresponding effort stalled).

>

> That's what is much more urgent than attempting to slice up the maths functionality of sagelib.

>

Apr 20, 2024, 3:15:36 PMApr 20

to sage-devel

On Saturday, April 20, 2024 at 12:56:30 AM UTC-7 Martin R wrote:

do I understand correctly that common lisp (via maxima) is the main dependency that prevents sagemath from being pip-installable?

No.

For one, SageMath is already pip-installable.

That was one of the first deliverables of the modularization project, completed in 2021.

It's documented in our README: https://github.com/sagemath/sage/blob/develop/README.md#alternative-installation-using-pypi

However, installing Sage in this way is equivalent to installing the full Sage distribution from source in the normal way. (This is what https://pypi.org/project/sage-conf/ does.)

It takes very long. This alone makes it simply not suitable as a dependency of other pip-installable projects.

In another message, you asked:

> 1.) Is there an example for someone who did not want to use sage because of some dependency of the math library? Or at least a possible reason? [...] But this begs the question: who profits from cutting the math library into pieces (which look very arbitrary to me and have a curious emphasis on discrete math topics)?

If you do allow me another example from discrete math: Sage has a lot of very good code in "sage.graphs" that provides functionality that is not available from other Python libraries.

But to potential projects that would need this functionality, the proposition to have to depend on the monolithic project SageMath, with hours of compilation time and/or dependencies on system packages that are obviously unrelated to the needed functionality (boost, brial, ecl, eclib, ecm, fflas_ffpack, flint, libgd, gap, giac, givaro, glpk, gmp, gsl, iml, lcalc, libbraiding, libhomfly, libpng, linbox, m4ri, m4rie, mpc, mpfi, mpfr, ntl, pari, rw, singular, symmetrica), is simply a showstopper.

Apr 20, 2024, 4:01:33 PMApr 20

to sage-...@googlegroups.com

On 20 April 2024 19:34:49 BST, Matthias Koeppe <matthia...@gmail.com> wrote:

>On Saturday, April 20, 2024 at 12:56:30 AM UTC-7 Martin R wrote:

>

>do I understand correctly that common lisp (via maxima) is the main

>dependency that prevents sagemath from being pip-installable?

>

>

>No.

>

>For one, SageMath is already pip-installable.

>That was one of the first deliverables of the modularization project,

>completed in 2021.

>See https://wiki.sagemath.org/ReleaseTours/sage-9.4#Alternative_installation_methods_using_pip

>

>It's documented in our README:

>https://github.com/sagemath/sage/blob/develop/README.md#alternative-installation-using-pypi

>

>However, installing Sage in this way is equivalent to installing the full

>Sage distribution from source in the normal way.

<https://pypi.org/project/sagemath-standard/#description>

Building sagemath-standard has a large number of system packages as prerequisites. See https://doc.sagemath.org/html/en/installation/source.html#linux-recommended-installation for partial lists for various systems.

That is, many packages are not built. What's being built is not a Sage distribution, it is sagelib.

> (This is

>what https://pypi.org/project/sage-conf/ does.)

>It takes very long. This alone makes it simply not suitable as a dependency

>of other pip-installable projects.

>

>

>In another message, you asked:

>> 1.) Is there an example for someone who did not want to use sage because

>of some dependency of the math library? Or at least a possible reason?

>[...] But this begs the question: who profits from cutting the math library

>into pieces (which look very arbitrary to me and have a curious emphasis on

>discrete math topics)?

>

>If you do allow me another example from discrete math: Sage has a lot of

>very good code in "sage.graphs" that provides functionality that is not

>available from other Python libraries.

>

>But to potential projects that would need this functionality, the

>proposition to have to depend on the monolithic project SageMath, with

>hours of compilation time and/or dependencies on system packages that are

>obviously unrelated to the needed functionality (boost, brial, ecl, eclib,

>ecm, fflas_ffpack, flint, libgd, gap, giac, givaro, glpk, gmp, gsl, iml,

>lcalc, libbraiding, libhomfly, libpng, linbox, m4ri, m4rie, mpc, mpfi,

>mpfr, ntl, pari, rw, singular, symmetrica), is simply a showstopper.

>

(To "but macOS?" I have a reply: "ask Apple to provide you a package manager, it's the best OS, right, you pay for it a lot of money, or learn to use Homebrew like Mac users usually do...")

Anyway, a normal Python pypi-installable package comes with binary wheels, i e. things are pre-built, and it's merely matter of downloading these to get a functional package. Few minutes on a fast network, not hours.

By choosing to be an exception in the Python world,

Sage obviously does something quite wrong.

Dima

Apr 20, 2024, 6:01:04 PMApr 20

to sage-devel

By choosing to be an exception in the Python world,

Sage obviously does something quite wrong.

Can someone who is not Dima or Matthias explain to us how it is possible that they both are claiming to represent the normal Python way of doing things? There have been numerous statements by both of them about this, which makes it sound like there are two pieces to it (modularization but also "de-vendoring"), and I can only assume that it would be possible for both to occur simultaneously. It would be helpful for this to be clarified, though, so that one knows precisely what *piece* of their proposals represent the "normal Python ecosystem".

That said, "normal Python" is not necessarily as relevant for those who would *only* want Sage, or at least mostly so. Having just another Python package might lead us to implementing powers as ** instead of ^, which would be a regression, or needing namespaces for almost everything, which again limits the value of Sage qua Sage.

Apr 20, 2024, 6:23:12 PMApr 20

to sage-...@googlegroups.com

On Sat, 2024-04-20 at 15:01 -0700, kcrisman wrote:

>

> Can someone who is not Dima or Matthias explain to us how it is possible

> that they both are claiming to represent the normal Python way of doing

> things? There have been numerous statements by both of them about this,

> which makes it sound like there are two pieces to it (modularization but

> also "de-vendoring"), and I can only assume that it would be possible for

> both to occur simultaneously.

It is, but in its current incarnation, the modularization relies
>

> Can someone who is not Dima or Matthias explain to us how it is possible

> that they both are claiming to represent the normal Python way of doing

> things? There have been numerous statements by both of them about this,

> which makes it sound like there are two pieces to it (modularization but

> also "de-vendoring"), and I can only assume that it would be possible for

> both to occur simultaneously.

heavily on the sage distribution vendoring. Conflict arises because the

modularization is cited as a blocker whenever someone wants to pare

down or disentangle some aspect of the sage distribution. It's not the

modularization per se that we object to. Personally, I would like it to

succeed, but not at the expense of everything else.

Apr 21, 2024, 4:25:11 AMApr 21

to sage-devel

On Saturday, April 20, 2024 at 3:01:04 PM UTC-7 kcrisman wrote:

"normal Python" is not necessarily as relevant for those who would *only* want Sage, or at least mostly so. Having just another Python package might lead us to implementing powers as ** instead of ^, which would be a regression, or needing namespaces for almost everything, which again limits the value of Sage qua Sage.

I understand this concern, and I can reassure you that the modularization project keeps all functionality of the Sage application intact; that includes the surface language using the Sage preparser.

Apr 21, 2024, 4:25:29 AMApr 21

to sage-devel

On Saturday, April 20, 2024 at 3:23:12 PM UTC-7 Michael Orlitzky wrote:

in its current incarnation, the modularization relies

heavily on the sage distribution vendoring. Conflict arises because the

modularization is cited as a blocker whenever someone wants to pare

down or disentangle some aspect of the sage distribution. It's not the

modularization per se that we object to. Personally, I would like it to

succeed, but not at the expense of everything else

Michael, I think you may be using too much jargon to get your point across to the general readership of this list.

Let's maybe use this opportunity to make this as concrete as possible and explain it in the most plain terms.

What entanglement are you concerned about?

Apr 21, 2024, 5:30:15 AMApr 21

to sage-devel

Dear Matthias,

This doesn't make sense to me. Why would you separate mathematics into packages that have no more external dependencies from others, which at the same time may grow internal dependencies over time?

I can imagine that it would make sense to make as much as possible into runtime dependencies - you wrote below that building the dependencies takes a lot of time. Maybe that's the core problem, I don't know.

However, to me the current proposal really looks like modularization just for its own sake. I wouldn't (and for that reason I didn't) mind if this comes without much cost, but this is not the case. I am quite sure that the mere existence of the packages will influence decisions whether to duplicate code or not.

Martin

Besides, I would prefer if we would not make projects natural persons, and I would prefer if we could stick to speaking about ourselves, rather than "the sage developers", whoever that might be.

Apr 21, 2024, 10:42:24 AMApr 21

to sage-devel

On Saturday, April 20, 2024 at 5:01:04 PM UTC-5 kcrisman wrote:

Can someone who is not Dima or Matthias explain to us how it is possible that they both are claiming to represent the normal Python way of doing things? There have been numerous statements by both of them about this, which makes it sound like there are two pieces to it (modularization but also "de-vendoring"), and I can only assume that it would be possible for both to occur simultaneously. It would be helpful for this to be clarified, though, so that one knows precisely what *piece* of their proposals represent the "normal Python ecosystem.

For the statements in this thread, I don't see any contradictions about the definition of the "normal Python way of doing things". My understanding of that term is to post **self-contained** binary wheels to PyPI for all supported platforms that install in a minute or two with no compilation (as Dima wrote), as well as a source-code only package that serves as a rarely-used backup if no suitable binary is available. (The self-contained bit is important; for example, on Linux here are the only external libraries a binary wheel is allowed to link to.)

So far, we only post a source-code only package "sagemath-standard" on PyPI, and so "pip install sagemath-standard" is basically equivalent to downloading the tarball and running "configure/make". As Matthias says "It takes very long. This alone makes it simply not suitable as a dependency of other pip-installable projects."

In particular, no one is arguing that Sage currently follows the "normal Python way", though we have made one step in that direction by posting the source-only package.

Best,

Nathan

Apr 21, 2024, 12:40:31 PMApr 21

to sage-devel

On Sunday, April 21, 2024 at 9:42:24 AM UTC-5 Nathan Dunfield wrote:

For the statements in this thread, I don't see any contradictions about the definition of the "normal Python way of doing things". My understanding of that term is to postself-containedbinary wheels to PyPI for all supported platforms that install in a minute or two with no compilation (as Dima wrote), as well as a source-code only package that serves as a rarely-used backup if no suitable binary is available. (The self-contained bit is important; for example, on Linux here are the only external libraries a binary wheel is allowed to link to.)

I hope it will help to acknowledge what might be regarded as the elephant in this room. That elephant consists of two facts:

1) *Users*, as opposed to *developers*, are very unlikely to be either willing to or capable of building a complex source-code only package which depends on external libraries, because doing that requires finding and installing (or building from source), all of the external libraries needed by the python package.

2) The use of a **self-contained **binary wheel on linux violates a core principle of linux packaging, namely that any library which is a runtime dependency of a package should be installed from the appropriate distro package.

Note that 2) also means that a pypi package developer must support the full range of different versions of these libraries which are provided by different linux distributions. Sage initially used the same principle in its design, but avoided the maintenance burden caused by trying to support all of these different library versions by providing its own libraries in SAGE_LOCAL. It has since gradually accepted more and more of the burden by allowing "system versions" of libraries wherever it seemed feasible.

In the "Python ecosystem" these issues have been worked around in a different way in order to make **self-contained** binary wheels a possibility, and thereby accommodate *users*. This is usually done with delocate on macOS or with auditwheel on linux. Those tools embed all of the required libraries inside the python package and adjust the rpaths in the python extension modules to make them use the embedded libraries instead of libraries which might or might not be installed on the host system. In the case of linux this also means compiling the libraries against a version of glibc which is older than the version found on any of the linux distributions supported by the package. The job of building libraries against old versions of glibc is done by building them on a manylinux docker image, which can be done on a CI runner. (On macOS one can specify a minimum target version of macOS, usually 10.12 these days.)

As a typical example (which recently created issues for Sage related to the jpeg library) you can look at the binary wheel for Pillow. If you unzip that wheel you will find a subdirectory named pillow.libs which contains: libbrotlicommon-3ecfe81c.so.1, libpng16-58efbb84.so.16.43.0, libbrotlidec-ba690955.so.1, libsharpyuv-20f78091.so.0.0.1, libfreetype-1f2cdfbb.so.6.20.1, libtiff-4da6744b.so.6.0.2, libharfbuzz-8b0b6641.so.0.60840.0,libwebp-850e2bec.so.7.1.8, libjpeg-f391b078.so.62.4.0, libwebpdemux-df9b36c7.so.2.0.14, liblcms2-e69eef39.so.2.0.16,libwebpmux-9fe05867.so.3.0.13, liblzma-13fa198c.so.5.4.5,libXau-154567c4.so.6.0.0, libopenjp2-05423b53.so , and libxcb-80c5a837.so.1.1.0. A linux packager would provide all of these by adding as dependencies all of the distro packages needed to provide these libraries. In the "Python ecosystem" these libraries are all built on manylinux docker images and then embedded in the binary wheel.

I don't see how Sage will be able to join the Python ecosystem without addressing this.

- Marc

Apr 21, 2024, 5:41:57 PMApr 21

to sage-devel

On Sunday, April 21, 2024 at 2:30:15 AM UTC-7 Martin R wrote:

I can imagine that it would make sense to make as much as possible into runtime dependencies - you wrote below that building the dependencies takes a lot of time. Maybe that's the core problem, I don't know.

If you want to know, then I recommend to read the sections of the Developer's Guide where

- the different types of dependencies are explained, as well as

- the techniques to reduce and manage such dependencies.

Apr 21, 2024, 5:42:08 PMApr 21

to sage-devel

On Saturday, April 20, 2024 at 1:01:33 PM UTC-7 Dima Pasechnik wrote:

Anyway, a normal Python pypi-installable package comes with binary wheels, i e. things are pre-built, and it's merely matter of downloading these to get a functional package. Few minutes on a fast network, not hours.

By choosing to be an exception in the Python world,

Sage obviously does something quite wrong.

I'll note that providing binary wheels is actually one of the deliverables of the modularization project.

- See for example https://pypi.org/project/sagemath-categories/#files

The meta-ticket "Distribution as wheels" (https://github.com/sagemath/sage/issues/31251) describes the next steps, including a PR that is waiting for review.

Apr 21, 2024, 5:42:37 PMApr 21

to sage-devel

On Sunday, April 21, 2024 at 2:30:15 AM UTC-7 Martin R wrote:

Why would you separate mathematics into packages that have no more external dependencies from others, which at the same time may grow internal dependencies over time?

Let's just go through the list of distribution packages and their dependencies for concreteness. (All depend on **sagemath-categories** and thus on the basic arithmetic libraries gmp, mpc, mpfr, gmpy2)

- non-Python dependency: *symmetrica* (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-combinat/pyproject.toml.m4#L65)

- non-Python dependency: *boost *(https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-graphs/pyproject.toml.m4#L73)

- non-Python dependency: *gsl* (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules/pyproject.toml.m4#L109)

- Python build requirement: **numpy**

- non-Python dependency (via **sagemath-gap**): *gap* (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-gap/pyproject.toml.m4#L52**)**

- non-Python dependency (via **sagemath-modules**): *gsl* (see above)

- non-Python dependency (via **sagemath-glpk**): *glpk*

- non-Python dependency (via **sagemath-modules**): *gsl* (see above)

- Python runtime dependency: **pplpy**

- non-Python dependency (via **sagemath-modules**): *gsl* (see above)

- non-Python dependencies (via **sagemath-singular**, **sagemath-flint**, **sagemath-ntl**, **sagemath-pari**): singular, flint, ntl, pari (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-pari/pyproject.toml.m4#L61)

- Python dependency (via **sagemath-singular**, **sagemath-flint**, **sagemath-pari**): *cypari2*

- Python dependency: **scipy**

- non-Python dependencies: *ecl*, *maxima*, *singular* https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-symbolics/pyproject.toml.m4#L89

- non-Python dependencies (via **sagemath-flint**, **sagemath-ntl**, **sagemath-modules**): *flint*, *ntl*, *pari*, *gsl*

- Python dependencies: **mpmath**, **sympy**, **cypari2, numpy**

Apr 21, 2024, 5:44:10 PMApr 21

to sage-devel

On Saturday, April 20, 2024 at 1:01:33 PM UTC-7 Dima Pasechnik wrote:

On 20 April 2024 19:34:49 BST, Matthias Koeppe <matthia...@gmail.com> wrote:

SageMath is already pip-installable.

>That was one of the first deliverables of the modularization project,

>completed in 2021.

>See https://wiki.sagemath.org/ReleaseTours/sage-9.4#Alternative_installation_methods_using_pip

>

>It's documented in our README:

>https://github.com/sagemath/sage/blob/develop/README.md#alternative-installation-using-pypi

>

>However, installing Sage in this way is equivalent to installing the full

>Sage distribution from source in the normal way.

No, it is not equivalent, far from it. One can read on

<https://pypi.org/project/sagemath-standard/#description>

Building sagemath-standard has a large number of system packages as prerequisites. See https://doc.sagemath.org/html/en/installation/source.html#linux-recommended-installation for partial lists for various systems.

That is, many packages are not built. What's being built is not a Sage distribution, it is sagelib.

Dima, as you well know, I designed and implemented all of these facilities related to the PyPI distribution method of SageMath as part of the modularization project. What you quote from https://pypi.org/project/sagemath-standard/#description -- I wrote it. It's hard to understand what the purpose of your message "correcting" me on these things could possibly be.

The distribution **sagemath-standard** does provide the Sage library.

But it **declares a required build-time and runtime dependency on https://pypi.org/project/sage-conf/**, which when installed, builds the Sage distribution in a subdirectory of ~/.sage.

(Note also that one of the disputed PRs, https://github.com/sagemath/sage/pull/37138, seeks to remove this declared dependency that implements this mechanism.)

>If you do allow me another example from discrete math: Sage has a lot of

>very good code in "sage.graphs" that provides functionality that is not

>available from other Python libraries.

>

>But to potential projects that would need this functionality, the

>proposition to have to depend on the monolithic project SageMath, with

>hours of compilation time and/or dependencies on system packages that are

>obviously unrelated to the needed functionality (boost, brial, ecl, eclib,

>ecm, fflas_ffpack, flint, libgd, gap, giac, givaro, glpk, gmp, gsl, iml,

>lcalc, libbraiding, libhomfly, libpng, linbox, m4ri, m4rie, mpc, mpfi,

>mpfr, ntl, pari, rw, singular, symmetrica), is simply a showstopper.

This is not true, either. First, a lot of these packages that you list are available on systems, and thus there is no need to build them.

(To "but macOS?" I have a reply: "ask Apple to provide you a package manager, it's the best OS, right, you pay for it a lot of money, or learn to use Homebrew like Mac users usually do...")

Note that I wrote "hours of compilation time and/or dependencies on system packages".

So there was nothing to correct here. My point stands.

What's the purpose of your comment?

Anyway, a normal Python pypi-installable package comes with binary wheels, i e. things are pre-built, and it's merely matter of downloading these to get a functional package. Few minutes on a fast network, not hours.

By choosing to be an exception in the Python world,

Sage obviously does something quite wrong.

Also the purpose of this part of your message is not clear to me, Dima.

As you well know, **preparing binary wheels is another deliverable of the modularization project. **What Sage may have been doing wrong --- the modularization project has been fixing it.

- For the modularized distributions that are already available, these binary wheels have already been up on PyPI for a long time, see https://pypi.org/project/sagemath-objects/#files and https://pypi.org/project/sagemath-categories/#files

- https://github.com/sagemath/sage/pull/37503 (to be merged) adds wheels for the macOS arm64 architecture to what it is built and deployed to PyPI on every release

- https://github.com/sagemath/sage/pull/37099 (needs review) and https://github.com/sagemath/sage/pull/36525, https://github.com/sagemath/sage/pull/36730 (in prep) add wheels for more distributions

An overview of this effort of making wheels available: Meta-PR https://github.com/sagemath/sage/issues/31251

I repeat my invitation for others to join me in this important effort.

Apr 21, 2024, 5:51:33 PMApr 21

to sage-...@googlegroups.com

On 2024-04-20 15:33:51, Matthias Koeppe wrote:

> Michael, I think you may be using too much jargon to get your point across

> to the general readership of this list.

>

> Let's maybe use this opportunity to make this as concrete as possible and

> explain it in the most plain terms.

> What entanglement are you concerned about?

For example, we have several tickets that are disputed because they
> Michael, I think you may be using too much jargon to get your point across

> to the general readership of this list.

>

> Let's maybe use this opportunity to make this as concrete as possible and

> explain it in the most plain terms.

> What entanglement are you concerned about?

will use ./bootstrap and data from the sage distribution (build/pkgs)

to generate the pyproject.toml files for the modular components. This

ensures that the components cannot truly be separated, not only from

the other components, but from the mini-distro in build/pkgs that a

large chunk of us hate.

Contrast with some other parts of sage that have been modularized:

* pplpy

* memory-allocator

* cypari

* cysignals

* primecountpy

These have been successfully disentangled from both the sage library

and the sage distribution and I don't think anyone has complaints

about them. They've been moved to separate repositories which, while

not strictly necessary, is certainly proof that they are in fact

disentangled. We can typically depend on stable versions of them and

install/test them independently. They do not depend on nonstandard or

bleeding-edge features of the python ecosystem.

Apr 22, 2024, 3:12:05 AMApr 22

to sage-devel

On Sunday, April 21, 2024 at 7:01:04 AM UTC+9 kcrisman wrote:

By choosing to be an exception in the Python world,

Sage obviously does something quite wrong.

Can someone who is not Dima or Matthias explain to us how it is possible that they both are claiming to represent the normal Python way of doing things? There have been numerous statements by both of them about this, which makes it sound like there are two pieces to it (modularization but also "de-vendoring"), ...

Sage consists of two pieces: the sage library and the **sage package**s (aka spkgs defined in sage_root/build/pkgs). The **sage distribution** is roughly the sum of the sage packages. The sage library is a big **python package** (sage.rings etc.). The modularization project, led by Matthias, is splitting the sage library and adding (pip-installable) **distribution package**s (defined in sage_root/pkgs). Michael is using the term "modularization" to mean several things: (1) using a system package (a **distro package **installed in a **linux distribution**, ubuntu for example) instead of a sage package (2) using a system python package instead of a sage package (3) splitting some portion of the sage library to a separate python package, which is installed as a sage package, examples are cypari, cysignals, etc.

Note that **"package**"s and "**distribution**"s appearing in the above paragraph in **bold face** all refer to different things.

Many sage packages (known as "pip package" and "wheel packages" ) just install python packages from pypi. Dima wants to reduce the number of such sage packages (prominently, jupyter and friends), and seems to call this "the normal Python way of doing things". You call this "de-vendoring".

Matthias' "normal Python way of doing things" is to follow the Python packaging standard in designing the distribution packages.

Philosophically, there is no reason that Michael's "modularization", Dima's "de-vendoring" , Tobias' "conda support", and Gonzalo's "sagemath distro package" conflict with Matthias' modularization. But they do in technical details in disputed PRs.

Apr 22, 2024, 5:22:36 AMApr 22

to sage-devel

Thank you for this list.

I still don't see why you would name these distributions as you do, and why you collect them as you do. For example, as far as I know, symmetrica is currently essentially only used by symmetric functions, Schubert polynomials. So, if symmetrica is such a burden to install, why would you want to install it if you don't need them?

On the other hand, combinatorial species will, after this summer, heavily depend on gap. Does this mean that you would want to move this part into sage-groups? Apart from the strange naming, I don't think that it will make anybody happy if dependencies change frequently.

Best wishes,

Martin

Apr 23, 2024, 5:07:37 AMApr 23

to sage-devel

In reply to the comment (https://github.com/sagemath/sage/pull/36676#issuecomment-2067371677)

>> My fear would be that at some point there is a request not
to use symbolics in some module, because Lisp is hard to install on
some system.

>That should not happen. Modularization is downstream to the sage library. Yes, we are restructuring some parts of the sage library to fit with modularization. But modularization should never be an obstacle in developing the sage library. If it ever be, the sage community might drop the modularization project.

We already witness multiple instances where the modularization project is cited as a reason not to merge certain PRs that only touch sage-the-library. How does this reality fit this view?

Moreover, once the modularization is in place, it will impose wide-ranging constraints on what functions from other modules you are able to use. Currently, you can use `sage.x` in `sage.y` for an arbitrary combination of `x` and `y`. The modularization project restricts this severely and you will only be able to use modules `x` that are declared as a dependency of `y` (the arrows in the picture in https://github.com/sagemath/sage/pull/35095).

Apr 23, 2024, 8:35:10 AMApr 23

to sage-devel

>> My fear would be that at some point there is a request not to use symbolics in some module, because Lisp is hard to install on some system.>That should not happen. Modularization is downstream to the sage library. Yes, we are restructuring some parts of the sage library to fit with modularization. But modularization should never be an obstacle in developing the sage library. If it ever be, the sage community might drop the modularization project.

We already witness multiple instances where the modularization project is cited as a reason not to merge certain PRs that only touch sage-the-library. How does this reality fit this view?

I meant the sage library as a collection of mathematical modules. If a certain module did not but somehow would develop to rely on the mathematical functionality of another module, then the design of the modularization should embrace the development. The splitting of the sage library in the present modularization reflects the present reality of the separation of different mathematical parts of the sage library. But of course the reality may change in the future as we develop the sage library. Then the modularization should reflect the change, not block the change. However this is all about the future, unrelated to the present disputed PRs, including yours.

Apr 23, 2024, 9:14:05 AMApr 23

to sage-...@googlegroups.com

On Sun, Apr 21, 2024 at 10:42 PM Matthias Koeppe <matthia...@gmail.com> wrote:

On Sunday, April 21, 2024 at 2:30:15 AM UTC-7 Martin R wrote:Why would you separate mathematics into packages that have no more external dependencies from others, which at the same time may grow internal dependencies over time?Let's just go through the list of distribution packages and their dependencies for concreteness. (All depend onsagemath-categoriesand thus on the basic arithmetic libraries gmp, mpc, mpfr, gmpy2)

these are seemingly incomplete:

sagemath-combinat- non-Python dependency:symmetrica(https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-combinat/pyproject.toml.m4#L65)

non-python: depends on GAP, givaro, too

sagemath-graphs

non-python: depends on GAP, givaro, too

sagemath-modules- non-Python dependency:gsl(https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules/pyproject.toml.m4#L109)- Python build requirement:numpy

non-python: linbox, flas_ffpac, too ?

sagemath-groups- non-Python dependency (viasagemath-gap):gap(https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-gap/pyproject.toml.m4#L52)- non-Python dependency (viasagemath-modules):gsl(see above)sagemath-polyhedra- non-Python dependency (viasagemath-glpk):glpk- non-Python dependency (viasagemath-modules):gsl(see above)- Python runtime dependency:pplpysagemath-schemes- non-Python dependency (viasagemath-modules):gsl(see above)- non-Python dependencies (viasagemath-singular,sagemath-flint,sagemath-ntl,sagemath-pari): singular, flint, ntl, pari (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-pari/pyproject.toml.m4#L61)- Python dependency (viasagemath-singular,sagemath-flint,sagemath-pari):cypari2- Python dependency:scipysagemath-symbolics- non-Python dependencies:ecl,maxima,singularhttps://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-symbolics/pyproject.toml.m4#L89- non-Python dependencies (viasagemath-flint,sagemath-ntl,sagemath-modules):flint,ntl,pari,gsl- Python dependencies:mpmath,sympy,cypari2, numpy

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/41d11d3f-d5e5-4bf5-9629-2ef17ce4d6b1n%40googlegroups.com.