Urgent and important: Please vote on disputed PR #36964 (next step of the modularization project)

[Matthias Koeppe]

Dear all:

As an alternative to the proposal to back out the PR https://github.com/sagemath/sage/pull/36964 whose disputed dependency PR https://github.com/sagemath/sage/pull/36676 which had not reached the required 2:1 supermajority of the dispute-resolution process (it currently only has a simple majority of 7 votes in favor, 5 votes against) --- I am asking for your votes on that dependency PR https://github.com/sagemath/sage/pull/36676 to heal the process. This will avoid further delays and disruptions.

What is the modularization project? The Sage developer community has long been aware of the severe problems that the monolithic design of Sage has brought. See in particular the lively 2016 sage-devel thread "How we develop Sage" (https://groups.google.com/g/ sage-devel/c/29ndCD8z94k) initiated by William. In its current incarnation, "modularization project" refers to my proposal from May 2020,

- to use modern Python packaging ("PEP 517/518/660") and Python 3's "implicit namespace packages" to 

- break the Sage library into separately buildable and installable "distribution packages"

- while keeping the structure of the source tree mostly unchanged, monolithic, for the convenience of the Sage developer community.

For the project, hundreds of tickets/PRs have been prepared and merged over the past 4 years, see the Meta-ticket https://github.com/sagemath/sage/issues/29705 for a list.

Has the Sage community been informed and consulted regarding the modularization project? Yes, in addition to the normal review that all tickets/PRs underwent:

- I have given detailed presentations about the project in SageDays 110 (2020), 112.358 (2022), 120 (2023), see https://github.com/sagemath/sage/issues/29705 for links.

- A chapter of the Sage Developer Guide, https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#packaging-the-sage-library, provides a detailed description of the design

- I have posted numerous times to sage-devel, most recently the series "SageMath modularization project: The five by five" (2023-06). See https://github.com/sagemath/sage/issues/29705 for links to all of these.

- Specifically, in the post "Modularization project: V. The blocs" (https://groups.google.com/g/sage-devel/c/kiB32zP3xD4/m/GJ0qF7TTAAAJ, 2023-06), I outlined the design of the pip-installable packages such as sagemath-combinat, sagemath-graphs, sagemath-flint, sagemath-plot etc. 

- And in my 2023-11 post https://groups.google.com/g/sage-devel/c/kiB32zP3xD4/m/GJ0qF7TTAAAJ in the same thread, I asked: 

> Ready for review: A restructuring of our "all.py" files along these dependencies in https://github.com/sagemath/sage/pull/36676. This is an opportunity to review the contents of the proposed distributions implemented in Mega-PR https://github.com/sagemath/sage/pull/35095 (~50 kLOC changes, not open for review). As https://github.com/sagemath/sage/pull/36676 rewrites all "all.py" files, it is also an opportunity for a deliberate coding style decision for these files. I welcome all constructive discussions in the PR.

What does the PR https://github.com/sagemath/sage/pull/36676 do? Per its title, "Restructure sage.*.all for modularization, replace relative by absolute imports". The PR is "mostly harmless": There are no user-visible changes; it's just a bunch of imports that are moved around. It includes no policy change of any kind; it only executes a design that was previously reviewed and carefully documented in separate PRs. Nothing permanent or irreversible is done here. The new files provide the top-level namespaces needed for doctesting modularized installations of Sage.

Has it been reviewed? Yes, David Coudert and John Palmieri did a detailed review. This was completed on November 15, 2023 --- over 5 months ago.

How did this PR become "disputed"? Back in November, one commenter floated an (untested) alternative design (https://github.com/sagemath/sage/pull/36676#pullrequestreview-1726079717); I explained in https://github.com/sagemath/sage/pull/36676#issuecomment-1806873154 why it's not suitable. Commenter demanded that the previously reviewed and documented design is reopened for discussion, https://github.com/sagemath/sage/pull/36676#issuecomment-1863667919. 

What are the concerns that have been made known during the voting process for this PR (March/April 2024)? I will not attempt to paraphrase, but here are links to some posts so that you can find the discussion.

- Gonzalo Tornaria https://github.com/sagemath/sage/pull/36676#issuecomment-2048350399, https://github.com/sagemath/sage/pull/36676#issuecomment-2048848093

- Michael Orlitzky https://github.com/sagemath/sage/pull/36676#issuecomment-2048600337

- Martin Rubey https://github.com/sagemath/sage/pull/36676#issuecomment-2050427818

How to vote: By posting a comment on the PR https://github.com/sagemath/sage/pull/36676.

Matthias

[Michael Orlitzky]

On 2024-04-18 14:18:37, Matthias Koeppe wrote:
> Dear all:
>
> As an alternative to the proposal to back out the

> PR https://github.com/sagemath/sage/pull/36964 whose *disputed dependency

> PR https://github.com/sagemath/sage/pull/36676 which had not reached the

> required 2:1 supermajority *of the dispute-resolution process *(it
> currently only has a simple majority of 7 votes in favor, 5 votes against)*

> --- I am asking for your votes on that dependency PR
> https://github.com/sagemath/sage/pull/36676 to heal the process. This will
> avoid further delays and disruptions.

This doesn't circumvent the issue. Voting on PRs with disputed
dependencies is nonsensical and counterproductive. We were told they
wouldn't be merged. I have no idea what 36964 is about and if I'm for
or against it because it's a waste of time until its dependencies are
accepted/rejected. Only then (after another two weeks or whatever) can
the vote on 36964 be considered meaningful.

[Martin R]

Dear Matthias!

> What is the modularization project? The Sage developer community has long been aware of the severe problems that the monolithic design of Sage has brought. See in particular the lively 2016 sage-devel thread "How we develop Sage" (https://groups.google.com/g/sage-devel/c/29ndCD8z94k) initiated by William.

I looked at a few of the messages in this thread (there are something like hundred messages, I feel unable to read them all).  I would like to make two remarks

* 26 persons participated in this thread, roughly 10 of these are still active

* many of the messages were quite critical regarding modularization of the math library in sage

* it seems to me that at least a big part of the thread is about modularization of the *infrastructure* of sage, not the math library.

In any case, I do not see consensus in this thread.  Of course, there may be other threads that I am not aware of.

I was not there at the time, and only very recently realized some potential consequences of the direction the modularization might take as indicated in https://github.com/sagemath/sage/pull/36964, and formulated some questions there.

To my surprise, at least Volker issued similar concerns in the thread above.  I could not find satisfying answers to his questions either.  Also, I must say, that I find "Don't fear. This has been thought through carefully, and you can read about it in the Sage developer's guide." in place of an answer on the edge of abuse.  It is at least condescending.

For convenience of discussion, I am reproducing my questions below. 

Martin

1.) Is there an example for someone who did not want to use sage because of some dependency of the math library?  Or at least a possible reason?  @kwankyu's comment above suggests that having something in the "wrong" distribution wouldn't be a big deal.  But this begs the question: who profits from cutting the math library into pieces (which look very arbitrary to me and have a curious emphasis on discrete math topics)?

My fear would be that at some point there is a request not to use symbolics in some module, because Lisp is hard to install on some system.

(I don't think this fear is unjustified: in the section of the developer guide you pointed to, I find

> The imports of the symbolic functions ceil() and floor() can likely be replaced by the artithmetic functions [integer_floor()](https://doc.sagemath.org/html/en/reference/rings_standard/sage/arith/misc.html#sage.arith.misc.integer_floor) and [integer_ceil()](https://doc.sagemath.org/html/en/reference/rings_standard/sage/arith/misc.html#sage.arith.misc.integer_ceil).

OK, so some user of that module happily replaces the two functions.  Now, I come along and would like to replace some other implementation by a call to something defined in symbolics.  But that would be breaking a promise to the user, so it would be really hard to justify.

In fact, this happened to me already, in some sense.  I noticed a function definition in `sage.modular.multiple_zeta`  with misleading documentation, which could be replaced by a call to code in `sage.combinat`.  However, this is *already* hard to do, because it might affect performance (which is a very valid point in my opinion).  I think it would be extremely bad to make it even harder.

2.) If this is about dependencies on other software, why aren't the distributions named after these dependencies?  (Of course, at some point dependencies might change, for example, there might be a switch from glpk to scip.)

Before I read - by chance - `distribution = sagemath-graphs` somewhere, I thought one would "modularize" things like the repl, user interfaces, and perhaps some low level stuff.  But it seems to me now that this is really about the modularization of the mathematics.

Also, I find it hard to believe that it is about dependencies, because the stuff in `abstract_tree.py` and friends has no dependencies on external software (unless you want to LaTeX them, perhaps).

[Matthias Koeppe]

Michael, note that in my message I asked for a vote on that dependency https://github.com/sagemath/sage/pull/36676.

[Matthias Koeppe]

On Friday, April 19, 2024 at 5:08:02 AM UTC-7 Martin R wrote:

> What is the modularization project? The Sage developer community has long been aware of the severe problems that the monolithic design of Sage has brought. See in particular the lively 2016 sage-devel thread "How we develop Sage" (https://groups.google.com/g/sage-devel/c/29ndCD8z94k) initiated by William.

I looked at a few of the messages in this thread (there are something like hundred messages, I feel unable to read them all).  I would like to make two remarks

* 26 persons participated in this thread, roughly 10 of these are still active

* many of the messages were quite critical regarding modularization of the math library in sage

* it seems to me that at least a big part of the thread is about modularization of the *infrastructure* of sage, not the math library.

In any case, I do not see consensus in this thread.

Of course there was not a "consensus"'; at that time there was not even a plan.

I was citing the thread, as I indicated, to support my sentence, "The Sage developer community has long been aware of the severe problems that the monolithic design of Sage has brought."

[Michael Orlitzky]

On Fri, 2024-04-19 at 09:46 -0700, Matthias Koeppe wrote:
>
> Michael, note that in my message I asked for a vote on that dependency
> https://github.com/sagemath/sage/pull/36676.
>

Even if 36676 gets approval, 36964 must be reverted. It was not
meaningfully voted upon.

[Matthias Koeppe]

On Friday, April 19, 2024 at 5:08:02 AM UTC-7 Martin R wrote:

2.) If this is about dependencies on other software, why aren't the distributions named after these dependencies?

Martin, I have answered this already when you asked it in the PR: Some are.

Note that the description of the PR where you asked this question contains the list of the distributions: https://github.com/sagemath/sage/pull/36676

"We restructure the all.py files for modularization. Guided by the technical constraints outlined in https://groups.google.com/g/sage-devel/c/ozh-7ZZ848s, https://github.com/sagemath/sage/pull/35095 defines distribution packages (pip-installable packages) sagemath-{brial, combinat, eclib, flint, gap, giac, glpk, graphs, groups, homfly, lcalc, libbraiding, libecm, linbox, modules, mpmath, ntl, pari, plot, polyhedra, schemes, singular, standard-no-symbolics, symbolics}."

My June 2023 sage-devel post https://groups.google.com/g/sage-devel/c/kiB32zP3xD4 explained that the design use "... 3 types of distribution packages:

- Packages that are named after a basic mathematical structure but may cover a wide range of generalizations/applications of this structure.

- Packages that are named after an upstream library. 
- Packages that are named after a technical functionality."

[Martin R]

On Friday 19 April 2024 at 20:08:51 UTC+2 Matthias Koeppe wrote:

On Friday, April 19, 2024 at 5:08:02 AM UTC-7 Martin R wrote:

2.) If this is about dependencies on other software, why aren't the distributions named after these dependencies?

Martin, I have answered this already when you asked it in the PR: Some are.

Matthias, this does not answer my question.  Maybe it is clearer if I ask: why do you introduce distributions sage-graphs, sage-combinat, sage-categories etc.

Martin

[Martin R]

A follow-up question: do I understand correctly that common lisp (via maxima) is the main dependency that prevents sagemath from being pip-installable?

All the best,

Martin

[Volker Braun]

Yes in a perfect world, but then you don't get a gold star for satisfying some purity test. We should just do the minimal amount of work to get us where we want to be. Lets focus on the direction to go and not too much on the process.

[Michael Orlitzky]

On Sat, 2024-04-20 at 02:22 -0700, Volker Braun wrote:
> Yes in a perfect world, but then you don't get a gold star for satisfying
> some purity test. We should just do the minimal amount of work to get us
> where we want to be. Lets focus on the direction to go and not too much on
> the process.

We don't know where we want to be. To figure that out, we should hold
some kind of vote or something.

[Dima Pasechnik]

On 20 April 2024 08:56:30 BST, 'Martin R' via sage-devel <sage-...@googlegroups.com> wrote:
>A follow-up question: do I understand correctly that common lisp (via
>maxima) is the main dependency that prevents sagemath from being
>pip-installable?

pip install sagemath-standard

already works in a venv on a box with enough deps
installed. Takes lot of time. To fix this,
anyhow, someone has to make their hands dirty and do to Lisp interface what's done to Pari/GP (cypari)
PPL (pplpy), primecount (primecountpy), etc.
Apart from Lisp, there is GAP (with the corresponding effort stalled).

That's what is much more urgent than attempting to slice up the maths functionality of sagelib.

Dima

[Dima Pasechnik]

Hi Volker,

On Sat, Apr 20, 2024 at 10:22 AM Volker Braun <vbrau...@gmail.com> wrote:

Yes in a perfect world, but then you don't get a gold star for satisfying some purity test. We should just do the minimal amount of work to get us where we want to be. Lets focus on the direction to go and not too much on the process.

the keywords are "where we want to be" and "direction". 

The revert should happen as there is no agreement on this.

It would help if we first sort out and agree on these, then we can resume the normal operations.

IMHO it would help if you stopped touching these disputed PRs until this these are done, or at least for a set period of time,

say a month or two.

Dima

 

On Friday, April 19, 2024 at 7:18:03 PM UTC+2 Michael Orlitzky wrote:

On Fri, 2024-04-19 at 09:46 -0700, Matthias Koeppe wrote:
>
> Michael, note that in my message I asked for a vote on that dependency
> https://github.com/sagemath/sage/pull/36676.
>

Even if 36676 gets approval, 36964 must be reverted. It was not
meaningfully voted upon.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/b9d41da8-57ec-4542-a5d8-7f2690849a49n%40googlegroups.com.

[Matthias Koeppe]

On Friday, April 19, 2024 at 12:34:06 PM UTC-7 Martin R wrote:

why do you introduce distributions sage-graphs, sage-combinat, sage-categories etc.

Let's follow the link included in my previous message to my June 2023 sage-devel post https://groups.google.com/g/sage-devel/c/kiB32zP3xD4 and see what's written there.

"Packages that are named after a basic mathematical structure but may cover a wide range of generalizations/applications of this structure. People who work in a specialized research area will of course recognize what structures they need. But the down-to-earth naming also creates discoverability for a broader audience. Not many more distribution packages than these 7 are needed:
- sagemath-combinat: everything combinatorial except for graphs.
- sagemath-graphs: also provides posets, designs, abstract complexes, quivers, etc.
- sagemath-groups
- sagemath-modules: also has matrices, tensors, homology, coding theory, etc.
- sagemath-polyhedra: also provides fans, hyperplane arrangements, etc.
- sagemath-schemes
- sagemath-symbolics"

In short, I introduce these packages to create discoverability for potential consumers of portions of the Sage library. (pip-installable packages are often named after the functionality that they provide.)

[Michael Orlitzky]

On Sat, 2024-04-20 at 10:07 +0100, Dima Pasechnik wrote:
>
> Apart from Lisp, there is GAP (with the corresponding effort stalled).
>
> That's what is much more urgent than attempting to slice up the maths functionality of sagelib.
>

Also the ancient copy of ginac/pynac we bundle.

[Matthias Koeppe]

On Saturday, April 20, 2024 at 12:56:30 AM UTC-7 Martin R wrote:

do I understand correctly that common lisp (via maxima) is the main dependency that prevents sagemath from being pip-installable?

No.

For one, SageMath is already pip-installable. 

That was one of the first deliverables of the modularization project, completed in 2021.

See https://wiki.sagemath.org/ReleaseTours/sage-9.4#Alternative_installation_methods_using_pip

It's documented in our README: https://github.com/sagemath/sage/blob/develop/README.md#alternative-installation-using-pypi

However, installing Sage in this way is equivalent to installing the full Sage distribution from source in the normal way. (This is what https://pypi.org/project/sage-conf/ does.)

It takes very long. This alone makes it simply not suitable as a dependency of other pip-installable projects.

In another message, you asked:

> 1.) Is there an example for someone who did not want to use sage because of some dependency of the math library?  Or at least a possible reason? [...] But this begs the question: who profits from cutting the math library into pieces (which look very arbitrary to me and have a curious emphasis on discrete math topics)?

If you do allow me another example from discrete math: Sage has a lot of very good code in "sage.graphs" that provides functionality that is not available from other Python libraries. 

But to potential projects that would need this functionality, the proposition to have to depend on the monolithic project SageMath, with hours of compilation time and/or dependencies on system packages that are obviously unrelated to the needed functionality (boost, brial, ecl, eclib, ecm, fflas_ffpack, flint, libgd, gap, giac, givaro, glpk, gmp, gsl, iml, lcalc, libbraiding, libhomfly, libpng, linbox, m4ri, m4rie, mpc, mpfi, mpfr, ntl, pari, rw, singular, symmetrica), is simply a showstopper.

[Dima Pasechnik]

On 20 April 2024 19:34:49 BST, Matthias Koeppe <matthia...@gmail.com> wrote:
>On Saturday, April 20, 2024 at 12:56:30 AM UTC-7 Martin R wrote:
>
>do I understand correctly that common lisp (via maxima) is the main
>dependency that prevents sagemath from being pip-installable?
>
>
>No.
>
>For one, SageMath is already pip-installable.
>That was one of the first deliverables of the modularization project,
>completed in 2021.
>See https://wiki.sagemath.org/ReleaseTours/sage-9.4#Alternative_installation_methods_using_pip
>
>It's documented in our README:
>https://github.com/sagemath/sage/blob/develop/README.md#alternative-installation-using-pypi
>
>However, installing Sage in this way is equivalent to installing the full
>Sage distribution from source in the normal way.

No, it is not equivalent, far from it. One can read on
<https://pypi.org/project/sagemath-standard/#description>

Building sagemath-standard has a large number of system packages as prerequisites. See https://doc.sagemath.org/html/en/installation/source.html#linux-recommended-installation for partial lists for various systems.

That is, many packages are not built. What's being built is not a Sage distribution, it is sagelib.

> (This is
>what https://pypi.org/project/sage-conf/ does.)
>It takes very long. This alone makes it simply not suitable as a dependency
>of other pip-installable projects.
>
>
>In another message, you asked:
>> 1.) Is there an example for someone who did not want to use sage because
>of some dependency of the math library? Or at least a possible reason?
>[...] But this begs the question: who profits from cutting the math library
>into pieces (which look very arbitrary to me and have a curious emphasis on
>discrete math topics)?
>
>If you do allow me another example from discrete math: Sage has a lot of
>very good code in "sage.graphs" that provides functionality that is not
>available from other Python libraries.
>
>But to potential projects that would need this functionality, the
>proposition to have to depend on the monolithic project SageMath, with
>hours of compilation time and/or dependencies on system packages that are
>obviously unrelated to the needed functionality (boost, brial, ecl, eclib,
>ecm, fflas_ffpack, flint, libgd, gap, giac, givaro, glpk, gmp, gsl, iml,
>lcalc, libbraiding, libhomfly, libpng, linbox, m4ri, m4rie, mpc, mpfi,
>mpfr, ntl, pari, rw, singular, symmetrica), is simply a showstopper.
>

This is not true, either. First, a lot of these packages that you list are available on systems, and thus there is no need to build them.
(To "but macOS?" I have a reply: "ask Apple to provide you a package manager, it's the best OS, right, you pay for it a lot of money, or learn to use Homebrew like Mac users usually do...")

Anyway, a normal Python pypi-installable package comes with binary wheels, i e. things are pre-built, and it's merely matter of downloading these to get a functional package. Few minutes on a fast network, not hours.
By choosing to be an exception in the Python world,
Sage obviously does something quite wrong.

Dima

[kcrisman]

By choosing to be an exception in the Python world,
Sage obviously does something quite wrong. 

Can someone who is not Dima or Matthias explain to us how it is possible that they both are claiming to represent the normal Python way of doing things?  There have been numerous statements by both of them about this, which makes it sound like there are two pieces to it (modularization but also "de-vendoring"), and I can only assume that it would be possible for both to occur simultaneously.  It would be helpful for this to be clarified, though, so that one knows precisely what *piece* of their proposals represent the "normal Python ecosystem".

That said, "normal Python" is not necessarily as relevant for those who would *only* want Sage, or at least mostly so.  Having just another Python package might lead us to implementing powers as ** instead of ^, which would be a regression, or needing namespaces for almost everything, which again limits the value of Sage qua Sage.

[Michael Orlitzky]

On Sat, 2024-04-20 at 15:01 -0700, kcrisman wrote:
>
> Can someone who is not Dima or Matthias explain to us how it is possible
> that they both are claiming to represent the normal Python way of doing
> things? There have been numerous statements by both of them about this,
> which makes it sound like there are two pieces to it (modularization but
> also "de-vendoring"), and I can only assume that it would be possible for
> both to occur simultaneously.

It is, but in its current incarnation, the modularization relies
heavily on the sage distribution vendoring. Conflict arises because the
modularization is cited as a blocker whenever someone wants to pare
down or disentangle some aspect of the sage distribution. It's not the
modularization per se that we object to. Personally, I would like it to
succeed, but not at the expense of everything else.

[Matthias Koeppe]

On Saturday, April 20, 2024 at 3:01:04 PM UTC-7 kcrisman wrote:

"normal Python" is not necessarily as relevant for those who would *only* want Sage, or at least mostly so.  Having just another Python package might lead us to implementing powers as ** instead of ^, which would be a regression, or needing namespaces for almost everything, which again limits the value of Sage qua Sage.

I understand this concern, and I can reassure you that the modularization project keeps all functionality of the Sage application intact; that includes the surface language using the Sage preparser. 

[Matthias Koeppe]

On Saturday, April 20, 2024 at 3:23:12 PM UTC-7 Michael Orlitzky wrote:

in its current incarnation, the modularization relies
heavily on the sage distribution vendoring. Conflict arises because the
modularization is cited as a blocker whenever someone wants to pare
down or disentangle some aspect of the sage distribution. It's not the
modularization per se that we object to. Personally, I would like it to
succeed, but not at the expense of everything else

Michael, I think you may be using too much jargon to get your point across to the general readership of this list. 

Let's maybe use this opportunity to make this as concrete as possible and explain it in the most plain terms.

What entanglement are you concerned about?

[Martin R]

Dear Matthias,

This doesn't make sense to me.  Why would you separate mathematics into packages that have no more external dependencies from others, which at the same time may grow internal dependencies over time?

I can imagine that it would make sense to make as much as possible into runtime dependencies - you wrote below that building the dependencies takes a lot of time.  Maybe that's the core problem, I don't know.

However, to me the current proposal really looks like modularization just for its own sake.  I wouldn't (and for that reason I didn't) mind if this comes without much cost, but this is not the case.  I am quite sure that the mere existence of the packages will influence decisions whether to duplicate code or not.

Martin

Besides, I would prefer if we would not make projects natural persons, and I would prefer if we could stick to speaking about ourselves, rather than "the sage developers", whoever that might be.

[Nathan Dunfield]

On Saturday, April 20, 2024 at 5:01:04 PM UTC-5 kcrisman wrote:

Can someone who is not Dima or Matthias explain to us how it is possible that they both are claiming to represent the normal Python way of doing things?  There have been numerous statements by both of them about this, which makes it sound like there are two pieces to it (modularization but also "de-vendoring"), and I can only assume that it would be possible for both to occur simultaneously.  It would be helpful for this to be clarified, though, so that one knows precisely what *piece* of their proposals represent the "normal Python ecosystem.

For the statements in this thread, I don't see any contradictions about the definition of the "normal Python way of doing things".  My understanding of that term is to post self-contained binary wheels to PyPI for all supported platforms that install in a minute or two with no compilation (as Dima wrote), as well as a source-code only package that serves as a rarely-used backup if no suitable binary is available.  (The self-contained bit is important; for example, on Linux here are the only external libraries a binary wheel is allowed to link to.)

So far, we only post a source-code only package "sagemath-standard" on PyPI, and so "pip install sagemath-standard" is basically equivalent to downloading the tarball and running "configure/make".   As Matthias says "It takes very long. This alone makes it simply not suitable as a dependency of other pip-installable projects."

In particular, no one is arguing that Sage currently follows the "normal Python way", though we have made one step in that direction by posting the source-only package.

Best,

Nathan

[Marc Culler]

On Sunday, April 21, 2024 at 9:42:24 AM UTC-5 Nathan Dunfield wrote:

For the statements in this thread, I don't see any contradictions about the definition of the "normal Python way of doing things".  My understanding of that term is to post self-contained binary wheels to PyPI for all supported platforms that install in a minute or two with no compilation (as Dima wrote), as well as a source-code only package that serves as a rarely-used backup if no suitable binary is available.  (The self-contained bit is important; for example, on Linux here are the only external libraries a binary wheel is allowed to link to.)

I  hope it will help to acknowledge what might be regarded as the elephant in this room.  That elephant consists of two facts:

1) Users, as opposed to developers, are very unlikely to be either willing to or capable of building a complex source-code only package which depends on external libraries, because doing that requires finding and installing (or building from source), all of the external libraries needed by the python package.

2) The use of a self-contained binary wheel on linux violates a core principle of linux packaging, namely that any library which is a runtime dependency of a package should be installed from the appropriate distro package.

Note that 2) also means that a pypi package developer must support the full range of different versions of these libraries which are provided by different linux distributions.  Sage initially used the same principle in its design, but avoided the maintenance burden caused by trying to support all of these different library versions by providing its own libraries in SAGE_LOCAL.  It has since gradually accepted more and more of the burden by allowing "system versions" of libraries wherever it seemed feasible.

In the "Python ecosystem" these issues have been worked around in a different way in order to make self-contained binary wheels a possibility, and thereby accommodate users.  This is usually done with delocate on macOS or with auditwheel on linux.  Those tools embed all of the required libraries inside the python package and adjust the rpaths in the python extension modules to make them use the embedded libraries instead of libraries which might or might not be installed on the host system.  In the case of linux this also means compiling the libraries against a version of glibc which is older than the version found on any of the linux distributions supported by the package.  The job of building libraries against old versions of glibc is done by building them on a manylinux docker image, which can be done on a CI runner.  (On macOS one can specify a minimum target version of macOS, usually 10.12 these days.)

As a typical example (which recently created issues for Sage related to the jpeg library) you can look at the binary wheel for Pillow.  If you unzip that wheel you will find a subdirectory named pillow.libs which contains: libbrotlicommon-3ecfe81c.so.1,  libpng16-58efbb84.so.16.43.0, libbrotlidec-ba690955.so.1, libsharpyuv-20f78091.so.0.0.1, libfreetype-1f2cdfbb.so.6.20.1, libtiff-4da6744b.so.6.0.2, libharfbuzz-8b0b6641.so.0.60840.0,libwebp-850e2bec.so.7.1.8, libjpeg-f391b078.so.62.4.0, libwebpdemux-df9b36c7.so.2.0.14, liblcms2-e69eef39.so.2.0.16,libwebpmux-9fe05867.so.3.0.13, liblzma-13fa198c.so.5.4.5,libXau-154567c4.so.6.0.0, libopenjp2-05423b53.so , and libxcb-80c5a837.so.1.1.0.  A linux packager would provide all of these by adding as dependencies all of the distro packages needed to provide these libraries.  In the "Python ecosystem" these libraries are all built on manylinux docker images and then embedded in the binary wheel.

I don't see how Sage will be able to join the Python ecosystem without addressing this.

- Marc

[Matthias Koeppe]

On Sunday, April 21, 2024 at 2:30:15 AM UTC-7 Martin R wrote:

I can imagine that it would make sense to make as much as possible into runtime dependencies - you wrote below that building the dependencies takes a lot of time.  Maybe that's the core problem, I don't know.

If you want to know, then I recommend to read the sections of the Developer's Guide where 

- the different types of dependencies are explained, as well as

- the techniques to reduce and manage such dependencies. 

Links in https://github.com/sagemath/sage/pull/36753#issuecomment-2013517117

[Matthias Koeppe]

On Saturday, April 20, 2024 at 1:01:33 PM UTC-7 Dima Pasechnik wrote:

Anyway, a normal Python pypi-installable package comes with binary wheels, i e. things are pre-built, and it's merely matter of downloading these to get a functional package. Few minutes on a fast network, not hours.
By choosing to be an exception in the Python world,
Sage obviously does something quite wrong.

I'll note that providing binary wheels is actually one of the deliverables of the modularization project. 

- See for example https://pypi.org/project/sagemath-categories/#files

The meta-ticket "Distribution as wheels" (https://github.com/sagemath/sage/issues/31251) describes the next steps, including a PR that is waiting for review.

[Matthias Koeppe]

On Sunday, April 21, 2024 at 2:30:15 AM UTC-7 Martin R wrote:

Why would you separate mathematics into packages that have no more external dependencies from others, which at the same time may grow internal dependencies over time?

Let's just go through the list of distribution packages and their dependencies for concreteness. (All depend on sagemath-categories and thus on the basic arithmetic libraries gmp, mpc, mpfr, gmpy2)

sagemath-combinat 

- non-Python dependency: symmetrica (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-combinat/pyproject.toml.m4#L65)

sagemath-graphs

- non-Python dependency: boost (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-graphs/pyproject.toml.m4#L73)

sagemath-modules

- non-Python dependency: gsl (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules/pyproject.toml.m4#L109)

- Python build requirement: numpy

sagemath-groups

- non-Python dependency (via sagemath-gap): gap (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-gap/pyproject.toml.m4#L52)

- non-Python dependency (via sagemath-modules): gsl (see above)

sagemath-polyhedra

- non-Python dependency (via sagemath-glpk): glpk

- non-Python dependency (via sagemath-modules): gsl (see above)

- Python runtime dependency: pplpy

sagemath-schemes

- non-Python dependency (via sagemath-modules): gsl (see above)

- non-Python dependencies (via sagemath-singular, sagemath-flint, sagemath-ntl, sagemath-pari): singular, flint, ntl, pari (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-pari/pyproject.toml.m4#L61)

- Python dependency (via sagemath-singular, sagemath-flint, sagemath-pari): cypari2

- Python dependency: scipy

sagemath-symbolics

- non-Python dependencies: ecl, maxima, singular https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-symbolics/pyproject.toml.m4#L89

- non-Python dependencies (via sagemath-flint, sagemath-ntl, sagemath-modules): flint, ntl, pari, gsl

- Python dependencies: mpmath, sympy, cypari2, numpy

[Matthias Koeppe]

On Saturday, April 20, 2024 at 1:01:33 PM UTC-7 Dima Pasechnik wrote:

On 20 April 2024 19:34:49 BST, Matthias Koeppe <matthia...@gmail.com> wrote:
 SageMath is already pip-installable.
>That was one of the first deliverables of the modularization project,
>completed in 2021.
>See https://wiki.sagemath.org/ReleaseTours/sage-9.4#Alternative_installation_methods_using_pip
>
>It's documented in our README:
>https://github.com/sagemath/sage/blob/develop/README.md#alternative-installation-using-pypi
>
>However, installing Sage in this way is equivalent to installing the full
>Sage distribution from source in the normal way.

No, it is not equivalent, far from it. One can read on
<https://pypi.org/project/sagemath-standard/#description>

Building sagemath-standard has a large number of system packages as prerequisites. See https://doc.sagemath.org/html/en/installation/source.html#linux-recommended-installation for partial lists for various systems.

That is, many packages are not built. What's being built is not a Sage distribution, it is sagelib.

Dima, as you well know, I designed and implemented all of these facilities related to the PyPI distribution method of SageMath as part of the modularization project. What you quote from https://pypi.org/project/sagemath-standard/#description -- I wrote it. It's hard to understand what the purpose of your message "correcting" me on these things could possibly be.

The distribution sagemath-standard does provide the Sage library.

But it declares a required build-time and runtime dependency on https://pypi.org/project/sage-conf/, which when installed, builds the Sage distribution in a subdirectory of ~/.sage.

(Note also that one of the disputed PRs, https://github.com/sagemath/sage/pull/37138, seeks to remove this declared dependency that implements this mechanism.)

>If you do allow me another example from discrete math: Sage has a lot of
>very good code in "sage.graphs" that provides functionality that is not
>available from other Python libraries.
>
>But to potential projects that would need this functionality, the
>proposition to have to depend on the monolithic project SageMath, with
>hours of compilation time and/or dependencies on system packages that are
>obviously unrelated to the needed functionality (boost, brial, ecl, eclib,
>ecm, fflas_ffpack, flint, libgd, gap, giac, givaro, glpk, gmp, gsl, iml,
>lcalc, libbraiding, libhomfly, libpng, linbox, m4ri, m4rie, mpc, mpfi,
>mpfr, ntl, pari, rw, singular, symmetrica), is simply a showstopper.

This is not true, either. First, a lot of these packages that you list are available on systems, and thus there is no need to build them.
(To "but macOS?" I have a reply: "ask Apple to provide you a package manager, it's the best OS, right, you pay for it a lot of money, or learn to use Homebrew like Mac users usually do...")

Note that I wrote "hours of compilation time and/or dependencies on system packages".

So there was nothing to correct here. My point stands. 

What's the purpose of your comment?

Anyway, a normal Python pypi-installable package comes with binary wheels, i e. things are pre-built, and it's merely matter of downloading these to get a functional package. Few minutes on a fast network, not hours.
By choosing to be an exception in the Python world,
Sage obviously does something quite wrong.

Also the purpose of this part of your message is not clear to me, Dima.

As you well know, preparing binary wheels is another deliverable of the modularization project. What Sage may have been doing wrong --- the modularization project has been fixing it.

- For the modularized distributions that are already available, these binary wheels have already been up on PyPI for a long time, see https://pypi.org/project/sagemath-objects/#files and https://pypi.org/project/sagemath-categories/#files

- https://github.com/sagemath/sage/pull/37503 (to be merged) adds wheels for the macOS arm64 architecture to what it is built and deployed to PyPI on every release

- https://github.com/sagemath/sage/pull/37099 (needs review) and https://github.com/sagemath/sage/pull/36525, https://github.com/sagemath/sage/pull/36730 (in prep) add wheels for more distributions

An overview of this effort of making wheels available: Meta-PR https://github.com/sagemath/sage/issues/31251

I repeat my invitation for others to join me in this important effort.

[Michael Orlitzky]

On 2024-04-20 15:33:51, Matthias Koeppe wrote:
> Michael, I think you may be using too much jargon to get your point across
> to the general readership of this list.
>
> Let's maybe use this opportunity to make this as concrete as possible and
> explain it in the most plain terms.
> What entanglement are you concerned about?

For example, we have several tickets that are disputed because they
will use ./bootstrap and data from the sage distribution (build/pkgs)
to generate the pyproject.toml files for the modular components. This
ensures that the components cannot truly be separated, not only from
the other components, but from the mini-distro in build/pkgs that a
large chunk of us hate.

Contrast with some other parts of sage that have been modularized:

* pplpy
* memory-allocator
* cypari
* cysignals
* primecountpy

These have been successfully disentangled from both the sage library
and the sage distribution and I don't think anyone has complaints
about them. They've been moved to separate repositories which, while
not strictly necessary, is certainly proof that they are in fact
disentangled. We can typically depend on stable versions of them and
install/test them independently. They do not depend on nonstandard or
bleeding-edge features of the python ecosystem.

[Kwankyu Lee]

On Sunday, April 21, 2024 at 7:01:04 AM UTC+9 kcrisman wrote:

By choosing to be an exception in the Python world,
Sage obviously does something quite wrong. 

Can someone who is not Dima or Matthias explain to us how it is possible that they both are claiming to represent the normal Python way of doing things?  There have been numerous statements by both of them about this, which makes it sound like there are two pieces to it (modularization but also "de-vendoring"), ...

Sage consists of two pieces: the sage library and the sage packages (aka spkgs defined in sage_root/build/pkgs). The sage distribution is roughly the sum of the sage packages. The sage library is a big python package (sage.rings etc.). The modularization project,  led by Matthias,  is splitting the sage library and adding (pip-installable) distribution packages (defined in sage_root/pkgs). Michael is using the term "modularization" to mean several things: (1) using a system package (a distro package installed in a linux distribution, ubuntu for example) instead of a sage package (2) using a system python package instead of a sage package (3) splitting some portion of the sage library to a separate python package, which is installed as a sage package, examples are cypari, cysignals, etc. 

Note that "package"s and "distribution"s appearing in the above paragraph in bold face all refer to different things. 

Many sage packages (known as "pip package" and "wheel packages" ) just install python packages from pypi. Dima wants to reduce the number of such sage packages (prominently, jupyter and friends), and seems to call this "the normal Python way of doing things". You call this "de-vendoring".

Matthias' "normal Python way of doing things" is to follow the Python packaging standard in designing the distribution packages.   

Philosophically, there is no reason that Michael's "modularization", Dima's "de-vendoring" , Tobias' "conda support", and Gonzalo's "sagemath distro package" conflict with Matthias' modularization. But they do in technical details in disputed PRs.

 

[Martin R]

Thank you for this list.

I still don't see why you would name these distributions as you do, and why you collect them as you do.  For example, as far as I know, symmetrica is currently essentially only used by symmetric functions, Schubert polynomials.  So, if symmetrica is such a burden to install, why would you want to install it if you don't need them?

On the other hand, combinatorial species will, after this summer, heavily depend on gap.  Does this mean that you would want to move this part into sage-groups?  Apart from the strange naming, I don't think that it will make anybody happy if dependencies change frequently.

Best wishes,

Martin

[Tobia...@gmx.de]

In reply to the comment (https://github.com/sagemath/sage/pull/36676#issuecomment-2067371677)

>> My fear would be that at some point there is a request not to use symbolics in some module, because Lisp is hard to install on some system.

>That should not happen. Modularization is downstream to the sage library. Yes, we are restructuring some parts of the sage library to fit with modularization. But modularization should never be an obstacle in developing the sage library. If it ever be, the sage community might drop the modularization project.

We already witness multiple instances where the modularization project is cited as a reason not to merge certain PRs that only touch sage-the-library. How does this reality fit this view?

Moreover, once the modularization is in place, it will impose wide-ranging constraints on what functions from other modules you are able to use. Currently, you can use `sage.x` in `sage.y` for an arbitrary combination of `x` and `y`. The modularization project restricts this severely and you will only be able to use modules `x` that are declared as a dependency of `y` (the arrows in the picture in https://github.com/sagemath/sage/pull/35095).

[Kwankyu Lee]

>> My fear would be that at some point there is a request not to use symbolics in some module, because Lisp is hard to install on some system.

>That should not happen. Modularization is downstream to the sage library. Yes, we are restructuring some parts of the sage library to fit with modularization. But modularization should never be an obstacle in developing the sage library. If it ever be, the sage community might drop the modularization project.

We already witness multiple instances where the modularization project is cited as a reason not to merge certain PRs that only touch sage-the-library. How does this reality fit this view?

I meant the sage library as a collection of mathematical modules. If a certain module did not but somehow would develop to rely on the mathematical functionality of another module, then the design of the modularization should embrace the development. The splitting of the sage library  in the present modularization reflects the present reality of the separation of different mathematical parts of the sage library. But of course the reality may change in the future as we develop the sage library. Then the modularization should reflect the change, not block the change. However this is all about the future, unrelated to the present disputed PRs, including yours.      

[Dima Pasechnik]

On Sun, Apr 21, 2024 at 10:42 PM Matthias Koeppe <matthia...@gmail.com> wrote:

On Sunday, April 21, 2024 at 2:30:15 AM UTC-7 Martin R wrote:

Why would you separate mathematics into packages that have no more external dependencies from others, which at the same time may grow internal dependencies over time?

Let's just go through the list of distribution packages and their dependencies for concreteness. (All depend on sagemath-categories and thus on the basic arithmetic libraries gmp, mpc, mpfr, gmpy2)

these are seemingly incomplete:

sagemath-combinat 

- non-Python dependency: symmetrica (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-combinat/pyproject.toml.m4#L65)

non-python: depends on GAP, givaro, too

 

sagemath-graphs

- non-Python dependency: boost (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-graphs/pyproject.toml.m4#L73)

non-python:  depends on GAP, givaro, too

 

sagemath-modules

- non-Python dependency: gsl (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules/pyproject.toml.m4#L109)

- Python build requirement: numpy

non-python: linbox, flas_ffpac, too ?

sagemath-groups

- non-Python dependency (via sagemath-gap): gap (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-gap/pyproject.toml.m4#L52)

- non-Python dependency (via sagemath-modules): gsl (see above)

sagemath-polyhedra

- non-Python dependency (via sagemath-glpk): glpk

- non-Python dependency (via sagemath-modules): gsl (see above)

- Python runtime dependency: pplpy

sagemath-schemes

- non-Python dependency (via sagemath-modules): gsl (see above)

- non-Python dependencies (via sagemath-singular, sagemath-flint, sagemath-ntl, sagemath-pari): singular, flint, ntl, pari (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-pari/pyproject.toml.m4#L61)

- Python dependency (via sagemath-singular, sagemath-flint, sagemath-pari): cypari2

- Python dependency: scipy

sagemath-symbolics

- non-Python dependencies: ecl, maxima, singular https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-symbolics/pyproject.toml.m4#L89

- non-Python dependencies (via sagemath-flint, sagemath-ntl, sagemath-modules): flint, ntl, pari, gsl

- Python dependencies: mpmath, sympy, cypari2, numpy

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/41d11d3f-d5e5-4bf5-9629-2ef17ce4d6b1n%40googlegroups.com.

[Marc Culler]

On Tuesday, April 23, 2024 at 4:07:37 AM UTC-5 Tobias wrote:

In reply to the comment (https://github.com/sagemath/sage/pull/36676#issuecomment-2067371677)

 once the modularization is in place, it will impose wide-ranging constraints on what functions from other modules you are able to use. Currently, you can use `sage.x` in `sage.y` for an arbitrary combination of `x` and `y`. The modularization project restricts this severely and you will only be able to use modules `x` that are declared as a dependency of `y` (the arrows in the picture in https://github.com/sagemath/sage/pull/35095).

This comment seems to me to reflect a basic misunderstanding of the goal of modularization.  The author is assuming that Sage users will have to make choices about which parts of Sage to install and find many things missing.  But Sage users should, as they do now, install all of Sage -- everything should be available to a user who does a normal install of Sage.  The projects that will really benefit from modularization will be those that provide their own limited mathematical context.  Developers of such projects will be able to choose which parts of Sage are relevant to their specific context.  Those parts of Sage can be provided for their users without having to embed a huge monolithic environment into a relatively small project.  For example, people studying 3-manifolds mainly need to work with simplicial complexes, groups, polynomials, number fields and interval arithmetic.  They do not need many other large parts of Sage, such as abstract manifolds (since all 3-manifolds can be triangulated).

This is basically a tautological argument, saying that modularization will cause modularization.  It does not reflect the reality for users.

- Marc

[Matthias Koeppe]

On Tuesday, April 23, 2024 at 6:14:05 AM UTC-7 Dima Pasechnik wrote:

On Sun, Apr 21, 2024 at 10:42 PM Matthias Koeppe <matthia...@gmail.com> wrote:

Let's just go through the list of distribution packages and their dependencies for concreteness. (All depend on sagemath-categories and thus on the basic arithmetic libraries gmp, mpc, mpfr, gmpy2)

these are seemingly incomplete:

What makes that seem to you?

 

sagemath-combinat 

- non-Python dependency: symmetrica (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-combinat/pyproject.toml.m4#L65)

non-python: depends on GAP, givaro, too

No. Your source?

 

 

sagemath-graphs

- non-Python dependency: boost (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-graphs/pyproject.toml.m4#L73)

non-python:  depends on GAP, givaro, too

No. Source?

 

sagemath-modules

- non-Python dependency: gsl (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules/pyproject.toml.m4#L109)

- Python build requirement: numpy

non-python: linbox, flas_ffpac, too ?

No. Source?

[Martin R]

If I understand correctly, the current proposal does not mind if some things don't work or could be replaced without too much effort.  For example, Dima might have referred to the fact that OrderedPartitions.cardinality uses gap, even though it is in sagemath-combinat.

The gap dependency in `designs.database` (which is in sagemath-graphs) and `matrices.latin` (which is in sagemath-combinat) might have been overlooked.

Martin

[Matthias Koeppe]

On Tuesday, April 23, 2024 at 5:35:10 AM UTC-7 Kwankyu Lee wrote:

I meant the sage library as a collection of mathematical modules. If a certain module did not but somehow would develop to rely on the mathematical functionality of another module, then the design of the modularization should embrace the development. The splitting of the sage library  in the present modularization reflects the present reality of the separation of different mathematical parts of the sage library. But of course the reality may change in the future as we develop the sage library. Then the modularization should reflect the change, not block the change.

Exactly. The design of the modularized distributions can evolve, and like everything in Sage, such changes will be in the hands of the Sage developer community.

[Dima Pasechnik]

On Tue, Apr 23, 2024 at 3:31 PM Matthias Koeppe

<matthia...@gmail.com> wrote:
>
> On Tuesday, April 23, 2024 at 6:14:05 AM UTC-7 Dima Pasechnik wrote:
>
> On Sun, Apr 21, 2024 at 10:42 PM Matthias Koeppe <matthia...@gmail.com> wrote:
>
> Let's just go through the list of distribution packages and their dependencies for concreteness. (All depend on sagemath-categories and thus on the basic arithmetic libraries gmp, mpc, mpfr, gmpy2)
>
>
> these are seemingly incomplete:
>
>
> What makes that seem to you?
>
>
> sagemath-combinat
> - non-Python dependency: symmetrica (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-combinat/pyproject.toml.m4#L65)
>
> non-python: depends on GAP, givaro, too
>
>
> No. Your source?

in

src/sage/combinat//designs/block_design.py

you can see

lazy_import('sage.libs.gap.libgap', 'libgap')
lazy_import('sage.rings.finite_rings.finite_field_constructor', 'FiniteField')

>
>
>
>
> sagemath-graphs
> - non-Python dependency: boost (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-graphs/pyproject.toml.m4#L73)
>
>
> non-python: depends on GAP, givaro, too
>
>
> No. Source?

similar to the above

>
>
> sagemath-modules
> - non-Python dependency: gsl (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules/pyproject.toml.m4#L109)
> - Python build requirement: numpy
>
>
> non-python: linbox, flas_ffpac, too ?
>
>
> No. Source?

same as above, basically

These modules don't function in any meaningful way without
dependencies I listed.

Dima

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/f54a1d55-b7c2-47a8-9245-d470a63091fan%40googlegroups.com.

[Matthias Koeppe]

On Monday, April 22, 2024 at 2:22:36 AM UTC-7 Martin R wrote:

I still don't see why you would name these distributions as you do, and why you collect them as you do.

Above I explained, "I introduce these packages to create discoverability for potential consumers of portions of the Sage library. (pip-installable packages are often named after the functionality that they provide.)"

For example, as far as I know, symmetrica is currently essentially only used by symmetric functions, Schubert polynomials.  So, if symmetrica is such a burden to install, why would you want to install it if you don't need them?

symmetrica, as a single dependency on its own, is not "such a burden to install".

Dependencies can become a burden simply if there are too many of them.

I have defined the distributions with an eye on keeping the number of distributions and the complexity low.

But there's no dogma. If the community finds it useful, then a distribution *sagemath-symmetrica could later be split out from sagemath-combinat and can become either a required dependency or an optional dependency of sagemath-combinat.

On the other hand, combinatorial species will, after this summer, heavily depend on gap.  Does this mean that you would want to move this part into sage-groups? 

If it has a build-time dependency on sage.libs.gap, then it would likely be shipped by sagemath-gap. If it's merely a runtime dependency, it does not really matter from a technical point of view which distribution ships it. (Once more I refer to the chapter in the Developer Guide that explains the different types of dependencies,  https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#dependencies-and-distribution-packages)

It could then be advertised, for example, as an "extra" of sagemath-combinat so that it can be installed as "pip install 'sagemath-combinat[species]'" or "pip install 'sagemath-combinat[gap]'", like the existing definitions of extras that make it possible to do "pip install 'sagemath-combinat[modules]'" for typical algebraic combinatorics features. https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-combinat/pyproject.toml.m4#L40

I gave more examples for the use of such extras in my 2023-06 sage-devel post "Modularization project: V. The blocs", https://groups.google.com/g/sage-devel/c/kiB32zP3xD4

 

Apart from the strange naming, I don't think that it will make anybody happy if dependencies change frequently.

Hard to tell what exactly you find strange about the naming if you're not more specific.

In https://github.com/sagemath/sage/pull/36964#discussion_r1564162445 and below, you already expressed surprise that trees and posets are in sagemath-graphs and that symmetric functions are in sagemath-combinat.

I'll just say that the existing design is the result of years of work that has taken all of the following into account:

- the technical constraints of Python packaging, 

- the actual dependencies of the Sage library, 

- discoverability by users,

- attribution and visibility for upstream libraries/projects,

- the overall complexity,

- and the potential for connecting the new distributions to communities of users and developers.

But as I said above, there's no dogma as to what exactly goes into what distribution, and the Sage developer community will be able to change it over time, like everything in Sage.

[Matthias Koeppe]

On Tuesday, April 23, 2024 at 8:38:13 AM UTC-7 Martin R wrote:

If I understand correctly, the current proposal does not mind if some things don't work or could be replaced without too much effort.  For example, Dima might have referred to the fact that OrderedPartitions.cardinality uses gap, even though it is in sagemath-combinat.

The gap dependency in `designs.database` (which is in sagemath-graphs) and `matrices.latin` (which is in sagemath-combinat) might have been overlooked.

Dependencies come in different flavors, or strengths. See again https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#dependencies-and-distribution-packages

Here's a tl;dr version:

Build-time dependencies: If a Cython module uses cimport to pull in anything from .pxd files, these files must be either part of the portion shipped as the distribution being built, or the distribution that provides these files must be declared as a build-time dependency.

Module-level runtime dependencies: Any import statements at the top level of a Python or Cython module are executed when the module is imported. Hence, the imported modules must be part of the distribution, or provided by another distribution – which then must be declared as a run-time dependency.

Other runtime dependencies: If import statements are used within a method, the imported module is loaded the first time that the method is called. Hence the module defining the method can still be imported even if the module needed by the method is not present. 

It is then a question whether a run-time dependency should be declared. If the method needing that import provides core functionality, then probably yes. 

But if it only provides what can be considered “optional functionality”, then probably not, and in this case it will be up to the user to install the distribution enabling this optional functionality. It is possible to declare such optional runtime dependencies as "extras", which provides a way to give a nickname to a distribution that can be installed as an add-on.

Doctest-only dependencies: Doctests often use examples constructed using functionality provided by other portions of the Sage library. This kind of integration testing is one of the strengths of Sage; but it also creates extra dependencies. We can deal with them using the same mechanism that we use for making doctests conditional on the presence of optional libraries: using "# optional - FEATURE" (or "# needs FEATURE") directives in the doctests. Adding these directives will allow developers to test the distribution separately, without requiring all of Sage to be present.

Looking at your examples:

$ head -n 4 src/sage/combinat/matrices/latin.py

# sage_setup: distribution = sagemath-combinat
# sage.doctest: needs sage.combinat sage.groups sage.modules
r"""
Latin Squares

So here you see that the module is shipped by the distribution sagemath-combinat, but the functionality in this module depends at runtime on the features "sage.combinat sage.groups sage.modules".

$ head -n 3 src/sage/combinat/designs/database.py
# sage_setup: distribution = sagemath-graphs
r"""
Database of small combinatorial designs

$ git grep '# needs' src/sage/combinat/designs/database.py

src/sage/combinat/designs/database.py:        sage: is_difference_matrix(M,G,8,1)                                             # needs sage.rings.finite_rings
src/sage/combinat/designs/database.py:        sage: _ = designs.difference_matrix(56,8)                                       # needs sage.rings.finite_rings
src/sage/combinat/designs/database.py:        sage: G,M = DM_57_8_1()                                                         # needs sage.rings.finite_rings sage.schemes
src/sage/combinat/designs/database.py:        sage: is_difference_matrix(M,G,8,1)                                             # needs sage.rings.finite_rings sage.schemes
src/sage/combinat/designs/database.py:        sage: _ = designs.difference_matrix(57,8)                                       # needs sage.rings.finite_rings sage.schemes
src/sage/combinat/designs/database.py:        sage: _ = designs.difference_matrix(75,8)                                       # needs sage.rings.finite_rings
src/sage/combinat/designs/database.py:        sage: G,M = DM_273_17_1()                                                       # needs sage.schemes
src/sage/combinat/designs/database.py:        sage: is_difference_matrix(M,G,17,1)                                            # needs sage.schemes
src/sage/combinat/designs/database.py:        sage: _ = designs.difference_matrix(273,17)                                     # needs sage.schemes
src/sage/combinat/designs/database.py:        sage: G,M = DM_993_32_1()                                                       # needs sage.schemes
src/sage/combinat/designs/database.py:        sage: is_difference_matrix(M,G,32,1)                                            # needs sage.schemes
src/sage/combinat/designs/database.py:        sage: _ = designs.difference_matrix(993,32)                                     # needs sage.schemes
src/sage/combinat/designs/database.py:        sage: RBIBD = RBIBD_120_8_1()                                                   # needs sage.modules
src/sage/combinat/designs/database.py:        sage: is_pairwise_balanced_design(RBIBD,120,[8])                                # needs sage.modules
src/sage/combinat/designs/database.py:        sage: for i in range(17):                                                       # needs sage.modules
src/sage/combinat/designs/database.py:        sage: _ = designs.balanced_incomplete_block_design(120,8)                       # needs sage.modules
src/sage/combinat/designs/database.py:        sage: B2 = Hypergraph(BIBD_45_9_8(from_code=True))      # not tested            # needs sage.rings.finite_rings
src/sage/combinat/designs/database.py:        sage: B2.is_isomorphic(B)               # not tested                            # needs sage.rings.finite_rings
src/sage/combinat/designs/database.py:        sage: D = IncidenceStructure(BIBD_79_13_2())                                    # needs sage.libs.gap

So here you see that some of the individual doctests are marked as dependent on additional features. These are the mildest kind of dependencies.

[Matthias Koeppe]

On Tuesday, April 23, 2024 at 10:32:22 AM UTC-7 Dima Pasechnik wrote:

in
src/sage/combinat//designs/block_design.py

you can see

lazy_import('sage.libs.gap.libgap', 'libgap')
lazy_import('sage.rings.finite_rings.finite_field_constructor', 'FiniteField')

$ git blame src/sage/combinat//designs/block_design.py

fdbe7f7e3348 (Matthias Koeppe    2023-07-12 10:53:08 -0700   65) lazy_import('sage.libs.gap.libgap', 'libgap')
fdbe7f7e3348 (Matthias Koeppe    2023-07-12 10:53:08 -0700   66) lazy_import('sage.matrix.matrix_space', 'MatrixSpace')
fdbe7f7e3348 (Matthias Koeppe    2023-07-12 10:53:08 -0700   67) lazy_import('sage.modules.free_module', 'VectorSpace')
fdbe7f7e3348 (Matthias Koeppe    2023-07-12 10:53:08 -0700   68) lazy_import('sage.rings.finite_rings.finite_field_constructor', 'FiniteField')

What you see there is the result of work in the modularization project, using one of the techniques documented in https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#dependencies-and-distribution-packages ("Reducing module-level run-time dependencies:") to reduce dependencies or to make dependencies milder.

[Dima Pasechnik]

On 23 April 2024 18:41:34 BST, Matthias Koeppe <matthia...@gmail.com> wrote:
>On Tuesday, April 23, 2024 at 10:32:22 AM UTC-7 Dima Pasechnik wrote:
>
>in
>src/sage/combinat//designs/block_design.py
>
>you can see
>
>lazy_import('sage.libs.gap.libgap', 'libgap')
>lazy_import('sage.rings.finite_rings.finite_field_constructor',
>'FiniteField')
>
>

>*$ git blame src/sage/combinat//designs/block_design.py*

>
>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 65)
>lazy_import('sage.libs.gap.libgap', 'libgap')
>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 66)
>lazy_import('sage.matrix.matrix_space', 'MatrixSpace')
>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 67)
>lazy_import('sage.modules.free_module', 'VectorSpace')
>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 68)
>lazy_import('sage.rings.finite_rings.finite_field_constructor',
>'FiniteField')
>
>What you see there is the result of work in the modularization project,
>using one of the techniques documented
>in https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#dependencies-and-distribution-packages
>("Reducing module-level run-time dependencies:") to reduce dependencies or
>to make dependencies milder.

The problem is that without these imports, lazy or not, this module is basically useless.


>
>

[Matthias Koeppe]

That's correct, but that's not a "problem". 

It's part of a deliberate design, explained in detail in my 2023-06 sage-devel posts. 

From "Modularization project: IV. The rules" (https://groups.google.com/g/sage-devel/c/ozh-7ZZ848s):

3. Distribution packages declare build dependencies and runtime dependencies (on other Python distribution packages). These two can be entirely unrelated to each other. In addition to the runtime dependencies, it is possible to declare extra dependencies – either those necessary for running tests or for providing advertised extra features. But there is no such thing as an "optional build dependency" or "conditional compilation". The set of modules that form a distribution package is static. There is no place in the name (or metadata) of a wheel package that could indicate different "configurations".

4. As a result of 3, it is possible to create distributions that contain some modules that cannot be imported because some of their dependencies are missing. That's OK; they can become importable simply by the presence of other distributions, in particular those declared as extra dependencies. All of this is discovered at the time of importing a module; there is no ahead-of-time linking step of any sort.

From "Modularization project: V. The blocs" (https://groups.google.com/g/sage-devel/c/kiB32zP3xD4):

We should not attempt to define a distribution package for every possible community or subfield of mathematics that Sage supports.
The proposed design instead introduces 3 types of distribution packages: [...]

Each of the distribution packages can define a list of extras (nicknames for sets of other distribution packages that provide additional advertised features). When using pip to install a distribution, users can use square bracket notation to add (adjoin?) these extras.

[Dima Pasechnik]

On 23 April 2024 19:13:44 BST, Matthias Koeppe <matthia...@gmail.com> wrote:
>On Tuesday, April 23, 2024 at 11:06:12 AM UTC-7 Dima Pasechnik wrote:
>
>
>
>On 23 April 2024 18:41:34 BST, Matthias Koeppe <matthia...@gmail.com>
>wrote:
>>*$ git blame src/sage/combinat//designs/block_design.py*
>>
>>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 65)
>>lazy_import('sage.libs.gap.libgap', 'libgap')
>>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 66)
>>lazy_import('sage.matrix.matrix_space', 'MatrixSpace')
>>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 67)
>>lazy_import('sage.modules.free_module', 'VectorSpace')
>>fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 68)
>>lazy_import('sage.rings.finite_rings.finite_field_constructor',
>>'FiniteField')
>>
>>What you see there is the result of work in the modularization project,
>>using one of the techniques documented
>>in
>https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#dependencies-and-distribution-packages
>>("Reducing module-level run-time dependencies:") to reduce dependencies or
>>to make dependencies milder.
>
>The problem is that without these imports, lazy or not, this module is
>basically useless.
>
>
>That's correct, but that's not a "problem".

It is a problem, because such a module doesn't do what is says on the label.
Basically such a module is a semi-random part of sagelib, and it would not be fully functional until one installs a good chunk, probably well over half, of sagelib.

I am not saying it is totally useless, this sort of partition, but I don't see the point of distributing such pieces, as Python wheels, as something useful on their own.

Application builders - who, according to Marc, are the only ones to get benefits from this design - can just as well build the needed part of sagelib from source, they don't need these barely useful wheels.

>
>It's part of a deliberate design, explained in detail in my 2023-06
>sage-devel posts.
>

>From "*Modularization project: IV. The rules*"
>(https://groups.google.com/g/sage-devel/c/ozh-7ZZ848s):
>*3.* Distribution packages declare build dependencies and runtime
>dependencies
><https://sagemath-tobias.netlify.app/developer/packaging_sage_library.html#dependencies-and-distribution-packages> (on

>other Python distribution packages). These two can be entirely unrelated to
>each other. In addition to the runtime dependencies, it is possible to
>declare extra dependencies – either those necessary for running tests

><https://sagemath-tobias.netlify.app/developer/packaging_sage_library.html#doctest-only-dependencies> or
>for providing advertised
><https://sagemath-tobias.netlify.app/developer/packaging_sage_library.html#other-runtime-dependencies>
>extra
><https://sagemath-tobias.netlify.app/developer/packaging_sage_library.html#other-runtime-dependencies>
>features
><https://sagemath-tobias.netlify.app/developer/packaging_sage_library.html#other-runtime-dependencies>.

>But there is no such thing as an "optional build dependency" or
>"conditional compilation". The set of modules that form a distribution
>package is static. There is no place in the name (or metadata) of a wheel
>package that could indicate different "configurations".
>

>*4. *As a result of 3, it is possible to create distributions that contain

>some modules that cannot be imported because some of their dependencies are
>missing. That's OK; they can become importable simply by the presence of
>other distributions, in particular those declared as extra dependencies.
>All of this is discovered at the time of importing a module; there is no
>ahead-of-time linking step of any sort.
>

>From "*Modularization project: V. The blocs*"
>(https://groups.google.com/g/sage-devel/c/kiB32zP3xD4):
>*We should not attempt to define a distribution package for every possible
>community or subfield of mathematics that Sage supports.*

[Martin R]

Dear Matthias!

You mentioned several times, that discoverability is an important aspect.  Do you have any evidence to support that?

Wouldn't people in the python world who need a serious amount of math know of sage anyway, and then, if they cannot rely on all of sage because that is too large, use, for example, `citation.get_systems` to see whether they can do without some dependencies?

I don't see how splitting the sage math library into pieces helps discoverability.  I admit, however, that I cannot really think of any serious use of any of the functionality of sage outside of math, where people would know that what they need is, say combinatorics or graph theory.  I guess it would be quite insane to install sage or parts of sage to get an implementation of a particular algorithm in a non-math environment.

Martin

[Kwankyu Lee]

Wouldn't people in the python world who need a serious amount of math know of sage anyway, and then, if they cannot rely on all of sage because that is too large, use, for example, `citation.get_systems` to see whether they can do without some dependencies?

 I think they would do `pip install sagemath-graphs` if they need graphs functionality.

[Oscar Benjamin]

On Tue, 23 Apr 2024 at 15:27, Marc Culler <marc....@gmail.com> wrote:
>
> The projects that will really benefit from modularization will be those that provide their own limited mathematical context. Developers of such projects will be able to choose which parts of Sage are relevant to their specific context. Those parts of Sage can be provided for their users without having to embed a huge monolithic environment into a relatively small project.

Is the benefit in this case mainly about reduced disk/network usage?

I could imagine other theoretical benefits like maybe some parts could
be installed natively on Windows or some parts might be easier to
provide binaries for etc.

Are there any indicative numbers for what the size would be when
installing some useful portion of Sage vs installing the whole of
Sage?

--
Oscar

[Dima Pasechnik]

On Wed, Apr 24, 2024 at 2:08 PM Kwankyu Lee <ekwa...@gmail.com> wrote:
>
> Wouldn't people in the python world who need a serious amount of math know of sage anyway, and then, if they cannot rely on all of sage because that is too large, use, for example, `citation.get_systems` to see whether they can do without some dependencies?
>
>
> I think they would do `pip install sagemath-graphs` if they need graphs functionality.

"Too large" is relative. What was "too large" few years ago is normal
now, and RAM only gets cheaper, and 32-bit systems
get irrelevant.

And they'd be often quite unhappy, as e.g. basically nothing in
src/sage/graphs/generators/classical_geometries.py
will work without sage.rings or GAP.
Many other parts of sage.graphs use things in sage.combinat, as well
as cliquer, bliss/nauty, etc.

Dima

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/8dd2c1ce-7811-4ada-94cc-075d5b6ab93bn%40googlegroups.com.

[Marc Culler]

I think that CyPari ;and CyPari2 provide a relevant example.

Some background ... CyPari is a PyPi package with binary wheels which predates and was the starting point for Sage's cypari2 (hence the 2 in the name).   The basis for CyPari was Sage's pari module.  That module was modified to make it independent from the rest of Sage so that Sage's Pari support could be provided as a component of SnapPy. Binary wheels for CyPari are available for Windows, macOS and manylinux.  The current versions of those wheels are statically linked against Pari 2.15.4

The binary wheel for CyPari weighs in at 7MB.  Sage's CyPari2 is available as a binay wheel for macOS and manylinux and the size is of the same order of magnitude.  When they are unpacked and installed the sizes of these python packages are respectively 19MB and 7.5MB (cypari2 is linked against the dynamic libraries libgmp and libpari, which together are about 12MB and which are external to the python package).  A full Sage installation is about 5 gigabytes.  With some significant omissions and compression of the documentation, the macOS app squeezes that down to 3.4GB.

So the size of this one significant self-contained component of sage is about 200 times smaller than the full sage installation and it could be made installable on Windows with some additional work which has already been done for a very closely related package.

- Marc

--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/mqgtkLr2gXY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAHVvXxQUanPY%3D1svhf7Q8xDFD5BroD9wTLRc1-wmFC3vQhBMRg%40mail.gmail.com.

[Dima Pasechnik]

On Wed, Apr 24, 2024 at 3:37 PM Marc Culler <marc....@gmail.com> wrote:
>
> I think that CyPari ;and CyPari2 provide a relevant example.
>
> Some background ... CyPari is a PyPi package with binary wheels which predates and was the starting point for Sage's cypari2 (hence the 2 in the name). The basis for CyPari was Sage's pari module. That module was modified to make it independent from the rest of Sage so that Sage's Pari support could be provided as a component of SnapPy. Binary wheels for CyPari are available for Windows, macOS and manylinux. The current versions of those wheels are statically linked against Pari 2.15.4
>
> The binary wheel for CyPari weighs in at 7MB. Sage's CyPari2 is available as a binay wheel for macOS and manylinux and the size is of the same order of magnitude. When they are unpacked and installed the sizes of these python packages are respectively 19MB and 7.5MB (cypari2 is linked against the dynamic libraries libgmp and libpari, which together are about 12MB and which are external to the python package). A full Sage installation is about 5 gigabytes. With some significant omissions and compression of the documentation, the macOS app squeezes that down to 3.4GB.
>
> So the size of this one significant self-contained component of sage is about 200 times smaller than the full sage installation and it could be made installable on Windows with some additional work which has already been done for a very closely related package.

cypari2 is indeed self-contained (with libpari and libgmp added to the
count), and provides a Pari-Python interface.
This full installation takes about 35 Mb. But it's not a relevant
example, cause it's a very limited functionality.

This is unlike sagemath-graphs, etc, which are, as I wrote, very far
from being self-contained. Many of them, with all of its dependencies
installed, will be blow up to a good half of the whole Sage, if not
more.
So there are no 10000% savings on bandwidth here, more like 30%-50%, or less.

Dima

>
> - Marc
>
>
> On Wed, Apr 24, 2024 at 8:48 AM Oscar Benjamin <oscar.j....@gmail.com> wrote:
>>
>> On Tue, 23 Apr 2024 at 15:27, Marc Culler <marc....@gmail.com> wrote:
>> >
>> > The projects that will really benefit from modularization will be those that provide their own limited mathematical context. Developers of such projects will be able to choose which parts of Sage are relevant to their specific context. Those parts of Sage can be provided for their users without having to embed a huge monolithic environment into a relatively small project.
>>
>> Is the benefit in this case mainly about reduced disk/network usage?
>>
>> I could imagine other theoretical benefits like maybe some parts could
>> be installed natively on Windows or some parts might be easier to
>> provide binaries for etc.
>>
>> Are there any indicative numbers for what the size would be when
>> installing some useful portion of Sage vs installing the whole of
>> Sage?
>>
>> --
>> Oscar
>>
>> --
>> You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/mqgtkLr2gXY/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAHVvXxQUanPY%3D1svhf7Q8xDFD5BroD9wTLRc1-wmFC3vQhBMRg%40mail.gmail.com.
>
> --

> You received this message because you are subscribed to the Google Groups "sage-devel" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CALcZXREFctqVF8Hr1B%2Bmz9WfhV9Dspop5EmWN7Q%2BsYdJLvnh4w%40mail.gmail.com.

[Oscar Benjamin]

On Wed, 24 Apr 2024 at 15:37, Marc Culler <marc....@gmail.com> wrote:
>
> I think that CyPari ;and CyPari2 provide a relevant example.
>
> Some background ... CyPari is a PyPi package with binary wheels which predates and was the starting point for Sage's cypari2 (hence the 2 in the name). The basis for CyPari was Sage's pari module. That module was modified to make it independent from the rest of Sage so that Sage's Pari support could be provided as a component of SnapPy. Binary wheels for CyPari are available for Windows, macOS and manylinux. The current versions of those wheels are statically linked against Pari 2.15.4
>
> The binary wheel for CyPari weighs in at 7MB. Sage's CyPari2 is available as a binay wheel for macOS and manylinux and the size is of the same order of magnitude. When they are unpacked and installed the sizes of these python packages are respectively 19MB and 7.5MB (cypari2 is linked against the dynamic libraries libgmp and libpari, which together are about 12MB and which are external to the python package). A full Sage installation is about 5 gigabytes. With some significant omissions and compression of the documentation, the macOS app squeezes that down to 3.4GB.
>
> So the size of this one significant self-contained component of sage is about 200 times smaller than the full sage installation and it could be made installable on Windows with some additional work which has already been done for a very closely related package.

Thanks Marc. This seems like a good example of a useful part of Sage
that can be extracted to something much smaller than Sage.

Presumably though a hypothetical person who wants CyPari2 but not all
of Sage can already just use CyPari so that person is already well
served. Is the plan that CyPari2 would effectively replace CyPari?
Then the benefit is not needing to maintain CyPari separately from
sagelib's CyPari2 dependency?

I am wondering how representative the CyPari case is compared to other
parts of Sage. There seems to be some disagreement about what parts of
Sage would go where but not much discussion about what it means in
terms of disk footprint, portability etc i.e. the factors that make
the modularisation useful (maybe I misunderstand what the intended
benefits are).

I know that a full Sage install takes 5GB but I have no real
understanding of what it is that takes up all that space or how that
can break down into usable pieces with a smaller footprint.

At least disk footprint is something that can be quantified though:
what is the disk usage of different virtual environments that contain
some useful collection of parts of Sage?

--
Oscar

[Marc Culler]

On Wed, Apr 24, 2024 at 2:26 PM Oscar Benjamin <oscar.j....@gmail.com> wrote:

Presumably though a hypothetical person who wants CyPari2 but not all of Sage can already just use CyPari so that person is already well served.

That hypothetical person could also use CyPari2 if they didn't care about memory leaks and they were not running Windows.

I am wondering how representative the CyPari case is compared to other parts of Sage.

That is a good question.  Dima says it is not representative.

I think there are different types of hypothetical people: users and developers.  Users should just install all of sage.  But one copy of a software package that size should be enough.  Developers producing smaller more specialized packages would benefit from being able to embed or require as dependencies some parts of Sage, without having to turn a modestly sized package into a monster.

- Marc

[Nathan Dunfield]

On Wednesday, April 24, 2024 at 2:26:37 PM UTC-5 Oscar Benjamin wrote:

Thanks Marc. This seems like a good example of a useful part of Sage
that can be extracted to something much smaller than Sage.

Presumably though a hypothetical person who wants CyPari2 but not all
of Sage can already just use CyPari so that person is already well
served.

Correct.

 

Is the plan that CyPari2 would effectively replace CyPari? Then the benefit is not needing to maintain CyPari separately from
sagelib's CyPari2 dependency?

No, rather CyPari is an example of the sort of free-standing Python package that could be extracted from Sage.  Modularization would create more of these, in varying sizes and levels of granularity.

Some level of modularization is necessary to address what William Stein described last year as:

The fundamental and massive problem that I think SageMath has is that it is not part of the  Python ecosystem,

by which I mean that there is no good way to do "pip install sagemath-[foo]", in sufficient generality.

PROBLEM: SageMath is not part of the Python ecosystem.

DEFINITION: A piece of software is part of the Python ecosystem, if you can do "pip install <software-name>" on

basically the same platforms as the intersection of where you can install scipy/numpy/matplotlib/pandas,

and with somewhat comparable resource usage (i.e., installing Sagemath can't use 100x of the time/space of

the above, as that would be unfair).

On a related note, the reason that CyPari2 and CyPari are still separate relates to what Marc mentioned earlier about tension between two models of installing software: the "Linux distro way" using a system-level package manager (where there is typically only one version of any given library on the system) and the "Python pip" way (where all needed libraries are statically linked into the wheel).  CyPari2 follows the first, CyPari the second.  (This story is further complicated by the fact that, from a user's perspective, conda is like "Python pip" in that it is orthogonal to any system libraries, but developing packages for conda is akin to building them for a Linux distro.)

Of course, most projects in the scientific Python community support both models, but there is technical overhead in doing so, which I believe is the root of some of the current conflict.

Best,

Nathan

[Dima Pasechnik]

There isn't a big problem to set up a GitHub wheel builder for
CyPari2, so there is not really a sea of difference here in this
sense.

Also, probably it should be mentioned that linux distro way/pip way
can be very nicely combined using a python venv.
So one can use system packages, but add up more packages, and, if
needed, override system packages with other versions.

>
> Of course, most projects in the scientific Python community support both models, but there is technical overhead in doing so, which I believe is the root of some of the current conflict.
>
> Best,
>
> Nathan
>

> --

> You received this message because you are subscribed to the Google Groups "sage-devel" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/60a60056-f8e9-417e-b03a-1dcfcbc3c6ebn%40googlegroups.com.

[Nathan Dunfield]

On Wednesday, April 24, 2024 at 4:25:41 PM UTC-5 Dima Pasechnik wrote:

On Wed, Apr 24, 2024 at 9:29 PM Nathan Dunfield wrote:
> On a related note, the reason that CyPari2 and CyPari are still separate relates to what Marc mentioned earlier about tension between two models of installing software: the "Linux distro way" using a system-level package manager (where there is typically only one version of any given library on the system) and the "Python pip" way (where all needed libraries are statically linked into the wheel). CyPari2 follows the first, CyPari the second. (This story is further complicated by the fact that, from a user's perspective, conda is like "Python pip" in that it is orthogonal to any system libraries, but developing packages for conda is akin to building them for a Linux distro.)

There isn't a big problem to set up a GitHub wheel builder for
CyPari2, so there is not really a sea of difference here in this
sense.

Dima,

Good point, I had forgotten that Matthias did exactly that when he released the most recent CyPari2:

https://github.com/sagemath/cypari2/blob/master/.github/workflows/dist.yml

and the binary wheels for CyPari2 on PyPI are indeed statically linked and self-contained.  So CyPari2 is a good example of a Python package the fully supports both modes.  (Unlike CyPari, which only supports "Python pip", though unlike CyPari2 it does work on Windows.)

 

Also, probably it should be mentioned that linux distro way/pip way
can be very nicely combined using a python venv.
So one can use system packages, but add up more packages, and, if
needed, override system packages with other versions.

Agreed!

Best,

Nathan

[Matthias Koeppe]

On Wednesday, April 24, 2024 at 1:07:44 AM UTC-7 Martin R wrote:

You mentioned several times, that discoverability is an important aspect.  Do you have any evidence to support that?

I mentioned "discoverability" in the context of how I have named the distributions.

 

Wouldn't people in the python world who need a serious amount of math know of sage anyway,

What is a "serious" amount of math?

and then, if they cannot rely on all of sage because that is too large, use, for example, `citation.get_systems` to see whether they can do without some dependencies?

What do you mean by, "whether they can do without some dependencies"?

That's exactly the point of the modularization: 

- To enable people to use parts of Sage without some [actually, most!] of the dependencies. 

[...] I admit, however, that I cannot really think of any serious use of any of the functionality of sage outside of math, where people would know that what they need is, say combinatorics or graph theory.

What do you mean by "outside of math"?

 

  I guess it would be quite insane to install sage or parts of sage to get an implementation of a particular algorithm in a non-math environment.

Yes, "quite insane" would be a good description of doing this (reusing what is implemented in Sage) with the status quo, the monolithic Sage.

That's exactly the point. 

- From the viewpoint of users: It just makes no sense to use a small part of Sage: Because of space and time and friction (and limited portability).

- From the viewpoint of contributors of new code that could be reused: It just makes no sense to trap it in Sage, where it cannot be reused sanely.

Modularization makes it at least "less insane", or perhaps "kind of acceptable", or sometimes even "reasonable".

Any of these degrees of improvement of "sanity" would already be a win.

[Matthias Koeppe]

On Wednesday, April 24, 2024 at 6:48:30 AM UTC-7 Oscar Benjamin wrote:

Is the benefit in this case mainly about reduced disk/network usage?

I could imagine other theoretical benefits like maybe some parts could
be installed natively on Windows or some parts might be easier to
provide binaries for etc.

Oscar, thanks for bringing up these important points.

Yes, the modularization -- namely the ability to separately build and run and test portions of the Sage library -- also facilitates porting to new platforms in several ways.

- If nothing else, because it enables an incremental approach to porting: I don't have to first make the whole thing build before I can run and test a portion.

- More importantly, it varies very much by the dependency packages how hard it is to port them to a new platform; and how much *interest* in the developer community of these packages there is in porting.

Yes, native Windows would clearly be a very important target.

In another direction: I have started a port of Sage to pyodide, the distribution of Python for WebAssembly (WASM), which makes Python runnable directly in the browser. I can already run and test the modularized distributions **sagemath-objects**, **sagemath-categories** there. 

See https://github.com/sagemath/sage/wiki/Sage-10.3-Release-Tour#porting-to-the-pyodide-web-assembly-platform

As I wrote there: "Several upstream projects that Sage uses already have ongoing Pyodide or WASM/Emscripten efforts: The SymPy Live Shell and the interactive shell on the NumPy.org front page use JupyterLite, running Pyodide in a Web Worker in the browser. Also PARI/GP runs in the browser using a WASM/Emscripten port. Giac also has an Emscripten port."

[Martin R]

On Thursday 25 April 2024 at 05:13:37 UTC+2 Matthias Koeppe wrote:

On Wednesday, April 24, 2024 at 1:07:44 AM UTC-7 Martin R wrote:

You mentioned several times, that discoverability is an important aspect.  Do you have any evidence to support that?

I mentioned "discoverability" in the context of how I have named the distributions.

Sorry that my question was not clear enough.  Do you have evidence, that this naming enhances discoverability, and that this enhanced discoverability would be worthwhile, since it comes with a cost (as outlined above)?

 

Wouldn't people in the python world who need a serious amount of math know of sage anyway,

What is a "serious" amount of math?

You know it when you see it.  What I mean is, roughly, that it certainly does not make sense to use sage as a package if you need a few graph algorithms (like shortest paths or some such), because then you'd be better served with using a specialised library or perhaps copy the code and adapt it. You might want to use sage as a package, if you want to do serious enumeration of arbitrary combinatorial objects.  But then you will need gap, symmetrica, nauty and maxima or fricas anyway.

 

and then, if they cannot rely on all of sage because that is too large, use, for example, `citation.get_systems` to see whether they can do without some dependencies?

What do you mean by, "whether they can do without some dependencies"?

 

That's exactly the point of the modularization: 

- To enable people to use parts of Sage without some [actually, most!] of the dependencies.

The only point I am critical of is the splitting of the math library into arbitrary pseudo-mathematical parts (i.e., sage-combinat ... sage-symbolics, as listed above), which has nothing to do with dependencies.

 

[...] I admit, however, that I cannot really think of any serious use of any of the functionality of sage outside of math, where people would know that what they need is, say combinatorics or graph theory.

What do you mean by "outside of math"?

I agree that my terminology is not good.  I tried to make a distinction between research involving math and the - to me unknown - rest.  I find it hard to imagine that any mathematician would bother installing anything else but all of sage.  So, maybe I should have written: "I cannot think of .... sage in an environment, where dependencies are show-stopper, ...".

  I guess it would be quite insane to install sage or parts of sage to get an implementation of a particular algorithm in a non-math environment.

Yes, "quite insane" would be a good description of doing this (reusing what is implemented in Sage) with the status quo, the monolithic Sage.

That's exactly the point. 

- From the viewpoint of users: It just makes no sense to use a small part of Sage: Because of space and time and friction (and limited portability).

However, it *does* make sense to copy small parts of sage.  If I need an implementation of some combinatorial algorithm, chances are extremely good that I can simply use ?? to go to the code, and copy the relevant part.

 

- From the viewpoint of contributors of new code that could be reused: It just makes no sense to trap it in Sage, where it cannot be reused sanely.

 

I doubt this.  Do you have any evidence for that?

Modularization makes it at least "less insane", or perhaps "kind of acceptable", or sometimes even "reasonable".

Any of these degrees of improvement of "sanity" would already be a win.

I disagree. "less insane" or "kind of acceptable" would be a loss, not a win, since the kind of modularization discussed in this message comes with a big cost.

Martin

[Nathan Dunfield]

On Wednesday, April 24, 2024 at 10:14:09 PM UTC-5 Matthias Koeppe wrote:

Yes, native Windows would clearly be a very important target.

As a data point, downloads of our stand-alone SnapPy app, which is about as high level pure math as it gets, are 60% higher for Windows than macOS.

 

In another direction: I have started a port of Sage to pyodide, the distribution of Python for WebAssembly (WASM), which makes Python runnable directly in the browser. I can already run and test the modularized distributions **sagemath-objects**, **sagemath-categories** there.

It would be amazing if a decent portion of Sage could be run in the browser this way, e.g. to have the occasional HW assignment that needs Sage without the overhead of using something like CoCalc. 

Best,

Nathan

 

[Nathan Dunfield]

On Thursday, April 25, 2024 at 2:17:31 AM UTC-5 Martin R wrote:

I agree that my terminology is not good.  I tried to make a distinction between research involving math and the - to me unknown - rest.  I find it hard to imagine that any mathematician would bother installing anything else but all of sage. 

As mentioned upthread, CyPari is one of the few examples of something that's been modularized out of Sage.  While it's small compared to Sage, it can still do everything Pari can, which is a lot.   Marc and I broke out CyPari so we could use it in SnapPy (https://snappy.computop.org), whose users are, at a guess, 90% mathematicians, 9% physicists,  1% other.   The most recent version stand-alone (Sage-free) version of SnapPy has been downloaded 1,200 times on Windows and macOS.  That's a lot of mathematicians who are already installing only part of Sage.

Another example is large-scale pure math computation on clusters.  Because of Sage's size and the nature of distributive file systems, the time to startup Sage can be 30 seconds or more, which complicates things if you want to do 100,000 calculations that are only 10 seconds each.  I was recently at a workshop on computational topology, and several researchers there regarded using Sage in this context as a non-starter, in one case they were completely changing their approach to avoid using Sage. 

Best,

Nathan

[Dima Pasechnik]

On Thu, Apr 25, 2024 at 1:28 PM Nathan Dunfield <nat...@dunfield.info> wrote:
>
> On Wednesday, April 24, 2024 at 10:14:09 PM UTC-5 Matthias Koeppe wrote:
>
> Yes, native Windows would clearly be a very important target.
>
>
> As a data point, downloads of our stand-alone SnapPy app, which is about as high level pure math as it gets, are 60% higher for Windows than macOS.

That's not for native Windows, that's for WSL, right?

Essential components of sagelib such as GAP, Singular, don't run on
native Windows (on Cygwin, yes, but
we know by now, Cygwin is too flaky for Sage to work) and I don't
think anyone is keen on
doing the hard work to port it.

This puts native Windows support into the area of wishful thinking.

Dima

[Martin R]

Another example is large-scale pure math computation on clusters.  Because of Sage's size and the nature of distributive file systems, the time to startup Sage can be 30 seconds or more, which complicates things if you want to do 100,000 calculations that are only 10 seconds each.

I agree that this an extremely important point.

Martin

[Marc Culler]

On Thu, Apr 25, 2024 at 8:28 AM Dima Pasechnik <dim...@gmail.com> wrote:

On Thu, Apr 25, 2024 at 1:28 PM Nathan Dunfield <nat...@dunfield.info> wrote:
>
> On Wednesday, April 24, 2024 at 10:14:09 PM UTC-5 Matthias Koeppe wrote:
>
> Yes, native Windows would clearly be a very important target.
>
>
> As a data point, downloads of our stand-alone SnapPy app, which is about as high level pure math as it gets, are 60% higher for Windows than macOS.

That's not for native Windows, that's for WSL, right?

Wrong!  SnapPy runs as a native Windows app.

We build Pari with msys2 and mingw on a CI runner.

Essential components of sagelib such as GAP, Singular, don't run on
native Windows (on Cygwin, yes, but
we know by now, Cygwin is too flaky for Sage to work) and I don't
think anyone is keen on
doing the hard work to port it.

Msys2 and mingw are not so flaky, and use more or less the same toolchain as Cygwin.  There are many ports of gnu libraries to msys2 which can be installed with their package manage (pacman).  I am not convinced that GAP, Singuler, etc. could not be built with mingw.

- Marc

[Dima Pasechnik]

E.g. GAP uses sbrk, fork (which are not in msys2/mingw), you'll need to port their GC too.
Doable, but very, very far from easy.



>
>- Marc
>

[Marc Culler]

On Thursday, April 25, 2024 at 8:28:48 AM UTC-5 Dima Pasechnik wrote:

Essential components of sagelib such as GAP, Singular, don't run on
native Windows

I was amused to find the following statement on the GAP forum from 2005:

  >  While porting GAP to use native Win32 calls is doable, basically src/system.c is the only place

   > that needs lots of changes, it is certainly a nontrivial and time-consuming task. (and one needs

   > to be a bit of an expert in programming to do this, IMHO)

The author was someone from the Netherlands by the name of Dima Pasechnik.  :^)

- Marc 

[Marc Culler]

Another amusing quote, this time from the sbrk man page on macOS:

> The brk and sbrk functions are historical curiosities left over from
 > earlier days before the advent of virtual memory management.

That seems to be a paraphrase of the FreeBSD man page, which says:

> The brk() and sbrk() functions are legacy interfaces from before the advent of modern virtual
> memory management. They are deprecated and not present on the arm64 or riscv architectures.

> The mmap(2) interface should be used to allocate pages instead.

Given that GAP runs on arm64, I suspect that it doesn't use sbrk in an essential way anymore.  Of course fork is a different story.  But there are surely lots of examples of programs that use fork which have been ported to Windows.  I would guess that emacs is among them.  (It has been ported to Windows - I assume it uses fork on posix systems).

- Marc

[Marc Culler]

The GAP project provides a native Windows installer.  So, evidently, it is possible to build GAP for Windows.  They do not seem to provide build instructions for Windows, however.

- Marc

[dim...@gmail.com]

On Thu, Apr 25, 2024 at 08:09:48AM -0700, Marc Culler wrote:
> The GAP project provides a native Windows installer

> <https://github.com/gap-system/gap/releases/download/v4.13.0/gap-4.13.0-x86_64.exe>.

> So, evidently, it is possible to build GAP for Windows. They do not seem
> to provide build instructions for Windows, however.

it uses Cygwin, packaged in.

Dima

[dim...@gmail.com]

On Thu, Apr 25, 2024 at 07:03:29AM -0700, Marc Culler wrote:
> On Thursday, April 25, 2024 at 8:28:48 AM UTC-5 Dima Pasechnik wrote:
>
> Essential components of sagelib such as GAP, Singular, don't run on
> native Windows
>
>
> I was amused to find the following statement on the GAP forum

> <https://www.gap-system.org/ForumArchive2/2005/000999.html> from 2005:

>
> > While porting GAP to use native Win32 calls is doable, basically
> src/system.c is the only place
> > that needs lots of changes, it is certainly a nontrivial and
> time-consuming task. (and one needs
> > to be a bit of an expert in programming to do this, IMHO)
>

> The author was someone from the Netherlands by the name of *Dima
> Pasechnik. :^)*

It was me, yes. And I used to know from what end
you have to approach a Windows machine. :-)
But not to the point of knowing exactly how to change fork() and sbrk(),
(and mmap()) into whatever functions with 15 arguments you have to use
on Win32 as their replacements (they already have about 10 versions of
spawn to use in place of fork).

Note that since 2005 GAP has changed quite a bit, too.
They made a go at making it multithreaded (HPC GAP), and that made code
harder to deal with (HPC GAP is still beta).
Instead of GAP's native GC (with its sbrk/mmap), HPC GAP uses Boehm GC, which does run on native
Windows. But it's a beta...

Oh, and someone died porting GAP to Windows, some years ago.

Dima


>
> - Marc

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/b0784b4b-ab38-4ff7-b5f7-d9cc47472b95n%40googlegroups.com.

[TB]

On 25/04/2024 15:28, Nathan Dunfield wrote:

In another direction: I have started a port of Sage to pyodide, the distribution of Python for WebAssembly (WASM), which makes Python runnable directly in the browser. I can already run and test the modularized distributions **sagemath-objects**, **sagemath-categories** there.

It would be amazing if a decent portion of Sage could be run in the browser this way, e.g. to have the occasional HW assignment that needs Sage without the overhead of using something like CoCalc. 

Although SageMathCell does not run locally, it does run in the browser. There are examples of Sage exercises in this book and more on the about page of SageMathCell. Having a completely offline version of parts of Sage that can run in the browser with WASM will be wonderful indeed.

Regards,

TB

[Dima Pasechnik]

On Thu, Apr 25, 2024 at 4:36 PM <dim...@gmail.com> wrote:
>
> On Thu, Apr 25, 2024 at 07:03:29AM -0700, Marc Culler wrote:
> > On Thursday, April 25, 2024 at 8:28:48 AM UTC-5 Dima Pasechnik wrote:
> >
> > Essential components of sagelib such as GAP, Singular, don't run on
> > native Windows
> >
> >
> > I was amused to find the following statement on the GAP forum
> > <https://www.gap-system.org/ForumArchive2/2005/000999.html> from 2005:

you might also like to know that in 2000 I asked whether we can have libgap :P
https://www.gap-system.org/ForumArchive/Pasechni.1/Dmitrii.1/using_GA.1/1.html

my 1st message to GAP Forum was sent in 1993:
https://www.gap-system.org/ForumArchive/Pasechni.1/Dmitrii.1/Re__brea.1/1.html

[Marc Culler]

On Fri, Apr 26, 2024 at 8:02 AM Dima Pasechnik <dim...@gmail.com> wrote:

On Thu, Apr 25, 2024 at 4:36 PM <dim...@gmail.com> wrote:

you might also like to know that in 2000 I asked whether we can have libgap :P

https://www.gap-system.org/ForumArchive/Pasechni.1/Dmitrii.1/using_GA.1/1.htmlIt

Thank you!  It is a struggle to convince people to make software reusable.  We are still struggling, but I do think there has been progress since the turn of the century.

- Marc

[Matthias Koeppe]

Hi Sage developers, 

Since I posted my request to urgently vote on the modularization PRs, the big revert (https://github.com/sagemath/sage/pull/37796) was merged into Sage 10.4.beta4.

The modularization PRs have now been re-created (thanks, Julian, for your help with this). 

I'm now asking you to vote on the new PRs; it's important – participation matters!

- https://github.com/sagemath/sage/pull/37900 (Restructure sage.*.all for modularization, replace relative by absolute imports). (As I explained, the PR is "mostly harmless": There are no user-visible changes; it's just a bunch of imports that are moved around. It includes no policy change of any kind; it only executes a design that was previously reviewed and carefully documented in separate PRs. Nothing permanent or irreversible is done here. The new files provide the top-level namespaces needed for doctesting modularized installations of Sage.)

- https://github.com/sagemath/sage/pull/37901 (Add # sage_setup: distribution directives to all files, remove remaining # coding: utf-8).

I'm responding to a few messages that were posted here in this sage-devel thread in the meantime and I did not have a chance to respond to earlier.

On Thursday, April 25, 2024 at 6:28:48 AM UTC-7 Dima Pasechnik wrote:

> On Wednesday, April 24, 2024 at 10:14:09 PM UTC-5 Matthias Koeppe wrote:
> Yes, native Windows would clearly be a very important target.

Essential components of sagelib such as GAP, Singular, don't run on native Windows (on Cygwin, yes [...]) and I don't think anyone is keen on doing the hard work to port it. This puts native Windows support into the area of wishful thinking.

Yes, porting software to new platforms is hard (thanks all for the detailed and entertaining discussion regarding GAP). 

But Dima's message is ignoring the very point of why we are talking about porting to new platforms in this thread: 

The modularization project enables us to port those parts of Sage to new platforms for which there is interest to port, without being held back by those parts and libraries for which porting is too hard or in which there is no interest.

On Thursday, April 25, 2024 at 5:00:33 PM UTC-7 TB wrote:

On 25/04/2024 15:28, Nathan Dunfield wrote:

In another direction: I have started a port of Sage to pyodide, the distribution of Python for WebAssembly (WASM), which makes Python runnable directly in the browser. I can already run and test the modularized distributions **sagemath-objects**, **sagemath-categories** there.

It would be amazing if a decent portion of Sage could be run in the browser this way, e.g. to have the occasional HW assignment that needs Sage without the overhead of using something like CoCalc. 

Although SageMathCell does not run locally, it does run in the browser. There are examples of Sage exercises in this book and more on the about page of SageMathCell. Having a completely offline version of parts of Sage that can run in the browser with WASM will be wonderful indeed.

Yes, pyodide will enable running portions of Sage completely offline, i.e. in serverless mode. There is currently a lot of momentum in the scientific computing community for developing such deployments, see for example the post https://blog.pyodide.org/posts/marimo/ on the port of the (very impressive!) reactive Python notebook marimo to pyodide.

On Thursday, April 25, 2024 at 5:45:33 AM UTC-7 Nathan Dunfield wrote:

Another example is large-scale pure math computation on clusters.  Because of Sage's size and the nature of distributive file systems, the time to startup Sage can be 30 seconds or more, which complicates things if you want to do 100,000 calculations that are only 10 seconds each.  I was recently at a workshop on computational topology, and several researchers there regarded using Sage in this context as a non-starter, in one case they were completely changing their approach to avoid using Sage. 

Indeed, starting up sage (when installed from a shared volume) not only takes long, but it also incurs a huge load on the cluster's network from network file system operations for loading all the Python modules. This can degrade other jobs' performance. (My student's jobs using Sage on our HPC cluster were occasionally flagged by the admins for this.)

 

On Thursday, April 25, 2024 at 12:17:31 AM UTC-7 Martin R wrote:

On Thursday 25 April 2024 at 05:13:37 UTC+2 Matthias Koeppe wrote:

On Wednesday, April 24, 2024 at 1:07:44 AM UTC-7 Martin R wrote:

You mentioned several times, that discoverability is an important aspect.  Do you have any evidence to support that?

I mentioned "discoverability" in the context of how I have named the distributions.

Sorry that my question was not clear enough.  Do you have evidence, that this naming enhances discoverability, and that this enhanced discoverability would be worthwhile, since it comes with a cost (as outlined above)?

You are asking about the enhanced discoverability compared to what? 

I'm not aware of an alternative proposal how to name the distributions.

 

Wouldn't people in the python world who need a serious amount of math know of sage anyway,

What is a "serious" amount of math?

You know it when you see it.

What I mean is, roughly, that it certainly does not make sense to use sage as a package if you need a few graph algorithms (like shortest paths or some such), because then you'd be better served with using a specialised library

There's a false dichotomy regarding size happening here. 

There's a huge "dynamic range" between the size of "a few algorithms" and the size of the whole Sage library (in particular, when considering the dependencies). And everything that's interesting is happening between these two extremes. I have designed the modularized distributions to provide a meaningful and practical compromise regarding the granularity: A relatively small number of distributions, each with few dependencies, each not too big, and each with a clear and easy to explain scope and purpose.

or perhaps copy the code and adapt it.

Well, copying parts of the source code is certainly some kind of reusability, and it is allowed by our license. But it's the lowest grade of reusability and is well-known to be problematic for maintainability etc. 

In the modularization project, I am aiming for a higher grade of reusability; one that places Sage within best practices for reusable software in the Python "ecosystem".

You might want to use sage as a package, if you want to do serious enumeration of arbitrary combinatorial objects.  But then you will need gap, symmetrica, nauty and maxima or fricas anyway.

Yes, there exist use cases that use more libraries/dependencies than other use cases. That's fine (you just install some more of the modularized distributions), but the existence of examples with many dependencies should not hold back the benefits of the modularization for the use cases that need few dependencies. (Even the example that you gave only depends on a rather small fraction of the whole Sage library and its dependencies.)

 

and then, if they cannot rely on all of sage because that is too large, use, for example, `citation.get_systems` to see whether they can do without some dependencies?

What do you mean by, "whether they can do without some dependencies"?

 

That's exactly the point of the modularization: 

- To enable people to use parts of Sage without some [actually, most!] of the dependencies.

The only point I am critical of is the splitting of the math library into arbitrary pseudo-mathematical parts (i.e., sage-combinat ... sage-symbolics, as listed above), which has nothing to do with dependencies.

Martin, first of all, it's not "arbitrary" and it's certainly not "pseudo-"anything. 

The splitting is the result of my careful and detailed work over the years, which included going through the entirety of the Sage library line by line.

You asked before about the dependencies; and I explained them in detail in https://groups.google.com/g/sage-devel/c/mqgtkLr2gXY/m/xzBiXoHbAAAJ and https://groups.google.com/g/sage-devel/c/mqgtkLr2gXY/m/bVzb0CYaBQAJ (which you responded to by thanking me for the list).

I'm not sure what your declaration that the mathematical parts have "nothing to do with dependencies" could possibly be doing in this discussion after this.

 

[...] I admit, however, that I cannot really think of any serious use of any of the functionality of sage outside of math, where people would know that what they need is, say combinatorics or graph theory.

What do you mean by "outside of math"?

I agree that my terminology is not good.  I tried to make a distinction between research involving math and the - to me unknown - rest.  I find it hard to imagine that any mathematician would bother installing anything else but all of sage.

Ever since the symbolic calculus facilities were added to SAGE in 2005(?), the project has not been one that is purely focused on mathematical research. 

Just look at the mission statement of Sage, "Creating a Viable Open Source Alternative to Magma, Maple, Mathematica, and MATLAB". Let's ignore for a moment whether and how it should be updated. It certainly does not say, "Creating a Viable Open Source Alternative to mathematicians' use of Magma, Maple, Mathematica, and MATLAB".

But I am glad that you brought up this viewpoint because there's indeed something substantial to discuss. For example, one of the weakest points of Sage, the maintenance status of our symbolic calculus system, is arguably like this because of the current self-isolation of the Sage project in a community of developers focused on research mathematics unrelated to symbolic calculus. Similar effects of self-isolation can be noted in many other parts of our codebase.

  So, maybe I should have written: "I cannot think of .... sage in an environment, where dependencies are show-stopper, ...".

That's OK, as nothing in the modularization project will force anyone do change how they use Sage.

And with the confidence of the developer who has done the bulk of the modularization work in the Sage library, I can also say that the cost for Sage developers to work with the modularized Sage library is very low and is far outweighed by the opportunities that the modularization provides for our community.