[debian-science] Modularized sagemath packages: proof of concept

[Matthias Koeppe]

https://lists.debian.org/debian-science/2024/07/msg00023.html

[Kwankyu Lee]

That is an encouraging news.

To speed up the process, we have to pass

https://github.com/sagemath/sage/pull/37900

https://github.com/sagemath/sage/pull/37901

(by the way, that is a different Tobias :-)

[Matthias Koeppe]

On Monday, August 26, 2024 at 9:55:38 PM UTC-7 Kwankyu Lee wrote:

That is an encouraging news.

To speed up the process, we have to pass

https://github.com/sagemath/sage/pull/37900

https://github.com/sagemath/sage/pull/37901

Actually, the next step is to merge https://github.com/sagemath/sage/pull/36380

(Demote brial (= polybori) from standard to optional, add distribution sagemath-brial, enlarge sagemath-objects, sagemath-categories).

This PR is yet another one that went into "disputed" status.

All, please vote to get this important step unstuck.

 

[Dima Pasechnik]

This PR bundles together totally unrelated things.

What do sagemath-objects and sagemath-categories have to do with demoting brial to optional?

Dima

[Matthias Koeppe]

On Friday, September 27, 2024 at 1:02:50 PM UTC-7 Dima Pasechnik wrote:

This PR bundles together totally unrelated things.

No, it does not.

 

What do sagemath-objects and sagemath-categories have to do with demoting brial to optional?

sagemath-objects and sagemath-categories are basic distributions that are at the bottom of the dependency graph (see diagram in https://github.com/sagemath/sage/pull/35095).

Now the distribution sagemath-brial depends on a larger set of extension modules than what is currently shipped in sagemath-objects and sagemath-categories. (What is currently shipped is a set of modules that was ready as of Sage 9.6; https://github.com/sagemath/sage/issues/29865).

Hence I have set the scope of this PR to enlarge sagemath-objects and sagemath-categories to be more useful. This includes but is not limited to be suitable as a build dependency for sagemath-brial.

[Martin R]

I don't understand this PR.  The diagram you link to indicates that sagemath-categories is almost at the bottom, whereas sagemath-symbolics is almost at the top of the hierarchy that you propose.  However, the PR includes a file

src/sage/calculus/all__sagemath_categories.py

from sage.calculus.functional import diff, derivative, expand, simplify, taylor

and many similar files, e.g. src/sage/combinat/all__sagemath_categories.py, src/sage/groups/all__sagemath_groups.py, importing things like
from sage.groups.generic import (discrete_log, discrete_log_rho, discrete_log_lambda, linear_relation, multiple, multiples, order_from_multiple)

All of these are then imported by src/sage/all__sagemath_categories.py

I don't think that I understand the purpose of sagemath_categories.

Martin

[Dima Pasechnik]

If this is the case then the brial related PR ought to come after the sagemath-objects and -categories stuff PR. (Or maybe more than one PR).

It would be easier for reviewers, and moreover easier to find reviewers.

[Kwankyu Lee]

On Saturday, September 28, 2024 at 6:25:07 AM UTC+9 axio...@yahoo.de wrote:

... All of these are then imported by src/sage/all__sagemath_categories.py

I don't think that I understand the purpose of sagemath_categories.

Do you know the purpose of files like `all__xxx`? The purpose of these files and the purpose of the distribution package sagemath_categories are distinct.

I wonder which of them you don't understand.

[Kwankyu Lee]

Everyone here,

According to the recent survey, people complain lengthy technical discussions happening in sage-devel, that many of them find irrelevant to their interests.

If a thread in sage-devel involves a technical discussion, how about opening a new Discussion in https://github.com/sagemath/sage/discussions/categories/general

and just leave a link here?

[Dima Pasechnik]

Just to add that the PR as it is now changes over 100 files, and about 2000 lines of code.
And it goes about 4 different topics.

Thanks, Martin, for taking time looking into it. It seems that it puts about 50% or more of Sage into sagemath-categories, cause it includes SR (with calculus), polynomial rings, etc. One may ask whether it's worthwhile to have at all in this form.

Dima

[Matthias Koeppe]

On Friday, September 27, 2024 at 2:25:07 PM UTC-7 axio...@yahoo.de wrote:

The diagram you link to indicates that sagemath-categories is almost at the bottom, whereas sagemath-symbolics is almost at the top of the hierarchy that you propose.

That's right.

 

  However, the PR includes a file

src/sage/calculus/all__sagemath_categories.py

from sage.calculus.functional import diff, derivative, expand, simplify, taylor

and many similar files, e.g. src/sage/combinat/all__sagemath_categories.py, src/sage/groups/all__sagemath_groups.py, importing things like
from sage.groups.generic import (discrete_log, discrete_log_rho, discrete_log_lambda, linear_relation, multiple, multiples, order_from_multiple)

All of these are then imported by src/sage/all__sagemath_categories.py

Yes, that's right. That's the design.

 

I don't think that I understand the purpose of sagemath_categories.

Per its tagline, this distribution provides "Sage categories, basic rings, polynomials, functions".

[Matthias Koeppe]

On Friday, September 27, 2024 at 2:41:21 PM UTC-7 Dima Pasechnik wrote:

If this is the case then the brial related PR ought to come after the sagemath-objects and -categories stuff PR. (Or maybe more than one PR).

No, the PR has a suitable scope, and it has already been reviewed.

[Matthias Koeppe]

On Friday, September 27, 2024 at 7:02:17 PM UTC-7 Dima Pasechnik wrote:

It seems that it puts about 50% or more of Sage into sagemath-categories, cause it includes SR (with calculus), polynomial rings, etc. One may ask whether it's worthwhile to have at all in this form.

No, SR is not in sagemath-categories.

[Martin R]

I don't see why `simplify` should be in something named `sagemath_categories`.

How is it decided whether a function should be a function in `Sage categories, basic rings, polynomials, functions`?

Martin

[Martin R]

I think that "technical discussion" refers to "discussion about a technical detail".

However, this PR introduces a new kind of file, `all__sagemath_categories.py`, into (almost?) *every* subdirectory of src.  Thus, it seems to me that this affects almost all developers - after all these files have to be maintained once they are there.  I doubt that there are many people who understand the specification of these files.

For example, I see that PowerSeriesRing is imported, but LazyPowerSeriesRing is not.  I don't understand the difference.

I admit that I didn't see these files before.  On the version on my computer, they are (with very few exceptions) only in the top level.  I thought that they were about the distribution packages, because that's what's mentioned in the top lines.

Apart from that, there are rumours that some people are blocked by Matthias, I don't know whether they can read his pull requests.

Martin

[Kwankyu Lee]

On Saturday, September 28, 2024 at 3:52:51 PM UTC+9 axio...@yahoo.de wrote:

However, this PR introduces a new kind of file, `all__sagemath_categories.py`, into (almost?) *every* subdirectory of src.  Thus, it seems to me that this affects almost all developers - after all these files have to be maintained once they are there.

There are "all.py" files throughput the sage library. Did you ever care about them? 

For modularization of the sage library, these "all__xxx" files were introduced to cope with the situation where only part of the sage library is installed by a user of sage distribution packages. If you didn't care about "all.py" before, you still don't need to care about "all__" files.

 

[Martin R]

Yes, I had to care about the `all.py` files whenever I added something that should be exposed to the user.  Eg., Bijectionist, LazyXXXSeriesRing, GrowthDiagram, whatever.

So, at the very least I would like to know what decides whether something goes into `all__sagemath_categories` and what doesn't.  Given the examples, I am unable to figure it out.

I must say that I find it impolite and disrespectful how questions concerning modularizations are dismissed.

Martin

[Matthias Koeppe]

On Friday, September 27, 2024 at 11:52:51 PM UTC-7 axio...@yahoo.de wrote:

this PR introduces a new kind of file, `all__sagemath_categories.py`,

Not, it's not a new kind of file. These files have been around since Sage 9.6 (https://github.com/sagemath/sage/issues/29865).

The main purpose of these files is to define the global environment for doctesting, as documented in https://doc.sagemath.org/html/en/developer/doctesting.html#options-for-testing-in-virtual-environments

For example, I see that PowerSeriesRing is imported, but LazyPowerSeriesRing is not.  I don't understand the difference.

The file pkgs/sagemath-categories/MANIFEST.in.m4 defines which modules are shipped by the distribution package sagemath-categories.

The following lines there make the PowerSeriesRing available.

> include sage/rings/power_series_mpoly.p*

> include sage/rings/power_series_poly.p*
> include sage/rings/power_series_ring_element.p*
> include sage/rings/power_series_ring.p*

The modules defining LazyPowerSeriesRing are not included. This distinction is then reflected in the all__sagemath_categories.py file as well: Can't import from a module that is not shipped.

 

I thought that they were about the distribution packages, because that's what's mentioned in the top lines.

That's right! "all__sagemath_categories.py" defines the global environment for testing the distribution package sagemath-categories.

[Dima Pasechnik]

On Sat, Sep 28, 2024 at 3:02 AM Matthias Koeppe
<matthia...@gmail.com> wrote:
>
> On Friday, September 27, 2024 at 2:25:07 PM UTC-7 axio...@yahoo.de wrote:
>
> The diagram you link to indicates that sagemath-categories is almost at the bottom, whereas sagemath-symbolics is almost at the top of the hierarchy that you propose.
>
>
> That's right.
>
>
> However, the PR includes a file
>
> src/sage/calculus/all__sagemath_categories.py
> from sage.calculus.functional import diff, derivative, expand, simplify, taylor
>
> and many similar files, e.g. src/sage/combinat/all__sagemath_categories.py, src/sage/groups/all__sagemath_groups.py, importing things like
> from sage.groups.generic import (discrete_log, discrete_log_rho, discrete_log_lambda, linear_relation, multiple, multiples, order_from_multiple)
>
> All of these are then imported by src/sage/all__sagemath_categories.py
>
>
> Yes, that's right. That's the design.

Yes, and it's called spaghetti design. Making things only more
complicated, confusing, and difficult to work with.
Covering the contents of sagelib with ~20 (or ~30) confusingly named
"distributions" (sagemath-categories and sagemath-objects two of
them), most of them containing well over 50% of sagelib functionality,
with contents of each of them not discussed anywhere - apart from the
current thread here, it seems.

One important side effect of this is that with this, a change in
sagelib which would work in the current design
might break some of these ~20 distributions, meaning that one would
need to run ~20 times more testing rounds, for each of these
distributions,
and understand why a breakage happens, say, in distribution A.
And then, fixing the breakage in distribution A might induce a
breakage in distribution B,
as they overlap in hard to keep in mind, spaghetti-like, ways. So
you'd in turn have to fix B, etc. ad nauseam, slowing
development down even more.

It's bad design, it will only slow down the development, it should not
go forward as is.

A much more meaningful design, and an obvious improvement, would be to
have a (say) sagemath-smallcore, where, say, 70% of
sagelib functionality is, with the remaining parts based off this
sagemath-smallcore.
So if you don't touch anything in sagemath-smallcore, then you don't
break seemingly unrelated "distributions".

>
>
> I don't think that I understand the purpose of sagemath_categories.
>
>
> Per its tagline, this distribution provides "Sage categories, basic rings, polynomials, functions".

The contents is not the purpose. What is its purpose, what does it
provide as a coherent software component, not as a collection of
seemingly unrelated things ?

Dima

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/58728851-e053-4923-a433-36077490a781n%40googlegroups.com.

[Martin R]

 

A much more meaningful design, and an obvious improvement, would be to
have a (say) sagemath-smallcore, where, say, 70% of
sagelib functionality is, with the remaining parts based off this
sagemath-smallcore. 

I agree.  This sounds more reasonable to me.  I could also imagine to have three layers:

* a core distribution with absolutely minimal dependencies and only dependencies which have proved stable on all supported platforms

* a distribution, or possibly even a few distributions, which exclude some possibly difficult dependencies

* all of sagemath

Martin

[Matthias Koeppe]

On Saturday, September 28, 2024 at 11:24:46 AM UTC-7 Dima Pasechnik wrote:

[...] fixing the breakage in distribution A might induce a

breakage in distribution B,
as they overlap in hard to keep in mind, spaghetti-like, ways.

No, the distributions do not overlap. They are disjoint.

[Matthias Koeppe]

On Saturday, September 28, 2024 at 12:28:30 PM UTC-7 axio...@yahoo.de wrote:

 I could also imagine to have three layers:

* a core distribution with absolutely minimal dependencies and only dependencies which have proved stable on all supported platforms

This is exactly sagemath-categories. It has absolutely minimal dependencies.

[Oscar Benjamin]

Would it make sense to give this a different name like Dima's
suggestion of sagemath-core or something else like sagemath-base,
sagemath-minimal etc?

I haven't looked at what is in sagemath-categories but it sounds like
its purpose is to be a smaller more manageable part of the Sage
software rather than to support any clearly contained branch of
mathematics. That seems very reasonable to me but it helps everyone if
the names for packages line up with their purposes.

--
Oscar

[Matthias Koeppe]

On Saturday, September 28, 2024 at 5:21:56 PM UTC-7 oscar.j....@gmail.com wrote:

On Sun, 29 Sept 2024 at 00:22, Matthias Koeppe <matthia...@gmail.com> wrote:
> On Saturday, September 28, 2024 at 12:28:30 PM UTC-7 axio...@yahoo.de wrote:
>
> I could also imagine to have three layers:
>
> * a core distribution with absolutely minimal dependencies and only dependencies which have proved stable on all supported platforms
>
> This is exactly sagemath-categories. It has absolutely minimal dependencies.

Would it make sense to give this a different name like Dima's
suggestion of sagemath-core or something else like sagemath-base,
sagemath-minimal etc?

I think all of these options are much less expressive than the current choice.

Anything named -minimal could describe pretty much any arbitrary choice. (It's "minimal" for *what*?)

Likewise -core could mean pretty much anything. There are many other things that are arguably the "core" of Sage.

And sagemath-categories is definitely not the -base; below it in the dependency graph there is sagemath-objects (an even smaller subset).

 

[Kwankyu Lee]

I welcome this discussion on  the design of modularized distribution packages.

Perhaps Matthias has his design fixed in his plan. But I believe the modularization project would succeed only when many sage developers understand and accept the design.

Hence it is crucial that we have open discussions on the design with all interested developers participating.

So I opened a github Discussion on the topic: https://github.com/sagemath/sage/discussions/38735

  

[Dima Pasechnik]

On 29 September 2024 01:50:39 BST, Matthias Koeppe <matthia...@gmail.com> wrote:
>On Saturday, September 28, 2024 at 5:21:56 PM UTC-7 oscar.j....@gmail.com
>wrote:
>
>On Sun, 29 Sept 2024 at 00:22, Matthias Koeppe <matthia...@gmail.com>
>wrote:
>> On Saturday, September 28, 2024 at 12:28:30 PM UTC-7 axio...@yahoo.de
>wrote:
>>
>> I could also imagine to have three layers:
>>
>> * a core distribution with absolutely minimal dependencies and only
>dependencies which have proved stable on all supported platforms
>>
>> This is exactly sagemath-categories. It has absolutely minimal
>dependencies

So it's not minimal, as it contains sagemath-objects

.
>
>Would it make sense to give this a different name like Dima's
>suggestion of sagemath-core or something else like sagemath-base,
>sagemath-minimal etc?
>
>
>I think all of these options are much less expressive than the current
>choice.

The current choice is confusing, for a number of reasons:

* as it invokes wrong associations: "oh, it's category theory, but with more algebra - it must be Abelian categories there. Sage does Abelian categories, wow!"

* the name does not stand out at all from the rest of distributions to indicate that it is something minimal/core/base and contained (almost) everywhere else

* it arbitrarily names the distribution after one of its parts (sage.categories), and ignores the rest of its content.

>
>Anything named *-minimal* could describe pretty much any arbitrary choice.

>(It's "minimal" for *what*?)

A minimal element in the poset of distributions,
obviously.


>Likewise *-core* could mean pretty much anything. There are many other

>things that are arguably the "core" of Sage.

"core" is also bringing a better intuition than categories. Core is a good name for an element of a poset which is unique in the sense that it's one you cannot avoid while travelling from minimal to maximal elements in the dependency tree.


>And *sagemath-categories* is definitely not the *-base*; below it in the

>dependency graph there is sagemath-objects (an even smaller subset).

Perhaps sagemath-foundations is better?

Should we hold a vote?


Dima

>
>

[Martin R]

> I could also imagine to have three layers:
>
> * a core distribution with absolutely minimal dependencies and only dependencies which have proved stable on all supported platforms
>
> This is exactly sagemath-categories. It has absolutely minimal dependencies.

Would it make sense to give this a different name like Dima's
suggestion of sagemath-core or something else like sagemath-base,
sagemath-minimal etc?

I think all of these options are much less expressive than the current choice.

Anything named -minimal could describe pretty much any arbitrary choice. (It's "minimal" for *what*?)

Likewise -core could mean pretty much anything. There are many other things that are arguably the "core" of Sage.

And sagemath-categories is definitely not the -base; below it in the dependency graph there is sagemath-objects (an even smaller subset).

Thank you for clarifying the intention.  I would never have guessed this meaning from the name `sagemath-categories`.  I think that `sagemath--minimal-dependencies` would be clear, wouldn't it?

I have now two more questions:

* why are some classes without any dependencies missing from what's currently named `sagemath-categories`?  I don't know how to do a systematic search, so I only checked the ones I know well, and `GrowthDiagram` is not there, but is pure python.

* wouldn't it be better to tag dependencies, rather than having to explicitly put something into what's currently `sagemath-categories`?  I imagine that it will be a constant source of confusion whether to put some new functionality into all.py or all__sagemath_categories.py

Best wishes,

Martin

[Martin R]

Please don't fragment this discussion.  It seems productive to me currently.

[Kwankyu Lee]

On Sunday, September 29, 2024 at 5:31:15 PM UTC+9 axio...@yahoo.de wrote:

Please don't fragment this discussion.  It seems productive to me currently.

According to the recent survey, visitors to sage-devel seem to get tired of lengthy technical discussions with only a few people involved. This thread is not too long yet, but is likely to be so.

I don't mean to fragment the discussion. Either all of you move to the github Discussion that I opened for you, or ignore it and continue here. My suggestion is experimental.   

 

[Matthias Koeppe]

On Sunday, September 29, 2024 at 12:15:04 AM UTC-7 Dima Pasechnik wrote:

On 29 September 2024 01:50:39 BST, Matthias Koeppe <matthia...@gmail.com> wrote:
>On Saturday, September 28, 2024 at 5:21:56 PM UTC-7 oscar.j....@gmail.com
>wrote:
>
>On Sun, 29 Sept 2024 at 00:22, Matthias Koeppe <matthia...@gmail.com>
>wrote:
>> On Saturday, September 28, 2024 at 12:28:30 PM UTC-7 axio...@yahoo.de
>wrote:
>>
>> I could also imagine to have three layers:
>>
>> * a core distribution with absolutely minimal dependencies and only
>dependencies which have proved stable on all supported platforms
>>
>> This is exactly sagemath-categories. It has absolutely minimal
>dependencies

So it's not minimal, as it contains sagemath-objects

When one talks about dependencies in this context, one means the recursive dependencies on that are not just other Sage distribution packages.

The dependencies of both sagemath-objects and sagemath-categories are gmpy2 (and its non-Python dependency libraries gmp, mpfr, mpc) and memory_allocator and nothing else.

 

>I think all of these options are much less expressive than the current
>choice.

The current choice is confusing, for a number of reasons:

* as it invokes wrong associations: "oh, it's category theory, but with more algebra - it must be Abelian categories there. Sage does Abelian categories, wow!"

Hardly. It uses the word "categories" in the same way that "sage.categories" does.

 

* the name does not stand out at all from the rest of distributions to indicate that it is something minimal/core/base and contained (almost) everywhere else

* it arbitrarily names the distribution after one of its parts (sage.categories), and ignores the rest of its content.

No, it's not arbitrary. It's named from the viewpoint of users: sagemath-categories does provide the full set of Sage's categories. It's suitable for users to explore Sage's categories and their relationships. The distribution does not provide the concrete implementations of the parents/elements that lie in these categories -- because often that depends on additional libraries. 

The other contents of sagemath-categories are provided in similar spirit. For example, there are the Function objects from sage.functions -- but only in a generic, dispatching role. Actual implementations of most functions depend on various libraries. Likewise, polynomials are there only in a generic implementation; the specialized implementations and higher-level functionality (e.g., Gröbner bases) are provided by other distributions as they depend on various libraries.

>Anything named *-minimal* could describe pretty much any arbitrary choice.
>(It's "minimal" for *what*?)

A minimal element in the poset of distributions,
obviously.

That's hardly what people would expect.

Perhaps sagemath-foundations is better?

No. "Foundations" can also mean anything to everyone.

Should we hold a vote?

No, a vote is unlikely to be useful until more people have gained much more experience with the design of the modularization and the contents of its distributions.

And for sure doing any such renaming games will be outside of the scope of the PR in question. https://github.com/sagemath/sage/pull/36380 should not wait for it.

[Matthias Koeppe]

On Sunday, September 29, 2024 at 1:30:38 AM UTC-7 axio...@yahoo.de wrote:

I think that `sagemath--minimal-dependencies` would be clear, wouldn't it?

Again fails the "minimal for what" test; and fails to distinguish sagemath-categories from sagemath-objects (which has the same dependencies).

 

I have now two more questions:

* why are some classes without any dependencies missing from what's currently named `sagemath-categories`?  I don't know how to do a systematic search, so I only checked the ones I know well, and `GrowthDiagram` is not there, but is pure python.

"Some classes are missing" is really far from a description of sagemath-categories. It's really much, much smaller than that.

GrowthDiagram, like most of sage.combinat, will be shipped in sagemath-combinat (which sits on top of sagemath-categories in the dependencies graph).

 

* wouldn't it be better to tag dependencies, rather than having to explicitly put something into what's currently `sagemath-categories`?  I imagine that it will be a constant source of confusion whether to put some new functionality into all.py or all__sagemath_categories.py

There's no guesswork involved, and there is no potential for confusion because the source files from which stuff is imported are tagged by "# sage_setup: distribution" lines. It just has to be kept in sync; and the automated tests ensure that.

(I'll also note that the "all.py" files are in very poor shape in Sage; in the past, too little thought has been put into what belongs into these files. See Meta-ticket: deprecate useless or misleading functions in the global namespace, https://github.com/sagemath/sage/issues/25383)

[Martin R]

So, it turns out that

> * a core distribution with absolutely minimal dependencies and only dependencies which have proved stable on all supported platforms
>
> This is exactly sagemath-categories. It has absolutely minimal dependencies

is not what I tried to say.  What I tried to say is that the core distribution should have minimal dependencies, but contain as much of sage as possible.

I don't see any advantage of restricting to a smaller subset.  What do you have in mind people would do with just the stuff in `categories`?

Martin

[Kwankyu Lee]

There are two factors that forms a distribution package: (a) mathematical contents of the code and (b) dependencies (external software packages) that supports the code.
Some distribution packages are named after mathematical contents and others are named after the main dependency.

Distribution packages high in the hierarchy tree are named after the mathematical contents

Distribution packages low in the hierarchy tree are named after the main dependency

sagemath-categories is at the base of those distribution packages named after mathematical contents. I think the name is good.

 

... What I tried to say is that the core distribution should have minimal dependencies, but contain as much of sage as possible.

More  code requires more dependencies. The "core distribution" as you describe is not possible.

...  What do you have in mind people would do with just the stuff in `categories`?

I don't think people would do anything useful just with sagemath-categories. It exists to serve as the common base of all mathematical distribution packages. 

[Martin R]

... What I tried to say is that the core distribution should have minimal dependencies, but contain as much of sage as possible.

More  code requires more dependencies. The "core distribution" as you describe is not possible.

I don't understand this, and I don't understand why some "functionality" (as mentioned above, e.g. "simplify" and "taylor") is in sagemath-categories.

I think it is problematic if we deploy a scheme that not even all developers understand.  In particular, the description given by Matthias, reproduced below, makes no sense to me at all.

> The other contents of sagemath-categories are provided in similar spirit. For example, there are the Function objects from sage.functions -- but only in a generic, dispatching role. Actual implementations of most functions depend on various libraries. Likewise, polynomials are there only in a generic implementation; the specialized implementations and higher-level functionality (e.g., Gröbner bases) are provided by other distributions as they depend on various libraries.

Martin

[Dima Pasechnik]

On Wed, Oct 2, 2024 at 6:54 AM 'Martin R' via sage-devel
<sage-...@googlegroups.com> wrote:
>
> So, it turns out that
> > * a core distribution with absolutely minimal dependencies and only dependencies which have proved stable on all supported platforms
> >
> > This is exactly sagemath-categories. It has absolutely minimal dependencies
>
> is not what I tried to say. What I tried to say is that the core distribution should have minimal dependencies, but contain as much of sage as possible.
>
> I don't see any advantage of restricting to a smaller subset. What do you have in mind people would do with just the stuff in `categories`?

With sagemath-categories as it is now, one can play a bit with
polynomials, and (I guess) Sage's Sets.
As one cannot do anything meaningful with sage.categories alone, as
it's just Sage's "abstract nonsense", without any implementations of
objects in these categories, sagemath-categories distro contains some
more or less random, concrete, implementations of maths objects, e.g.
Sage's polynomial rings. (And so a substantial part of sage.rings, I
presume)

So how is this absolute minimum? What if the user doesn't need polynomials?
It must be, by the current design of distributions, imported in any
Sage-dependent piece of code - by virtue of
being in almost every sagemath-* distribution.

By the way, Sage's categories is a bit of misnaming, I think --- but
that's another story.
It's not really an implementation of a bit of category theory in the
mathematical sense, it's actually just types,
in the CS sense, as we don't have there (not in the generality of
category theory for sure) anything like functors, natural
transformations, Yoneda lemma, etc.
That's Sage's type system that's there in sagemath-categories, more or
less. And so let's not add there semi-random pieces, unless it's
renamed.

Dima

>
> Martin
>
> On Wednesday 2 October 2024 at 02:25:38 UTC+2 Matthias Koeppe wrote:
>
> On Sunday, September 29, 2024 at 1:30:38 AM UTC-7 axio...@yahoo.de wrote:
>
>
> I think that `sagemath--minimal-dependencies` would be clear, wouldn't it?
>
>
> Again fails the "minimal for what" test; and fails to distinguish sagemath-categories from sagemath-objects (which has the same dependencies).
>
>
> I have now two more questions:
>
> * why are some classes without any dependencies missing from what's currently named `sagemath-categories`? I don't know how to do a systematic search, so I only checked the ones I know well, and `GrowthDiagram` is not there, but is pure python.
>
>
> "Some classes are missing" is really far from a description of sagemath-categories. It's really much, much smaller than that.
>
> GrowthDiagram, like most of sage.combinat, will be shipped in sagemath-combinat (which sits on top of sagemath-categories in the dependencies graph).
>
>
> * wouldn't it be better to tag dependencies, rather than having to explicitly put something into what's currently `sagemath-categories`? I imagine that it will be a constant source of confusion whether to put some new functionality into all.py or all__sagemath_categories.py
>
>
> There's no guesswork involved, and there is no potential for confusion because the source files from which stuff is imported are tagged by "# sage_setup: distribution" lines. It just has to be kept in sync; and the automated tests ensure that.
>
> (I'll also note that the "all.py" files are in very poor shape in Sage; in the past, too little thought has been put into what belongs into these files. See Meta-ticket: deprecate useless or misleading functions in the global namespace, https://github.com/sagemath/sage/issues/25383)
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/5123f648-b830-4010-b016-5c1afd687b25n%40googlegroups.com.

[Dima Pasechnik]

On Wed, Oct 2, 2024 at 12:43 AM Matthias Koeppe

<matthia...@gmail.com> wrote:
>
> On Sunday, September 29, 2024 at 12:15:04 AM UTC-7 Dima Pasechnik wrote:
>
> On 29 September 2024 01:50:39 BST, Matthias Koeppe <matthia...@gmail.com> wrote:
> >On Saturday, September 28, 2024 at 5:21:56 PM UTC-7 oscar.j....@gmail.com
> >wrote:
> >
> >On Sun, 29 Sept 2024 at 00:22, Matthias Koeppe <matthia...@gmail.com>
> >wrote:
> >> On Saturday, September 28, 2024 at 12:28:30 PM UTC-7 axio...@yahoo.de
> >wrote:
> >>
> >> I could also imagine to have three layers:
> >>
> >> * a core distribution with absolutely minimal dependencies and only
> >dependencies which have proved stable on all supported platforms
> >>
> >> This is exactly sagemath-categories. It has absolutely minimal
> >dependencies
>
> So it's not minimal, as it contains sagemath-objects
>
>
> When one talks about dependencies in this context, one means the recursive dependencies on that are not just other Sage distribution packages.
>
> The dependencies of both sagemath-objects and sagemath-categories are gmpy2 (and its non-Python dependency libraries gmp, mpfr, mpc) and memory_allocator and nothing else.

You forgot polynomials (and thus quite a bit of sage.rings), and few
other, unrelated to the Sage typing system, things you put there.

>
>
> >I think all of these options are much less expressive than the current
> >choice.
>
> The current choice is confusing, for a number of reasons:
>
> * as it invokes wrong associations: "oh, it's category theory, but with more algebra - it must be Abelian categories there. Sage does Abelian categories, wow!"
>
>
> Hardly. It uses the word "categories" in the same way that "sage.categories" does.
>
>
> * the name does not stand out at all from the rest of distributions to indicate that it is something minimal/core/base and contained (almost) everywhere else
>
> * it arbitrarily names the distribution after one of its parts (sage.categories), and ignores the rest of its content.
>
>
> No, it's not arbitrary. It's named from the viewpoint of users: sagemath-categories does provide the full set of Sage's categories. It's suitable for users to explore Sage's categories and their relationships. The distribution does not provide the concrete implementations of the parents/elements that lie in these categories -- because often that depends on additional libraries.

They do provide a random subset of concrete implementations - e.g. polynomials.

>
> The other contents of sagemath-categories are provided in similar spirit. For example, there are the Function objects from sage.functions -- but only in a generic, dispatching role. Actual implementations of most functions depend on various libraries. Likewise, polynomials are there only in a generic implementation; the specialized implementations and higher-level functionality (e.g., Gröbner bases) are provided by other distributions as they depend on various libraries.
>
> >Anything named *-minimal* could describe pretty much any arbitrary choice.
> >(It's "minimal" for *what*?)
>
> A minimal element in the poset of distributions,
> obviously.
>
>
> That's hardly what people would expect.

Do you think that "categories" conveys the meaning of "import it
everywhere" better than "minimal"?

>
> Perhaps sagemath-foundations is better?
>
>
> No. "Foundations" can also mean anything to everyone.

At least it does not mis-specify what's there, whereas "categories"
certainly does.

>
> Should we hold a vote?
>
>
> No, a vote is unlikely to be useful until more people have gained much more experience with the design of the modularization and the contents of its distributions.

I don't think it's wise to subject people to more messy misnamed
things than it is absolutely necessary.
Once people are used to bad designs and wrong names (assuming there is
anyone left working with them!),
it's pretty hard to make changes.

>
> And for sure doing any such renaming games will be outside of the scope of the PR in question. https://github.com/sagemath/sage/pull/36380 should not wait for it.

#36380 is first of all waiting for, requested by more than one
reviewer, splitting off the part related to brial - as the latter is a
really urgent and unrelated to sagemath-categories thing.

Once it's split off, we can weigh the merits of the proposed design easier.

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/5fc801ca-70b3-4739-9023-0468822c0eabn%40googlegroups.com.

[Matthias Koeppe]

On Tuesday, October 1, 2024 at 10:54:36 PM UTC-7 axio...@yahoo.de wrote:

So, it turns out that

> * a core distribution with absolutely minimal dependencies and only dependencies which have proved stable on all supported platforms
>
> This is exactly sagemath-categories. It has absolutely minimal dependencies

is not what I tried to say.  What I tried to say is that the core distribution should have minimal dependencies, but contain as much of sage as possible.

I don't see any advantage of restricting to a smaller subset.

That the much smaller subset actually works is a key technological achievement of the modularization project.

The small size is favorable for maintainability, for porting to new platforms, etc.

 

  What do you have in mind people would do with just the stuff in `categories`?

As the dependency diagram shows -- it works as the common dependency of various other distributions. 

Likewise, it will also suffice as the build dependency of user packages.

[Matthias Koeppe]

On Wednesday, October 2, 2024 at 1:57:48 AM UTC-7 Dima Pasechnik wrote:

> And for sure doing any such renaming games will be outside of the scope of the PR in question. https://github.com/sagemath/sage/pull/36380 should not wait for it.

#36380 is first of all waiting for, requested by more than one
reviewer, splitting off the part related to brial - as the latter is a
really urgent and unrelated to sagemath-categories thing.

I have already corrected you about this; why repeat it? 

It's not "unrelated" to sagemath-categories. sagemath-brial needs a larger sagemath-categories as a build dependency.

[Matthias Koeppe]

On Wednesday, October 2, 2024 at 1:57:48 AM UTC-7 Dima Pasechnik wrote:

They do provide a random subset of concrete implementations - e.g. polynomials.

It's not "random". 

What is included is the result of years of my work on code and tests, to find a viable subset that can be separately built, run, and tested.

[Matthias Koeppe]

On Wednesday, October 2, 2024 at 1:57:48 AM UTC-7 Dima Pasechnik wrote:

I don't think it's wise to subject people to more messy misnamed

things than it is absolutely necessary.
Once people are used to bad designs and wrong names (assuming there is
anyone left working with them!),
it's pretty hard to make changes.

All of these things are just denunciations of the quality of my work, based on zero first-hand experience by Pasechnik. 

I will not respond further; a discussion is not possible in an environment of disrespect.

[Matthias Koeppe]

On Wednesday, October 2, 2024 at 1:08:37 AM UTC-7 axio...@yahoo.de wrote:

... What I tried to say is that the core distribution should have minimal dependencies, but contain as much of sage as possible.

More  code requires more dependencies. The "core distribution" as you describe is not possible.

I don't understand this, and I don't understand why some "functionality" (as mentioned above, e.g. "simplify" and "taylor") is in sagemath-categories.

I think it is problematic if we deploy a scheme that not even all developers understand.

"All developers" is an impossible threshold.

There's a simple and important principle of Open Source: 

Trust those who are doing the work.

[Oscar Benjamin]

Thanks Matthias for explaining more about the design. I hadn't
considered the possibility that sagemath-categories could be a
sufficient build-time dependency for some packages. In pip/PyPI land
having a smaller build-time dependency can make a big difference
because builds are all in isolated environments.

I suggested sagemath-minimal before as a somewhat throw away name. I
can see now that this does not make sense for what sagemath-categories
is in the design.

In my mind it could make sense for there to be a sagemath-minimal
package which from a user perspective represents a "minimal
installation of Sage" in comparison to say a package called
sagemath-full. The implication of that though is that the package
provides enough usable features that a user who only wants to use
widely used basic features of Sage would be able to use
sagemath-minimal to do what they want. It sounds like
sagemath-categories does not provide that. I think that the same
problem applies to the name sagemath-core because again to me that
name implies some level of usability from an end-user perspective.

It sounds like sagemath-categories is what I would have perhaps called
sagemath-base where the implication of "base" is:

- You need to install the base before you can install anything else.
- The base is not useful on its own but only as a foundation for other parts.

Looking at the diagram (in
https://github.com/sagemath/sage/pull/35095) is the part on the right
"sagemath-repl"?

I'm assuming that it is separate from categories because it depends on
ipython or something but is it likely that anyone would want to
install sagemath-repl without also installing sagemath-categories?

If not then would it make sense for sagemath-repl to depend on
sagemath-categories rather than sagemath-objects and then would there
still be a need for sagemath-objects and
sagemath-environment/sage-conf?

Something that is very unclear to me is what the sizes of these things
are. For example, how much disk space does it take to install
sagemath-mpmath plus dependencies? How much larger or smaller is that
than just installing sagemath-categories?

--
Oscar

[Kwankyu Lee]

In my mind it could make sense for there to be a sagemath-minimal
package which from a user perspective represents a "minimal
installation of Sage" in comparison to say a package called
sagemath-full.

sagemath-standard is your sagemath-full.

I think distribution packages in the current design are "building blocks", each of them contains a minimal subset of the sage library and dependencies that make the distribution package work. For example, sagemath-combinat is supposed to contain combinatorics portion of the sage library but deciding specifically which files and dependencies to include depends on what combinations make the distribution package work. Matthias did this hard work.

Hence the distribution packages in the current design were designed so from developers' point of view. A "building block" distribution package should be minimal in the sense that splitting it into two or more meaningful distribution packages that work is not possible. I don't know if the distribution packages in the current design achieved this or not.

We may define "user" distribution packages combining some subset of the building blocks, from users' point of view. Here I think your sagemath-minimal distribution package makes sense. It may provide a "sage" with elementary mathematics that all advanced mathematics depend. Likewise, we may define "user" distribution package sagemath-number-theory, and so forth.

To realize the "user" distribution packages, we should have all the "building block" distribution packages in sage first.

[Martin R]

> It may provide a "sage" with elementary mathematics that all advanced mathematics depend

There is no boundary between "elementary" and "advanced" mathematics.  Also, there is no boundary between "discrete mathematics" and "complex analysis" or "numerics".

> There's a simple and important principle of Open Source: Trust those who are doing the work.

I cannot quite pin it down, but this sounds fishy to me.  Maybe because in this particular case it's not just about work, maybe also because there are some aspects I find plain wrong.

Martin

[Kwankyu Lee]

I can imagine this:

sagemath-number-theory won't have the coding theory portion of the sage library. Hence developers for sagemath-number-theory could test their favorite distribution package quicker.

[Martin R]

I think that this would be a major step backwards.  Currently, if I make changes in one part of sage (right now: polynomials), I am sometimes a bit surprised where I am breaking code (in the case at hand: braid groups).  Of course, this is rarely surprising once I look at the code, but I don't have all places on my mind all the time.

It seems to me that one of the main innovations of sage (among other CAS) was to make writing doctests really easy *and* keep them together with the code *and* make testing them *while developing* very straightforward.

The final of these three points would be broken by encouraging developers to test only the parts of sage they are interested in, even if only slightly so.  This can be seen in packages that were developed for sage, but not integrated, and used mostly for a relatively specific task.  An example I have in mind is the ore_algebra package.

Martin

[Kwankyu Lee]

It seems to me that one of the main innovations of sage (among other CAS) was to make writing doctests really easy *and* keep them together with the code *and* make testing them *while developing* very straightforward.

True!

 

The final of these three points would be broken by encouraging developers to test only the parts of sage they are interested in, even if only slightly so.  This can be seen in packages that were developed for sage, but not integrated, and used mostly for a relatively specific task.  An example I have in mind is the ore_algebra package.

We should not encourage that. 

When I develop some code, thinking to merge it eventually to sage. I first run tests only in the files that I touch. Later when the code is mature, I run all the tests in sage locally (on mac). Later after I push the code to a PR, the ci run all the tests again (on ubuntu). Later when the PR is merged and a new release of sage is made, the ci run all the tests again on many platforms.

I am imagining this:

Suppose a number theorist developer installed sagemath-number-theory on his platform. (You are not the developer :-) He develops some code. He tests his code by running doctests in the files he touched. Later when the code is mature, he runs all the tests in sagemath-number-theory. This is the step that gets quicker. Later when the code is pushed to a PR, the ci tests the code with all the doctests of the whole sage library. 

[Dima Pasechnik]

And I have repeatedly explained that no matter how  brial and sagemath-categories are related, one can perfectly either
have a completely independent PR for brial, or
base the brial one on a PR that deals with updates to sagemath-whatever distribution(s).

If X depends on Y, make a PR for Y, and a PR for X which depends on the PR for Y and is based on the
branch of the latter. That's the normal way to deal with such things, in particular as in this case X and Y are nontrivial.


Besides, what's so special about an updated sagemath-categories distribution so that you absolutely must base the brial update on them?
Distributions are just wrappers for code, after all. 
Instead of using -distributions magic only you, and perhaps, Tobias, know (as witnessed on the PR which does a similar job, but for brial,
<https://github.com/sagemath/sage/pull/38712>), you can merely use a few imports that need to be used, regardless of the state of sagemath-categories.


Dima

[Matthias Koeppe]

On Wednesday, October 2, 2024 at 1:08:37 AM UTC-7 axio...@yahoo.de wrote:

the description given by Matthias, reproduced below, makes no sense to me at all.

> The other contents of sagemath-categories are provided in similar spirit. For example, there are the Function objects from sage.functions -- but only in a generic, dispatching role. Actual implementations of most functions depend on various libraries. Likewise, polynomials are there only in a generic implementation; the specialized implementations and higher-level functionality (e.g., Gröbner bases) are provided by other distributions as they depend on various libraries.

To understand it better, you may, after all, need to study the "theory of dependencies", https://github.com/sagemath/sage/pull/36753#issuecomment-2013579329

[Matthias Koeppe]

On Wednesday, October 2, 2024 at 2:48:10 PM UTC-7 oscar.j....@gmail.com wrote:

In my mind it could make sense for there to be a sagemath-minimal
package which from a user perspective represents a "minimal
installation of Sage" in comparison to say a package called
sagemath-full. The implication of that though is that the package
provides enough usable features that a user who only wants to use
widely used basic features of Sage would be able to use
sagemath-minimal to do what they want. It sounds like
sagemath-categories does not provide that. I think that the same
problem applies to the name sagemath-core because again to me that
name implies some level of usability from an end-user perspective.

Yes, there's definitely room for one or more such distributions below what is currently called sagemath-standard (= all standard features of Sage). There is no technical difficulty in putting together such distributions (without content, just dependencies). A discussion among users/developers what the scope of such distributions should be would be very valuable, but would have relatively little urgency.

 

Looking at the diagram (in
https://github.com/sagemath/sage/pull/35095) is the part on the right
"sagemath-repl"?

Yes.

 

I'm assuming that it is separate from categories because it depends on
ipython or something but is it likely that anyone would want to
install sagemath-repl without also installing sagemath-categories?

Yes. sagemath-repl contains also the Sage doctesting facility. It is used for doctesting sagemath-objects, the smaller distribution below sagemath-categories.

If not then would it make sense for sagemath-repl to depend on
sagemath-categories rather than sagemath-objects and then would there
still be a need for sagemath-objects and
sagemath-environment/sage-conf?

Anything that uses environment variables or configuration variables goes through sagemath-environment.

A distribution without a dependency on that therefore declares that it is "configuration-free". This is valuable by itself.

 

Something that is very unclear to me is what the sizes of these things
are. For example, how much disk space does it take to install
sagemath-mpmath plus dependencies? How much larger or smaller is that
than just installing sagemath-categories?

sagemath-mpmath is not a good example because the need for it will go away with Remove outdated mpmath Sage backend (https://github.com/sagemath/sage/pull/38113)

But here are the sizes of the wheels as of the PR in question (https://github.com/sagemath/sage/pull/36380). (These are not "repaired wheels", i.e., they do not contain the shared libraries that they depend on.)

-rw-r--r--  1 mkoeppe  staff    100021 Oct  1 17:31 venv/var/lib/sage/wheels/sagemath_bliss-10.5b6-cp312-cp312-macosx_14_0_x86_64.whl
-rw-r--r--  1 mkoeppe  staff    623865 Oct  1 17:48 venv/var/lib/sage/wheels/sagemath_brial-10.5b6-cp312-cp312-macosx_14_0_x86_64.whl
-rw-r--r--  1 mkoeppe  staff  12898188 Oct  1 17:39 venv/var/lib/sage/wheels/sagemath_categories-10.5b6-cp312-cp312-macosx_14_0_x86_64.whl
-rw-r--r--  1 mkoeppe  staff    168193 Oct  1 17:31 venv/var/lib/sage/wheels/sagemath_coxeter3-10.5b6-cp312-cp312-macosx_14_0_x86_64.whl
-rw-r--r--  1 mkoeppe  staff    120728 Oct  1 17:30 venv/var/lib/sage/wheels/sagemath_environment-10.5b6-py3-none-any.whl
-rw-r--r--  1 mkoeppe  staff     34979 Oct  1 17:31 venv/var/lib/sage/wheels/sagemath_mcqd-10.5b6-cp312-cp312-macosx_14_0_x86_64.whl
-rw-r--r--  1 mkoeppe  staff    210888 Oct  1 17:31 venv/var/lib/sage/wheels/sagemath_meataxe-10.5b6-cp312-cp312-macosx_14_0_x86_64.whl
-rw-r--r--  1 mkoeppe  staff   4677952 Oct  1 17:32 venv/var/lib/sage/wheels/sagemath_objects-10.5b6-cp312-cp312-macosx_14_0_x86_64.whl
-rw-r--r--  1 mkoeppe  staff    449903 Oct  1 17:33 venv/var/lib/sage/wheels/sagemath_repl-10.5b6-py3-none-any.whl
-rw-r--r--  1 mkoeppe  staff     68201 Oct  1 17:31 venv/var/lib/sage/wheels/sagemath_sirocco-10.5b6-cp312-cp312-macosx_14_0_x86_64.whl
-rw-r--r--  1 mkoeppe  staff  98114018 Oct  1 17:47 venv/var/lib/sage/wheels/sagemath_standard-10.5b6-cp312-cp312-macosx_14_0_x86_64.whl
-rw-r--r--  1 mkoeppe  staff    149635 Oct  1 17:31 venv/var/lib/sage/wheels/sagemath_tdlib-10.5b6-cp312-cp312-macosx_14_0_x86_64.whl
 

Note the size difference between the current sagemath-standard and sagemath-{objects,categories,repl,environment}. 

Happy to provide more detail.

Matthias

[Dima Pasechnik]

However, they'd need to test all packages which use number theory, and coding theory is one of them.
(and testing the whole sagelib is a weaker, insufficient test here, compared to testing a bunch of packages in isolation).

Besides, Sage has enough possibilities to test parts of sagelib, it has been the case long ago.





>

[Dima Pasechnik]

Why is it quicker than running ./sage -t src/sage/lfunctions src/sage/rings/
(perhaps few other should be added) on a full sagelib install?

sagemath-number-theory is probably 70-80% of sagelib.
There is very little to win here, at expense of quite a bit of extra
complexity of all these sagemath-* distributions.
And extra complexity to run wider tests, for sagelib, as it's not
installed by default.

For a wider testing, we should not assume that free and generous
GitHub CI allowance we enjoy now is there to stay.
Anytime Microsoft, the owner of github, would want extra cash, this
will be over in no time, like it was the case with GitLab,
and before that with a quite popular, back then, dedicated CI service
(forgot the name).

Dima

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/f8ae2625-bff2-411f-9070-00619a4bd593n%40googlegroups.com.

[Matthias Koeppe]

On Thursday, October 3, 2024 at 6:11:48 AM UTC-7 Kwankyu Lee wrote:

It seems to me that one of the main innovations of sage (among other CAS) was to make writing doctests really easy *and* keep them together with the code *and* make testing them *while developing* very straightforward.

True!

 

The final of these three points would be broken by encouraging developers to test only the parts of sage they are interested in, even if only slightly so.  This can be seen in packages that were developed for sage, but not integrated, and used mostly for a relatively specific task.  An example I have in mind is the ore_algebra package.

We should not encourage that. 

And we aren't. Integration testing is always there.

[Matthias Koeppe]

On Thursday, October 3, 2024 at 5:26:09 AM UTC-7 axio...@yahoo.de wrote:

It seems to me that one of the main innovations of sage (among other CAS) was to make writing doctests really easy *and* keep them together with the code *and* make testing them *while developing* very straightforward.

Yes, and this structure is preserved by the modularization project.

[Matthias Koeppe]

On Thursday, October 3, 2024 at 2:47:20 AM UTC-7 axio...@yahoo.de wrote:

> There's a simple and important principle of Open Source: Trust those who are doing the work.

I cannot quite pin it down,

I think it's worth to engage in a bit more of reflection.

but this sounds fishy to me.  Maybe because in this particular case it's not just about work, maybe also because there are some aspects I find plain wrong.

What are these "aspects", and do any of them have any bearing on the PR in question, https://github.com/sagemath/sage/pull/36380, whose specific scope is to add a new distribution sagemath-brial and to enlarge the existing distributions sagemath-objects, sagemath-categories?

[Matthias Koeppe]

On Thursday, October 3, 2024 at 1:16:12 PM UTC-7 Dima Pasechnik wrote:

what's so special about an updated sagemath-categories distribution so that you absolutely must base the brial update on them?
Distributions are just wrappers for code, after all. 

"Wrappers"? No.

 

Instead of using -distributions magic only you, and perhaps, Tobias, know (as witnessed on the PR which does a similar job, but for brial,
<https://github.com/sagemath/sage/pull/38712>), you can merely use a few imports that need to be used, regardless of the state of sagemath-categories.

No. One can only import modules that have been shipped.

The current sagemath-categories does not ship enough modules.

Hence the scope of https://github.com/sagemath/sage/pull/38712, which combines enlarging sagemath-categories with the sample motivating application, namely creating sagemath-brial.

[Matthias Koeppe]

On Thursday, October 3, 2024 at 1:18:44 PM UTC-7 Dima Pasechnik wrote:

Why is it quicker than running ./sage -t src/sage/lfunctions src/sage/rings/
(perhaps few other should be added) on a full sagelib install?

You are missing the point. 

The main point of testing a modularized distribution is not speed but to protect against modularization regressions.

Testing the modularized distributions is not the same as passing a subset of files to the doctester.

  

And extra complexity to run wider tests, for sagelib, as it's not
installed by default.

Also wrong. There are no changes to what is installed by default.

 

For a wider testing, we should not assume that free and generous

GitHub CI allowance we enjoy now is there to stay. [...]

Fear about suddenly disappearing resources for our CI is misplaced.

Moreover, it has no bearing whatsoever on the PR in question, https://github.com/sagemath/sage/pull/36380, whose specific scope is to add a new distribution sagemath-brial and to enlarge the existing distributions sagemath-objects, sagemath-categories.

[Dima Pasechnik]

On Thu, Oct 3, 2024 at 11:01 PM Matthias Koeppe
<matthia...@gmail.com> wrote:
>
> On Thursday, October 3, 2024 at 1:18:44 PM UTC-7 Dima Pasechnik wrote:
>
> Why is it quicker than running ./sage -t src/sage/lfunctions src/sage/rings/
> (perhaps few other should be added) on a full sagelib install?
>
>
> You are missing the point.
>
> The main point of testing a modularized distribution is not speed but to protect against modularization regressions.
> Testing the modularized distributions is not the same as passing a subset of files to the doctester.

I am not missing the point. Of course testing each of the distributions

is not the same as passing a subset of files to the doctester.

We are taking about a particular example - of developing a piece of
number-theoretic functionality, and how
using only sagemath-number-theory distribution might benefit the developer.
It was suggested that it might be quicker with sagemath-number-theory,
as it would only test a part of the sagelib.
I pointed out that an option to test a part is already here, for long time.
And on the other hand, as at some point one would need to test the
whole sagelib,
it's more work with the modularised system, not less
My point is that the benefits of modularisation are unclear, there are
more tests to run, more complexity
to understand, for seemingly no gain.

> And extra complexity to run wider tests, for sagelib, as it's not
> installed by default.
>
>
> Also wrong. There are no changes to what is installed by default.

Really? So one cannot even build/install sagemath-number-theory
without building/installing the whole sagelib?
(once again, we are talking about a hypothetical example of someone
installing sagemath-number-theory
and developing it)
And what's the point of all that "modularisation" then? Looks like
just extra baggage to be saddled with.
And running 10 or 20 times more tests.

> For a wider testing, we should not assume that free and generous
> GitHub CI allowance we enjoy now is there to stay. [...]
>
> Fear about suddenly disappearing resources for our CI is misplaced.

Really? Do we have a binding, continuing, contract for free provision
of CI services certifying to this?
Or perhaps a sizeable amount of MSFT stock to be able to pressure for
CI availability? :-)
Given that lately I have to prove to GutHub that I am entitled to an
academic discount (i.e. a free account with a bit more
than the default free account) every year, or even more often (and
GitHub keeps sending me bills as if it's not aware of
my status), this looks like the days of free GitHub might be numbered.

>
> Moreover, it has no bearing whatsoever on the PR in question, https://github.com/sagemath/sage/pull/36380, whose specific scope is to add a new distribution sagemath-brial and to enlarge the existing distributions sagemath-objects, sagemath-categories.

No, why? This thread is a discussion as to sagemath-categories should
proceed as outlined in #36380.

By the way, it seems that the only way forward with #36380 is for me
or someone else to split the brial part off,
as you refuse to do so for an unclear reason.

Dima

[Kwankyu Lee]

We are observing disadvantage of a long thread.

At some point, other participants forget the context in which a participant write a comment.

Dima and I were talking about a fictional sagemath-number-theory distribution package.

I imagined a user/developer who installed sagemath-number-theory, and is developing some mathematical code.  

I claimed that running all doctests in sagemath-number-theory on his local platform would take less time than running all doctests in sagemath-standard.

> On Thursday, October 3, 2024 at 1:18:44 PM UTC-7 Dima Pasechnik wrote:
>
> Why is it quicker than running ./sage -t src/sage/lfunctions src/sage/rings/
> (perhaps few other should be added) on a full sagelib install?

 I didn't claim that.

 

On Thu, Oct 3, 2024 at 11:01 PM Matthias Koeppe 

> The main point of testing a modularized distribution is not speed but to protect against modularization regressions.

> Testing the modularized distributions is not the same as passing a subset of files to the doctester.

Matthias is talking about testing modularized distributions to check if they work together well. This will be done in github ci.

We are taking about a particular example - of developing a piece of

number-theoretic functionality, and how
using only sagemath-number-theory distribution might benefit the developer.
It was suggested that it might be quicker with sagemath-number-theory,
as it would only test a part of the sagelib.
I pointed out that an option to test a part is already here, for long time.
And on the other hand, as at some point one would need to test the
whole sagelib,

Yes. The one is github ci.

 

it's more work with the modularised system, not less

On the project level, It takes more work to provide more products (distribution packages) to users. 

On an individual developer level, it is less work if he can focus on "a smaller sage library" (sagemath-number-theory) 

My point is that the benefits of modularisation are unclear, there are
more tests to run, more complexity
to understand, for seemingly no gain.

Like our github ci, "more tests to run, more complexity to understand" is a burden upon the core developers maintaining the project.

The motivation of the modularization project is to reduce the burden who only needs some portion of the sage library,

and wants to use and develop the portion within the python ecosystem.

An individual mathematician who only needs some portion of the sage library may install only his favorite distribution package.

There are obvious benefits of using smaller software package: less time in installing and testing, less number of failure points in installing and testing, less cognitive stress, ...

 

[Travis Scrimshaw]

We are observing disadvantage of a long thread.

It is also very useful to have it all in one thread though.

 

it's more work with the modularised system, not less

On the project level, It takes more work to provide more products (distribution packages) to users. 

I completely agree with this point.

On an individual developer level, it is less work if he can focus on "a smaller sage library" (sagemath-number-theory) 

This is directly contradicted by your previous point as the individual developer needs to understand and check (!!) how the changes affect all of the other components, and how different configurations of components can require additional code complications or adding "magic" things (e.g., "# needs XYZ"). In our current model, the developer still is just focused on the section of code they want to change.

My point is that the benefits of modularisation are unclear, there are
more tests to run, more complexity
to understand, for seemingly no gain.

Like our github ci, "more tests to run, more complexity to understand" is a burden upon the core developers maintaining the project.

As I mentioned above, this is a burden on *all* developers. There also becomes an additional barrier of core developers having to explain to new developers why they need to add stuff to get some complex test to pass.

The motivation of the modularization project is to reduce the burden who only needs some portion of the sage library,

and wants to use and develop the portion within the python ecosystem.

We should be more judicious about cutting off pieces as dependencies that are useful to the broader community. I continue to remain highly skeptical that the proposed modularization will have a net positive impact on how Sage interacts with the Python ecosystem, such as increased developer contributions to Sage or use of the modular parts of Sage.

An individual mathematician who only needs some portion of the sage library may install only his favorite distribution package.

There are obvious benefits of using smaller software package: less time in installing and testing, less number of failure points in installing and testing, less cognitive stress, ...

Cognitive stress has no relation to the size of a software package, but instead the complexity of its installation. Consequently, having more pieces means the user needs to make more decisions about what to install and increases the number of points of failure due to (often nontrivial and subtle) interactions. The user also doesn't care about testing.

Best,

Travis

[Marc Culler]

On Thursday, October 3, 2024 at 8:05:09 PM UTC-5 Kwankyu Lee wrote:

The motivation of the modularization project is to reduce the burden who only needs some portion of the sage library,

and wants to use and develop the portion within the python ecosystem.

I would say that the motivation is to make it possible for  a developer to include a self-contained portion of sage in a separate project without having to make that project as large as a full Sage distribution.

 

An individual mathematician who only needs some portion of the sage library may install only his favorite distribution package.

As I have said so many times before, apparently to no avail, I strongly disagree with this point of view.  It is not reasonable to expect an individual mathematician, or student, to be able to decide which portions of Sage are needed, nor even to know which portions can be installed separately.  An individual who wants to use Sage should just install Sage.  All of it.  And that installation should be very easy.  It should certainly not require deciding which portions of Sage might be needed in which contexts. 

- Marc

[G. M.-S.]

I wholly agree with Marc.

For me, this is another instance of a user point of view versus a developer point of view.

Coming from someone who tries to make SageMath easily available for ordinary people, Marc's views are all the more valuable IMHO.

Guillermo

[Robert Bruner]

I do not usually weigh in because I am no expert, but I do use sage frequently. For me, Marc's point is key:

" An individual who wants to use Sage should just install Sage."

I would rather do math than spend my time learning the ins and outs of a software package. I nearly said this in response to some past posts making the same point. I agree with them too.

Robert Bruner

________________________________________
From: sage-...@googlegroups.com <sage-...@googlegroups.com> on behalf of G. M.-S. <list...@gmail.com>
Sent: Sunday, October 6, 2024 11:41 AM
To: sage-...@googlegroups.com
Subject: Re: [sage-devel] Re: [debian-science] Modularized sagemath packages: proof of concept

[EXTERNAL]

I wholly agree with Marc.

For me, this is another instance of a user point of view versus a developer point of view.

Coming from someone who tries to make SageMath easily available for ordinary people, Marc's views are all the more valuable IMHO.

Guillermo

On Sun, 6 Oct 2024 at 17:24, Marc Culler <marc....@gmail.com<mailto:marc....@gmail.com>> wrote:

On Thursday, October 3, 2024 at 8:05:09 PM UTC-5 Kwankyu Lee wrote:
The motivation of the modularization project is to reduce the burden who only needs some portion of the sage library,
and wants to use and develop the portion within the python ecosystem.

I would say that the motivation is to make it possible for a developer to include a self-contained portion of sage in a separate project without having to make that project as large as a full Sage distribution.

An individual mathematician who only needs some portion of the sage library may install only his favorite distribution package.

As I have said so many times before, apparently to no avail, I strongly disagree with this point of view. It is not reasonable to expect an individual mathematician, or student, to be able to decide which portions of Sage are needed, nor even to know which portions can be installed separately. An individual who wants to use Sage should just install Sage. All of it. And that installation should be very easy. It should certainly not require deciding which portions of Sage might be needed in which contexts.

- Marc

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com<mailto:sage-devel+...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CANnG18_%2BiVEiMLyfwECMvZq5aCFsnB5Jd6K7uVD_LRczKKvOjw%40mail.gmail.com<https://groups.google.com/d/msgid/sage-devel/CANnG18_%2BiVEiMLyfwECMvZq5aCFsnB5Jd6K7uVD_LRczKKvOjw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Kwankyu Lee]

On Monday, October 7, 2024 at 12:24:04 AM UTC+9 marc....@gmail.com wrote:

I would say that the motivation is to make it possible for a developer to include a self-contained portion of sage in a separate project without having to make that project as large as a full Sage distribution.

OK. That is perhaps a major case that one needs a portion of the sage library.  I can also imagine a user installing a smaller sage library just because it is enough for him/her. 

 

...  It is not reasonable to expect an individual mathematician, or student, to be able to decide which portions of Sage are needed, nor even to know which portions can be installed separately.  An individual who wants to use Sage should just install Sage.  All of it.  And that installation should be very easy.  It should certainly not require deciding which portions of Sage might be needed in which contexts. 

Sure.Those users don't need to care about the modularization project. 

On the other hand, who would be the users of the distribution packages for whatever need? I wonder how they overlap with sage developers.

If the sage developers, who are mostly mathematicians, cannot foresee any use of distribution packages for them, then I think the modularization project is doomed.

[Oscar Benjamin]

On Mon, 7 Oct 2024 at 06:05, Kwankyu Lee <ekwa...@gmail.com> wrote:
>
> On Monday, October 7, 2024 at 12:24:04 AM UTC+9 marc....@gmail.com wrote:
>
> > I would say that the motivation is to make it possible for a developer to include a self-contained portion of sage in a separate project without having to make that project as large as a full Sage distribution.
>
> OK. That is perhaps a major case that one needs a portion of the sage library. I can also imagine a user installing a smaller sage library just because it is enough for him/her.

I suspect that if Sage was modularised then over time people would
work on making some parts more portable so that some subset of Sage
would be natively installable in WASM or on Windows etc. Greater
portability would make Sage more attractive for use as a software
dependency. There would also be a strong incentive to try to carve out
a meaningful subset of Sage for end-users that was more portable but I
don't know how realistic it could be to achieve that.

--
Oscar

[Kwankyu Lee]

On Monday, October 7, 2024 at 6:38:34 PM UTC+9 oscar.j....@gmail.com wrote:

... There would also be a strong incentive to try to carve out
a meaningful subset of Sage for end-users that was more portable ...

"portable" sounds a good adjective to characterize distribution packages. Another may be "light-weight".

I can't really imagine that a distribution package is used for a non-mathematical project, though no one can be certain about the future.

As for a mathematical project, I may imagine a mathematician who wrote a python script that does a graph-theoretic computation. To run the script in a seminar, she quickly pip-installs sagemath-graphs to a computer in the seminar room and then shows off her computation. I don't know if this is realistic either...

 

[Marc Culler]

On Monday, October 7, 2024 at 12:05:25 AM UTC-5 Kwankyu Lee wrote:

On the other hand, who would be the users of the distribution packages for whatever need? I wonder how they overlap with sage developers.

A concrete example of a useful standalone Sage module is CyPari2.  By including CyPari within SnapPy we are able to make it possible to compute number theoretic invariants of hyperbolic manifolds.  We are unable to use Sage's CyPari2 because we need to support Windows. But, as Oscar says, it should be easier to port individual modules to Windows than all of Sage.  That is the case with CyPari.

As another example, it would be great for SnapPy if we could also include Sage's interval arithmetic code, packaged as a standalone module.

- Marc

[kcrisman]

For me, this is another instance of a user point of view versus a developer point of view.

Coming from someone who tries to make SageMath easily available for ordinary people, Marc's views are all the more valuable IMHO.

An individual mathematician who only needs some portion of the sage library may install only his favorite distribution package.

As I have said so many times before, apparently to no avail, I strongly disagree with this point of view.  It is not reasonable to expect an individual mathematician, or student, to be able to decide which portions of Sage are needed, nor even to know which portions can be installed separately.  An individual who wants to use Sage should just install Sage.  All of it.  And that installation should be very easy.  It should certainly not require deciding which portions of Sage might be needed in which contexts. 

+1 to both of these. 

[Dima Pasechnik]

On Mon, Oct 7, 2024 at 10:38 AM Oscar Benjamin
<oscar.j....@gmail.com> wrote:
>
> On Mon, 7 Oct 2024 at 06:05, Kwankyu Lee <ekwa...@gmail.com> wrote:
> >
> > On Monday, October 7, 2024 at 12:24:04 AM UTC+9 marc....@gmail.com wrote:
> >
> > > I would say that the motivation is to make it possible for a developer to include a self-contained portion of sage in a separate project without having to make that project as large as a full Sage distribution.
> >
> > OK. That is perhaps a major case that one needs a portion of the sage library. I can also imagine a user installing a smaller sage library just because it is enough for him/her.
>
> I suspect that if Sage was modularised then over time people would
> work on making some parts more portable so that some subset of Sage
> would be natively installable in WASM or on Windows etc.

given a number of essential Sage dependencies which don't work on
(native) Windows,
such as GAP and Singular, native Windows port probably won't happen -
unless funding amounting
to several person-years can be found to fix these, by people working
full-time on this for years.
The situation with WASM is similar. (I heard about GAP people
experimenting with WASM, but it's very preliminary).

> Greater
> portability would make Sage more attractive for use as a software
> dependency. There would also be a strong incentive to try to carve out
> a meaningful subset of Sage for end-users that was more portable but I
> don't know how realistic it could be to achieve that.

This is realistic, but I doubt that the current modularisation project
is of any help.
What is realistic, and what modularisation has reportedly made more
difficult, is packaging Sage for Linux distros.

Dima

>
> --
> Oscar

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAHVvXxTu5rD%3DSJCvEJQPupA%2Bc2sxS79AEjNdkkoXOUOwfodzzw%40mail.gmail.com.

[Dima Pasechnik]

"pip-install networkx", and do (some) graph theory with it, yes.
Or "<your-OS-package-manager> install sagemath", and use it for anything.

If we concentrated on facilitating the latter, rather than on
distribution packages, it could have been there now.

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/c7a28dfa-2456-406e-8abd-a83a9beccbcen%40googlegroups.com.

[Dima Pasechnik]

Sage has a few things which can serve as interval arithmetic code,
e.g. its interface to flint/arb (arb used to be a separate package).
The latter may be used separately - https://pypi.org/project/python-flint
(not sure how easy for you would be to use it in SnapPy)

Dima


>
> - Marc

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/b5d5eeb2-238f-4520-9715-4b6d3704ba65n%40googlegroups.com.

[William Stein]

Hi,

There are a number of big chunks of functionality in Sage, like interval arithmetic (I was just going to say this and Marc beat me to it), and the sage preparser, which could be valuable as separate packages that Sage uses.

License of Sage is GPLv3, whereas the vast majority of the scientific Python (etc.) ecosystem is BSD/MIT/Apache/etc (e.g., sympy: https://www.sympy.org/en/index.html).   Thus most of the broader Python ecosystem won't touch any Sage packages with a 10 foot pole, even if they are useful and better than what is available.  For example, many would select sympy over a similar part of sage, just because of the license.   I don't think there is anything to be done about this.  It's just something to be aware of, which will forever significantly limit the impact of Sage on the broader Python ecosystem.

 - William

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/b5d5eeb2-238f-4520-9715-4b6d3704ba65n%40googlegroups.com.

--

William (http://wstein.org)

[Nils Bruin]

On Monday 7 October 2024 at 06:26:52 UTC-7 marc....@gmail.com wrote:

A concrete example of a useful standalone Sage module is CyPari2.  By including CyPari within SnapPy we are able to make it possible to compute number theoretic invariants of hyperbolic manifolds.  We are unable to use Sage's CyPari2 because we need to support Windows. But, as Oscar says, it should be easier to port individual modules to Windows than all of Sage.  That is the case with CyPari.

As another example, it would be great for SnapPy if we could also include Sage's interval arithmetic code, packaged as a standalone module.

Both CyPari and interval arithmetic are examples of packages in Sage that interface with other libraries: CyPari with GP/Pari and interval arithmetic with MPFI or (if you prefer ball arithmetic) ARB. So these are examples of functionality that is already available in stand-alone form, but for which access is made more convenient by a python interface that was developed as part of Sage. There is a good example out there, in the form of MPMath, of a python library that does not *depend* on sage but can work well with Sage if it is present. It was not developed as part of sage but was developed while sage was definitely one of the users.

There are other examples, like CyPari, that started out as part of Sage and were later spun off as independent units.

I have yet to see a convincing example where chopping up core architecture of sagemath (like the coercion framework, the category framework, etc) leads to usable bits for other projects (that get used!) It would be interesting to see something like that, particularly to evaluate what direction modularization should take.

[Michael Orlitzky]

On Mon, 2024-10-07 at 07:55 -0700, William Stein wrote:
> Hi,
>
> There are a number of big chunks of functionality in Sage, like interval
> arithmetic (I was just going to say this and Marc beat me to it), and the
> sage preparser, which could be valuable as separate packages that Sage uses.
>
> License of Sage is GPLv3, whereas the vast majority of the scientific
> Python (etc.) ecosystem is BSD/MIT/Apache/etc (e.g., sympy:
> https://www.sympy.org/en/index.html). Thus most of the broader Python
> ecosystem won't touch any Sage packages with a 10 foot pole, even if they
> are useful and better than what is available. For example, many would
> select sympy over a similar part of sage, just because of the license. I
> don't think there is anything to be done about this. It's just something
> to be aware of, which will forever significantly limit the impact of Sage
> on the broader Python ecosystem.

You didn't say that it does, just that they think it does, but... how
would Sage being GPLv3 affect these people in any way?

The GPLv3 is nearly irrelevant for end users. It only stops you from
doing pathological things like modifying the source code, compiling the
result, shipping the binaries, and then refusing to publish the source
code to those binaries. Adding a sage-* package to your dependencies in
pyproject.toml does not even constitute distribution.

[William Stein]

Dima: 

If we concentrated on facilitating the latter, rather than on
distribution packages, it could have been there now.

+1

Nils:

> I have yet to see a convincing example where chopping up core architecture of sagemath (like the coercion framework, the category framework, etc) leads to usable bits for other projects (that get used!) It would be interesting to see something like that, particularly to evaluate what direction modularization should take.

Licensing is a critical part of evaluating this.  For example, mpmath is BSD licensed.  Even if chopping up the core of Sage produced things that are useful, a lot of projects wouldn't touch them due to the GPLv3 license.  (Networkx used to be GPL licensed and they somehow managed to relicense as BSD.)     Michael Orlitzky dismissed my comments about licensing in this thread, and I really wish he were correct. 

William

--

William (http://wstein.org)

[Michael Orlitzky]

On Mon, 2024-10-07 at 11:47 -0700, William Stein wrote:
>
> Licensing is a critical part of evaluating this. For example, mpmath is
> BSD licensed. Even if chopping up the core of Sage produced things that
> are useful, a lot of projects wouldn't touch them due to the GPLv3
> license. (Networkx used to be GPL licensed and they somehow managed to
> relicense as BSD.) Michael Orlitzky dismissed my comments about
> licensing in this thread, and I really wish he were correct.
>

I'm not being dismissive, I'm just trying to understand what the
problem is. Does the license stop people from doing things that they
want to do, or is the issue that people don't understand licenses?
They're very different problems.

[Dima Pasechnik]

I more than agree with Michael here. The fears about GPL are old
groundless fears nowadays.
(unless you want to get rich quick, see
https://pivot-to-ai.com/2024/10/04/ex-coiner-y-combinator-startup-bro-dawg-i-chatgptd-the-license-cant-be-bothered-with-legal/)

Mind you, Mathematica (!) bundles Flint (which is GPL, and depends on
GPLd libraries).
(see e.g. https://wolfram.com/broadcast/video.php?c=104&v=3815&ob=title&o=ASC&disp=grid&p=53)

Dima

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/a7e73053f605427829f8158a53513433b7ca8def.camel%40orlitzky.com.

[Dima Pasechnik]

Sage crucially depends on GPLd things: GMP, mpfr, mpc, Flint, Pari,
GAP, Singular, ppl. It's a bit moot to wish that Sage was BSD
licensed,
in view of this.
I also don't get your point about mpmath - as it also depends on GMP etc.

>
> William
>
>
> --
> William (http://wstein.org)
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CACLE5GDB9Yz%3DhUnOG5rVNWStWK-CyyTTW4iDTx2mF%2BL_QNW8fA%40mail.gmail.com.

[Dima Pasechnik]

If could be that particular "angel investors" have a beef with GPL
(some surely do),
and push this through by only funding things
which don't use GPL.

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/e6274d873c64cefe3dba4e1e122cf2cb6f4b5d77.camel%40orlitzky.com.

[Oscar Benjamin]

On Mon, 7 Oct 2024 at 20:19, Dima Pasechnik <dim...@gmail.com> wrote:
>
> Mind you, Mathematica (!) bundles Flint (which is GPL, and depends on
> GPLd libraries).

FLINT is LGPL as are its dependencies GMP and MPFR.

I'm no expert on license terms but Wikipedia says:

The main difference between the GPL and the LGPL is that the latter
allows the work to be linked with (in the case of a library, "used
by") a non-(L)GPLed program, regardless of whether it is licensed
under a license of GPL family or other licenses.

That suggests that LGPL rather than GPL is needed for Mathematica to
bundle FLINT in the way that it does.

--
Oscar

[Dima Pasechnik]

Indeed, Flint is dual licensed under GPL and LGPL - so why don't we
re-license sagelib under LGPL then. E.g. that's how gmpy2 is licensed:

sage: gmpy2.license()
'The GMPY2 source code is licensed under LGPL 3 or later. The
supported versions of the GMP, MPFR, and MPC libraries are also
licensed under LGPL 3 or later.'

I don't know whether it'd need a lot of individual approvals, but
GPL->LGPL is certainly done quite often. E.g. GMP was relicenced under
LGPL from version 6 on.

Dima

>
> --
> Oscar

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAHVvXxSqfrUkMSwOwg7zmfFUtDPFh34G7neiZHgbAEZ0_ibghw%40mail.gmail.com.

[Edgar Costa]

> Mind you, Mathematica (!) bundles Flint (which is GPL, and depends on
GPLd libraries).

This is wrong. Flint is LGPLv3, which is what enables Mathematica to link to it.

[David Lowry-Duda]

On 22:35 Mon 07 Oct 2024, Dima Pasechnik wrote:
> Indeed, Flint is dual licensed under GPL and LGPL - so why don't we
> re-license sagelib under LGPL then.

> ...[SNIP]...

> I don't know whether it'd need a lot of individual approvals, but
> GPL->LGPL is certainly done quite often. E.g. GMP was relicenced under
> LGPL from version 6 on.

Yes, relicensing GPL->LGPL would require approval from every contributor. Or at least every contributor to the parts of sagelib that would be made available under LGPL. I think getting approval from all contributors would be very hard, maybe impossible.

Also, the parts of sagelib that would be available under LGPL can't include GPL code. Disentangling this seems hard.

- DLD

[Nils Bruin]

Summarizing what I've seen come by here:

 - the examples we have of bits of software  developed as part of sage that ended up as library components of other projects are peripheral, interfacing parts that were spun off into independent libraries.

 - we don't have examples of core functionality of sage getting adopted as a component of another project

 - licensing under GPL may or may not be an obstruction to this.

 - relicensing core parts of sage may well be impossible because it would involve tracking down all relevant contributors to agree to relicensing and that may be infeasible (plus, each individual developer on that list would have a veto).

Against that background: what purpose would modularizing the core of sage serve and what benefit do we think to derive from it? It's a nice idea that sagemath code would be reused elsewhere, but against the backdrop that we have here I'm not so sure it's realistic to expect this (and I'd wonder what benefit *sage* would have from this).

Note that the message quoted at the start of this thread:

https://lists.debian.org/debian-science/2024/07/msg00023.html

wasn't about code reuse at all: it was from a debian packager who took source distributions from pypi to build "manageable" chunks. Quite frankly, I don't see why a pypi package is more manageable if it is split up (wouldn't one monolithic pypi distribution work just as well?), but if this splitting up makes packaging sagemath more palatable for linux distributions then I think we have a real win. Once distributions are happy packaging sage in a conventional way, perhaps people could easily generate flatpaks or snap packages as well, which should be more stable (they'll be huge, though).

I could see how there are other arguments for splitting sage up: components like jupyter and much of the docbuilding infrastructure is only very weakly tied to the sagemath codebase: communication takes place through protocols and file formats rather than through an API. So packaging sagelib without these and then perhaps a distribution that bundles sagelib with a jupyter server for convenience could make things easier (but it might also be that we're already very close to doing that anyway, and perhaps we literally do that)

[Marc Culler]

On Tuesday, October 8, 2024 at 1:23:55 PM UTC-6 Nils Bruin wrote:

 - the examples we have of bits of software  developed as part of sage that ended up as library components of other projects are peripheral, interfacing parts that were spun off into independent libraries.

 - we don't have examples of core functionality of sage getting adopted as a component of another project

The most obvious reason for this is that it too hard for a project to adopt core functionality of Sagewith the current organization of Sage.  Making that easier was one of the main motivations for the modularization project.  So this is circular reasoning.

 

Against that background:

Do you mean against the background based on circular reasoning?

what purpose would modularizing the core of sage serve and what benefit do we think to derive from it?

Making it practical, or at least possible, for core functionality of sage to be reused.  That has always been the purpose.

 

Once distributions are happy packaging sage in a conventional way, perhaps people could easily generate flatpaks or snap packages as well, which should be more stable (they'll be huge, though).

The Sage_macOS binary package is a (compressed) 1GB download which expands to about 3.5GB.  A 1GB download is not huge these days.  The size of the raw Sage github repository is about 620MB.  After building Sage it swells to about 10GB.  A snap or app-image would use squashfs, which is compressed, even after installation.  So it shouldn't be larger than, say, 2GB.  (Flatpaks are not compressed after installation.)

Pypi packages have a default size limit of 100MB per file and 10GB per project.

 

- Marc

[Nils Bruin]

On Tuesday 8 October 2024 at 13:20:54 UTC-7 marc....@gmail.com wrote:

On Tuesday, October 8, 2024 at 1:23:55 PM UTC-6 Nils Bruin wrote:

 - the examples we have of bits of software  developed as part of sage that ended up as library components of other projects are peripheral, interfacing parts that were spun off into independent libraries.

 - we don't have examples of core functionality of sage getting adopted as a component of another project

The most obvious reason for this is that it too hard for a project to adopt core functionality of Sagewith the current organization of Sage.  Making that easier was one of the main motivations for the modularization project.  So this is circular reasoning.

Indeed, it being hard to use the sagemath core in other projects would be an obstacle to doing so. However, it is not obvious to me that this is the only obstruction. It could be that there is simply no demand, regardless of how easy it is. And, as pointed out, licensing could be another obstruction. So it seems to me that the opportunity of benefit to sage from use in other projects is an unknown quantity at the moment.

On the other hand, it seems there are other ways in which making sage more modular may bring benefits. As you're pointing out, sage still fits within 10GB in source, so it looks like sagemath could just be one pypi package. And yet, a debian packager is enthusiastic about being able to do it in smaller bits.

Personally, I've seen sage develop as this monolithic, tightly interlinked library, where even the order of imports is delicate because of circularities. It seems to me that splitting that up comes at a considerable cost to actually perform. Particularly if, in addition, we get extra load imposed because all doctests now have to declare their prerequisites.

To get buy-in from a lot of other developers, there will need to be a clear and obvious benefit to implementing this modularization. I think it will be much easier to get that buy-in by exhibiting some sure or nearly sure pay-off from the investment. The hope for a post-fact pay-off in the form of code reuse will probably be a bit too nebulous for many developers to put up with the extra costs.

[Oscar Benjamin]

> > On Tuesday 8 October 2024 at 13:20:54 UTC-7 marc....@gmail.com wrote:
>
> On Tuesday, October 8, 2024 at 1:23:55 PM UTC-6 Nils Bruin wrote:
>

> > Pypi packages have a default size limit of 100MB per file and 10GB per project.
>

> As you're pointing out, sage still fits within 10GB in source, so it looks like sagemath could just be one pypi package.

I think that you have misunderstood the limits that Marc was referring
to. The 100MB file limits means that a single project installer file
should be less than 100MB. The 10GB project limit means that all
installer files for each Python version and each OS for all versions
of a project should not add up to more than 10GB. A single Sage
installer file for one OS, Sage version and CPython version comes
close to the total file size limit allowed for any project by default
in PyPI.

You can see the release history for python-flint here:
https://pypi.org/project/python-flint/#history

You can see the file sizes for the most recent python-flint release here:
https://pypi.org/project/python-flint/0.7.0a5/#files

The total of those file sizes summed over all releases of python-flint
must not exceed 10GB (without special permission from PyPI).

Currently python-flint does not provide binaries for Windows or Linux
on ARM. Doing so would almost double the total file size per release.
As of CPython 3.13 there are free-threaded builds. Providing binaries
for those could end up doubling the total file size as well.

--
Oscar

[Nils Bruin]

On Tuesday 8 October 2024 at 15:40:07 UTC-7 oscar.j....@gmail.com wrote:

> As you're pointing out, sage still fits within 10GB in source, so it looks like sagemath could just be one pypi package.

I think that you have misunderstood the limits that Marc was referring
to. The 100MB file limits means that a single project installer file
should be less than 100MB. The 10GB project limit means that all
installer files for each Python version and each OS for all versions
of a project should not add up to more than 10GB. A single Sage
installer file for one OS, Sage version and CPython version comes
close to the total file size limit allowed for any project by default
in PyPI.

 

Thanks for the clarification. So Pypi is wholly unsuitable for distributing a monolithic sagelib. If we want sagelib distributed through pypi then it *has* to be chopped up in order to make it fit. Distributing sagelib through pypi sounds like a pretty good idea, given that pypi is the de facto standard for distributing python software. So it sounds like considerations for reuse in other projects might not even be so relevant for motivating modularization: we already need it to make sage fit onto pypi. But that might also change the consideration for how sagemath needs to be chopped up. For fitting onto pypi, it's sufficient it gets chopped up in roughly equal-sized parts that can be built separately, in some specified order.

[Dima Pasechnik]

I read on Python Discource a proposal to allow externally hosted

wheels on PyPI. With PyPI only hosting metadata and

checksums. I imagine this will lift the size constraints, if accepted.

Dima

[Gonzalo Tornaría]

FWIW, the source distribution of sagemath 10.4 (from pypy) is about 20M, see https://pypi.org/project/sagemath-standard/#files

We build the sagemath package for void linux from this source alone, obtaining a binary package of about 55M, see https://voidlinux.org/packages/?arch=x86_64&q=sagemath.

The binary package is compressed with zstd. It expands to 548M

$ du -sh /usr/lib/python3.12/site-packages/sage
548M /usr/lib/python3.12/site-packages/sage

of which 154M is cython debug information (I'm actually not sure if it's necessary to ship it):

$ du -sh /usr/lib/python3.12/site-packages/sage/cython_debug/
154M /usr/lib/python3.12/site-packages/sage/cython_debug/

This is what could be called "a monolithic sagelib". Everything else is external dependencies and they can be in separate packages.

Just as a point of comparision: scipy 1.14.1 source distribution is 59M (https://pypi.org/project/scipy/#files) and the binary package for void linux is 19M (https://voidlinux.org/packages/?arch=x86_64&q=python3-scipy).

Note, however, that some dependencies of sagemath are huge. For instance, gap 4.13.1 complete source is more than 500M, and our binary package for void linux is 209M (https://voidlinux.org/packages/?arch=x86_64&q=gap).

Maybe it's more useful to split off the big components of sagemath into separate packages (e.g. gap) than try to split sagemath in pieces. And this is partly already done!  Indeed gap packages for most distros are already available.

IMHO, the main difficulty with packaging sagemath for a distro is not "sagelib is monolithic" but rather that "sage-the-distro is too tightly coupled, almost monolithic". Indeed, sagelib is too rigid in the specific versions of dependencies it requires. This is a hard problem, but we are learning and getting better at this. However, building sagelib in isolation of sage-the-distro is not a clearly supported configuration and I think only a few distro packages really use it, no end users.

I think that sagemath without e.g. polynomial rings or without groups, etc. would no longer be sagemath. If one accepts this, it means pari, singular, gap, are hard dependencies of sagemath. And thus it seems the only reasonable way to offer binary wheels of sagemath on pypi is that there are corresponding pypi packages for pari, singular, gap. IOW, distribute a monolithic sagelib via pypi / do not distribute a monolithic sage-the-distro via pypi.

Best,

Gonzalo

[Gonzalo Tornaría]

On Monday, October 7, 2024 at 10:26:52 AM UTC-3 marc....@gmail.com wrote:

On Monday, October 7, 2024 at 12:05:25 AM UTC-5 Kwankyu Lee wrote:

On the other hand, who would be the users of the distribution packages for whatever need? I wonder how they overlap with sage developers.

A concrete example of a useful standalone Sage module is CyPari2.  By including CyPari within SnapPy we are able to make it possible to compute number theoretic invariants of hyperbolic manifolds.  We are unable to use Sage's CyPari2 because we need to support Windows. But, as Oscar says, it should be easier to port individual modules to Windows than all of Sage.  That is the case with CyPari.

I used to think cypari2 was a good example of how to modularize sagemath; take a good useful piece that is really standalone and package it in pypi so it can be used / developed / tested / etc. to improve adoption.

For me cypari2 works really nice and it's not particularly difficult to package (except it broke with pari 2.17, but of course having this as a standalone package makes it much easier to fix it). What is the problem with windows?

Best,

Gonzalo

[Marc Culler]

On Tue, Oct 8, 2024 at 8:52 PM 'Gonzalo Tornaría' via sage-devel
<sage-...@googlegroups.com> wrote:

> For me cypari2 works really nice and it's not particularly difficult to package (except it broke with pari 2.17, but of course having this as a standalone package makes it much easier to fix it). What is the problem with windows?

The problem with Windows is just that cypari2 does not support it.
CyPari works fine on Windows. There is some trickiness required to
get cysignals to work on Windows. This is because sage's
implementation of sig_on calls setjmp and expects a signal handler to
call longjmp, but that scheme fails on Windows where each signal
handler runs in a separate thread with its own stack. However, CyPari
has a workaround for this.

- Marc

PS There is another problem with cypari2, completely unrelated to this
thread, which prevents SnapPy from using it, even on platforms which
are supported by cypari2. That problem is that huge memory leaks were
introduced in cypari2 by a redesign of the Python Gen class to try to
keep the Pari GEN managed by a Python Gen object on the Pari stack as
long as possible. Those leaks make cypari2 unusable for us. CyPari
has reverted to the original model, where each Pari GEN is moved to
the Pari heap when its wrapper Gen is constructed, and is freed from
the Pari heap in the wrapper's __dealloc__ method. That fixes the
leaks.

[Tobia...@gmx.de]

Currently, when you try to package sage on a new distro, you need to have all dependencies installed before you can even explore building sage. The modularization effort could be helpful in this regard, because it enables a more incremental approach where you first package a smaller subset of the dependencies, and then build (and distribute) only a subset of sage. I think this is what excited people on the debian mailing list. However, this is actually mostly a limitation of the current build system - once you can say: please compile only these parts of sage for which the correct dependencies are installed, you no longer get an advantage of modularization in this regard.

As for the main body of the modularization project, I agree there is a conflict between the needs of users and developers:

- As a user, I'm okay with having a core sage project and then install certain extensions/add-ons/plugins to extend the functionality (if needed). This is a pretty common approach to split large projects, and users shouldn't be too confused (at least once it's easy enough to discover in the docs which addon one needs to install to enable a certain feature).

- As a developer, I want a completely independent piece of sage code that I can directly use in my own project without pulling in a massive amount of transitive dependencies. In particular, these smaller modules need to be compiled with minimal dependencies and also their runtime interface needs be self-contained. And as Marc said, these modules need to be available on many platforms.

Perhaps it's less of a "conflict" but just that developer-focused modules need more work than user-focused ones. For example, the sagemath-bliss is a nice add-on that fulfills the needs of a user. But it cannot really be used in some other projects because you need to pass a sage graph to it. So to make it more useful someone needs to put more effort in it, and rework the public interface to be useful without references to the rest of the sage project.

Here is a concrete proposal to move forward with the modularization effort:

1) Create more extension distributions that users can install to complement the core functionality. The requirement is that these extensions should compile with the required external dependencies installed and with sagemath-standard installed. Ideally, they should be on pypi so that one can simply install them with `pip install sagemath-xyz`, and distributions are not required to ship all of them. The goal here is to make sagemath-standard leaner and easier to install (and compile) by moving some less-often-used external dependencies to external modules.

2) Spearhead some developer-focused modules. Perhaps, start with what Marc needs for snappy, and bundle exactly this part of sage in a new distribution. The goal here would be to see what problems you encounter and how to overcome them (eg what I foresee are a lot of circular import issues and that our current build system needs be overhauled to support building on windows).

3) Convert more and more of the user-focused distributions to developer-focused ones.

A perhaps rather important question is, who actually wants to work on any aspect of this?

[Dima Pasechnik]

specifically, this is PEP 759 – External Wheel Hosting (https://peps.python.org/pep-0759)

and there

https://peps.python.org/pep-0759/#addressing-pypi-limits

[Gonzalo Tornaría]

On Wednesday, October 9, 2024 at 2:17:34 AM UTC-3 marc....@gmail.com wrote:

The problem with Windows is just that cypari2 does not support it.
CyPari works fine on Windows. There is some trickiness required to
get cysignals to work on Windows. This is because sage's
implementation of sig_on calls setjmp and expects a signal handler to
call longjmp, but that scheme fails on Windows where each signal
handler runs in a separate thread with its own stack. However, CyPari
has a workaround for this.

Thanks for the explanation. As far as I know, cysignals is another instance of a component originally developed for sagemath, about maybe 20 years ago, then separated into a standalone package. In theory, this separation should make it easier to support other OS, but the fact that windows support hasn't materialized yet is a sign that this is not so clear. The design may be showing its age, but it seems to me to be fundamentally intrinsic to sagemath. There are other problems with cysignals (e.g. https://github.com/sagemath/sage/issues/37026).

PS There is another problem with cypari2, completely unrelated to this
thread, which prevents SnapPy from using it, even on platforms which
are supported by cypari2. That problem is that huge memory leaks were
introduced in cypari2 by a redesign of the Python Gen class to try to
keep the Pari GEN managed by a Python Gen object on the Pari stack as
long as possible. Those leaks make cypari2 unusable for us. CyPari
has reverted to the original model, where each Pari GEN is moved to
the Pari heap when its wrapper Gen is constructed, and is freed from
the Pari heap in the wrapper's __dealloc__ method. That fixes the
leaks.

The pari stack is also fundamentally intrinsic to pari. I'm guessing there's a tension here between memory and performance.

Both issues seem to be important, but are contained in separate pypi packages. It doesn't seem that lack of modularization is what prevents them from working well on windows.

Best,

Gonzalo

 

[Marc Culler]

On Wednesday, October 9, 2024 at 9:32:27 AM UTC-6 Gonzalo Tornaría wrote:

As far as I know, cysignals is another instance of a component originally developed for sagemath, about maybe 20 years ago, then separated into a standalone package. In theory, this separation should make it easier to support other OS, but the fact that windows support hasn't materialized yet is a sign that this is not so clear.

Windows support *did* materialize.  The modified cysignals module included with CyPari *does* work on Windows.  And that also happened many years ago.  What did not happen was for the Sage cysignals project to incorporate the changes that make it work on Windows and to produce Windows packages on PyPi.  I always assumed that the reason for that was that there would be no point in doing so,  given that Sage is the only user of cysignals (with CyPari as the sole exception) and Sage doesn't support Windows.  Moreover, adding Windows support adds a maintenance burden.  I think that the cysignals project never intended to produce anything that would work on non-posix systems.  I think the same is true of all projects under the Sage umbrella, although it is obviously not the case for the "python ecosystem" in general.

 - Marc

[Oscar Benjamin]

I actually just took a look at cysignals to see whether it would make
sense for python-flint to use it but immediately decided that the
answer was no once I saw that there were no Windows wheels on PyPI.

--
Oscar

[Dima Pasechnik]

PRs to cysignals are most welcome. It should not be impossible to hook up a Windows CI and wheel builder in cysignals, too.

[William Stein]

For a timely example of how PyPI size limits can be very real, JupyterLab's is hitting this right now:

https://github.com/jupyterlab/jupyterlab/issues/16859

ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
         Project size too large. Limit for project 'jupyterlab' total size is 10
         GB. See https://pypi.org/help/#project-size-limit

For them, they are hitting a total size limit of 10 GB.

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/79e2d59c-162d-4162-acb7-7ee7ae958324n%40googlegroups.com.

--

William (http://wstein.org)