What was/is/will be the purpose of maintaining the Sage distribution?

[Matthias Koeppe]

A previous sage-devel thread led to this question, which I think is important and timely to discuss.

**What was/is/will be the *purpose* of maintaining the Sage distribution?**
(Historically; as of today; and looking forward by a year or two.)

Here are some of my thoughts on this question:

1. Ease of installation.

Historically, an important purpose was to make loads of software packages, including many poorly maintained math software, easy to install.
In particular -- easy to install for a user on a shared machine ("big department compute server") without root access.
And in particular -- on shared machines with long outdated distributions ("Department IT installed it when the machine was bought, 10 years ago.")

As of today, it is plausible that such situations still exist. There are definitely still "shared compute server" situations (in particular, HPC clusters) where users cannot use container technology such as Docker and lxc, and therefore cannot use Linux distribution packaging solutions (archlinux, debian, ...). Overall I don't think we have sufficient insights about our worldwide user base to be sure. Here the Sage distribution still may have a value for users. Another option is non-root installable packaging: essentially, conda-forge (although nix and guix may be other options). But as I wrote earlier, there are still loose ends regarding Sage development in a pure conda environment (note that it is still marked as "experimental" in https://doc.sagemath.org/html/en/installation/conda.html#using-conda-to-provide-all-dependencies-for-the-sage-library-experimental), and also optional packages are missing.

Going forward, if the loose ends are ironed out (I'm mixing metaphors for comical effect) and missing packages added, I think that Sage installation, use, and development can be fully supported on top of conda-forge. This takes engagement and work; and this could certainly be done faster if a decision to abandon the Sage distribution on a specific release / date is made, because then substantial resources are freed. A time frame of a year or two could be realistic.

(I am also working on another deployment path using Python wheels, but this is much more work than just fixing the remaining conda-forge problems.)


2. Control of dependencies and the "many upstreams - many downstreams" problem.

For Sage developers, the Sage distribution is a way to have direct control over all dependencies -- that's the Sage distribution's role as a "reference distribution".

(This role has been weakened since we introduced the spkg-configure mechanism of working with system packages, of course. This is an activity that does make sense one package at a time, but details such as how strict we are in what we accept from the system matter; see Michael's thoughts in his message.)

But still the following points apply:

a) If a critical bug is discovered, we can patch it and don't have to rely on people and infrastructure "outside the project" to fix things for us.
Of course, we have been very lucky that packagers for many distributions have been consistently highly engaged with the project; but this is not something that we can take for granted.

b) And, of course, more Sage developers can become contributors to the packaging communities; but there is the real danger that taking care of both upstream development *and* downstream packaging for the same project can lead to developer burnout.

c) There is a danger that our project's endorsing of a particular packaging solution (e.g. conda-forge) could alienate highly engaged packagers for other systems. And if left unchecked, it could also lead to the re-introduction of code that is too tightly coupled with the specifics of conda-forge packaging.

[Dima Pasechnik]

On Thu, Apr 27, 2023 at 2:15 AM Matthias Koeppe
<matthia...@gmail.com> wrote:
>
> A previous sage-devel thread led to this question, which I think is important and timely to discuss.
>
> **What was/is/will be the *purpose* of maintaining the Sage distribution?**
> (Historically; as of today; and looking forward by a year or two.)
>
> Here are some of my thoughts on this question:
>
> 1. Ease of installation.
>
> Historically, an important purpose was to make loads of software packages, including many poorly maintained math software, easy to install.
> In particular -- easy to install for a user on a shared machine ("big department compute server") without root access.
> And in particular -- on shared machines with long outdated distributions ("Department IT installed it when the machine was bought, 10 years ago.")
>
> As of today, it is plausible that such situations still exist.

There is no "mystery HPC cluster" on the list of platforms supported
by Sage. I don't see a point in "what if" argument like this.
We know it's not possible to build a modern gcc using gcc5 or whatever
outdated thing there can be, and let us leave this
potential (and very small) segment alone.

> There are definitely still "shared compute server" situations (in particular, HPC clusters) where users cannot use container technology such as Docker and lxc, and therefore cannot use Linux distribution packaging solutions (archlinux, debian, ...). Overall I don't think we have sufficient insights about our worldwide user base to be sure. Here the Sage distribution still may have a value for users. Another option is non-root installable packaging: essentially, conda-forge (although nix and guix may be other options). But as I wrote earlier, there are still loose ends regarding Sage development in a pure conda environment (note that it is still marked as "experimental" in https://doc.sagemath.org/html/en/installation/conda.html#using-conda-to-provide-all-dependencies-for-the-sage-library-experimental), and also optional packages are missing.
>
> Going forward, if the loose ends are ironed out (I'm mixing metaphors for comical effect) and missing packages added, I think that Sage installation, use, and development can be fully supported on top of conda-forge. This takes engagement and work; and this could certainly be done faster if a decision to abandon the Sage distribution on a specific release / date is made, because then substantial resources are freed. A time frame of a year or two could be realistic.
>
> (I am also working on another deployment path using Python wheels, but this is much more work than just fixing the remaining conda-forge problems.)
>
>
> 2. Control of dependencies and the "many upstreams - many downstreams" problem.
>
> For Sage developers, the Sage distribution is a way to have direct control over all dependencies -- that's the Sage distribution's role as a "reference distribution".
>
> (This role has been weakened since we introduced the spkg-configure mechanism of working with system packages, of course. This is an activity that does make sense one package at a time, but details such as how strict we are in what we accept from the system matter; see Michael's thoughts in his message.)

At this point, you start contradicting yourself. There is no value,
only added costs of maintainance, in carrying on with spkgs for which
viable alternatives are available, and such packages may be dropped
from Sage, one at a time, with no loss whatsoever.
Primary candidates here are python3 and gcc, packages you refuse to
drop due to "lack of greater plan".
But there is no greater plan needed here, these are just dead meat
which only gets used in automated testing, if at all.

We have spent years running in a vicious circle of endless upgrades of
packages we can use from the OS, such a waste.

So my proposal is to drop python3 and gcc spkgs immediately, and
gfortran upon a bit of investigation (only needed on macOS).
The next in line will be jupyter and friends.

>
> But still the following points apply:
>
> a) If a critical bug is discovered, we can patch it and don't have to rely on people and infrastructure "outside the project" to fix things for us.
> Of course, we have been very lucky that packagers for many distributions have been consistently highly engaged with the project; but this is not something that we can take for granted.
>
> b) And, of course, more Sage developers can become contributors to the packaging communities; but there is the real danger that taking care of both upstream development *and* downstream packaging for the same project can lead to developer burnout.

We already have developer burnout, I myself am a prime example.

Dima

>
> c) There is a danger that our project's endorsing of a particular packaging solution (e.g. conda-forge) could alienate highly engaged packagers for other systems. And if left unchecked, it could also lead to the re-introduction of code that is too tightly coupled with the specifics of conda-forge packaging.
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/fd00a6ea-5874-4bd3-9fcd-ec1462543760n%40googlegroups.com.

[kcrisman]

As of today, it is plausible that such situations still exist.

I am wondering about such situations existing in less-resourced areas globally (which would include less-resourced parts of developed countries).  One big advantage of Sage-the-distribution historically was the ability to make USB drives that had the complete thing (maybe also a Linux VM?) on them, from which one could boot.

It strikes me that many arguments for removing the distribution along these lines (not the developer side, which is also important) are akin to those arguments which assume one should "just" use a remote option for Sage at all times.  Yes, that has been seriously made on multiple occasions, though usually not on this list.  But even "post-pandemic" there are still plenty of reliable high-speed internet deserts even where I live on the US East Coast, much less around the world.  I wouldn't want to use CoCalc without a fairly new computer.

Likewise, there are plenty of people using 5-10 year old computers who, in principle, could be afforded Sage access, but for our continued upgrading.  (Again, see below for the developer side.)  Arguments about how they should upgrade or face security issues are fine, but in practice (whether for financial or other reasons) this is not how humans respond to those incentives, and presumably at least some of them might benefit from Sage.  A lot of the paradigm discussed on this list (but not all, for sure) focuses SO MUCH on people who have access to fairly recent technology, and that simply doesn't obtain.

As an example, how old of a Windows computer could one install the current Sage on?  I mean from scratch - not necessarily from source - using WSL, which I guess is now the main supported way to do so?  What about the Cygwin installer - does it still exist in older versions on sagemath.org mirrors, what does that support?  How easy is it for someone who does NOT know about compiling to install Sage on a not-too-recent Windows machine?  I bet it's easy to install the various M's ...

In any case, it would be very helpful for people who may be actively using Sage in less-resourced environment to chime in here.

Moving to the developer side:

a) If a critical bug is discovered, we can patch it and don't have to rely on people and infrastructure "outside the project" to fix things for us.
Of course, we have been very lucky that packagers for many distributions have been consistently highly engaged with the project; but this is not something that we can take for granted.

This is basically why William started Sage in the first place.  (Well, one reason!)  When I still had time to be an active developer, this was a major source of necessary work.  It's true that a lot of packages are now more responsive (or have been canned/subsumed into Sage), but presumably it could still be a problem, especially with some extremely math-specific packages that might not regularly update in a platform-agnostic way.  That said, presumably Python and gcc are no longer in the situation where we need to actively maintain a lot of patches to them.

 

b) And, of course, more Sage developers can become contributors to the packaging communities; but there is the real danger that taking care of both upstream development *and* downstream packaging for the same project can lead to developer burnout.

This (whether connected to upstream packaging or not) is really the most powerful argument for radical decoupling.  (Similarly to the GH transition.)  Clearly R fell in this category.  Reading the other thread did not really clarify for me whether python3 or gcc fell into this category, and I don't think it will be helpful to revisit that right now.  In any case, this should be weighed against Sage ease of access. 

One thing that might help all of this is having older versions of Sage *binaries* for such platforms readily available for download (as many of our upstream packages in fact do).  I don't think we are.  In fact, https://www.sagemath.org/mirrors.html was kind of scary - a lot of mirrors don't seem to have anything at all.  I will assume I missed a thread (quite likely that I did) that we were dropping binary support via mirrors completely, which needless to say would make my suggestion difficult to implement.  I do think it is the best way to provide quite fully-featured versions of Sage to people with less-modern setups (who probably now simply don't use Sage because they can't, or stick with older versions they already have, which we see from time to time on the support list) while still allowing for dropping some of this support when it sucks up too much developer effort.

[William Stein]

Hi,

To what extent does or could Conda with a little more work solve most
of these problems? There are some notes below from me poking around,
and I'm very optimistic.

I looked at

https://doc.sagemath.org/html/en/installation/conda.html

and I would love some further discussion of that and whether with
enough help it could be viable.
For example, on my M1 mac I just tried what seems to me to be the most
obvious first thing to do
based on the instructions:

(base) wstein@max ~ % mamba create -n sage sage python=3.11

and it fails with "└─ sage is uninstallable because there are no
viable options"
Obviously I'm going to nex try "mamba create -n sage sage", which works, but
that's not what our docs say to do. Incidentally, it took about a
minute to download
and install everything (and took 5.8GB disk instead of the 3GB disk of
the Sage mac app).
Then a few minutes of me being confused if I should do
"mamba activate sage" or "conda activate sage", and finally I typed "sage" and
strangely it just shows nothing at all while it mysteriously takes
about a minute
for sage to start the first time (on my M1 max laptop with SSD). Sage then
starts and works fine. Subsequent sage startups are very fast (e.g., 1 second).
What is it doing that first time, and why is it silent? It's very unnerving.

I also think this section
https://doc.sagemath.org/html/en/installation/conda.html#using-conda-to-provide-all-dependencies-for-the-sage-library-experimental
called "Using conda to provide all dependencies for the Sage library
(experimental)" is pretty exciting!

For many years when I gave talks about Sage, I had a slide: "What is
Sage", with two points:

1. A distribution of open source math software
2. A new library tying everything together

I definitely only started 1 out of necessity because nothing existed
at the time. My hope is that
at this point in time conda is good enough that maybe it could totally
solve 1, and we can focus on 2?

In any case, I think that migrating from "Sage the distribution" to
solving a lot of the misc environment issues
via conda would be very analogous to switching to Github, instead of
maintaining our own issue tracker.
I.e., if you want the latest version of sage on Ubuntu 22.04 (say),
then our recommendation is "use conda",
and we put effort into making Sage-via-conda extremely good. If you
want some random version of sage,
then you can use system packages.

For CoCalc.com, the key thing we need is a way to have self-contained
stable installations of sage-9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8,
10.0, etc. all on the same Ubuntu system, all at once, and have them
not get screwed up when we do normal system updates. Doing major
Ubuntu version updates (e.g., 20.04 --> 22.04) doesn't have to be
supported.
My impression is that conda potentially solves this problem at least
as well as sage-the-distribution does right now.

-- William

PS Thanks again to the people who put so much work into packaging sage
and its dependencies for conda!

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/8b0c2df5-c3be-406c-a1fd-fd0506e5f946n%40googlegroups.com.



--
William (http://wstein.org)

[Michael Orlitzky]

On Thu, 2023-04-27 at 05:12 -0700, kcrisman wrote:
>
> As an example, how old of a Windows computer could one install the current

> Sage on? ...

>
> In any case, it would be very helpful for people who may be actively using

> Sge in less-resourced environment to chime in here.
>

My desktop is 19 years old and my laptop is now 15. Switching between
branches can cause 20 hours of recompile time. If I'm lucky and don't
need to use the result, I can cut that in half with -O0. If this were
spent doing something useful, it would be more tolerable, but it's
spent fetching, compiling, and installing packages that are already on
my system. The test suite can take another full day to run -- some of
that is useful, but a lot is not. This is the biggest impediment to the
use and development of sage on an old system.

Upgrading (say) OpenSSL on the system is a sunk cost because it happens
anyway. The only question is whether or not I *also* have to repeatedly
rebuild some highly specific version that sage wants. The problem is
not only the SPKGs, but the way sagelib is designed around them;
eliminating the sage distribution would force us to address those
indirect inefficiencies in addition to immediately eliminating the
direct ones.

(I'm planning upgrades in the next year or two, but it will be to
relatively low power RISC-V hardware. There are moral, legal,
environmental, and other non-financial reasons why people use "old"
hardware. But of course the financial reasons are very real too.)

[Isuru Fernando]

> it fails with "└─ sage   is uninstallable because there are no viable options"

We don't have a python 3.11 version of sage in conda yet. I started a PR manually as the automatic update

failed for some reason.

> What is it doing that first time, and why is it silent?  It's very unnerving.

macOS 10.15+ does some shady things when users request to run "untrusted" applications. For eg:

% clang hello.c
% time ./a.out     
Hello world!
./a.out  0.00s user 0.00s system 0% cpu 0.380 total
% time ./a.out
Hello world!
./a.out  0.00s user 0.00s system 62% cpu 0.006 total

See https://lapcatsoftware.com/articles/catalina-executables.html for a possible explanation.

Sage loads hundreds of dynamic libraries not all at the same time, so macOS sends multiple requests

to Apple servers.

> and took 5.8GB disk instead of the 3GB disk of the Sage mac app).

Yes, conda packages usually come with batteries included which means packages come with their

optional build time dependencies installed. That's usually not a big deal for other packages, but

Sage is special in that it has tons of dependencies.

As usual, the biggest hurdle to making things work more seamlessly is manpower.

Most of the niche packages that sage depends on are maintained by me and Julian and improvements

to supporting conda in the sage build system are mostly Matthias and Tobias.

Isuru

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CACLE5GCTmMFcJw_mw2_UJKu4H6nvhOCJ0t5sbUoT1a090sh2aw%40mail.gmail.com.

[William Stein]

Isuru,

Thanks for answering all my questions. I just want to reiterate that
I'm thrilled with what you are doing and greatly appreciate it!

William

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CA%2B01voOHYPmYOUm8_2AAoPcPibXbHS3sZ_-BM9PfwoJ%2BXpu4zg%40mail.gmail.com.



--
William (http://wstein.org)

[Michael Orlitzky]

On Thu, 2023-04-27 at 05:49 -0700, William Stein wrote:
> Hi,
>
> To what extent does or could Conda with a little more work solve most
> of these problems? There are some notes below from me poking around,
> and I'm very optimistic.

This isn't the first time the idea has come up. Burcin got pretty far
with it using Gentoo Prefix in place of Conda:

https://groups.google.com/g/sage-devel/c/XFJn3jGVBG8

Nix could also work. All three serve roughly the same purpose.

[Matthias Koeppe]

On Thursday, April 27, 2023 at 5:49:50 AM UTC-7 William Stein wrote:

To what extent does or could Conda with a little more work solve most

of these problems? [...]

I also think this section
https://doc.sagemath.org/html/en/installation/conda.html#using-conda-to-provide-all-dependencies-for-the-sage-library-experimental
called "Using conda to provide all dependencies for the Sage library
(experimental)" is pretty exciting!

Yes, I think this mode of installation should be the future default of Sage for developers.

In any case, I think that migrating from "Sage the distribution" to
solving a lot of the misc environment issues
via conda would be very analogous to switching to Github, instead of
maintaining our own issue tracker.

Indeed, and just like in our successful transition to GitHub, a clean planned switchover to the new model is the most effective solution. 

Attempts to do this gradually (such as the attempt of a soft transition from Trac to GitLab by means of maintaining a Trac<->GitLab gateway) fail because of the concave costs on the path from one end to the other.

So, as I said, I would welcome a clear decision by the community to do so -- with a target date or target release number.

[Matthias Koeppe]

On Thursday, April 27, 2023 at 10:07:44 AM UTC-7 Isuru Fernando wrote:

> and took 5.8GB disk instead of the 3GB disk of the Sage mac app).

Yes, conda packages usually come with batteries included which means packages come with their

optional build time dependencies installed. That's usually not a big deal for other packages, but

Sage is special in that it has tons of dependencies.

I haven't checked the details for conda, but from what I see on some distros, I think there are a few particular packages that cause this:

- our use of libgd (see https://github.com/sagemath/sage/pull/35477) that pulls in a huge set of (unused) graphics libraries

- our use of CAS packages that are not separable from their GUI components (in particular GIAC)

 

[William Stein]

On Thu, Apr 27, 2023 at 1:21 PM Matthias Koeppe
<matthia...@gmail.com> wrote:
>
> On Thursday, April 27, 2023 at 5:49:50 AM UTC-7 William Stein wrote:
>
> To what extent does or could Conda with a little more work solve most
> of these problems? [...]
> I also think this section
> https://doc.sagemath.org/html/en/installation/conda.html#using-conda-to-provide-all-dependencies-for-the-sage-library-experimental
> called "Using conda to provide all dependencies for the Sage library
> (experimental)" is pretty exciting!
>
>
> Yes, I think this mode of installation should be the future default of Sage for developers.

AWESOME and thanks for the clarification! I'm strongly supportive of
this, and I agree that
switching to GitHub from Trac was a good model for how to approach
this problem.

-- William

>
> In any case, I think that migrating from "Sage the distribution" to
> solving a lot of the misc environment issues
> via conda would be very analogous to switching to Github, instead of
> maintaining our own issue tracker.
>
>
> Indeed, and just like in our successful transition to GitHub, a clean planned switchover to the new model is the most effective solution.
>
> Attempts to do this gradually (such as the attempt of a soft transition from Trac to GitLab by means of maintaining a Trac<->GitLab gateway) fail because of the concave costs on the path from one end to the other.
>
> So, as I said, I would welcome a clear decision by the community to do so -- with a target date or target release number.
>
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/26656418-5f72-4330-bb6c-dd4c968de928n%40googlegroups.com.



--
William (http://wstein.org)

[Matthias Koeppe]

On Thursday, April 27, 2023 at 5:12:45 AM UTC-7 kcrisman wrote:

As an example, how old of a Windows computer could one install the current Sage on?  I mean from scratch - not necessarily from source - using WSL, which I guess is now the main supported way to do so?

One needs Windows >= 10 and a CPU with the necessary virtualization support.

 

 What about the Cygwin installer - does it still exist in older versions on sagemath.org mirrors, what does that support?  How easy is it for someone who does NOT know about compiling to install Sage on a not-too-recent Windows machine? 

It's dead.

 

One thing that might help all of this is having older versions of Sage *binaries* for such platforms readily available for download (as many of our upstream packages in fact do).  I don't think we are.

We decided to get out of the business to publish binaries; we weren't doing it well, and numerous distributions and third-party binaries filled the role.

[Isuru Fernando]

True. Guix and spack could also work.

I guess there are several requirements we need

1. Support for all major OS/architecture combinations

2. Easy to build for rare OS/architecture combinations

3. Possible to install as a non-root user

4. Binaries are available for non-root

AFAIK, there's no package manager that has all 4 above and we will have to pick which

requirements are more important than others.

Isuru

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/0a85f27c909f8d8f185801af2228e6331bcad544.camel%40orlitzky.com.

[Matthias Koeppe]

On Thursday, April 27, 2023 at 2:54:47 PM UTC-7 Isuru Fernando wrote:

On Thu, Apr 27, 2023 at 1:00 PM Michael Orlitzky <mic...@orlitzky.com> wrote:

[...] Gentoo Prefix [...] Nix [...]

Guix and spack could also work.

I guess there are several requirements we need

1. Support for all major OS/architecture combinations

2. Easy to build for rare OS/architecture combinations

3. Possible to install as a non-root user

4. Binaries are available for non-root

AFAIK, there's no package manager that has all 4 above and we will have to pick which

requirements are more important than others.

I believe 1/3/4 are the most important and conda-forge suppports them.

And there's also the very important:

5. Its relevance in the Python dev world.

... where conda-forge is the only choice (other than PyPI).

 

[kcrisman]

(I'm planning upgrades in the next year or two, but it will be to
relatively low power RISC-V hardware. There are moral, legal,
environmental, and other non-financial reasons why people use "old"
hardware. But of course the financial reasons are very real too.)

Agreed on all points.  Access reasons too. 

[John H Palmieri]

On Thursday, April 27, 2023 at 2:54:47 PM UTC-7 Isuru Fernando wrote:

I guess there are several requirements we need

1. Support for all major OS/architecture combinations

2. Easy to build for rare OS/architecture combinations

3. Possible to install as a non-root user

4. Binaries are available for non-root

This is a good list. We also need to agree about what fits into category 1. As far as platforms for developers are concerned, does plain OS X count, or plain OS X + Xcode command-line tools, or OS X + homebrew, or OS X + conda, etc.?

[Dr. David Kirkby]

On Thu, 27 Apr 2023 at 15:49, Michael Orlitzky <mic...@orlitzky.com> wrote:

The test suite can take another full day to run -- some of
that is useful, but a lot is not. This is the biggest impediment to the
use and development of sage on an old system.

To me at least, it would be unwise not run the test suite. 

If you are choosing to use 15-20 year old hardware, you can not reasonably to handle a large modern program like Sagemath. More modern machines than that get thrown in skips. 😂

Dave. 

--

Dr. David Kirkby,
Kirkby Microwave Ltd,
drki...@kirkbymicrowave.co.uk
https://www.kirkbymicrowave.co.uk/
Telephone 01621-680100./ +44 1621 680100

Registered in England & Wales, company number 08914892.
Registered office:
Stokes Hall Lodge, Burnham Rd, Althorne, Chelmsford, Essex, CM3 6DT, United Kingdom

[Michael Orlitzky]

On Fri, 2023-04-28 at 18:06 +0100, Dr. David Kirkby wrote:
>
> To me at least, it would be unwise not run the test suite.
>
> If you are choosing to use 15-20 year old hardware, you can not reasonably
> to handle a large modern program like Sagemath. More modern machines than
> that get thrown in skips. 😂
>

I have ~1,000 other modern packages built from source on these machines
and hack on many of them. The hardware is as fast as the day I bought
it. The issue is not that sage is modern, rather that it's still built
like a pet project from 2005.

The test suite is an entirely different topic where your criticism is
more valid. There are a few different issues there, all pretty
irrelevant to this discussion:

* We have many redundant tests
* Doctests inherently require redundancy and are relatively slow
* We have tests for bugs that were fixed in upstream projects
and are already checked by the upstream test suite
* The "too long" warning is outrageously high
* The "too long" warning uses "wall time", which is meaningless
as an objective measure of how much computation is done. Someone 
with newer hardware can easily introduce a "fast" test that 
takes over a minute for me to run.

Beyond that, whatever time it takes to run the test suite is necessary
and I'm not complaining about it. It's the unnecessary bit that's
annoying.

[Matthias Koeppe]

On Thursday, April 27, 2023 at 1:24:01 AM UTC-7 Dima Pasechnik wrote:

The next in line will be jupyter and friends.

I agree with this point; specifically the front-end parts of Jupyter/JupyterLab, which run in a separate Python process (and therefore can be installed in a completely separate tree / venv). For this idea, we have the the long-open metaticket https://github.com/sagemath/sage/issues/30306 

But this observation can be generalized. 

For any package from which we only use executables or data, not a shared library, we can install it in any way we want -- without compromising the Sage distribution's ability to use already installed system packages.

So we can switch from our own installation scripts to using conda-forge for these!

I've written up the details in https://github.com/sagemath/sage/issues/35583

I'm counting about 40 packages (not counting the Jupyter frontend components) for which we can make a safe and easy withdrawal from packaging activities, WITHOUT making the Sage distribution harder to install, and avoiding the "concave cost" along the path that I mentioned.

(Note that gcc/gfortran do not fall into this category of packages because of their runtime library components.)

[Volker Braun]

+1 to just using conda for all basic infrastructure packages if the host doesn't have them. Just put a conda env in the PATH and done.

In my experience these work really well. I've only gotten issues with conda when you veer off the trodden path too much, e.g. I've had nothing but trouble installing ruby from conda-forge and then trying to build gems on top of that (ruby has various hard-coded compiler configuration paths, some of which point to the wrong place). 

[Matthias Koeppe]

That's now https://github.com/sagemath/sage/pull/35585 (needs review); starting with optional packages, not standard infrastructure packages though.

[Matthias Koeppe]

In this PR (https://github.com/sagemath/sage/pull/35585), the three optional packagesinfo, valgrind, rubiks are switched from our custom installation scripts to a (binary) installation from conda-forge. 

This drops platform support for 32-bit Linux (see https://github.com/sagemath/sage/wiki/Sage-9.8-Release-Tour#availability-of-sage-98-and-installation-help) for these optional packages. We will need a decision if this OK.

A precedent for phasing out support for a platform in this way was set in 2021 (Sage 9.4), when we dropped support for optional packages on old Linux systems with GCC 4.x toolchains (https://wiki.sagemath.org/ReleaseTours/sage-9.4#Support_for_optional_packages_on_systems_with_gcc_4.x_dropped).

[TB]

On 29/04/2023 23:10, Matthias Koeppe wrote:

In this PR (https://github.com/sagemath/sage/pull/35585), the three optional packagesinfo, valgrind, rubiks are switched from our custom installation scripts to a (binary) installation from conda-forge. 

This drops platform support for 32-bit Linux (see https://github.com/sagemath/sage/wiki/Sage-9.8-Release-Tour#availability-of-sage-98-and-installation-help) for these optional packages. We will need a decision if this OK.

A precedent for phasing out support for a platform in this way was set in 2021 (Sage 9.4), when we dropped support for optional packages on old Linux systems with GCC 4.x toolchains (https://wiki.sagemath.org/ReleaseTours/sage-9.4#Support_for_optional_packages_on_systems_with_gcc_4.x_dropped).

On Saturday, April 29, 2023 at 12:28:22 AM UTC-7 Matthias Koeppe wrote:

That's now https://github.com/sagemath/sage/pull/35585 (needs review); starting with optional packages, not standard infrastructure packages though.

On Saturday, April 29, 2023 at 12:14:56 AM UTC-7 Volker Braun wrote:

+1 to just using conda for all basic infrastructure packages if the host doesn't have them. Just put a conda env in the PATH and done.

I have a naive question about the installation process. It is naive because I have read only part of the thread and linked PRs:

Some of the packages in #35585 (e.g. pandoc) are rather popular, and usually have support in a superset of Linux distros supported by Sage. Should `sage -i pandoc`, at least in an interactive session, first recommend to install it from the distro, and not default to conda? For example, if a user wants to customize an init file of the package or already installed most dependencies of a package from the distro, then the installing from the distro might be better. Have this been discussed before?

Regards,

TB

[Matthias Koeppe]

On Saturday, April 29, 2023 at 1:45:31 PM UTC-7 TB wrote:

On 29/04/2023 23:10, Matthias Koeppe wrote:

In this PR (https://github.com/sagemath/sage/pull/35585), the three optional packagesinfo, valgrind, rubiks are switched from our custom installation scripts to a (binary) installation from conda-forge. 

I have a naive question about the installation process. It is naive because I have read only part of the thread and linked PRs:

Some of the packages in #35585 (e.g. pandoc) are rather popular, and usually have support in a superset of Linux distros supported by Sage. Should `sage -i pandoc`, at least in an interactive session, first recommend to install it from the distro, and not default to conda?

Yes, that's exactly what this PR does. There are no changes to system package detection! 

Currently (without the PR), we do not have an install script for pandoc (it is a "dummy" package). So when the system package is not installed but the user insists to install it using Sage (for example by typing "make pandoc"), that is an error.

With the PR, "make pandoc" will use conda-forge to install it.

[Matthias Koeppe]

On Saturday, April 29, 2023 at 1:45:31 PM UTC-7 TB wrote:

Should `sage -i pandoc`, at least in an interactive session, first recommend to install it from the distro, and not default to conda? 

I missed this detail about "sage -i" in my response above.

This is an interesting idea, but is orthogonal to how the Sage distribution installs packages (whether using the Sage-specific install script or by running conda).

I wouldn't want "./sage -i pandoc" to ask for confirmation, but we could certainly have it print a note first and then users can ^C out of it if they want to try that.

"make pandoc" should definitely do neither.

The third way of installing optional packages, using "./configure --enable-pandoc",  already prints system package advice at the end of configure.

 

[Michael Orlitzky]

On Sat, 2023-04-29 at 13:10 -0700, Matthias Koeppe wrote:
>
> This drops platform support for 32-bit Linux (see
> https://github.com/sagemath/sage/wiki/Sage-9.8-Release-Tour#availability-of-sage-98-and-installation-help)
> for these optional packages. We will need a decision if this OK.
>

GNU info and Valgrind can be installed with the package manager on any
32-bit distro so there's no practical difference for those two.

[Matthias Koeppe]

Sure, but these just the first few packages. Better list: https://github.com/sagemath/sage/issues/35583

[Tobia...@gmx.de]

What is now the plan of action?

In my experience, the conda workflow is currently already more stable than using sage distribution. Conda so far never failed to install a dependency for me (but there were some minor hick-ups with the integration in sage from time to time), while I was always afraid to run `make` since almost always some sage dependency failed to build (partly due to me using wsl).

[Matthias Koeppe]

On Saturday, April 29, 2023 at 2:08:40 PM UTC-7 Matthias Koeppe wrote:

On Saturday, April 29, 2023 at 1:45:31 PM UTC-7 TB wrote:

Should `sage -i pandoc`, at least in an interactive session, first recommend to install it from the distro, and not default to conda? 

I wouldn't want "./sage -i pandoc" to ask for confirmation, but we could certainly have it print a note first and then users can ^C out of it if they want to try that.

I've opened https://github.com/sagemath/sage/issues/35669 for this

[Michael Orlitzky]

On Mon, 2023-05-22 at 12:50 -0700, Matthias Koeppe wrote:
>
> On Saturday, April 29, 2023 at 1:45:31 PM UTC-7 TB wrote:
>
> Should `sage -i pandoc`, at least in an interactive session, first
> recommend to install it from the distro, and not default to conda?
>

Eventually I think the "sage" script should be relieved of the
responsibility to install distro/conda packages (both processes are
well-documented), but suggesting the distro package first seems like
the best approach for the time being. It may suggest packages that
can't be used, but that's no different than what ./configure does.