#32532 - removing gcc and gfortran spkgs

[Dima Pasechnik]

https://trac.sagemath.org/ticket/32532 proposes to remove these packages as not needed, and a huge time sink for everyone involved.

Rationale: nowadays every platform that Sage supports has said tools (or their equivalents - e.g. clang/clang++ in case of macOS and FreeBSD) available, if not outright as a system package, but surely with a very minimal effort.

There is no need to carry this baggage forward.

Moreover, it seems that Sage  is the only Python library/platform around which (potentially) vendors gcc/g++/gfortran.

[François Bissey]

As someone who contributed to those (and literally created the gfortran package when I pushed for enabling clang support on OS X) I welcome their demise. I’ll review their removal with pleasure if you need it.

François

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq0hsaf3SR-78Wajb_OdYjZq1MyReRH2vcwSO%2BU7CzLtYw%40mail.gmail.com.

[Michael Orlitzky]

On Thu, 2021-09-23 at 23:17 +0100, Dima Pasechnik wrote:
> https://trac.sagemath.org/ticket/32532
> <https://trac.sagemath.org/ticket/32532#comment:21> proposes to remove

> these packages as not needed, and a huge time sink for everyone involved.
>

I'm sure this list changes every time I make it up, but here are the
pros/cons that I see:

Pro (in favor of removal):

1. gcc and gfortran are available everywhere; and the (recent) 
default behavior is to exit ./configure with an error if the
user does not have gcc installed on his system anyway.

2. It is always an error for end-users to install these SPKGs.
If you have a problem installing your "native" gcc/gfortran
packages, then we (on this list) will tell you how to solve
that problem; not to use the SPKG.

3. If all else fails, using homebrew, conda, nix, or something
similar is still preferable to the SPKG.

4. As you note, they require a bunch of developer time that is
more or less wasted, since no one should ever use them.

5. They're standard packages, so the (large) sources are shipped
as part of the (thus even larger) release tarballs.

And the cons (in favor of keeping them):

1. Inertia. If someone has these as part of his workflow, us removing
them is going to cause him some headaches, despite whatever moral
high ground we can claim.

2. As a truly last resort, anyone with a sage tree can try to
build the SPKG without having to learn conda or nix or how
to do a manual install into /usr/local.

3. Removing them means we'll have to re-think the detection that 
currently happens in the corresponding spkg-configure.m4 files.

4. Having them as SPKGs means that I can create a branch that 
upgrades the gcc/gfortran packages, and use Github actions
to easily test sage with future releases.

I think the main value is in (4), but personally it's not enough for
me. Testing a new gcc/gfortran is easy to do in a Gentoo container,
where you can just upgrade to the new gcc/gfortran, select it, and then
try to build sage. That won't let you test brand-new-gcc-on-an-ancient-
ubuntu to find unrelated ubuntu bugs, but those should not be the
responsibility of the sage developers. You'll be using "apt" to install
gcc/fortran on ubuntu anyway.

[John H Palmieri]

I presume that this is not an issue for linux, but maybe I'm wrong about that. In any case, I'm concerned about OS X.

- How well tested are the various standalone Fortran options for OS X? I mean, how well tested are they for building Sage? Gfortran coming from homebrew is already included in tox.ini, I assume conda too, but perhaps ​​https://github.com/fxcoudert/gfortran-for-macOS should be added, especially if we're going to advertise it.

- How well supported are the various Fortran options for older versions of OS X?

  John

[Matthias Koeppe]

I repeat my strong objections to the proposed change -- which does not solve any problems and only creates new ones.

1. When you refer to the "huge time sink", I think you are referring to painful memories from some distant past. But we have no current or recent problem with either the gcc or gfortran package. I fixed the last problems with the gfortran package in early 2020; there have not been any problems since.

2. We used the gcc spkg as recently as this Spring, for the 9.3 release, in order to support Fedora 34, which had just switched to gcc 11. No other solution was proposed or feasible at the time because several of our standard packages were not ready for this compiler, and a Sage release was already long overdue.

3. Your claims regarding availability of the tools on users' systems are not substantiated.

a) clang/clang++, other than the version on macOS, is NOT supported by the Sage distribution. Only recently we added tests for these compilers (https://trac.sagemath.org/ticket/30835), which revealed build problems that are still unresolved (https://trac.sagemath.org/ticket/32207).

b) Any talk about FreeBSD is unsubstantiated, there is no systematic effort to support it or even test this environment.

c) The use of any Fortran compiler other than homebrew's packaging of gfortran on macOS (and our gfortran spkg) is completely unexplored. Given the instability of homebrew -- as a rolling platform on which it is not possible to install a specific version, it would be reckless to tie ourselves to homebrew's gfortran as the only option. (An effort to support macports in https://trac.sagemath.org/ticket/31505 is stalled.)

d) Our build on top of conda has been broken for months.

4. I want to caution about a single-user-system centric point of view. Non-root users on a multi-user unix system, or on a locked-down corporate laptop CANNOT install a system package. By removing SPKGs and demanding that the corresponding system package is available, we would make the Sage distribution harder to install for users, thus reduce the value that the Sage distribution provides for non-root users.

5. The gcc spkg is specifically used in building the binary distribution. Any discussion of removing the gcc/gfortran spkgs without including a discussion of deployment as the binary package is incomplete. Availability of the compilers at runtime is a key feature that we have advertised through facilities such as the magic %cython directive and others.

6. (As I explained before in https://trac.sagemath.org/ticket/32060#comment:36) There is a crucial distinction between the gcc and the gfortran package: On systems with a too old or otherwise broken gcc, our gcc spkg often cannot be installed, so the gcc spkg is not very useful for dealing with old compilers (but see the successful recent use in the Fedora 34 case mentioned above). 

On the other hand, on systems with a working gcc but a missing gfortran, building our gfortran spkg is robust, and having it makes our distribution useful for a wider range of users.

7. (As I explained before in https://trac.sagemath.org/ticket/32532#comment:11) in particular, Python users / developers generally have a working gcc (because many Python packages include extension modules that need to be compiled) but they have no need to have gfortran installed. The reason is that the few Python packages that use gfortran (such as scipy) have high-quality deployments of wheels to PyPI, which are widely used. Sage, however, does not yet have a working strategy that would enable us to use wheels from PyPI. As you know, I have been working on a PyPI-based deployment solution for Sage in https://trac.sagemath.org/ticket/29039; this builds everything from source. 

Note that Python goes a long way to allow ordinary (non-root) users to install packages, either in a virtual environment or in the user scheme (pip install --user). gfortran, obviously, is not pip-installable -- so Python users would again need root access to install a system package to get the compiler.

The project of modularization and PyPI deployment (https://trac.sagemath.org/ticket/29705) is complicated enough. I would appreciate if we do not create artificial obstacles -- removing gfortran or whatever next package you assume is on "everyone's system".

Matthias

[François Bissey]

> On 24/09/2021, at 13:12, Matthias Koeppe <matthia...@gmail.com> wrote:
>
> c) The use of any Fortran compiler other than homebrew's packaging of gfortran on macOS (and our gfortran spkg) is completely unexplored. Given the instability of homebrew -- as a rolling platform on which it is not possible to install a specific version, it would be reckless to tie ourselves to homebrew's gfortran as the only option. (An effort to support macports in https://trac.sagemath.org/ticket/31505 is stalled.)
>

For the nitpick record.

This is not completely unexplored but I haven’t poured any activity there for months due to “too many things to do and not enough of me to go around”. I actually actively explored using alternative compiler (c/c++ and fortran).

Apart from sqlite that was detecting intrinsics in a stupid fashion at the time, sage did build with the intel C/C++ compiler but pari optimisation had to be turned off completely (pari compiled with -O0 or completely broken - no joke).

sage did build with the pgfortran (now nvfortran compiler) once an issue in openblas was fixed [I believe my fix made it upstream].

I don’t have a clear memory of the result with the intel fortran compiler, but I usually remember difficulties so I am going to assume there was none.

Optional packages were not covered in those exercises.

François

[Matthias Koeppe]

Thanks for the update on this, François; I have added a link to your post to our Fortran meta-ticket https://trac.sagemath.org/ticket/23926

[Michael Orlitzky]

On Thu, 2021-09-23 at 18:12 -0700, Matthias Koeppe wrote:
> I repeat my strong objections to the proposed change -- which does not
> solve any problems and only creates new ones.
>

2, 4, 6, and 7 are addressed by Conda, Nix, Guix, Homebrew, or even
Gentoo Prefix. These projects *want* to build entire distributions that
run out of your home directory.

> 5. The gcc spkg is specifically used in building the binary distribution.
> Any discussion of removing the gcc/gfortran spkgs without including a
> discussion of deployment as the binary package is incomplete. Availability
> of the compilers at runtime is a key feature that we have advertised
> through facilities such as the magic %cython directive and others.
>

Whoever is building the binary images could install a compiler into
SAGE_LOCAL with "dpkg-deb -x" or whatever before they start. There's no
need for it to come from an SPKG.

The fact that we even have to consider these issues supports the claim
that these packages are a maintenance burden.

And, veering off-topic:

Our binaries aren't really binaries in the traditional sense. We ship a
3GB image of an entire operating system. This could be done much easier
by... shipping an image of an entire operating system: QEMU,
VirtualBox, Docker, etc. You could tar up an entire Gentoo system with
sage installed and it would probably take up less space than our
existing binaries. Similarly, I'm sure we could tar up a Conda image
where sage was installed and tell people to download that. Provide a
launcher script and it would work out-of-the-box.

[William Stein]

(Staying off topic.)

I installed Isuru Fernando's Conda image of SageMath on my Macbook Pro
M1 "Apple Silicon" yesterday,
and it worked quickly with no trouble that I could see. Here's how
big the install is (decompressed):

wstein@Williams-MBP ~ % du -sch sagemath-forge
5.5G sagemath-forge
5.5G total

Compilers are included as part of this, so Cython does just work out
of the box as far as I can tell:

sage: cython('def f(n): return n+1')sage: f(10)
11
sage: !which clang
/Users/wstein/sagemath-forge/bin/clang
sage: !which gfortran
/Users/wstein/sagemath-forge/bin/gfortran

In any case, the momentum behind conda-forge is really impressive at
this point... though this particular
page on their website

https://conda-forge.org/feedstock-outputs/

listing all packages is really horrible, since it slowly loads a
massive list in random order, etc. It's funny
that it feels so broken as a result of them being so successful and
getting contributions! (It looks like there
might be around 14K packages listed there?)

Isuru was explaining to me earlier today that the bug that have caused
some trouble with
the sage-9.3 and sage-9.4 binaries are due to OpenBLAS build options.
The conda-forge maintainers also
hit these same problems with their openblas package (which sage in
conda just depends on), and
fixed that one single package a while ago. (DISCLAIMER: I'm
repeating this second hand so may
have some details wrong. ) In any case, conda seems to be doing a
surprisingly good job at solving
a really, really hard problem.

-- William

>
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/381f2f471070a1192a1b7ba79799bfb3eee1acbd.camel%40orlitzky.com.



--
William (http://wstein.org)

[John H Palmieri]

Item #2 actually happened. Can you cite posts from people successfully using the tools you mention to get around it?

[Matthias Koeppe]

On Thursday, September 23, 2021 at 7:48:35 PM UTC-7 Michael Orlitzky wrote:

Whoever is building the binary images could install a compiler into
SAGE_LOCAL with "dpkg-deb -x" or whatever before they start. There's no
need for it to come from an SPKG.

I don't think Debian packages are relocatable. "dpkg-deb -x" is specifically NOT for installing, but for inspecting.

So there goes another suggestion to replace something that works by something conjectural.

[Michael Orlitzky]

On Thu, 2021-09-23 at 20:32 -0700, John H Palmieri wrote:
> Item #2 actually happened. Can you cite posts from people successfully
> using the tools you mention to get around it?

People who solve the unnecessary problem in the obvious and reliable
way don't post about it. If you really doubt that these tools can be
used to work around it, you can download sage-9.3 on fedora-34 and try
it. But I don't think anyone is claiming that they don't work.

Item #2 happened, incidentally, only because we've copy & pasted so
many packages into sage pretending to be a linux distribution.
Eliminating that problem altogether begins right here, at the root of
the dependency tree.

[Matthias Koeppe]

On Thursday, September 23, 2021 at 7:48:35 PM UTC-7 Michael Orlitzky wrote:

2, 4, 6, and 7 are addressed by Conda, Nix, Guix, Homebrew, or even
Gentoo Prefix. 

Item 7 of my list, pip-installability, is definitely not addressed by any of these. From a Python dev viewpoint, a project that is not pip-installable does not exist. 

[Matthias Koeppe]

On Thursday, September 23, 2021 at 7:48:35 PM UTC-7 Michael Orlitzky wrote:

We ship a
3GB image of an entire operating system. This could be done much easier
by... shipping an image of an entire operating system: QEMU,
VirtualBox, Docker, etc. 

We don't even seem to have enough developer interest in this to keep the existing Docker build of Sage in good shape.

[Michael Orlitzky]

"or whatever" was meant to suggest that if this exact command won't
work, then something similar to it will. #5 is a legitimate problem,
but it's not a hard one.

[Matthias Koeppe]

On Thursday, September 23, 2021 at 7:48:35 PM UTC-7 Michael Orlitzky wrote:

You could tar up an entire Gentoo system with
sage installed and it would probably take up less space than our
existing binaries. 

Gentoo is not relocatable, so that would not be useful. gentoo-prefix is hardly production-ready last time I checked (https://trac.sagemath.org/ticket/29105), and there is no evidence anyone has ever built Sage on it.

[Michael Orlitzky]

The only concrete complaint I saw in #7 was "Python users would again
need root access to install a system package to get [gfortran]."

I admit to having very little understanding of your larger plan for
pip-installability and modularizing sagelib. I've read all of the
comments about it, but I still don't see what the gfortran SPKG has to
do with it.

[Matthias Koeppe]

The ticket https://trac.sagemath.org/ticket/29039 explains it.

gfortran is just one of the packages that is part of the build.

 

[Michael Orlitzky]

On Thu, 2021-09-23 at 21:18 -0700, Matthias Koeppe wrote:
> On Thursday, September 23, 2021 at 7:48:35 PM UTC-7 Michael Orlitzky wrote:
>
> > You could tar up an entire Gentoo system with
> > sage installed and it would probably take up less space than our
> > existing binaries.
> >
>
> Gentoo is not relocatable, so that would not be useful.
>

If the entire system is in the tarball, you can chroot right into the
extracted directory.

[Justin C. Walker]

I think my only concern with this is (g)fortran. The C/C++ compiler situation seems pretty stable, but doesn’t the fortran problem still exist? I do not use any package managers on my systems, so home-brew and the like aren’t useful to me.

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq0hsaf3SR-78Wajb_OdYjZq1MyReRH2vcwSO%2BU7CzLtYw%40mail.gmail.com.

--
Justin C. Walker, Curmudgeon-At-Large
Institute for the Absorption of Federal Funds
--------
Men are from Earth.
Women are from Earth.
Deal with it.
--------

[Matthias Koeppe]

On Thursday, September 23, 2021 at 9:41:56 PM UTC-7 jus...@mac.com wrote:

I think my only concern with this is (g)fortran. The C/C++ compiler situation seems pretty stable, but doesn’t the fortran problem still exist? I do not use any package managers on my systems, so home-brew and the like aren’t useful to me.

Yes, that's pretty much my point 3c.

[William Stein]

Hi Matthias,

Since Fortran is pretty central to this discussion, any chance you
could give a quick overview of the extent to which Sage depends on
Fortran in 2021?
I was reading about pyodide [1] recently and they had to work very
hard to build scipy for webassembly without using Fortran. I think
they get "far enough"
for their purposes using f2c, and by possibly disabling a handful of
modules. I'm curious what in Sage depends on Fortran these days, and
to what
extent it depends on Fortran.

For what it is worth, I remember making the first gfortran package for
Sage long ago, and at the time any sort of Fortran support on MacOS
was a nightmare.
I imagine things might be better today, due to Flang, which seems to
be a fortran frontend (like Clang) for LLVM. That seems likely to
work well on MacOS.

[1] https://pyodide.org/en/stable/
[2] https://releases.llvm.org/11.0.0/tools/flang/docs/index.html

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/68918d67-d3a0-4a02-a3e3-03fb1b4a9ffdn%40googlegroups.com.



--
William (http://wstein.org)

[Matthias Koeppe]

On Thursday, September 23, 2021 at 9:09:45 PM UTC-7 Michael Orlitzky wrote:

Item #2 happened, incidentally, only because we've copy & pasted so
many packages into sage pretending to be a linux distribution.
Eliminating that problem altogether begins right here, at the root of
the dependency tree.

I don't think so. The packages that were not ready for gcc 11 support at that time also were not ready upstream nor in other Linux distributions (see https://trac.sagemath.org/ticket/31786). The issue is not so much that we duplicate work by linux distributions but rather that Sage has effectively adopted many poorly maintained or semi-abandoned math packages. 

[François Bissey]

I have at least one close user/tester that builds sage-on-gentoo on a gentoo prefix (on a debian machine I think). Pretty much every beta/rc release get built and tested.
I used to work on gentoo-prefix on OS X for a while. I should try it again someday but I don’t feel like my little macbook pro is a suitable environment for the kind of always compiling that ends up happening if you are doing serious development rather than just installing it every once in a while to use it.

By the way switching to clang+gfortran-spkg over the gcc-spkg shaved 3 hours of building time of vanilla sage on my previous macbook. That was what took the most time to build by a very wide margin.

> On 24/09/2021, at 16:18, Matthias Koeppe <matthia...@gmail.com> wrote:
>
> Gentoo is not relocatable, so that would not be useful. gentoo-prefix is hardly production-ready last time I checked (https://trac.sagemath.org/ticket/29105), and there is no evidence anyone has ever built Sage on it.
>

François

[Matthias Koeppe]

On Thursday, September 23, 2021 at 9:54:19 PM UTC-7 wst...@gmail.com wrote:

Since Fortran is pretty central to this discussion, any chance you
could give a quick overview of the extent to which Sage depends on
Fortran in 2021?

No big surprises: It comes in from openblas, numpy, scipy; and suitesparse (a dependency of cvxopt).

openblas, of course, is not just used via these packages, but by many more of our C libraries.

numpy&scipy of course make high quality wheels available for all platforms, but we probably shouldn't try to build our C libraries against their vendored openblas...

I was reading about pyodide [1] recently and they had to work very
hard to build scipy for webassembly without using Fortran.

Yes, this is very impressive work. Likely will be important for edge-computing type deployments. If we ever want to have Sage run in such an environment, modularization is the way to go - so that we can focus on subsystems with minimal compiled library dependencies.

I imagine things might be better today, due to Flang, which seems to
be a fortran frontend (like Clang) for LLVM. That seems likely to
work well on MacOS.

We have a meta-ticket that tracks this: https://trac.sagemath.org/ticket/23926

But I have not followed the details.

[Matthias Koeppe]

On Thursday, September 23, 2021 at 10:17:13 PM UTC-7 François Bissey wrote:

By the way switching to clang+gfortran-spkg over the gcc-spkg shaved 3 hours of building time of vanilla sage on my previous macbook. That was what took the most time to build by a very wide margin.

Sure, nobody is arguing in favor of using the gcc spkg on macOS.

 

[Dima Pasechnik]

On Fri, 24 Sep 2021, 02:12 Matthias Koeppe, <matthia...@gmail.com> wrote:

I repeat my strong objections to the proposed change -- which does not solve any problems and only creates new ones.

1. When you refer to the "huge time sink", I think you are referring to painful memories from some distant past. But we have no current or recent problem with either the gcc or gfortran package. I fixed the last problems with the gfortran package in early 2020; there have not been any problems since.

I am referring to the time sink caused by the mere presence of gcc/gfortran spkg in Sage. Namely,

- support of outdated systems nobody uses (many tickets supporting past EOL systems recently)

- support of modern, but broken, systems (e.g. we just should not care that a particular Fedora version broke its gcc, it's not out problem, let us not try to bite more than we can chew)

- users who attempt to build gfortran as they don't have it installed - many fail, but I guess many more succeed,

and rebuild it for each new release

- our Linux and  macOS binary builds are increasingly useless, and a timesink for everyone involved, potential users too.

2. We used the gcc spkg as recently as this Spring, for the 9.3 release, in order to support Fedora 34, which had just switched to gcc 11. No other solution was proposed or feasible at the time because several of our standard packages were not ready for this compiler, and a Sage release was already long overdue.

nobody pushes us into not skipping one particular version.

I cannot imagine e.g. cpython or scipy delaying a release in such a situation.

Besides, building gcc from source is not too hard

with a modern C compiler available.

3. Your claims regarding availability of the tools on users' systems are not substantiated.

a) clang/clang++, other than the version on macOS, is NOT supported by the Sage distribution. Only recently we added tests for these compilers (https://trac.sagemath.org/ticket/30835), which revealed build problems that are still unresolved (https://trac.sagemath.org/ticket/32207).

#32207 is resolved, and ready for review :-)

b) Any talk about FreeBSD is unsubstantiated, there is no systematic effort to support it or even test this environment.

There is a FreeBSD sagemath package: https://www.freshports.org/math/sage, currently at 9.2.+patches.

It's of course much more fun to fool around with past-EOL linux systems, just cause it's easier with GitHub Actions.

c) The use of any Fortran compiler other than homebrew's packaging of gfortran on macOS (and our gfortran spkg) is completely unexplored. Given the instability of homebrew -- as a rolling platform on which it is not possible to install a specific version, it would be reckless to tie ourselves to homebrew's gfortran as the only option. (An effort to support macports in https://trac.sagemath.org/ticket/31505 is stalled.)

 

In questionable cases homebrew does offer a choice of versions (e.g. with python, or, indeed, gcc)

Besides, they are quick in fixing their problems, qucker than us.

As to other Fortran compilers' support - no, I did in the past look at flang, and currently look at NAG's nagfor support.

Unfortunately flang isn't really considered production quality yet (although perhaps it's getting there).

Good news about flang is that it's merged in the clang tree:

https://releases.llvm.org/11.0.0/tools/flang/docs/ReleaseNotes.html

d) Our build on top of conda has been broken for months.

So? I think it's good to burn the gcc/gfortan bridges, and instead fix the conda build.

4. I want to caution about a single-user-system centric point of view. Non-root users on a multi-user unix system, or on a locked-down corporate laptop CANNOT install a system package. By removing SPKGs and demanding that the corresponding system package is available, we would make the Sage distribution harder to install for users, thus reduce the value that the Sage distribution provides for non-root users.

our experience dealing with requests for help from HPC clusters here is that the ability to build gcc/gfortran

is mostly useless for them, as the systems tooling is old or just weird.

Locked down to the inability to install things corporate laptops running Linux or macOS?

I've never seen one.

5. The gcc spkg is specifically used in building the binary distribution. Any discussion of removing the gcc/gfortran spkgs without including a discussion of deployment as the binary package is incomplete. Availability of the compilers at runtime is a key feature that we have advertised through facilities such as the magic %cython directive and others.

 

yet one cannot reliably (or not at all) install optional packages  in such a scenario.

I consider the linux and macOS binary builds as mostly a timesink (see above)

6. (As I explained before in https://trac.sagemath.org/ticket/32060#comment:36) There is a crucial distinction between the gcc and the gfortran package: On systems with a too old or otherwise broken gcc, our gcc spkg often cannot be installed, so the gcc spkg is not very useful for dealing with old compilers (but see the successful recent use in the Fedora 34 case mentioned above). 

On the other hand, on systems with a working gcc but a missing gfortran, building our gfortran spkg is robust, and having it makes our distribution useful for a wider range of users.

At least you admit that the gcc spkg is useless, right?

7. (As I explained before in https://trac.sagemath.org/ticket/32532#comment:11) in particular, Python users / developers generally have a working gcc (because many Python packages include extension modules that need to be compiled) but they have no need to have gfortran installed. The reason is that the few Python packages that use gfortran (such as scipy) have high-quality deployments of wheels to PyPI, which are widely used. Sage, however, does not yet have a working strategy that would enable us to use wheels from PyPI. As you know, I have been working on a PyPI-based deployment solution for Sage in https://trac.sagemath.org/ticket/29039; this builds everything from source. 

 

Why does it build everything from source? E.g. numpy doesn't build Fortran compilers.

Why do we have to outdo other Python projects here?

Note that Python goes a long way to allow ordinary (non-root) users to install packages, either in a virtual environment or in the user scheme (pip install --user). gfortran, obviously, is not pip-installable -- so Python users would again need root access to install a system package to get the compiler.

The project of modularization and PyPI deployment (https://trac.sagemath.org/ticket/29705) is complicated enough. I would appreciate if we do not create artificial obstacles -- removing gfortran or whatever next package you assume is on "everyone's system".

 

As I said, we should  not try to bite more than  we can chew. No other Python project vendors a full-blown

Fortran compiler. Absence of a Fortran compiler on one's box should not be our problem, just as it's not the Numpy/SciPy's problem, or not an OpenBlas' problem.

Let's stop pretending we are a full-blown distribution.

Dima

Matthias

On Thursday, September 23, 2021 at 3:17:29 PM UTC-7 Dima Pasechnik wrote:

https://trac.sagemath.org/ticket/32532 proposes to remove these packages as not needed, and a huge time sink for everyone involved.

Rationale: nowadays every platform that Sage supports has said tools (or their equivalents - e.g. clang/clang++ in case of macOS and FreeBSD) available, if not outright as a system package, but surely with a very minimal effort.

There is no need to carry this baggage forward.

Moreover, it seems that Sage  is the only Python library/platform around which (potentially) vendors gcc/g++/gfortran.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/bfd20a60-70d5-49df-bfa3-9d53caa447f7n%40googlegroups.com.

[Michael Orlitzky]

The list of blockers there implicates,

* gcc
* eclib
* singular
* ntl
* gfortran
* libgd
* ecl
* lcalc
* fplll
* giac

all of which are available on Fedora 34, and pretty much everywhere.
lcalc and fplll have the smallest number of entries in
build/pkgs/$pkg/distros, with seven.

While they may not all have been patched upstream (lcalc upstream was
gone), I'm fairly sure they were all patched in Gentoo or sage-on-
gentoo before they were patched in sage itself -- especially if I
change that wording to "patched in a release." I'd also bet that Conda,
Nix, and friends had it all sorted out before sage did. And all without
wasting sage developers' time (except for those of us who maintain the
distro packages, too, and who got to do everything twice).

We are only concerned about gcc-11 support in any of those packages
because we're supporting their SPKGs. But, for a while, the SPKGs have
had superior alternatives. Delete the SPKGs and you might run into some
other corner cases, but you certainly aren't going to hit gcc-11 build
failures if you aren't building.

[Julien Puydt]

Hi,

Le vendredi 24 septembre 2021 à 12:23 +0100, Dima Pasechnik a écrit :
>
> On Fri, 24 Sep 2021, 02:12 Matthias Koeppe,
> <matthia...@gmail.com> wrote:
> > I repeat my strong objections to the proposed change -- which does
> > not solve any problems and only creates new ones.
> >
> > 1. When you refer to the "huge time sink", I think you are
> > referring to painful memories from some distant past. But we have
> > no current or recent problem with either the gcc or gfortran
> > package. I fixed the last problems with the gfortran package in
> > early 2020; there have not been any problems since.
>
> I am referring to the time sink caused by the mere presence of
> gcc/gfortran spkg in Sage. Namely,
>
> - support of outdated systems nobody uses (many tickets supporting
> past EOL systems recently)
>
> - support of modern, but broken, systems (e.g. we just should not
> care that a particular Fedora version broke its gcc, it's not out
> problem, let us not try to bite more than we can chew)
>
> - users who attempt to build gfortran as they don't have it installed
> - many fail, but I guess many more succeed,
> and rebuild it for each new release
>
> - our Linux and  macOS binary builds are increasingly useless, and a
> timesink for everyone involved, potential users too.
>
> >

I made the point years ago that sagemath should be broken in two:

- sagemath-the-software ;

- sagemath-the-distribution.

This thread is an illustration of why that would simplify matters:

- sagemath developpers could focus on making the sofware work (both
increasing its scope/efficiency and solving real issues) ;

- the various distribution packagers would handle their specific
problems, and only complain upstream when there's an actual problem.

Cheers,

J.Puydt

[Matthias Koeppe]

On Friday, September 24, 2021 at 4:23:32 AM UTC-7 Dima Pasechnik wrote:

On Fri, 24 Sep 2021, 02:12 Matthias Koeppe, <matthia...@gmail.com> wrote:

I repeat my strong objections to the proposed change -- which does not solve any problems and only creates new ones.

7. (As I explained before in https://trac.sagemath.org/ticket/32532#comment:11) in particular, Python users / developers generally have a working gcc (because many Python packages include extension modules that need to be compiled) but they have no need to have gfortran installed. The reason is that the few Python packages that use gfortran (such as scipy) have high-quality deployments of wheels to PyPI, which are widely used. Sage, however, does not yet have a working strategy that would enable us to use wheels from PyPI. As you know, I have been working on a PyPI-based deployment solution for Sage in https://trac.sagemath.org/ticket/29039; this builds everything from source. 

 

Why does it build everything from source? E.g. numpy doesn't build Fortran compilers.

Why do we have to outdo other Python projects here?

I have explained in the paragraph that you quoted (and previously in #32532). Wheels. They have wheels. We don't.

 

[Matthias Koeppe]

On Friday, September 24, 2021 at 6:28:50 AM UTC-7 Julien Puydt wrote:

I made the point years ago that sagemath should be broken in two:

- sagemath-the-software ;

- sagemath-the-distribution.

This thread is an illustration of why that would simplify matters

Yes, and many of us have worked on making progress in this direction.

But the approach of chipping away parts of the Sage distribution, declaring that this or that package is one everyone's system, does not help in any way with this.

Meta-ticket https://trac.sagemath.org/ticket/29705 lists many tasks that would actually help.

 

[Matthias Koeppe]

On Friday, September 24, 2021 at 4:23:32 AM UTC-7 Dima Pasechnik wrote:

On Fri, 24 Sep 2021, 02:12 Matthias Koeppe, <matthia...@gmail.com> wrote:

I repeat my strong objections to the proposed change -- which does not solve any problems and only creates new ones.

2. We used the gcc spkg as recently as this Spring, for the 9.3 release, in order to support Fedora 34, which had just switched to gcc 11. No other solution was proposed or feasible at the time because several of our standard packages were not ready for this compiler, and a Sage release was already long overdue.

nobody pushes us into not skipping one particular version.

I cannot imagine e.g. cpython or scipy delaying a release in such a situation.

Besides, building gcc from source is not too hard

with a modern C compiler available.

Bizarre that you think that it's easy for users to build gcc from source but it's somehow hard for us to maintain the package that builds gcc from source.

[Dima Pasechnik]

Maintaining is a task. Non-maintaining is no task to worry about. 

What's bizarre about this? 

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/01d0a820-7967-45e8-bd72-f75e3782ebaen%40googlegroups.com.

[Dima Pasechnik]

Are we supposed to reinvent wheels too (pun intended)?

You never explained why you went with building everything from source, and why you think it's a good idea (I don't think it is).

Do explain why we cannot have what they have.

 

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/bffb95ab-3407-487f-8e19-b8945f7e10c3n%40googlegroups.com.

[Matthias Koeppe]

On Friday, September 24, 2021 at 8:30:40 AM UTC-7 Dima Pasechnik wrote:

On Fri, Sep 24, 2021 at 3:54 PM Matthias Koeppe <matthia...@gmail.com> wrote:

On Friday, September 24, 2021 at 4:23:32 AM UTC-7 Dima Pasechnik wrote:

Why does it build everything from source? E.g. numpy doesn't build Fortran compilers.

Why do we have to outdo other Python projects here?

I have explained in the paragraph that you quoted (and previously in #32532). Wheels. They have wheels. We don't.

Are we supposed to reinvent wheels too (pun intended)?

Fortunately no -- Python wheels are sufficiently flexible. We would "just" need a strategy for deployment of Sage components as wheels, and people who work on it. There are many tasks and opportunities. Meta-ticket https://trac.sagemath.org/ticket/31251 is an entry point.

 

You never explained why you went with building everything from source, and why you think it's a good idea (I don't think it is).

Do explain why we cannot have what they have.

Oh, wheel deployment of Sage-the-distribution is waiting for review at https://trac.sagemath.org/ticket/31396

[Matthias Koeppe]

On Friday, September 24, 2021 at 8:27:07 AM UTC-7 Dima Pasechnik wrote:

On Fri, Sep 24, 2021 at 4:14 PM Matthias Koeppe <matthia...@gmail.com> wrote:

Bizarre that you think that it's easy for users to build gcc from source but it's somehow hard for us to maintain the package that builds gcc from source.

Maintaining is a task. Non-maintaining is no task to worry about. 

What's bizarre about this? 

Because you are shifting the work from our working spkg-install.in script to handholding users who attempt it on the mailing lists.

[Dima Pasechnik]

On Fri, Sep 24, 2021 at 5:41 AM 'Justin C. Walker' via sage-devel <sage-...@googlegroups.com> wrote:

I think my only concern with this is (g)fortran.  The C/C++ compiler situation seems pretty stable, but doesn’t the fortran problem still exist?  I do not use any package managers on my systems, so home-brew and the like  aren’t useful to me.

0) install gmp, libmpc (a.k.a. mpc), and mpfr from source (a cheating way is to get them from Homebrew, as proposed in

https://github.com/fxcoudert/gfortran-for-macOS/blob/master/build_package.md), in a directory called FOO, say.

Remove all the *.dylib files there, to make sure they are linked statically (we're building gcc/g++/gfortran that only depends on system dylibs).

1) download gcc-11.2.0 tarball, untar it, cd gcc-11.2.0

2) assuming dependencies are in FOO, run

./configure --prefix=/usr/local --with-gmp=$FOO/gmp --with-mpfr=$FOO/mpfr --with-mpc=$FOO/libmpc --disable-multilib --with-native-system-header-dir=/usr/include --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

3) make && sudo make install

4) done: now you have gfortran   in /usr/local/bin; Sage's ./configure should be able to pick it up, no need to build gfortran package for some months/years.

Dima

 

> On Sep 23, 2021, at 15:17 , Dima Pasechnik <dim...@gmail.com> wrote:
>
> https://trac.sagemath.org/ticket/32532 proposes to remove these packages as not needed, and a huge time sink for everyone involved.
>
> Rationale: nowadays every platform that Sage supports has said tools (or their equivalents - e.g. clang/clang++ in case of macOS and FreeBSD) available, if not outright as a system package, but surely with a very minimal effort.
>
> There is no need to carry this baggage forward.
> Moreover, it seems that Sage  is the only Python library/platform around which (potentially) vendors gcc/g++/gfortran.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq0hsaf3SR-78Wajb_OdYjZq1MyReRH2vcwSO%2BU7CzLtYw%40mail.gmail.com.

--
Justin C. Walker, Curmudgeon-At-Large
Institute for the Absorption of Federal Funds
--------
Men are from Earth.
Women are from Earth.
   Deal with it.
--------



--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/DCE3ACDD-C65D-40B5-8AD9-409400C863AE%40mac.com.

[Matthias Koeppe]

On Thursday, September 23, 2021 at 9:09:45 PM UTC-7 Michael Orlitzky wrote:

[Matthias'] Item #2 happened, incidentally, only because we've copy & pasted so

many packages into sage pretending to be a linux distribution.
Eliminating that problem altogether begins right here, at the root of
the dependency tree.

In another post:

> [Matthias's items] 2, 4, 6, and 7 are addressed by Conda, Nix, Guix, Homebrew, or even
> Gentoo Prefix. These projects *want* to build entire distributions that
> run out of your home directory.

These two comments together really are the crux of the discussion.

If we say that availability of gfortran in any of these user-installable distributions is the solution, then this applies to lots of other spkgs as well --- perhaps to ALL of our non-Python packages.

So essentially it is to say, let's stop maintaining the Sage distribution. 

This certainly *could* make sense for the project -- but a lot of work is needed to bring missing packages to these distributions: conda-forge does not have all of our optional packages, homebrew is missing a lot of packages. On the Sage side, also working on the tasks https://trac.sagemath.org/ticket/30914 (creating proper upstreams for some Sage-specific packages) will help.

The big problem is that the middle ground between "Complete Sage distribution that works in most use cases" and "No Sage distribution" is worse than both of the extremes. By removing spkgs one by one, we would make the Sage distribution less useful. 

So this is not a meaningful process.

 

[Dima Pasechnik]

We spoiled the users, they assumed that Sage is a distribution. Yes, there will be a short while they'll ask - and the answer will be: 

RTFM/apt-get install gfortran/brew install gcc/...

NumPy/SciPy mailing lists/support forums aren't full of questions on how to install a Fortran compiler, you know.

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/22649531-58ee-4cc8-a717-79c1d986b50an%40googlegroups.com.

[Matthias Koeppe]

On Friday, September 24, 2021 at 9:03:20 AM UTC-7 Dima Pasechnik wrote:

NumPy/SciPy mailing lists/support forums aren't full of questions on how to install a Fortran compiler, you know.

Yes because of wheels.

 

[Matthias Koeppe]

On Thursday, September 23, 2021 at 9:13:23 PM UTC-7 Michael Orlitzky wrote:

> > Whoever is building the binary images could install a compiler into
> > SAGE_LOCAL with [some system package command] or whatever before they start. There's no
> > need for it to come from an SPKG.

Well, we have several binary distributors now: (1) Whoever is making the binaries for Linux (I presume it is Volker); (2) the new macOS distribution by Marc Culler (https://github.com/3-manifolds/Sage_macOS/releases); (3) the relocatable wheels from my ticket https://trac.sagemath.org/ticket/31396 

Making gfortran each of these distributors' individual problems is a step in the wrong direction. 

[Dima Pasechnik]

On Fri, Sep 24, 2021 at 4:50 PM Matthias Koeppe <matthia...@gmail.com> wrote:

On Thursday, September 23, 2021 at 9:09:45 PM UTC-7 Michael Orlitzky wrote:

[Matthias'] Item #2 happened, incidentally, only because we've copy & pasted so
many packages into sage pretending to be a linux distribution.
Eliminating that problem altogether begins right here, at the root of
the dependency tree.

In another post:

> [Matthias's items] 2, 4, 6, and 7 are addressed by Conda, Nix, Guix, Homebrew, or even
> Gentoo Prefix. These projects *want* to build entire distributions that
> run out of your home directory.

These two comments together really are the crux of the discussion.

If we say that availability of gfortran in any of these user-installable distributions is the solution, then this applies to lots of other spkgs as well --- perhaps to ALL of our non-Python packages.

So essentially it is to say, let's stop maintaining the Sage distribution. 

Sage "distribution" is what's in build/ (and in upstream/). Many packages don't need to be vendored, like we stopped vendoring tar, and we can stop vendoring patch, gcc, gfortran, python3, and few other packages, very easily.

Of course these scripts still need to be maintained, so that's not "let's stop maintaining", it's "let's stop vendoring".

 

 

This certainly *could* make sense for the project -- but a lot of work is needed to bring missing packages to these distributions: conda-forge does not have all of our optional packages, homebrew is missing a lot of packages. On the Sage side, also working on the tasks https://trac.sagemath.org/ticket/30914 (creating proper upstreams for some Sage-specific packages) will help.

The big problem is that the middle ground between "Complete Sage distribution that works in most use cases" and "No Sage distribution" is worse than both of the extremes. By removing spkgs one by one, we would make the Sage distribution less useful. 

So this is not a meaningful process. 

 

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/323c5a9d-cdd4-42ee-b250-98d2c5caea4bn%40googlegroups.com.

[Matthias Koeppe]

On Friday, September 24, 2021 at 9:03:20 AM UTC-7 Dima Pasechnik wrote:

On Fri, Sep 24, 2021 at 4:43 PM Matthias Koeppe <matthia...@gmail.com> wrote:

you are shifting the work from our working spkg-install.in script to handholding users who attempt [install of gcc from source] on the mailing lists.

We spoiled the users, they assumed that Sage is a distribution. Yes, there will be a short while they'll ask - and the answer will be: 

RTFM/apt-get install gfortran/brew install gcc/...

You might recall that in 2020 I wrote the system-package recommendation scripts, a virtual beekeeper if you will, to reduce the workload of those answering the repeated questions by users on the mailing list regarding installation problems.

It would be unwise to make changes to the Sage distribution that provoke a new wave of installation problems.

[Dima Pasechnik]

On Fri, 24 Sep 2021, 17:30 Matthias Koeppe, <matthia...@gmail.com> wrote:

On Friday, September 24, 2021 at 9:03:20 AM UTC-7 Dima Pasechnik wrote:

On Fri, Sep 24, 2021 at 4:43 PM Matthias Koeppe <matthia...@gmail.com> wrote:

you are shifting the work from our working spkg-install.in script to handholding users who attempt [install of gcc from source] on the mailing lists.

We spoiled the users, they assumed that Sage is a distribution. Yes, there will be a short while they'll ask - and the answer will be: 

RTFM/apt-get install gfortran/brew install gcc/...

You might recall that in 2020 I wrote the system-package recommendation scripts, a virtual beekeeper if you will, to reduce the workload of those answering the repeated questions by users on the mailing list regarding installation problems.

My surname means "beekeeper" in Russian/Ukrainian, FYI :-P

Your recommendations get routinely ignored, as they are recommendations, and not laws - one can proceed to building, and ignore them. On the other hand, if you try to build Sage without any C compiler on the PATH, you'll be forced to install one, in no uncertain terms.

It would be unwise to make changes to the Sage distribution that provoke a new wave of installation problems.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/6ab95d5b-6e54-4e2a-aaf8-19e6f858c9a9n%40googlegroups.com.

[Matthias Koeppe]

On Friday, September 24, 2021 at 9:36:57 AM UTC-7 Dima Pasechnik wrote:

On Fri, 24 Sep 2021, 17:30 Matthias Koeppe, <matthia...@gmail.com> wrote:

You might recall that in 2020 I wrote the system-package recommendation scripts

 

Your recommendations get routinely ignored, as they are recommendations, and not laws - one can proceed to building, and ignore them. On the other hand, if you try to build Sage without any C compiler on the PATH, you'll be forced to install one, in no uncertain terms.

Yes, as you know I implemented this already in https://trac.sagemath.org/ticket/32060 ("configure: Change defaults to --with-system-gcc=force"), which is slated for inclusion in Sage 9.5.

There is no need to put the axe to the Sage distribution just because we want issue more forceful recommendations.

[Dima Pasechnik]

with a number of platforms needing a bump to gcc 11 (e.g. Apple's M1), it's better to stop fooling around with vendoring gcc all together; the earlier, the better.

  

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/24a4706e-aa55-4dc4-aab8-2d1276a20e04n%40googlegroups.com.

[William Stein]

On Fri, Sep 24, 2021 at 9:03 AM Dima Pasechnik <dim...@gmail.com> wrote

> We spoiled the users, they assumed that Sage is a distribution. Yes, there will be a short while they'll ask - and the answer will be:

> RTFM [...]

I don't want to do that to our users.  Sage was started to compete with [Magma | Mathematica | Maple], so the install process for Sage needed to compete with what [Magma | Mathematica | Maple] offers.   A few weeks ago, I was bummed that Ken Ribet [1] was lamenting on Facebook that he does NOT recommend that people install Sage, because it is too confusing or likely to be broken.  Something like [2, 3] but for Sage would go a long way to help.   

Hopefully going forward, we can spoil users of Sage even more than we already are. 

[1] A good example of a representative influential mathematician, e.g., he was the last president of the American Mathematical Society.  He uses MacOS.
[2] https://github.com/jupyterlab/jupyterlab_app
[3] https://groups.google.com/g/sage-devel/c/QeYle_D8Otc/m/_5Q8zHPLAwAJ?utm_medium=email&utm_source=footer

--
William (http://wstein.org)

[Dima Pasechnik]

On Fri, Sep 24, 2021 at 6:26 PM William Stein <wst...@gmail.com> wrote:

On Fri, Sep 24, 2021 at 9:03 AM Dima Pasechnik <dim...@gmail.com> wrote
> We spoiled the users, they assumed that Sage is a distribution. Yes, there will be a short while they'll ask - and the answer will be:
> RTFM [...]

I don't want to do that to our users.  Sage was started to compete with [Magma | Mathematica | Maple], so the install process for Sage needed to compete with what [Magma | Mathematica | Maple] offers.   A few weeks ago, I was bummed that Ken Ribet [1] was lamenting on Facebook that he does NOT recommend that people install Sage, because it is too confusing or likely to be broken.  Something like [2, 3] but for Sage would go a long way to help.   

Does Ken build Sage from source? I guess he does not, and

yes, Sage binaries on macOS in particular are total crap - in part thanks to "let's vendor everything moving" approach,

in part thanks to Apple being bad to developers.

Hopefully going forward, we can spoil users of Sage even more than we already are. 

Sage is trying to bite more than it can chew, and it is badly needs to slim down, before becoming a total crap nobody uses. 

If one wants to spoil (in the good way) Sage users on macOS, more brew formulas are needed for Sage packages.

Then a similar to Sage on Debian Sage package might be possible.

This is totally orthogonal to vendoring things needlessly.

[1] A good example of a representative influential mathematician, e.g., he was the last president of the American Mathematical Society.  He uses MacOS.
[2] https://github.com/jupyterlab/jupyterlab_app
[3] https://groups.google.com/g/sage-devel/c/QeYle_D8Otc/m/_5Q8zHPLAwAJ?utm_medium=email&utm_source=footer

--
William (http://wstein.org)

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CACLE5GB1PuUXJcU-d-PrOsYo1yL3BTeaJPsVsSq-3nGzT9095Q%40mail.gmail.com.

[William Stein]

On Fri, Sep 24, 2021 at 10:38 AM Dima Pasechnik <dim...@gmail.com> wrote:

On Fri, Sep 24, 2021 at 6:26 PM William Stein <wst...@gmail.com> wrote:

On Fri, Sep 24, 2021 at 9:03 AM Dima Pasechnik <dim...@gmail.com> wrote

 

yes, Sage binaries on macOS in particular are total crap - in part thanks to "let's vendor everything moving" approach,

in part thanks to Apple being bad to developers.

I assume you are talking about the official binaries that are distributed on Sagemath.org.  Fortunately, the Sage binaries on 

MacOS that are produced by the conda-forge devs are not total crap.   If you read

   https://groups.google.com/g/sage-devel/c/QeYle_D8Otc/m/_5Q8zHPLAwAJ?utm_medium=email&utm_source=footer

you'll see that Isuru Fernando quickly responded to my post with something I could copy/paste into the terminal on my Macbook

M1 ARM laptop.  I did so, waited for the download/install, and had what appeared to be a perfectly working self-contained pure userspace

copy of Sage on my M1 laptop (and it is not using x86_64 emulation).      Similarly, the "JupyterLab app" that sparked that thread 

is also a nice graphical MacOS installer for JupyterLab (via Electron) with the Python scientific stack bundled -- the guy 

who made it was introducing it in the JupyterLab weekly dev meeting, and within a few sentences of him telling us about it, 

I had installed and tested it and it  was working.  He also said that it wasn't particular hard for him to build, since so much work 

has gone into making things like conda-forge more mature.

 

Hopefully going forward, we can spoil users of Sage even more than we already are. 

Sage is trying to bite more than it can chew, and it is badly needs to slim down, before becoming a total crap nobody uses. 

If one wants to spoil (in the good way) Sage users on macOS, more brew formulas are needed for Sage packages.

Then a similar to Sage on Debian Sage package might be possible.

This is totally orthogonal to vendoring things needlessly.

The way to spoil users of Sage on MacOS (or anywhere) is to create a binary installer that work really well, similar to 

what the JupyterLab devs did in [3].  It seems to me that this doesn't necessarily involve Homebrew or Debian Sage at all, but

instead involve vendoring things, though that vendoring might better happen by joining forces with conda-forge, rather than 

duplicating their work.  

Isn't conda-forge also doing exactly the thing that you're arguing against as doing?  

I started Sage and its package system around the same time Travis Oliphant started conda, and it 

could have turned out that Sage's package system is the one that has 14000+ packages and everybody uses in the scientific

Python ecosystem.   It didn't turn out that way.  I'm glad some people such as Isuru in the Sage community have 

embraced conda-forge.

I hope we can focus on solving the big important problems, rather than whatever is being argued about in this thread.

Matthias is right that another important big challenge is "pip install sagemath", and it sounds to me like he has a viable

strategy to eventually solve that problem.  It's a completely different problem than "binary that is easy for Ken Ribet

to install on macOS", which conda-forge could solve.   And yet a different big problem is a "Sage on Microsoft Windows",

which conda-forge doesn't solve at all, and for which the cygwin solution isn't close to optimal.

In conflict with all of the above, I also personally wish there were a significantly smaller Sage core with much less 

dependencies, and which removes everything from the Sage that annoys Dima, and much more.    This is a difficult

technical challenge, since it would certainly involve changing core parts of the library.  E.g., it would be nice to have

a working Sage that doesn't depend on Maxima or GAP being present, but still starts up and is generally useful

for the rest of what Sage does.    Creating such a thing involves making significant changes to the assumptions that

can be made in the Sage library code about what they assume is available by default. 

 -- William

[1] A good example of a representative influential mathematician, e.g., he was the last president of the American Mathematical Society.  He uses MacOS.
[2] https://github.com/jupyterlab/jupyterlab_app
[3] https://groups.google.com/g/sage-devel/c/QeYle_D8Otc/m/_5Q8zHPLAwAJ?utm_medium=email&utm_source=footer

--
William (http://wstein.org)

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CACLE5GB1PuUXJcU-d-PrOsYo1yL3BTeaJPsVsSq-3nGzT9095Q%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq0hT_e-2Mq1B4MVqEq1P5a929HHUo0bq3z-%2BexK36diRA%40mail.gmail.com.

--

William (http://wstein.org)

[Nils Bruin]

On Friday, 24 September 2021 at 11:12:38 UTC-7 wst...@gmail.com wrote:

In conflict with all of the above, I also personally wish there were a significantly smaller Sage core with much less 

dependencies, and which removes everything from the Sage that annoys Dima, and much more.    This is a difficult

technical challenge, since it would certainly involve changing core parts of the library.  E.g., it would be nice to have

a working Sage that doesn't depend on Maxima or GAP being present, but still starts up and is generally useful

for the rest of what Sage does.    Creating such a thing involves making significant changes to the assumptions that

can be made in the Sage library code about what they assume is available by default. 

Be careful what you wish for! Although in this particular case you may not get it anyway, so it wouldn't be a problem. If there are different degrees to which you can have "sage installed" there's a whole new  slew of problems that arise: if someone has a problem in such a situation, they need to figure if they have a partial sagemath install and if so, which parts are missing and if their absence causes the problem. I think this would similarly lead to people not recommending to install sagemath. I think an inevitable corollary of having components of sage not mandatory is that support and maintenance of these components deteriorates.

Note that Ken not advising people to install sage doesn't necessarily mean he doesn't advise people to use it. For many people who are not particularly computer-literate, using cloud-based solutions nowadays may be a sensible option. I think for sage there is something called CoCalc that might do a reasonable job.

I do think it is important that we do package sage in a way that  a lot of people can productively install it, though, and especially that it's easy to get a development-capable installation. On a positive note, a student of mine was able to get  a source install on Windows via cygwin running recently. I understand it wasn't pleasant experience, but the student did succeed without outside help -- I was impressed. I would have advised WSL2, which worked quite well for me when I tried it on Win10. 

He did note that the Cygwin binary distributions available at https://github.com/sagemath/sage-windows/releases only go to 9.3 -- and there were some functional differences between 9.3 and 9.4 that lead to incompatibilities for https://github.com/nbruin/RiemannTheta that should eventually get incorporated in sage to resolve the 12-year old ticket https://trac.sagemath.org/ticket/6371. This is partially an experiment to see if offering new functionality initially as a separate pip-installable package improves development and early availability [like a preprint server ...]. One big downside I already found is that "sage -pip install ... --user" doesn't work. People need a LOT of knowledge to use a sagemath python module that is not installed in the main sage tree (which may not be writable for them!).

[Dima Pasechnik]

On Fri, 24 Sep 2021, 18:26 William Stein, <wst...@gmail.com> wrote:

On Fri, Sep 24, 2021 at 9:03 AM Dima Pasechnik <dim...@gmail.com> wrote
> We spoiled the users, they assumed that Sage is a distribution. Yes, there will be a short while they'll ask - and the answer will be:
> RTFM [...]

I don't want to do that to our users.  Sage was started to compete with [Magma | Mathematica | Maple], so the install process for Sage needed to compete with what [Magma | Mathematica | Maple] offers. 

we already have https://trac.sagemath.org/ticket/32060

which will be, from the point of view of 99.9% of the users, identical to what's proposed in the ticket we are discussing, #32532.

(the remaining 0.1% are people deliberately building gcc/gfortran as a part of Sage).

Thus on Matthias' side it's a bit strange to claim that with #32532 we will be hit by a barrage of users' questions. We will be, as much as, or as little as, we will be with #32060.

 A few weeks ago, I was bummed that Ken Ribet [1] was lamenting on Facebook that he does NOT recommend that people install Sage,

by the way, that's a very kind feedback indeed, let's just all lament on social media about entitled Sage users who contribute laments to Sage. :-)

 Maybe they can at least contribute to Sage donation campaign some money for developers' Apple fees and a few Apple M1 macs...

because it is too confusing or likely to be broken.  Something like [2, 3] but for Sage would go a long way to help.   

Hopefully going forward, we can spoil users of Sage even more than we already are. 

[1] A good example of a representative influential mathematician, e.g., he was the last president of the American Mathematical Society.  He uses MacOS.
[2] https://github.com/jupyterlab/jupyterlab_app
[3] https://groups.google.com/g/sage-devel/c/QeYle_D8Otc/m/_5Q8zHPLAwAJ?utm_medium=email&utm_source=footer

--
William (http://wstein.org)

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CACLE5GB1PuUXJcU-d-PrOsYo1yL3BTeaJPsVsSq-3nGzT9095Q%40mail.gmail.com.

[Matthias Koeppe]

On Friday, September 24, 2021 at 11:12:38 AM UTC-7 wst...@gmail.com wrote:

I also personally wish there were a significantly smaller Sage core with much less 

dependencies, and which removes everything from the Sage that annoys Dima, and much more.    This is a difficult

technical challenge, since it would certainly involve changing core parts of the library.  E.g., it would be nice to have

a working Sage that doesn't depend on Maxima or GAP being present, but still starts up and is generally useful

for the rest of what Sage does.    Creating such a thing involves making significant changes to the assumptions that

can be made in the Sage library code about what they assume is available by default. 

Yes, this is exactly what modularization is doing. https://trac.sagemath.org/ticket/29705

[Dima Pasechnik]

Yes, they surely are, and I am saying that we should not duplicate their (or other distros') efforts, and join forces instead.

The sooner dead weight packages such as gcc are removed from Sage, the better, not the least because they allow to keep dragging our feet...

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CACLE5GBS56zZrDks%2BAU7F9g0yky0MyNv_nUaLXoLY93W%2B2brEg%40mail.gmail.com.

[Matthias Koeppe]

It works by defining subsets (namespace packages) of the Sage library that are carefully designed around the dependencies on compiled (non-pip-installable) libraries.

For example, https://trac.sagemath.org/ticket/32432 defines a distribution sagemath-polyhedra with minimal dependencies. This work needs help in disentangling some basic library code (such as for integers, integer matrices, etc.) that depends on too many libraries simultaneously (PARI and FLINT and NTL and ....)

The benefit of constructing such distributions is: It has a clearly definable goal; when done, it can immediately be advertised to potential new users; and it does not compromise the usefulness of the Sage distribution for old users on the way there.

[kcrisman]

The way to spoil users of Sage on MacOS (or anywhere) is to create a binary installer that work really well,

Correct.  This is what all this discussion should be aiming toward - including a binary installer that *allows for compiling/installing of optional packages and Cython*.  (Or gives not just a meaningful error if you try to do that and fail, but a precise thing to click on to do it.)

The reason why building Sage from source is so important historically is that a newbie on any platform, WITH NO EXPERIENCE USING ANY COMMAND LINE TOOLS, was able to download something, open a Terminal, and get a version of Sage that worked on their particular platform.  And to easily update it to the *next* version, or use Cython, or whatever.

In the event, this didn't end up being the case long-term - on Mac, at least, largely due to Apple's doing annoying things and then partly due to our changing how to "upgrade".  But that is what one would want, to have a "viable open-source competitor to ..."

If we can decouple those things from having to build from source, great.  It's not clear to me that is the case, because there is so much fragmentation in the computer space.  That's not our problem, but it is if we want to attract as many users as possible - including ones who will not be able to follow the instructions in a few posts in this thread, or who have no choice but to stick on older "EOL" platforms - that is what is out there.  You can mock people on them, but many of them really cannot afford to upgrade or whoever is controlling their computers will not do it.

Obviously it would good to make things less work for both our very hard-working developers and for users.  If we can disentangle from Sage-the-distribution effectively, that is fine.  Please, let's not have to have Fortran - I guess we already have disentangled R, or else someone would have mentioned that as requiring Fortran?  

But so far, it's only been demonstrated that either 1) viable easier distros are from projects not in the Sage core codebase like the new Mac app that Nathan D.'s group has created (I know it wasn't just him, I just forget who else) or like Isuru's distribution 2) you need to be well within the Linux ecosystem and conversant with things that a large number of users will not be familiar with.  Including some who are highly computer-literate, but not shell-literate.  And having the gcc/gfortran, again historically, was a great way to fix that.  Apparently that is still the case in a number of situations, or we wouldn't be having this conversation.  It's a failsafe for those times, which keep creeping in, and unless Apple itself starts providing the full ecosystem (fat chance), we'll still be relying on the good graces of projects like Homebrew.

The point is made several times that an internet solution may work for this.  But I have to point out YET AGAIN that not everyone lives in happy internet-land with great connectivity all day long.  And Cocalc is a big app for some connectivity situations too ... so that's not the long-term answer.  As many of us discovered during the pandemic and all of a sudden even some colleagues in "the West" (whatever that means) did not have as good of internet as we thought.

[Matthias Koeppe]

On Friday, September 24, 2021 at 1:07:48 PM UTC-7 kcrisman wrote:

 I guess we already have disentangled R, or else someone would have mentioned that as requiring Fortran?  

No, we haven't even removed the R dependency yet. When I proposed downgrading R from its status as standard package (because of *actual* build problems of our R spkg at the time), it was met with opposition here on sage-devel. As a result, the R spkg currently has an in-between status: It's still a standard package but I made it "disablable" (./configure --disable-r). That was good enough for me as I can disable it in my pip-installable builds (source / wheels); and so can users who are running into build problems.

[Volker Braun]

On OSX I don't feel comfortable recommending anyone to run a script as root so homebrew can barf random files into the filesystem. For security and maintainability we really need to be able to install Sage without having unsigned and effectively unversioned dependencies.

My suggestion would be to replace the toolchain packages with a local conda install, i.e. keep the part where our configure checks if a suitable gcc/gfortran/... is installed. If not then pull down miniconda and install a working toolchain into $SAGE_LOCAL/conda, instead of trying to compile it. That way we can easily offload the toolchain maintenance effort, and keep all the benefits.

[Dima Pasechnik]

On Sun, Sep 26, 2021 at 12:53 PM Volker Braun <vbrau...@gmail.com> wrote:

On OSX I don't feel comfortable recommending anyone to run a script as root so homebrew can barf random files into the filesystem. For security and maintainability we really need to be able to install Sage without having unsigned and effectively unversioned dependencies.

I don't really see how homebrew is different from a rolling Linux distro. The core binaries are audited, etc etc.

Surely if you install arbitrary binary stuff (their "bottles") from a non-standard location, then you're at risk (most if not all Linux systems

also have similar high-risk mechanisms), but that's not what you need to build Sage with homebrew packages.

Certainly not if you use gfortran and other core packages such as gmp from Homebrew, then you're as secure as with a Linux distro, IMHO.

My suggestion would be to replace the toolchain packages with a local conda install, i.e. keep the part where our configure checks if a suitable gcc/gfortran/... is installed. If not then pull down miniconda and install a working toolchain into $SAGE_LOCAL/conda, instead of trying to compile it. That way we can easily offload the toolchain maintenance effort, and keep all the benefits.

 

I know nothing about security mitigations in conda-forge, are they significantly stricter than ones in Homebrew?

Anyway, I don't see much benefit from automating this pulling, I think it's better to tell the user upfront what they have to have installed

(including an option of using conda-forge, or mamba-forge), as an average Sage user should be fine with Sage from conda-forge,

it's more flexible than binary Sage installs.

Dima

On Friday, September 24, 2021 at 12:17:29 AM UTC+2 dim...@gmail.com wrote:

https://trac.sagemath.org/ticket/32532 proposes to remove these packages as not needed, and a huge time sink for everyone involved.

Rationale: nowadays every platform that Sage supports has said tools (or their equivalents - e.g. clang/clang++ in case of macOS and FreeBSD) available, if not outright as a system package, but surely with a very minimal effort.

There is no need to carry this baggage forward.

Moreover, it seems that Sage  is the only Python library/platform around which (potentially) vendors gcc/g++/gfortran.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/7ad09b74-dc26-473c-b44f-342ab84e01ddn%40googlegroups.com.

[Volker Braun]

On Sunday, September 26, 2021 at 11:34:33 PM UTC+2 dim...@gmail.com wrote:

I don't really see how homebrew is different from a rolling Linux distro.

Homebrew doesn't come from the OS vendor. No automatic security updates. There is no package management where the admin can look up if any suspicious file was installed by homebrew. You can't verify that the files in /usr/local are actually from homebrew, and not tampered with. The homebrew installation process is not cryptographically signed. I can go on if you want me to...

Conda, on the other hand, doesn't need root. Its as safe as any other user space app. You don't need admin permissions, and the standard unix permissions prevent you from screwing over other users on the system.

[Dima Pasechnik]

On Mon, 27 Sep 2021, 00:01 Volker Braun, <vbrau...@gmail.com> wrote:

On Sunday, September 26, 2021 at 11:34:33 PM UTC+2 dim...@gmail.com wrote:

I don't really see how homebrew is different from a rolling Linux distro.

Homebrew doesn't come from the OS vendor. No automatic security updates. There is no package management where the admin can look up if any suspicious file was installed by homebrew. You can't verify that the files in /usr/local are actually from homebrew, and not tampered with. The homebrew installation process is not cryptographically signed. I can go on if you want me to...

Homebrew need not be installed into /usr/local, and doesn't need root if one chooses not to install there.

Homebrew packages do come with sha256 hashes.

Not sure if it's any less secure if than conda if used this way.

Conda, on the other hand, doesn't need root. Its as safe as any other user space app. You don't need admin permissions, and the standard unix permissions prevent you from screwing over other users on the system.

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/0a523dcb-c97b-44f2-90cd-353d88f04a91n%40googlegroups.com.

[Nathan Dunfield]

On Friday, September 24, 2021 at 1:12:38 PM UTC-5 William Stein wrote:

I assume you are talking about the official binaries that are distributed on Sagemath.org.  Fortunately, the Sage binaries on 

MacOS that are produced by the conda-forge devs are not total crap.  

William,

There are two different Sage binaries for macOS referred to on SageMath.org.  The first, which is the recommended one is:

https://github.com/3-manifolds/Sage_macOS/releases

created by Marc Culler over the past year and is very far from total crap.  While it does not yet support the M1 natively, it does offer a code-signed and notarized app that is just as easy to install as Mathematica et al.   (Note that the JupyterLab App is *not* notarized so one gets the "macOS cannot verify that this app is free from malware" message and have to go through System Preferences to run the installer.)

The second is the old:

http://mirrors.mit.edu/sage/osx/intel/index.html

Best,

Nathan

[Samuel Lelievre]

2021-09-27 02:03:51 UTC, Nathan Dunfield:

Nathan,

I believe William is talking about "sage-forge" or "sagemath-forge"

as discussed recently on sage-devel in these threads:

https://groups.google.com/g/sage-devel/c/JYwHrmcqNhc

https://groups.google.com/g/sage-devel/c/QeYle_D8Otc

https://groups.google.com/g/sage-devel/c/NfUKjAaTcUg

I'm hoping to test it this week.  --Samuel

[Samuel Lelièvre]

2021-09-27 à 08:32 UTC, Samuel Lelievre:

>
> Nathan,
>
> I believe William is talking about "sage-forge" or "sagemath-forge"
> as discussed recently on sage-devel in these threads:
>
> https://groups.google.com/g/sage-devel/c/JYwHrmcqNhc
> https://groups.google.com/g/sage-devel/c/QeYle_D8Otc
> https://groups.google.com/g/sage-devel/c/NfUKjAaTcUg
>
> I'm hoping to test it this week. --Samuel

Scratch the first link (from 12 years ago).

[Volker Braun]

On Monday, September 27, 2021 at 1:29:23 AM UTC+2 dim...@gmail.com wrote:

Homebrew need not be installed into /usr/local, and doesn't need root if one chooses not to install there.

You can theoretically untar homebrew in a different directory, but at least when I tried that a couple of years ago I found myself up in a shit creek without a paddle. Have you actually tried? The installation document says:

However do yourself a favour and use the installer to install to the default prefix. Some things may not build when installed elsewhere. One of the reasons Homebrew just works relative to the competition is because we recommend installing here. Pick another prefix at your peril!

[Dima Pasechnik]

I am not insisting on Homebrew as the primary choice. If you think miniconda/conda-forge/mamba-forge is a better way, fine, let's go this way then.

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/67bd3615-2c52-4c19-b68a-70f2187e092cn%40googlegroups.com.