proposal - remove gcc, gfortran, python building/spkgs

[Dima Pasechnik]

It's high time we get rid of this annoyances; all the systems Sage
supports have C/C++/fortran
compilers capable of building Sage, and Python3 as well.

(and in the very rare cases where it's not available, it's usually a
problem solved
by using something like Conda;)

[David Roe]

I'm not that well informed of the consequences, but I'm generally supportive of removing gcc and (especially) gfortran.  We should point people at other resources to get a functional compiler if there's an issue.  I also give Dima's opinions on this a lot of weight, since I've seen him answer question after question from people struggling to build Sage.  So +1 from me.

Would this mean that Sage not longer installed our own python and we just used the system python to run Sage (or require people to install their own python first if the system python were too old)?  I've found it convenient to have all the python packages Sage installs be self contained in a folder in userspace, but I'm open to this change if it makes the build process smoother.  If we do this, we should add some documentation for Python's venv for Sage developers (like me) who are used to having things automatically self contained.

David

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq0c4rn62YdQ-U8m7rCCpqzSmeGh%3DO%2BMg99kVv25jDOJig%40mail.gmail.com.

[Matthias Koeppe]

Strong -1 on this.

Given the troubles that we have every time that a major gcc version shows up in distributions (we still do not have GCC 11 support -https://trac.sagemath.org/ticket/31786), and given the trouble that we had most recently with faulty homebrew packaging of python3, this would dramatically limit our options.

It would also make test tickets for upcoming releases, such as Python 3.10, in https://trac.sagemath.org/ticket/30767, infeasible.

[Justin C. Walker]

On the face of it, I am not happy with this. I do not use any package managers on any of my macOS systems, and really don’t want to get into the business of dealing with the things that go wrong there.

Unless I am missing something, I am a strong “-1” for this move.

Justin

--
Justin C. Walker, Curmudgeon at Large
Director
Institute for the Enhancement of the Director's income
-----------
--
They said it couldn't be done, but sometimes,
it doesn't work out that way.
- Casey Stengel
--

[Dima Pasechnik]

On Thu, Jun 24, 2021 at 7:16 PM Matthias Koeppe
<matthia...@gmail.com> wrote:
>
> Strong -1 on this.
> Given the troubles that we have every time that a major gcc version shows up in distributions (we still do not have GCC 11 support -https://trac.sagemath.org/ticket/31786),

This is usually a waste of time to support people trying to build gcc,
sorry. It often just does not work, as an older gcc is often
impossible to build
with a newer one.
Let them use Conda or clang, this is my answer (clang on Linux is
mature enough). Non-crazy distributions also allow installing
non-bleeding-edge gcc.

> and given the trouble that we had most recently with faulty homebrew packaging of python3, this would dramatically limit our options.
> It would also make test tickets for upcoming releases, such as Python 3.10, in https://trac.sagemath.org/ticket/30767, infeasible.

pyenv provides perfectly functioning Python3, many versions to choose
from. Or the official Python (for macOS).
Or Conda, again, is to the rescue.

>
> On Thursday, June 24, 2021 at 3:57:44 AM UTC-7 Dima Pasechnik wrote:
>>
>> It's high time we get rid of this annoyances; all the systems Sage
>> supports have C/C++/fortran
>> compilers capable of building Sage, and Python3 as well.
>>
>> (and in the very rare cases where it's not available, it's usually a
>> problem solved
>> by using something like Conda;)
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/c2ca9b40-d54c-476d-afdf-17a48b78a535n%40googlegroups.com.

[Matthias Koeppe]

On Thursday, June 24, 2021 at 12:24:10 PM UTC-7 Dima Pasechnik wrote:

clang on Linux is
mature enough)

As you know, we are currently not testing this at all. 

 

[Volker Braun]

What about replacing gcc with a script that installs the conda toolchain automatically. The distro gcc currently does not work for me, so its not THAT rare to get into problems. On the other hard, installing conda is more likely to work than compiling gcc from scratch.

[Michael Orlitzky]

On Thu, 2021-06-24 at 15:54 -0700, Volker Braun wrote:
> What about replacing gcc with a script that installs the conda toolchain
> automatically. The distro gcc currently does not work for me, so its not
> THAT rare to get into problems. On the other hard, installing conda is more
> likely to work than compiling gcc from scratch.
>
>

What does Conda do that a gcc-10 deb/rpm can't do?

I'm probably spoiled since Gentoo allows you to switch the default "cc"
compiler. Is the problem that e.g. gcc-10 and gcc-11 conflict? Or that
with both installed, the user has to specify CC/CXX?

[Matthias Koeppe]

On Thursday, June 24, 2021 at 4:24:10 PM UTC-7 Michael Orlitzky wrote:

On Thu, 2021-06-24 at 15:54 -0700, Volker Braun wrote:
> What about replacing gcc with a script that installs the conda toolchain
> automatically. The distro gcc currently does not work for me, so its not
> THAT rare to get into problems. On the other hard, installing conda is more
> likely to work than compiling gcc from scratch.

What does Conda do that a gcc-10 deb/rpm can't do?

It can be installed by unprivileged users. 

 

[jonatha...@googlemail.com]

I did a large computation on a redhat application server 1 1/2 years ago and I was very happy that I could just compile gcc to replace the version 4.x. There where a couple of problems though, but all of them could be resolved. I can't reproduce this however, as those people have thankfully provided gcc 10.2.0 in the meantime.

It's -1 from my side unless:

- there is a tool that allows installling gcc (or a tested compiler) without root privileges,

- it's well-documented and whatever the instructions, also a non-advanced person can follow it,

- there is still support for the issues with that (it doesn't have to be from our community)

In general, I think it is a good idea to join forces. If there is a community/tool that deals with these kind of issues already, we could invest our resources in other things.

Sidenote: I also think that administrators in universities etc. should provide a decent compiler, but I don't think we are going to change this. Our university network provides you a bunch of gcc versions and this makes life a lot easier for everyone.

[Dima Pasechnik]

On Fri, 25 Jun 2021 at 07:12, 'jonatha...@googlemail.com' via sage-devel <sage-...@googlegroups.com> wrote:

I did a large computation on a redhat application server 1 1/2 years ago and I was very happy that I could just compile gcc to replace the version 4.x. There where a couple of problems though, but all of them could be resolved. I can't reproduce this however, as those people have thankfully provided gcc 10.2.0 in the meantime.

I do not see an effort to maintain the ability (sometimes broken?) to build/install gcc and python 

as a core task for Sagemath. It is a burden on bots, on GH Actions we run, on people who actually maintain it, on users who try going this route without thinking.

Spin it out into an alternative to Conda if you so inclined.

Such a separation would make it clean for the user that it is not necessary in 99% of the cases, that it can be updated independently, etc.

I should have added the rest of Sage “toolchain” to the list.

It's -1 from my side unless:

- there is a tool that allows installling gcc (or a tested compiler) without root privileges,

- it's well-documented and whatever the instructions, also a non-advanced person can follow it,

- there is still support for the issues with that (it doesn't have to be from our community)

In general, I think it is a good idea to join forces. If there is a community/tool that deals with these kind of issues already, we could invest our resources in other things.

Sidenote: I also think that administrators in universities etc. should provide a decent compiler, but I don't think we are going to change this. Our university network provides you a bunch of gcc versions and this makes life a lot easier for everyone.

Matthias Koeppe schrieb am Freitag, 25. Juni 2021 um 03:59:20 UTC+2:

On Thursday, June 24, 2021 at 4:24:10 PM UTC-7 Michael Orlitzky wrote:

On Thu, 2021-06-24 at 15:54 -0700, Volker Braun wrote:


> What about replacing gcc with a script that installs the conda toolchain


> automatically. The distro gcc currently does not work for me, so its not


> THAT rare to get into problems. On the other hard, installing conda is more


> likely to work than compiling gcc from scratch.



What does Conda do that a gcc-10 deb/rpm can't do?

It can be installed by unprivileged users. 

 

--


You received this message because you are subscribed to the Google Groups "sage-devel" group.


To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/fcb07cc1-4a0c-422b-a7df-05e87215693dn%40googlegroups.com.

[jonatha...@googlemail.com]

I never worked with conda, but it looks like it's doable also for an inexperienced person.

At the moment we have a somewhat working setup for those poorly maintained systems. I don't want to remove it without a proper alternative. If the alternative is to document that one should/can use conda instead, then this is fine with me. There is no need to reinvent the wheel.

This is what it looks like from my point of view for poorly mainained systems without root priviliges:

Current situation: Some sort of support.

What I wouldn't like: No support.

What would be a nice resolution: Get conda running (if not already installed). There is plenty of help out there for this. Once you got conda running, we can help you.

[kcrisman]

My biggest concern on this is how flexible things like conda (which I don't know much about) is for different versions of Mac.  I presume that:

* If you are on Linux, you know how to use a package manager (though there may be some Ubuntu users who do not, I have known some)

* If you are on Windows, you will have one of the precompiled ready-to-go solutions we (?) provide

So, do these solutions work for all current versions of Mac?  And none of this "only the most recent Mac versions that Apple hasn't ended support for", because many academic and hobbyist users are not always in a position to upgrade those easily.  I've seen this many times (and not just with myself).  

Also, I'd like to hear from those who have tried to set up Sage installs in no/low-internet situations (e.g. in developing world).

[Dima Pasechnik]

On Thu, Jun 24, 2021 at 11:54 PM Volker Braun <vbrau...@gmail.com> wrote:
>
> What about replacing gcc with a script that installs the conda toolchain automatically.

automatically is too much, I think. Configure should error out if
there is no usable C/C++/gfortran compiler,
along with its toolchain friends patch, etc, and Python3 (just as we
do for tar and make, say).

A by-default off option that installs Conda and deps can be included.
But normally speaking, default system/conda/homebrew packages should do.

> The distro gcc currently does not work for me, so its not THAT rare to get into problems. On the other hard, installing conda is more likely to work than compiling gcc from scratch.
>
> On Thursday, June 24, 2021 at 12:57:44 PM UTC+2 dim...@gmail.com wrote:
>>
>> It's high time we get rid of this annoyances; all the systems Sage
>> supports have C/C++/fortran
>> compilers capable of building Sage, and Python3 as well.
>>
>> (and in the very rare cases where it's not available, it's usually a
>> problem solved
>> by using something like Conda;)
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/763f5e25-ec75-434c-b2dc-978e94457c86n%40googlegroups.com.

[Dima Pasechnik]

On Fri, Jun 25, 2021 at 2:07 PM kcrisman <kcri...@gmail.com> wrote:
>
> My biggest concern on this is how flexible things like conda (which I don't know much about) is for different versions of Mac. I presume that:
>
> * If you are on Linux, you know how to use a package manager (though there may be some Ubuntu users who do not, I have known some)
> * If you are on Windows, you will have one of the precompiled ready-to-go solutions we (?) provide

probably your best bet with Windows 10 is to install WSL 2 and be
basically on Linux.

>
> So, do these solutions work for all current versions of Mac? And none of this "only the most recent Mac versions that Apple hasn't ended support for", because many academic and hobbyist users are not always in a position to upgrade those easily. I've seen this many times (and not just with myself).

I think that Conda is more flexible than Sage in this sense, not less flexible.

>
> Also, I'd like to hear from those who have tried to set up Sage installs in no/low-internet situations (e.g. in developing world).

This is probably an orthogonal issue.
If you have a non-bleeding-edge Linux (not too old) you installed from
a DVD then you can use
a lot of packages for Sage.

[Michael Orlitzky]

Ok, that's a fair point, since it would be a regression to change it.
But considering that "gcc too new" is only a problem on bleeding-edge
linux distributions (and only for a limited time), how many people
would it truly affect?

IF you're on a shared machine, and IF you're running a bleeding edge
distro, and IF your administrator upgraded immediately to the latest
version, and IF your administrator won't install an older version of
gcc, and IF the latest sage isn't compatible with the latest gcc... the
intersection dwindles.

To argue for keeping gcc as an SPKG, we have to pile on even more: IF
you don't know how to compile/install gcc yourself in $HOME, and IF you
don't know how to use conda/nix/gentoo-prefix... then, there's an SPKG
for you. I don't think it's worth the trouble it causes.

[TB]

I have briefly read the thread, and the related threads, but first I
would like to say thanks to all those who do remote debugging by config
logs and general support for others.

Few questions below.

On 25/06/2021 16:27, Dima Pasechnik wrote:
> On Thu, Jun 24, 2021 at 11:54 PM Volker Braun <vbrau...@gmail.com> wrote:
>>
>> What about replacing gcc with a script that installs the conda toolchain automatically.
>
> automatically is too much, I think. Configure should error out if
> there is no usable C/C++/gfortran compiler,
> along with its toolchain friends patch, etc, and Python3 (just as we
> do for tar and make, say).
>
> A by-default off option that installs Conda and deps can be included.
> But normally speaking, default system/conda/homebrew packages should do.
>
>
>

I think it is common do have a working compiler as a dependency for
building a library, like the build-essential [1] package in Debian. I do
not know how difficult it is to have a Sage-compatible compiler suite on
a Mac (this mailing list gives some hints), but at many Linux distros I
suppose it is not too difficult. Is it?

Why does Sage builds gcc by default if it can not find it, instead of
what Dima proposes above? Only if ./configure is called with some flag
like "--with-compile-missing-gcc" it will try to build gcc, and
otherwise it will print the tip about installing system packages, or at
least those missing packages detected to be minimal required dependencies.

I recall Volker(?) once suggested to use nix for packaging. How does it
compare to conda for end-users? For development of Sage and optional
packages? Being free from a cooperation? For offline builds?

>> The distro gcc currently does not work for me, so its not THAT rare to get into problems. On the other hard, installing conda is more likely to work than compiling gcc from scratch.
>>
>> On Thursday, June 24, 2021 at 12:57:44 PM UTC+2 dim...@gmail.com wrote:
>>>
>>> It's high time we get rid of this annoyances; all the systems Sage
>>> supports have C/C++/fortran
>>> compilers capable of building Sage, and Python3 as well.
>>>
>>> (and in the very rare cases where it's not available, it's usually a
>>> problem solved
>>> by using something like Conda;)
>>
>> --
>> You received this message because you are subscribed to the Google Groups "sage-devel" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/763f5e25-ec75-434c-b2dc-978e94457c86n%40googlegroups.com.
>

Regards,
TB

[1] https://packages.debian.org/sid/build-essential

[TB]

This might be obvious for some people, but I just found that ./configure
--help gives:
--with-system-gcc={no|yes (default)|force (exit with an error if no
usable version is found)}
which mean changing the default to force is roughly what is described
above, and the same for other packages like gfortran.

[Matthias Koeppe]

On Friday, June 25, 2021 at 8:17:27 AM UTC-7 mathzeta2 wrote:

Why does Sage builds gcc by default if it can not find it, instead of
what Dima proposes above? Only if ./configure is called with some flag
like "--with-compile-missing-gcc" it will try to build gcc, and
otherwise it will print the tip about installing system packages, or at
least those missing packages detected to be minimal required dependencies.

We already issue hints at the end of configure regarding the installation of system packages. 

I would support strengthening these to errors for some key packages such as gcc (but not gfortran - because Xcode does not provide it) and python3, with the option of overriding them by configure options. We already have such options: --without-system-gcc, --without-system-python3.

This is a milder step than making lots of changes to our build system.

 

[Dima Pasechnik]

indeed - the biggest obstacle is that gcc/g++ is not the only game in
town on Linux, one can use
clang/clang++ as well, as this is only supported in some ad hoc manner
(via variables CC and CXX)

We should either create an option --with-cc_c++={gcc (default)|clang}
and further dispatch on this,
or get rid of --with-system-gcc completely, and rely on CC and CXX.
The latter is much simpler.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/225999bb-949f-fce2-9f0b-3d7b4054743f%40gmail.com.

[Matthias Koeppe]

I have opened https://trac.sagemath.org/ticket/32060#ticket (configure: Change defaults to --with-system-gcc=force, --with-system-python3=force) for this.

[Matthias Koeppe]

On Friday, June 25, 2021 at 8:49:22 AM UTC-7 Dima Pasechnik wrote:

We should either create an option --with-cc_c++={gcc (default)|clang} [...] or [...]

I do not support adding a nonstandard option of this kind. To select a compiler, a user needs to pass CC and CXX. This is standard practice.

[Dima Pasechnik]

well, in strictly GNU world, maybe :-)

I think that

CC=clang ./configure --with-system-gcc

it rather baffling - what should it do? Ignore CC value? Ignore 'gcc' part?

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/5e8a6ac5-d962-462d-b01e-e3881a38c4e1n%40googlegroups.com.

[Matthias Koeppe]

On Friday, June 25, 2021 at 9:51:08 AM UTC-7 Dima Pasechnik wrote:

I think that
CC=clang ./configure --with-system-gcc
it rather baffling - what should it do? Ignore CC value? Ignore 'gcc' part?

"--with-system-gcc" is the default, so users won't have to type it, so no danger of bafflement.

 

[Matthias Koeppe]

There is now a branch at https://trac.sagemath.org/ticket/32060 -- needs review.

[Nathan Dunfield]

On Thursday, June 24, 2021 at 1:16:29 PM UTC-5 Matthias Koeppe wrote:

Strong -1 on this.

Given the troubles that we have every time that a major gcc version shows up in distributions (we still do not have GCC 11 support -https://trac.sagemath.org/ticket/31786), and given the trouble that we had most recently with faulty homebrew packaging of python3, this would dramatically limit our options.

I have benefitted a lot from Sage providing the option of compiling its own gcc when building Sage for academic computer clusters.  Such clusters often run a very old OS with truly ancient compilers.  For example, the main one I use until a year ago was on Centos 6 with gcc 4.8 and  6.1; now it runs Centos 7 (first released 2014!) and gcc 9.3 is available.  With regards to conda, I'm not sure it's a cure-all: when Sage 9.0 came out this cluster's gcc was so old that it couldn't compile Sage's preferred gcc, so I tried with that gcc provided by conda. I still never managed to get Sage to build, IIRC due to competing system libraries and those in $CONDA_ROOT, so I just used conda to install all of Sage in binary form.  Which was very convenient, though I lost the ability to install optional spkg's as a result.

Still, I can see that it might be more robust to have users try to install gcc via a package manager rather than build it from source.  But on the flip side, having Sage build its own Python is surely increases the probability of a successful build.  If the users compiler and tool chain can't build Python, it almost certainly will choke on some much less robust component of Sage and the extra isolation of Sage having it's own Python surely prevents all sorts of headaches...

Best,

Nathan

[Dima Pasechnik]

On Sat, Jun 26, 2021 at 9:49 PM Nathan Dunfield <nat...@dunfield.info> wrote:
>
> On Thursday, June 24, 2021 at 1:16:29 PM UTC-5 Matthias Koeppe wrote:
>>
>> Strong -1 on this.
>> Given the troubles that we have every time that a major gcc version shows up in distributions (we still do not have GCC 11 support -https://trac.sagemath.org/ticket/31786), and given the trouble that we had most recently with faulty homebrew packaging of python3, this would dramatically limit our options.
>
>
> I have benefitted a lot from Sage providing the option of compiling its own gcc when building Sage for academic computer clusters. Such clusters often run a very old OS with truly ancient compilers. For example, the main one I use until a year ago was on Centos 6 with gcc 4.8 and 6.1; now it runs Centos 7 (first released 2014!) and gcc 9.3 is available. With regards to conda, I'm not sure it's a cure-all: when Sage 9.0 came out this cluster's gcc was so old that it couldn't compile Sage's preferred gcc, so I tried with that gcc provided by conda. I still never managed to get Sage to build, IIRC due to competing system libraries and those in $CONDA_ROOT, so I just used conda to install all of Sage in binary form. Which was very convenient, though I lost the ability to install optional spkg's as a result.

Conda can be set up to do Sage development,
see https://wiki.sagemath.org/Conda

>
> Still, I can see that it might be more robust to have users try to install gcc via a package manager rather than build it from source. But on the flip side, having Sage build its own Python is surely increases the probability of a successful build. If the users compiler and tool chain can't build Python, it almost certainly will choke on some much less robust component of Sage and the extra isolation of Sage having it's own Python surely prevents all sorts of headaches...
>
> Best,
>
> Nathan
>
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/0d31810f-7775-4da8-80f4-51cc0ff3f874n%40googlegroups.com.