Building a batteries-included Sage on macOS: gcc and gfortran

[Samuel Lelievre]

Dear sage-devel,

On macOS, alongside my attempts to build Sage

using as many Homebrew packages as possible,

I am trying to build a "batteries-included" Sage

completely independent of Homebrew.

I decided to configure with

    --with-system-gcc=no --with-system-gfortran=no

and was expecting the following outcome:

- Sage builds its own gcc, which contains gfortran

- Sage does not build an additional gfortran

but the actual outcome is:

- Sage builds its own gcc, which contains gfortran

- Sage then tries to build gfortran, which fails

  with an error indicating that Sage built its own gcc,

  which contains gfortran, so gfortran should not be built

Were my expectations legitimate and does this deserve a ticket?

[Matthias Koeppe]

That's a bug. Please open a ticket

[Dima Pasechnik]

On Fri, Jul 31, 2020 at 10:55 PM Matthias Koeppe
<matthia...@gmail.com> wrote:
>
> That's a bug. Please open a ticket
>
> On Friday, July 31, 2020 at 2:44:08 PM UTC-7, Samuel Lelievre wrote:
>>
>> Dear sage-devel,
>>
>> On macOS, alongside my attempts to build Sage
>> using as many Homebrew packages as possible,
>> I am trying to build a "batteries-included" Sage
>> completely independent of Homebrew.
>>
>> I decided to configure with
>>
>> --with-system-gcc=no --with-system-gfortran=no

--with-system-gcc must be yes on macOS, I think.
You actually cannot build Sage with "real" gcc/g++ on macOS,
you need system's gcc/g++, which is actually not gcc/g++, but clang/clang++

system-gcc will come from Xcode, not from Homebrew.

>>
>> and was expecting the following outcome:
>>
>> - Sage builds its own gcc, which contains gfortran
>> - Sage does not build an additional gfortran

see above. This won't fly.

>>
>> but the actual outcome is:
>>
>> - Sage builds its own gcc, which contains gfortran
>> - Sage then tries to build gfortran, which fails
>> with an error indicating that Sage built its own gcc,
>> which contains gfortran, so gfortran should not be built
>>
>> Were my expectations legitimate and does this deserve a ticket?

yes, but no, see above.


>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/21ff39a4-fcfd-4a69-947f-c4a0354b4ec9o%40googlegroups.com.

[Samuel Lelièvre]

2020-07-31 22:26 UTC, Dima Pasechnik:
>
> On Fri, Jul 31, 2020 at 10:55 PM Matthias Koeppe:

> >
> > That's a bug. Please open a ticket
> >
> > On Friday, July 31, 2020 at 2:44:08 PM UTC-7, Samuel Lelievre wrote:
> >>
> >> Dear sage-devel,
> >>
> >> On macOS, alongside my attempts to build Sage
> >> using as many Homebrew packages as possible,
> >> I am trying to build a "batteries-included" Sage
> >> completely independent of Homebrew.
> >>
> >> I decided to configure with
> >>
> >> --with-system-gcc=no --with-system-gfortran=no
>
> --with-system-gcc must be yes on macOS, I think.
> You actually cannot build Sage with "real" gcc/g++ on macOS,
> you need system's gcc/g++, which is actually not gcc/g++, but clang/clang++
>
> system-gcc will come from Xcode, not from Homebrew.

In fact, you can set `--with-system-gcc=no`, and Sage will build
its own `gcc` (using the system `clang` to do so), and then use
that `gcc` and its `gfortran` to build most other packages.

> >> and was expecting the following outcome:
> >>
> >> - Sage builds its own gcc, which contains gfortran
> >> - Sage does not build an additional gfortran
>
> see above. This won't fly.
>
> >> but the actual outcome is:
> >>
> >> - Sage builds its own gcc, which contains gfortran
> >> - Sage then tries to build gfortran, which fails
> >> with an error indicating that Sage built its own gcc,
> >> which contains gfortran, so gfortran should not be built
> >>
> >> Were my expectations legitimate and does this deserve a ticket?
>
> yes, but no, see above.

Turns out I obtained my desired outcome by
- configuring with `--with-system-gcc=no`
- leaving out `--with-system-gfortran` entirely.

The ability to do that means the ticket I asked about opening
is not an absolute necessity, but should I still open it?

[Dima Pasechnik]

On Fri, Jul 31, 2020 at 11:41 PM Samuel Lelièvre
<samuel....@gmail.com> wrote:
>
> 2020-07-31 22:26 UTC, Dima Pasechnik:
> >
> > On Fri, Jul 31, 2020 at 10:55 PM Matthias Koeppe:
> > >
> > > That's a bug. Please open a ticket
> > >
> > > On Friday, July 31, 2020 at 2:44:08 PM UTC-7, Samuel Lelievre wrote:
> > >>
> > >> Dear sage-devel,
> > >>
> > >> On macOS, alongside my attempts to build Sage
> > >> using as many Homebrew packages as possible,
> > >> I am trying to build a "batteries-included" Sage
> > >> completely independent of Homebrew.
> > >>
> > >> I decided to configure with
> > >>
> > >> --with-system-gcc=no --with-system-gfortran=no
> >
> > --with-system-gcc must be yes on macOS, I think.
> > You actually cannot build Sage with "real" gcc/g++ on macOS,
> > you need system's gcc/g++, which is actually not gcc/g++, but clang/clang++
> >
> > system-gcc will come from Xcode, not from Homebrew.
>
> In fact, you can set `--with-system-gcc=no`, and Sage will build
> its own `gcc` (using the system `clang` to do so), and then use
> that `gcc` and its `gfortran` to build most other packages.

Does this still work? Surely, some packages have to be built with
Xcode's clang/clang++.
What's the point of this?

>
> > >> and was expecting the following outcome:
> > >>
> > >> - Sage builds its own gcc, which contains gfortran
> > >> - Sage does not build an additional gfortran
> >
> > see above. This won't fly.
> >
> > >> but the actual outcome is:
> > >>
> > >> - Sage builds its own gcc, which contains gfortran
> > >> - Sage then tries to build gfortran, which fails
> > >> with an error indicating that Sage built its own gcc,
> > >> which contains gfortran, so gfortran should not be built
> > >>
> > >> Were my expectations legitimate and does this deserve a ticket?
> >
> > yes, but no, see above.
>
> Turns out I obtained my desired outcome by
> - configuring with `--with-system-gcc=no`
> - leaving out `--with-system-gfortran` entirely.
>
> The ability to do that means the ticket I asked about opening
> is not an absolute necessity, but should I still open it?

well, you're welcome - but it's certainly not a top priority.

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAEcArF3G74qOzFBC2nU2HU8%2B_uLLDO5UG00SENdivAACPTZBeA%40mail.gmail.com.

[Matthias Koeppe]

On Friday, July 31, 2020 at 3:51:45 PM UTC-7, Dima Pasechnik wrote:

On Fri, Jul 31, 2020 at 11:41 PM Samuel Lelièvre
<samuel....@gmail.com> wrote:
>
> 2020-07-31 22:26 UTC, Dima Pasechnik:

> > >> I decided to configure with
> > >>
> > >>     --with-system-gcc=no --with-system-gfortran=no
> >
> > --with-system-gcc must be yes on macOS, I think.
> > You actually cannot build Sage with "real" gcc/g++ on macOS,
> > you need system's gcc/g++, which is actually not gcc/g++, but clang/clang++
> >
> > system-gcc will come from Xcode, not from Homebrew.
>
> In fact, you can set `--with-system-gcc=no`, and Sage will build
> its own `gcc` (using the system `clang` to do so), and then use
> that `gcc` and its `gfortran` to build most other packages.

Does this still work?

Not "still", rather "again", i.e., since Sage 9.1.

This configuration is called "homebrew-macos-standard-gcc_spkg".

https://github.com/sagemath/sage/runs/910575934

Unfortunately, there are indeed several packages that do not succeed to build with the installed gcc, as you can see in this run.

Some of these build failures are already tracked.

For example: https://trac.sagemath.org/ticket/29613 for givaro.

I have marked it as "minor" because, as you say, there is little point in using the gcc from the SPKG when both the XCode "gcc" (clang) and the homebrew compilers are better options.

Samuel -- please do open the ticket - for the broken logic of the --with-system-XYZ arguments.

 

[Samuel Lelièvre]

2020-07-31 23:03 UTC, Matthias Koeppe:

>
> On Friday, July 31, 2020 at 3:51:45 PM UTC-7, Dima Pasechnik wrote:
>>

>> On Fri, Jul 31, 2020 at 11:41 PM Samuel Lelièvre:

>> >
>> > 2020-07-31 22:26 UTC, Dima Pasechnik:
>> > > >> I decided to configure with
>> > > >>
>> > > >> --with-system-gcc=no --with-system-gfortran=no
>> > >
>> > > --with-system-gcc must be yes on macOS, I think.
>> > > You actually cannot build Sage with "real" gcc/g++ on macOS,
>> > > you need system's gcc/g++, which is actually not gcc/g++, but clang/clang++
>> > >
>> > > system-gcc will come from Xcode, not from Homebrew.
>> >
>> > In fact, you can set `--with-system-gcc=no`, and Sage will build
>> > its own `gcc` (using the system `clang` to do so), and then use
>> > that `gcc` and its `gfortran` to build most other packages.
>>
>> Does this still work?
>
>
> Not "still", rather "again", i.e., since Sage 9.1.
>
> This configuration is called "homebrew-macos-standard-gcc_spkg".
>
> https://github.com/sagemath/sage/runs/910575934
>
> Unfortunately, there are indeed several packages that do not
> succeed to build with the installed gcc, as you can see in this run.
>
> Some of these build failures are already tracked.
> For example: https://trac.sagemath.org/ticket/29613 for givaro.
>
> I have marked it as "minor" because, as you say, there is little point
> in using the gcc from the SPKG when both the XCode "gcc" (clang)
> and the homebrew compilers are better options.

If Sage is built in /Applications/SageMath, that folder could be
copied to a different computer running the same version
of macOS, and hopefully everything would work regardless
of whether the other computer had Conda, Homebrew,
Apple's Command Line Tools for Developers, or Xcode.
Not completely sure about that but want to give it a try!

> Samuel -- please do open the ticket - for the broken logic of the
> --with-system-XYZ arguments.

I will do that. The issue is not macOS-specific if I understand
correctly: if one wanted to build Sage's with its own gcc and
gfortran on another platform, the gcc it builds would also have
gfortran and thus trying to build gfortran would complain that
it was already built as part of gcc, is that correct?

Is the following another instance of the same broken logic?

Configuring with
```
$ ./configure
--with-system-gmp=no \
--with-system-mpir=no \
--with-mp=mpir \
```
leads to both gmp and mpir being built.

Expected: mpir needs to be built but not gmp,
and thus we skip building gmp.

Workaround: leave out `--with-system-gmp`.

[Matthias Koeppe]

On Friday, July 31, 2020 at 5:12:23 PM UTC-7, Samuel Lelievre wrote:

Is the following another instance of the same broken logic?

Configuring with
```
$ ./configure
  --with-system-gmp=no \
  --with-system-mpir=no \
  --with-mp=mpir \
```
leads to both gmp and mpir being built.

Expected: mpir needs to be built but not gmp,
and thus we skip building gmp.

That's right -- and this reminded me that we already have a ticket for that: https://trac.sagemath.org/ticket/29620

I have added your gcc/gfortran example to it.

 

[Dima Pasechnik]

Sorry for shouting, but "batteries-included" monolithic monsters
don't work (certainly not long-term), in case of Sage it has become
increasingly obvious in the past ~10 years.
An illusion that it is sustainable has been, ironically, kept alive by
OpenDreamKit, where a number of people were full-time doing the
"dirty" system programming work.

Every package with spkg-configure.m4 should become part of Sage
"toolchain" and be removed from Sage proper. It's really become a
waste of everyone's time, duplication of effort to maintain things
like curl, libpng, gcc/gfortran, python, maxima, etc etc etc.

If one likes to do system programming, there are much better ways to
help Sage project along than maintaining the monolithic monster. E.g.,
removing dead bits, such as MPIR, Atlas, enabling system Python
packages.

Speaking of macOS, on modern macOS properly installing 3rd party
software into /Applications/ needs various Apple's blessings, unless
you turn off their security mechanisms. We badly need someone who can
use Xcode to revive Sage macOS app.
Besides, how about working with Homebrew formulae to offload more of
Sage packages there.
How about helping Conda's Sage distribution. (The latter potentially
very helpful for Conda on Linux, too).

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/84e903f3-4a79-4fc2-80e4-23d6cf4ea7a8o%40googlegroups.com.

[Samuel Lelievre]

2020-08-01 08:00:32 UTC, Dima Pasechnik:

>

> Sorry for shouting, but "batteries-included" monolithic  monsters

> don't work (certainly not long-term), in case of Sage it has become

> increasingly obvious in the past ~10 years.

> An illusion that it is sustainable has been, ironically, kept alive by

> OpenDreamKit, where a number of people were full-time doing the

> "dirty" system programming work.

>

> Every package with spkg-configure.m4 should become part of Sage

> "toolchain" and be removed from Sage proper. It's really become a

> waste of everyone's time, duplication of effort to maintain things

> like curl, libpng, gcc/gfortran, python, maxima, etc etc etc.

>

> If one likes to do system programming, there are much better ways to

> help Sage project along than maintaining the monolithic monster. E.g.,

> removing dead bits, such as MPIR, Atlas, enabling system Python

> packages.

>

> Speaking of macOS, on modern macOS properly installing 3rd party

> software into /Applications/ needs various Apple's blessings, unless

> you turn off their security mechanisms. We badly need someone who can

> use Xcode to revive Sage macOS app.

> Besides, how about working with Homebrew formulae to offload more of

> Sage packages there.

> How about helping Conda's Sage distribution. (The latter potentially

> very helpful for Conda on Linux, too).

Thanks Dima for suggesting better ways to spend my time.

I profoundly agree that I would be more useful if I could

- help prepare spkg-configure.m4 files for more packages

- help enable system Python packages

- help revive  the macOS app

- help with any or all of the Arch, Conda, Cygwin, Debian,

  Fedora, Guix, Homebrew, Nix, ZeroInstall distributions

but sadly I don't know how to do any of those.

If anybody can give a tutorial on any generally useful

but underpowered task that could take extra help,

such as one of the above, or any other one really,

I can make myself available at any time of day or night,

host the tutorial on my department's BigBlueButton server,

record the session if that would seem useful,

share the recording if the recorded people agree,

and try to start doing useful work.

I want to take this opportunity to thank you for your

relentless work on the spkg-configure.m4 front and on

many other fronts, in Sage and the wider math software

ecosystem, and for your invaluable and patient help with

install problems, crash reports, never tired of going

through build logs and crash logs to find the relevant

piece of information, make the appropriate suggestion,

ask the right follow-up question to figure out the

missing bits of information, and getting things going

when they would otherwise easily stall.

[Dima Pasechnik]

learn on the job - pick up e.g. https://trac.sagemath.org/ticket/28991
which already has a draft spkg-configure.m4 posted as attachment, create
a branch with it (needs a bit of tweaking, see comments there)
test it (run ./bootstrap && ./configure, study its output, config.log,
configure source (generated - this is really last resort)), bang head
on the keyboard, ask questions...

After it is done, move on to #29630 (which depends on #28991)

> - help enable system Python packages
> - help revive the macOS app

You'd be on your own here - but surely there is plenty on the net on how to
use Xcode, package apps, sign them, etc. See e.g.
https://medium.com/swlh/the-easiest-way-to-build-macos-installer-for-your-application-34a11dd08744

> - help with any or all of the Arch, Conda, Cygwin, Debian,
> Fedora, Guix, Homebrew, Nix, ZeroInstall distributions

I can offer a quick tour of https://github.com/sagemath/homebrew-science
with tweaked formulae.
I believe it's actually very easy to tweak Homebrew formulae - I did
this a bit, the
main difficulty is that they are in Ruby, and propose fixes upstream with PRs.

>
> but sadly I don't know how to do any of those.
>
> If anybody can give a tutorial on any generally useful
> but underpowered task that could take extra help,
> such as one of the above, or any other one really,
> I can make myself available at any time of day or night,
> host the tutorial on my department's BigBlueButton server,
> record the session if that would seem useful,
> share the recording if the recorded people agree,
> and try to start doing useful work.

see above. Needless to say, we can also do an informal Sage Day via
Google Meet or whatever.

>
> I want to take this opportunity to thank you for your
> relentless work on the spkg-configure.m4 front and on
> many other fronts, in Sage and the wider math software
> ecosystem, and for your invaluable and patient help with
> install problems, crash reports, never tired of going
> through build logs and crash logs to find the relevant
> piece of information, make the appropriate suggestion,
> ask the right follow-up question to figure out the
> missing bits of information, and getting things going
> when they would otherwise easily stall.

Thanks --- don't send me more papers, theses, and books to referee,
and joint papers to write,
for the time being :-)

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/57d0de6e-b549-4835-82b6-774f39ad0f2do%40googlegroups.com.

[Matthias Koeppe]

On Saturday, August 1, 2020 at 1:00:32 AM UTC-7, Dima Pasechnik wrote:

Besides, how about working with Homebrew formulae to offload more of
Sage packages there.
How about helping Conda's Sage distribution. (The latter potentially
very helpful for Conda on Linux, too).

+1 on the suggestion to help on conda-forge packaging. Ask Isuru Fernando how you can help.

This is, in my opinion, the most important direction for sage packaging - it will impact both macOS and Linux.

Regarding homebrew, my suggestion is to limit work to specific problems with packages that they already have -- in particular to report outdated or broken packages. Note that "homebrew/science" has been abandoned.

[Dima Pasechnik]

I did not suggest working on homebrew/science, I meant working on e.g. making their flint useful for Sage - their "core" flint lacks NTL interface.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/b46d7040-6d24-4f9a-8d10-42273699e64eo%40googlegroups.com. is