Question: why does /usr/bin/gcc get called during Sage startup?

[Marc Culler]

I discovered, by installing the Sage_macOS app on a pristine macOS system, that somehow, somewhere, in Sage's startup sequence there is a call to gcc.  This is true whether Sage is being started from a command line or a notebook.

On such a macOS system /usr/bin/gcc exists, but calling it causes a dialog to be posted which asks whether to download and install the Xcode "command line tools".

There is no need for a user to install, or be prompted to install, a C compiler in order to run Sage.  If we want to verify whether a C compiler is installed on the host system then we should check the return value of xcode-select -p rather than calling /usr/bin/gcc.

I am unable to find where this call occurs.  Do any of the Sage developers know which component of Sage could be calling /usr/bin/gcc on start up?

- Marc

[Dima Pasechnik]

It's not quite correct to say that a C, or other, compiler is not required to run Sage. Sage allows you to define, build, load, and run Cython extensions without leaving Sage, thus, it needs to call a compiler after cythonisation step.

It's another question why it's called on startup.
Might be macOS - only ?

Dima

>
>- Marc
>

[Gonzalo Tornaría]

https://github.com/sagemath/sage/blob/develop/src/bin/sage-env#L482 and L494

See: https://github.com/sagemath/sage/issues/14296 and https://github.com/sagemath/sage/commit/69213d74ead4e93687cf61f214b0d96dd3f9885a

Maybe you can workaround this by setting AS=as and LD=ld in sage-env-config.

HTH,

Gonzalo

[Marc Culler]

That was it!

Thank you Gonzalo; indeed, it helps a lot.  And your workaround is fine, since we don't support the -i option,  Even if we did, the default names for as, ld and ar are correct whenever the command line tools are installed.  So that block of code is completely irrelevant for the macOS platform.  This solves the problem.

- Marc

[Marc Culler]

On Tuesday, April 23, 2024 at 8:58:58 PM UTC-5 Dima Pasechnik wrot

It's not quite correct to say that a C, or other, compiler is not required to run Sage. Sage allows you to define, build, load, and run Cython extensions without leaving Sage, thus, it needs to call a compiler after cythonisation step.

It's another question why it's called on startup.

Yes. If a user were trying to compile cython code on a machine with no compiler it would make sense to post a warning and offer to install a compiler.  But it does not make sense to impose that warning and offer on a user who is not doing anything that needs a compiler.

 

Might be macOS - only ?

The xcode-select behavior is macOS only, of course.  But the code block that Gonzalo pointed us to is run for *every* invocation of Sage on every platform.  Its purpose is to determine whether the platform has non-standard names for the gcc assembler, linker or archiver, which evidently can occur in some rare cases.  It does this by calling $CC with a gcc-specific option which returns the name of one of these utiltiies (if $CC happens to be gcc).  While this is done for every invocation of sage, it is seems to be intended primarily for the sage -i command.

I wonder if the Sage community would be willing to reconsider the idea of having one entry point for every function provided by SageMath, with options determining which function is actually invoked.  This was a popular way of doing things at one time, but has been unfashionable for years as one can see, for example, by running man mkfs on linux.

Instead of having a single executable named sage with options like sage -python, sage -pip, sage -i, sage -gap, sage -singular etc. there could be separate executables: .sage, sage-python, sage-pip, sage-install sage-gap, sage-singular.  I think this would be clearer and simpler to implement.  It would be easier to make sure that only needed initializations are done - avoiding asking gcc for the name of its linker just because someone wants to run pip, for example.

- Marc

[Marc Culler]

On Tuesday, April 23, 2024 at 11:17:49 PM UTC-5 Marc Culler wrote:

I wonder if the Sage community would be willing to reconsider the idea of having one entry point for every function provided by SageMath,

I won't speak for the community, but I reconsidered this and I decided that it would be better to leave the structure of the sage script as it is.

- Marc

[Dima Pasechnik]

On Wed, Apr 24, 2024 at 4:45 AM Marc Culler <marc....@gmail.com> wrote:
>
> That was it!
>
> Thank you Gonzalo; indeed, it helps a lot. And your workaround is fine, since we don't support the -i option,

running Cython in sage prompt or in Sage Jupyter notebook has nothing
to do with -i option. One can call cythonize
in a regular Sage session (see, it builds a .so in /tmp):

sage: from sage.misc.cython import compile_and_load
sage: module = compile_and_load("def f(int n):\n return n*n")
sage: module.f(10)
100
sage: module
<module '_tmp_tmpbw1f5mpb_tmp_ys4atzvl_pyx_0' from
'/tmp/tmp6ce3su_y/spyx/_tmp_tmpbw1f5mpb_tmp_ys4atzvl_pyx/_tmp_tmpbw1f5mpb_tmp_ys4atzvl_pyx_0.cpython-311-x86_64-linux-gnu.so'>

> Even if we did, the default names for as, ld and ar are correct whenever the command line tools are installed. So that block of code is completely irrelevant for the macOS platform. This solves the problem.

Anyway, instead of calling these all the time (this was done in
https://github.com/sagemath/sage/issues/14296)
we can let ./configure to fill in @AS@, @LD@, etc templates in
build/bin/sage-build-env-config.in, and having the corresponding env.
vars.
initialised in build/bin/sage-build-env-config(.in).

Dima

>
> - Marc
>
>
>
> On Tuesday, April 23, 2024 at 8:59:12 PM UTC-5 Gonzalo Tornaría wrote:
>>
>> https://github.com/sagemath/sage/blob/develop/src/bin/sage-env#L482 and L494
>>
>> See: https://github.com/sagemath/sage/issues/14296 and https://github.com/sagemath/sage/commit/69213d74ead4e93687cf61f214b0d96dd3f9885a
>>
>> Maybe you can workaround this by setting AS=as and LD=ld in sage-env-config.
>>
>> HTH,
>> Gonzalo
>>
>>
>> On Tuesday, April 23, 2024 at 3:48:18 PM UTC-3 Marc Culler wrote:
>>>
>>> I discovered, by installing the Sage_macOS app on a pristine macOS system, that somehow, somewhere, in Sage's startup sequence there is a call to gcc. This is true whether Sage is being started from a command line or a notebook.
>>>
>>> On such a macOS system /usr/bin/gcc exists, but calling it causes a dialog to be posted which asks whether to download and install the Xcode "command line tools".
>>>
>>> There is no need for a user to install, or be prompted to install, a C compiler in order to run Sage. If we want to verify whether a C compiler is installed on the host system then we should check the return value of xcode-select -p rather than calling /usr/bin/gcc.
>>>
>>> I am unable to find where this call occurs. Do any of the Sage developers know which component of Sage could be calling /usr/bin/gcc on start up?
>>>
>>> - Marc
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/1fe1fca2-f228-45eb-9f43-a1d1be5d5895n%40googlegroups.com.

[Marc Culler]

On Wednesday, April 24, 2024 at 8:05:15 AM UTC-5 Dima Pasechnik wrote:

running Cython in sage prompt or in Sage Jupyter notebook has nothing
to do with -i option.

I realize that.  But it looked to me like those variables are being set in the sage-env script *primarily* to support sage -i.  Perhaps you are right that it is *primarily* to support compiling cython, but it doesn't look like it to me.  In any case, on macOS the default vaues work fine.

- Marc

[Dima Pasechnik]

sage -i actually calls make, on a makefile where everything is already
set up. So no, it's not for sage -i.

>
> - Marc
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/9d670ddb-ba2d-4c60-9fc2-5b0c1c209e3cn%40googlegroups.com.

[Marc Culler]

Well, it almost solved the problem.

It turns out that calling /usr/bin/gcc was not the only issue in sage-env.  That script also calls xcode-select.  On a system with no XCode app and no command line tools, calling gcc causes an error message to be printed to stderr and a dialog to open asking whether to install the command line tools.  Calling xcode-select on such a system prints the same error message but does not open the dialog. The error message appears in the terminal when running sage in a command line, which is annoying and/or alarming to someone with no plans to do anything involving compilation of C code.

The calls to xcode-select were added in PR#36599 in order to force XCode to use Apple's ld-classic linker instead of ld when their new version of ld was totally broken.  This is done by adding -ld_classic to LDFLAGS.

Note to people who worked on PR #36599 (@jhpalmieri and @mkoeppe): I think Apple's new linker is working now, so it is probably no longer necessary and not a good idea to force use of ld_classic.

- Marc

[Matthias Koeppe]

On Wednesday, April 24, 2024 at 6:11:23 AM UTC-7 Marc Culler wrote:

 But it looked to me like those variables are being set in the sage-env script *primarily* to support sage -i.  Perhaps you are right that it is *primarily* to support compiling cython, but it doesn't look like it to me.  In any case, on macOS the default values work fine.

There are three contexts in which setting these variables supports compilation:

- Yes, "sage -i" is one of these contexts. (In addition to sage-env, this context also sets other variables using build/bin/sage-build-env)

- "sage -pip install" is another context.

- And finally is runtime compilation (sage.misc.cython, sage.interfaces.tides)

I do think that we can 

- get rid of many of these variable settings in sage-env (such as LN, MKDIR, ... which were probably only needed for platforms that we stopped supporting long ago) and 

- conditionalize others to the case [ -z "$CC" -a -n "$SAGE_LOCAL" -a -x "$SAGE_LOCAL/bin/gcc" ] that is distinguished earlier in this script. 

I'll open a PR for this.

[Matthias Koeppe]

On Wednesday, April 24, 2024 at 8:19:03 AM UTC-7 Dima Pasechnik wrote:

sage -i actually calls make,

This part is true.

 

on a makefile where everything is already
set up. So no, it's not for sage -i.

This part is false.

Our makefiles (https://github.com/sagemath/sage/blame/develop/Makefile, https://github.com/sagemath/sage/blame/develop/build/make/Makefile.in) do not set such variables. 

The makefiles delegate package installation to build/bin/sage-spkg, which set up the build environment using sage-env and sage-build-env: https://github.com/sagemath/sage/blame/develop/build/bin/sage-spkg#L187

[John H Palmieri]

Hi Marc,

I just tried building Sage without `-ld-classic`. It builds, but I get warnings about "ignoring duplicate libraries", and those cause doctest failures. The lines could be modified to test whether xcode-select is present and executable first, or since Sage now does indeed build without `-ld-classic`, we could filter out the warnings when doctesting.

-- 

John

[Marc Culler]

Hi John,

I have noticed that .dylib files generated by Sage often have as many as 3 identical rpaths.  (When building the macOS app I remove all rpaths and replace them with a single rpath which is relative, meaning it starts with @loader_path.  Apple will not notarize app bundles with absolute rpaths.)

I wonder if the "duplicate libraries" warnings are caused by duplicate rpath entries in the mach files.  I will try to test that hypothesis.

Do you have any idea what could cause those duplicate rpaths?  Are there multiple -rpath options being passed to the compiler?

- Marc

--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/4s_5HmznHZM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/c81c9434-3b0c-4730-b3e8-6e36802f5224n%40googlegroups.com.

[John H Palmieri]

Take a look at the discussion at https://github.com/sagemath/sage/pull/36337, in particular https://github.com/sagemath/sage/pull/36337#issuecomment-1741293729. I saw both "duplicate rpath" warnings and "duplicate library" warnings. Some (maybe all, at least at that point) of the duplicate rpaths were removed by https://github.com/sagemath/sage/pull/36364, but I don't know how to prevent the duplicate libary warnings.

--

John

[John H Palmieri]

I also don't know when the linker was fixed and whether we need to support systems on which it is still broken. Anyway, if you want to open up a PR to avoid the call to xcode-select, I'd be happy to take a look. We could also revive some version of the branch at #36337 to filter out the warnings when doctesting.

--

John

[Marc Culler]

Thanks.

The xcode-select command always exists.

What works is to run the command with stderr redirected to /dev/null and then check $?.

I will open a PR.

- Marc

[Marc Culler]

This looks pretty complicated.

I used XCode 15.0.1 to build the macOS SageMath-10-3 app for Arm and XCode 13.4 for Intel.  I got "duplicate rpaths" and "duplicate libraries" warnings with 15.0.1 but not with 13.4. I pasted typical gcc commands (from cvxopt) below.  You can see that:

(1) Both commands contain duplicate rpath specifications but XCode 13.4 does not warn about them

(2) duplicate libraries and invalid library search paths only occur with XCode 15.01.

(3) With 15.0.1 the gcc command does not include the linker option -ld_classic, so the code in sage-env is not actually succeeding in using the  old linker.  (But the build works anyway.)

It is not clear to me that Sage is responsible for the duplicates, since it seems to depend on the XCode version, and it is not clear to me that the code in sage-env that aims to use the old linker has any effect, on spkgs anyway.

 

- Marc

XCode 15.0.1 on Arm:

-----------------------------------

gcc -bundle -undefined dynamic_lookup -L/private/var/tmp/sage-10.3-current/local/lib -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/lib -L/private/var/tmp/sage-10.3-current/local/lib -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/lib -Wl,-platform_version,macos,11.0,11.1 -L/Library/Developer/CommandLineTools/SDKs/macOSX.sdk/usr/lib -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/var/lib/sage/venv-python3.11.1/lib -L. -Wl,-rpath,. -L/private/var/tmp/sage-10.3-current/local/lib -L/private/var/tmp/sage-10.3-current/local/lib -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/lib -L/private/var/tmp/sage-10.3-current/local/lib -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/lib -Wl,-platform_version,macos,11.0,11.1 -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib -L/private/var/tmp/sage-10.3-current/local/lib -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/lib -L/private/var/tmp/sage-10.3-current/local/lib -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/lib -Wl,-platform_version,macos,11.0,11.1 -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib -O2 -mmacosx-version-min=11.0 build/temp.macosx-11.0-arm64-cpython-311/src/C/base.o build/temp.macosx-11.0-arm64-cpython-311/src/C/dense.o build/temp.macosx-11.0-arm64-cpython-311/src/C/sparse.o -L/private/var/tmp/sage-10.3-current/local/lib -L/private/var/tmp/sage-10.3-current/local/var/lib/sage/venv-python3.11.1/lib -lm -lopenblas -lopenblas -o build/lib.macosx-11.0-arm64-cpython-311/cvxopt/base.cpython-311-darwin.so
  ld: warning: duplicate -rpath '/private/var/tmp/sage-10.3-current/local/lib' ignored
  ld: warning: duplicate -rpath '/private/var/tmp/sage-10.3-current/local/lib' ignored
  ld: warning: duplicate -rpath '/private/var/tmp/sage-10.3-current/local/lib' ignored
  ld: warning: duplicate -rpath '/private/var/tmp/sage-10.3-current/local/lib' ignored
  ld: warning: duplicate -rpath '/private/var/tmp/sage-10.3-current/local/lib' ignored
  ld: warning: ignoring duplicate libraries: '-lopenblas'
  ld: warning: search path '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib' not found
  ld: warning: search path '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib' not found
  ld: warning: search path '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib' not found
  ld: warning: duplicate LC_RPATH are deprecated ('/private/var/tmp/sage-10.3-current/local/lib')

XCode 13.4 on Intel:

--------------------------------

gcc -bundle -undefined dynamic_lookup -L/private/var/tmp/sage-10.3-current/local/lib -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/lib -L/private/var/tmp/sage-10.3-current/local/lib -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/lib -Wl,-platform_version,macos,10.9,11.3 -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/var/lib/sage/venv-python3.11.1/lib -L. -Wl,-rpath,. -L/private/var/tmp/sage-10.3-current/local/lib -L/private/var/tmp/sage-10.3-current/local/lib -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/lib -L/private/var/tmp/sage-10.3-current/local/lib -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/lib -Wl,-platform_version,macos,10.9,11.3 -L/private/var/tmp/sage-10.3-current/local/lib -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/lib -L/private/var/tmp/sage-10.3-current/local/lib -Wl,-rpath,/private/var/tmp/sage-10.3-current/local/lib -Wl,-platform_version,macos,10.9,11.3 -O2 -mmacosx-version-min=10.9 -mno-avx -mno-avx2 -mno-bmi2 build/temp.macosx-10.9-x86_64-cpython-311/src/C/base.o build/temp.macosx-10.9-x86_64-cpython-311/src/C/dense.o build/temp.macosx-10.9-x86_64-cpython-311/src/C/sparse.o -L/private/var/tmp/sage-10.3-current/local/lib -L/private/var/tmp/sage-10.3-current/local/var/lib/sage/venv-python3.11.1/lib -lm -lopenblas -lopenblas -o build/lib.macosx-10.9-x86_64-cpython-311/cvxopt/base.cpython-311-darwin.so
  ld: warning: passed two min versions (10.9, 10.9) for platform macOS. Using 10.9.
  ld: warning: passed two min versions (10.9, 10.9) for platform macOS. Using 10.9.
  ld: warning: passed two min versions (10.9, 10.9) for platform macOS. Using 10.9.

 - Marc

[Marc Culler]

Here is a blog post which appears to corroborate my hypothesis that the duplicate libraries are being generated by XCode 15 and not by Sage's build system.

If the warnings are causing test failures it would probably be best to suppress them for now and somehow remember to re-enable them once the XCode bug gets fixed.

I still do not understand why adding -Wl,-ld_classic to LDFLAGS in sage-env does not cause the old linker to be used when building some spkgs.  Apparently the LDFLAGS environment variable is not used in those cases, but I don't know why not.

- Marc