Linker failure for gap_packages on arm64 macOS 15.3

[Marc Culler]

Wish Sage 10.6beta4 On an M1 macOS 15.3 system with CommandLineTools 16.2.0.0.1.1733547573, the gap_packages-14.13.1 build is failing for me.

(The build worked on an Intel system with XCode 16.0.0.0.1.1726343358).

The linker errors are (full log attached):

[spkg-install] gcc -o bin/aarch64-apple-darwin24-default64-kv9/crypting.so /var/folders/20/gvp6qws53xg610l741x6k_480000gn/T/gacXXXXXXX.r41KPjrB8W/56064_crypting.o -bundle -flat_namespace -bundle_loader
/private/var/tmp/sage-10.6-current/local/bin/gap -L/private/var/tmp/sage-10.6-current/local/lib -L/private/var/tmp/sage-10.6-current/local/lib -Wl,platform_version,macos,11.0,11.1 -L/Library/Developer/
CommandLineTools/SDKs/MacOSX.sdk/usr/lib
[spkg-install] Undefined symbols for architecture arm64:
[spkg-install]   "_ChangedBags", referenced from:
[spkg-install]       _FuncCRYPTING_SHA256_UPDATE in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       ...
[spkg-install]   "_ErrorQuit", referenced from:
[spkg-install]       _FuncCRYPTING_SHA256_UPDATE in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_HMAC in 56064_crypting.o
[spkg-install]   "_ImportGVarFromLibrary", referenced from:
[spkg-install]       _InitKernel in 56064_crypting.o
[spkg-install]   "_InitGVarFuncsFromTable", referenced from:
[spkg-install]       _InitLibrary in 56064_crypting.o
[spkg-install]   "_InitHdlrFuncsFromTable", referenced from:
[spkg-install]       _InitKernel in 56064_crypting.o
[spkg-install]   "_IsStringFuncs", referenced from:
[spkg-install]       _FuncCRYPTING_SHA256_UPDATE in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_HMAC in 56064_crypting

[spkg-install]   "_NewBag", referenced from:
[spkg-install]       _FuncCRYPTING_SHA256_INIT in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_HMAC in 56064_crypting.o
[spkg-install]   "_ObjInt_UInt", referenced from:
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       ...
[spkg-install]   "_SET_TYPE_OBJ", referenced from:
[spkg-install]       _FuncCRYPTING_SHA256_INIT in 56064_crypting.o
[spkg-install]   "_YoungBags", referenced from:
[spkg-install]       _FuncCRYPTING_SHA256_UPDATE in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_FINAL in 56064_crypting.o
[spkg-install]       _FuncCRYPTING_SHA256_HMAC in 56064_crypting.o
[spkg-install] ld: symbol(s) not found for architecture arm64
[spkg-install] clang: error: linker command failed with exit code 1 (use -v to see invocation)
[spkg-install] make[5]: *** [bin/aarch64-apple-darwin24-default64-kv9/crypting.so] Error 1
[spkg-install] ********************************************************************************
[spkg-install] Error building gap_packages-4.13.1
[spkg-install] ********************************************************************************

[Marc Culler]

The missing symbols are defined in libgap and I was able to build gap_packages after adding -lgap to LDFLAGS in spkg-install.in.

- Marc

[Marc Culler]

I don't have time for that right now.  But I will provide a patch for the 1-line fix that I used:

diff --git a/build/pkgs/gap_packages/spkg-install.in b/build/pkgs/gap_packages/spkg-install.in
index 7005cc3d322..157e093d86b 100644
--- a/build/pkgs/gap_packages/spkg-install.in
+++ b/build/pkgs/gap_packages/spkg-install.in
@@ -96,6 +96,7 @@ do
     echo "Building GAP package $pkg"
     CFLAGS="$CFLAGS -Wno-implicit-function-declaration"
     export CFLAGS
+    export LDFLAGS="$LDFLAGS -lgap"
     cd "$PKG_SRC_DIR/$pkg"
     ./configure "$GAP_ROOT"
     sdh_make -j1

- Marc

On Wed, Jan 29, 2025 at 11:42 AM Dima Pasechnik <dim...@gmail.com> wrote:

On Wed, Jan 29, 2025 at 11:39 AM Marc Culler <marc....@gmail.com> wrote:
>
> The missing symbols are defined in libgap and I was able to build gap_packages after adding -lgap to LDFLAGS in spkg-install.in.

yes, I can reproduce this with the latest beta5, on Apple clang
version 16.0.0 (clang-1600.0.26.4), on an arm64 (M1) box.

Will you do a PR to fix this?

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/sage-devel/b3249c92-08bf-4485-8eda-e32a5f42e459n%40googlegroups.com.

[Dima Pasechnik]

On Wed, Jan 29, 2025 at 11:39 AM Marc Culler <marc....@gmail.com> wrote:
>

> The missing symbols are defined in libgap and I was able to build gap_packages after adding -lgap to LDFLAGS in spkg-install.in.

yes, I can reproduce this with the latest beta5, on Apple clang
version 16.0.0 (clang-1600.0.26.4), on an arm64 (M1) box.

Will you do a PR to fix this?


>

[Dima Pasechnik]

On Wed, Jan 29, 2025 at 11:52 AM Marc Culler <marc....@gmail.com> wrote:
>
> I don't have time for that right now. But I will provide a patch for the 1-line fix that I used:
>
> diff --git a/build/pkgs/gap_packages/spkg-install.in b/build/pkgs/gap_packages/spkg-install.in
> index 7005cc3d322..157e093d86b 100644
> --- a/build/pkgs/gap_packages/spkg-install.in
> +++ b/build/pkgs/gap_packages/spkg-install.in
> @@ -96,6 +96,7 @@ do
> echo "Building GAP package $pkg"
> CFLAGS="$CFLAGS -Wno-implicit-function-declaration"
> export CFLAGS
> + export LDFLAGS="$LDFLAGS -lgap"
> cd "$PKG_SRC_DIR/$pkg"
> ./configure "$GAP_ROOT"
> sdh_make -j1

This is not a correct fix. libgap is not really involved here;
cypring's GAP kernel module
(that's what's compiled here) can be loaded either in libgap, or in
gap executable - and the latter
isn't linked to libgap.

The correct fix should be to clean up various CFLAGS and LDFLAGS, it seems.
Indeed, I cannot reproduce this on a stand-alone GAP 4.13.1 build
(with crypting package).

Here what the flags are with the successful stand-alone build:

dima@studio crypting % ./configure
Using settings from ../../sysinfo.gap
dima@studio crypting % make V=1
"/tmp/gap-4.13.1/gac" -d -p " " -P " " src/crypting.c -o
bin/aarch64-apple-darwin24-default64-kv9/crypting.so
gcc -pthread -g -O2 -fno-common -o
/var/folders/81/vt06lnj17rj0cpc9gdshy4q00000gr/T/gacXXXXXXX.4ENWuVAZjc/79411_crypting.o
-I/tmp/gap-4.13.1/build -I/tmp/gap-4.13.1/src -I/tmp/gap-4.13.1
-DUSE_GASMAN=1 -c src/crypting.c
gcc -o bin/aarch64-apple-darwin24-default64-kv9/crypting.so
/var/folders/81/vt06lnj17rj0cpc9gdshy4q00000gr/T/gacXXXXXXX.4ENWuVAZjc/79411_crypting.o
-bundle -flat_namespace -bundle_loader /tmp/gap-4.13.1/gap
rm -f /var/folders/81/vt06lnj17rj0cpc9gdshy4q00000gr/T/gacXXXXXXX.4ENWuVAZjc/79411_crypting.o
dima@studio crypting %

Dima

[Marc Culler]

On Wed, Jan 29, 2025 at 1:33 PM Dima Pasechnik <dim...@gmail.com> wrote:

 libgap is not really involved here;

cypring's GAP kernel module
(that's what's compiled here) can be loaded either in libgap, or in
gap executable - and the latter
isn't linked to libgap.

 

I am no expert in GAP.  But the code in crypting.c references symbols which are defined in libgap.  So to build crypting.so from crypting.o you have to link against libgap or something else which defines those same symbols.  After building with my patch, if I run the gap executable $SAGE_ROOT/local/bin/gap then I am able to load the crypting module:

% /Applications/SageMath-10-6.app/Contents/Frameworks/Sage.framework/Versions/Current/local/bin/gap
 ┌───────┐   GAP 4.13.1 of 2024-06-11
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: aarch64-apple-darwin24-default64-kv9
 Configuration:  gmp 6.3.0, GASMAN, readline
 Loading the library and packages ...
 Packages:   AClib 1.3.2, Alnuth 3.2.1, AtlasRep 2.1.8, AutPGrp 1.11,
             CRISP 1.4.6, Cryst 4.1.27, CrystCat 1.1.10, CTblLib 1.3.9,
             FactInt 1.6.3, FGA 1.5.0, GAPDoc 1.6.7, IO 4.8.2,
             IRREDSOL 1.4.4, LAGUNA 3.9.6, Polenta 1.3.10, Polycyclic 2.16,
             PrimGrp 3.4.4, RadiRoot 2.9, ResClasses 4.7.3, SmallGrp 1.5.3,
             Sophus 1.27, TomLib 1.2.11, TransGrp 3.6.5, utils 0.85
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap> LoadPackage("crypting");
────────────────────────────────────────────────────────────────────────────
Loading crypting 0.10.4 (Hashes and Crypto in GAP)
by Markus Pfeiffer (http://www.morphism.de/~markusp/).
maintained by:
   Markus Pfeiffer (http://www.morphism.de/~markusp/) and
   The GAP Team (sup...@gap-system.org).
Homepage: https://gap-packages.github.io/crypting/
Report issues at https://github.com/gap-packages/crypting/issues
────────────────────────────────────────────────────────────────────────────
true

That is all that I know about this.

- Marc

[Antonio Rojas]

See https://web.archive.org/web/20201218114127/https://trac.sagemath.org/ticket/27372 for some previous discussion about this

[Dima Pasechnik]

On Wed, Jan 29, 2025 at 3:16 PM Antonio Rojas <nqn...@gmail.com> wrote:
>
> See https://web.archive.org/web/20201218114127/https://trac.sagemath.org/ticket/27372 for some previous discussion about this

This discussion in #27372 (more up to date copy:
https://github.com/sagemath/sage/issues/27372) is misleading.
It's incorrect to link GAP kernel modules to libgap. E.g. my Gentoo
Linux system-wide GAP has

$ ldd /usr/lib64/gap/pkg/crypting/bin/amd64/crypting.so
linux-vdso.so.1 (0x00007fefecc2f000)
libc.so.6 => /lib64/libc.so.6 (0x00007fefeca0c000)
/lib64/ld-linux-x86-64.so.2 (0x00007fefecc31000)

The system also has libgap.so, which can perfectly fine load GAP
kernel modules.

Dima

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion visit https://groups.google.com/d/msgid/sage-devel/45bb25f5-a9f6-48d0-b0aa-9104662c6933n%40googlegroups.com.

[Dima Pasechnik]

On Wed, Jan 29, 2025 at 4:35 PM Marc Culler <marc....@gmail.com> wrote:
>
> Looking back at the failed command I actually think this is a clang bug which has nothing to do with GAP. The relevant option is:
> -bundle_loader /private/var/tmp/sage-10.6-current/local/bin/gap
> I think that option is broken in the latest clang.
> I think that option means that the linker should not complain about a missing symbol if that symbol is defined in the executable which loads the library, or in any library loaded by that executable. In the case of local/bin/gap I see that (of course) it loads libgap:

Why does it load libgap? That's a bug in the gap's building by Sage I
think (no idea how it happens though).
Out of the Sage tree build has the following linkage; see - no libgap linked.

% otool -L gap
gap:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
current version 1351.0.0)
/opt/homebrew/opt/gmp/lib/libgmp.10.dylib (compatibility
version 16.0.0, current version 16.0.0)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current
version 1.2.12)
/opt/homebrew/opt/readline/lib/libreadline.8.dylib
(compatibility version 8.2.0, current version 8.2.0)

as opposed to Sage's build:
% otool -L sage/local/bin/gap
sage/local/bin/gap:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
current version 1351.0.0)
/opt/homebrew/opt/gmp/lib/libgmp.10.dylib (compatibility
version 16.0.0, current version 16.0.0)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current
version 1.2.12)
/opt/homebrew/opt/readline/lib/libreadline.8.dylib
(compatibility version 8.2.0, current version 8.2.0)
/Volumes/dima/sage/local/lib/libgap.9.dylib (compatibility
version 10.0.0, current version 10.0.0)

Dima

> LC_LOAD_DYLIB: /usr/lib/libSystem.B.dylib
> LC_LOAD_DYLIB: @rpath/libgmp.10.dylib
> LC_LOAD_DYLIB: @rpath/libz.1.dylib
> LC_LOAD_DYLIB: @rpath/libreadline.8.dylib
> LC_LOAD_DYLIB: @rpath/libgap.9.dylib
> So the linker should not have complained about those missing symbols, much less have raised an error. And, in fact, there was no error on my Intel system with an older clang. However, I think that linking against libgap accomplishes the same thing, It lets the linker know that the symbol will exist when the crypting library is loaded. So I don't think that is a bad workaround, and I think it will not be necessary after the next release of XCode. GAP is doing nothing wrong, except to assume that the -bundle-loader option will work as advertised.
>
> - Marc.
>
> On Wed, Jan 29, 2025 at 3:21 PM Dima Pasechnik <dim...@gmail.com> wrote:

>>
>>
>>
>> On 29 January 2025 14:43:17 GMT-06:00, Marc Culler <marc....@gmail.com> wrote:
>> >On Wed, Jan 29, 2025 at 1:33 PM Dima Pasechnik <dim...@gmail.com> wrote:
>> >
>> >>
>> >> libgap is not really involved here;
>> >>
>> >cypring's GAP kernel module
>> >> (that's what's compiled here) can be loaded either in libgap, or in
>> >> gap executable - and the latter
>> >> isn't linked to libgap.
>> >
>> >
>> >I am no expert in GAP. But the code in crypting.c references symbols which
>> >are defined in libgap. So to build crypting.so from crypting.o you have to
>> >link against libgap or something else which defines those same symbols.
>>

>> GAP does dynamic symbol resolution when it loads an extension module.
>> On macOS, GAP does not link libgap; one can check with "otool -L".
>>
>> libgap does contain the internal symbols needed for kernel extension modules available in GAP the executable, and is equally able to load a GAP kernel extension module.
>>
>> If you link a GAP kernel module to libgap, and load it into GAP, the latter will also load libgap, and the hell can potentially break loose.
>> Even if it doesn't, an extra unneeded library would be loaded.
>>
>> crypting.so, the GAP kernel extension module, normally only links to a macOS system library, and nothing else.
>>
>> Dima

[Marc Culler]

On macOS, with both arm64: and x86_64

*  the gap executable loads libgap in SageMath 10.4, 10.5 and 10.6.

* crypting.so loads libgap in SageMath 10.4 and 10.6 but not 10.5.

On macOS x86_64

:

*  the gap executable loads libgap in SageMath 10.4, 10.5 and 10.6.

* crypting.so loads libgap in SageMath 10.4 and 10.5, but not 10.6.

- Marc

[Marc Culler]

Correction:  The first two bullets were for arm, not for intel.  I.e.

On macOS, arm64:

[Dima Pasechnik]

The most meaningful fix is to upgrade GAP to 4.14.1, where this should not happen.

See https://github.com/sagemath/sage/issues/39421