Darwin BuildSystem Bug

73 views
Skip to first unread message

Naville Zhang

unread,
Apr 18, 2024, 4:35:55 AMApr 18
to android-llvm
Hi:
We're compiling NDK downstream r26 on Darwin @ arm64.
In base_builders.LLVMBuilder.cmake_defines:
``LLVM_CODESIGNING_IDENTITY`` should not be set. I'll explain below:


On modern Darwin, the whole traditional linking / signing process goes:
  1. Linker generates an linker signature for the binary automatically
  2. Optionally, codesign the signature with a non-local(a.k.a. real) identity
  3. Other MachO modifications like install_name_tool, etc


When Step 2 is not executed:

  The binary is still signed with a local identity, which gets picked up by upcoming tools like install_name_tool that automatically re-applies the ad-hoc signature, therefore the binary is still able to function

However, if Step 2 is involved (Which is the case when the original buildsystem explicitly passed ``-`` as the signing identity) :
 The linker signature is lost, thus upcoming tools like install_name_tool won't be able to resign the binary. In fact, install_name_tool explicitly warns about this case. This results in a broken toolchain and especially breaks multi-stage building.

This works on X64_macOS host because X64 doesn't enforce codesigning, but breaks on arm64_macOS due to a valid codesign is required.

Naville Zhang

unread,
Apr 18, 2024, 4:43:19 AMApr 18
to android-llvm
Full experience:
Issue 1: 
the static version of libc++.a seems to be malformed and can't be linked, and the dynamic version misses @rpath and stuff, as such sequential building stage 1 bootstrapping compiler's llvm-tblgen will fail to execute due to missing @rpath.

And yes, ninja is invoked with LD_LIBRARY_PATH that (in theory) should fix the problem, unfortunately Ninja doesn't seem to inherit such env-var and this doesn't work (Ref: https://github.com/ninja-build/ninja/issues/1002 )

On some systems we own, dyld silently resolves back to the system shipped libc++.dylib and the execution continues, other systems throw an explicit dylib not found error and crashes. Currently unsure the cause of the differences

Bypass 1:
I patched stage1's cmake options to pass ``CMAKE_BUILD_RPATH`` to add extra @rpath to the built executables, which uses absolute path to reference libc++.dylib

Issue 2:
The problem in the OP
Bypass 2:
I removed ``LLVM_CODESIGNING_IDENTITY`` from the cmake options

Naville Zhang

unread,
Apr 18, 2024, 4:46:47 AMApr 18
to android-llvm
To be clear, just like issue 1, issue 2 also doesn't reproduce on my daily drive machine, which is developer configured. (I suspect ``sudo spctl developer-mode enable-terminal``  is the main cause).
But reproduces on our M1Pro MacStudio CI/CD machine which is pretty much stock-configured

Naville Zhang

unread,
Apr 18, 2024, 4:57:15 AMApr 18
to android-llvm
To verify OP:

hello.c
```c
int main(){
    return 0;
}
```


# Without explicit codesign:


➜  clang hello.c -o hello
➜  codesign -dvv hello    
Executable=hello
Identifier=hello
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=254 flags=0x20002(adhoc,linker-signed) hashes=5+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none
➜  install_name_tool -add_rpath / hello
➜  codesign -dvv hello                
Executable=/hello
Identifier=hello
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=254 flags=0x20002(adhoc,linker-signed) hashes=5+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none

Note how linker-signed flag is preserved




# Explicitly local codesign:

➜ clang hello.c -o hello
➜  codesign -dvv hello
Executable=/hello
Identifier=hello
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=254 flags=0x20002(adhoc,linker-signed) hashes=5+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none
➜  codesign -s - hello -f
hello: replacing existing signature
➜  codesign -dvv hello  
Executable=hello
Identifier=hello-555549448fe0cb26d0593be785a617e0b2982836
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=359 flags=0x2(adhoc) hashes=5+2 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12
➜  install_name_tool -add_rpath / hello
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: hello
➜  codesign -dvv hello                
Executable=/Users/naville/Downloads/hello
Identifier=hello-555549448fe0cb26d0593be785a617e0b2982836
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=359 flags=0x2(adhoc) hashes=5+2 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12


Note how linker-signed flag went away after the explicit ``codesign`` -s command  as well as the new "will invalidate the code signature" warning

Naville Zhang

unread,
Apr 19, 2024, 4:44:48 AMApr 19
to android-llvm
While we are at it, incremental build on Darwin results in:

``
linking ../lib/libncurses.6.dylib
ar -curv ../lib/libncurses.a XXXXXXXX
ar: ../lib/libncurses.a is a fat file (use libtool(1) or lipo(1) and ar(1) on it)
ar: ../lib/libncurses.a: Inappropriate file type or format
``
The best solution would be somehow detect ncurses has already been compiled and skip

Naville Zhang

unread,
Jul 25, 2024, 2:44:41 AM (2 days ago) Jul 25
to android-llvm
To follow-up, the static library malform issue is actually due to LLVM's llvm-ar doesn't support fat archives, only Apple's ar do.
For some reason LLVM's llvm/cmake/modules/UseLibtool.cmake didn't seem to work for libcxx and libcxxabi, so we ended-up rolling our own version that enforces libtool

Reply all
Reply to author
Forward
0 new messages