[llvm-dev] Building an LLVM cross-compiler

[Cág via llvm-dev]

Hi everyone,

I recently sent a message that may or may not have made it to the list,
on problems building an LLVM cross-compiler. Now the issue seems clear
to me and is irrelevant at this point, I would like to discuss the general
issue.

Suppose you have a host system, a Linux distribution on an x86_64 machine,
and want to build a cross-compiler for aarch64 (or any platform supported
by LLVM, other than an AMD-/Intel-based machine) to cross compile a Linux
distribution.

The process, in my opinion, should go like this:
1. Get the sources (llvm, lld, compiler-rt, libunwind, libcxx...).
2. Build an LLVM cross-compiler toolchain using native distribution's
compiler (i.e. build an x86_64 clang executable that targets aarch64).
3. Cross-compile libc and other libraries/dependencies to run the
userland.
4. Cross-compile the userland.

With LLVM it doesn't work:
1. You got the sources.
2. Built clang targetting, among others or only, aarch64/mips/etc.
3. Clang requires compiler-rt but you need to cross-compile compiler-rt
for the target platform. *You don't have a cross-compiler*.

It stops after that. To build compiler-rt you need C headers, libc
runtimes, *and libclang_rt.a*. You can't cross-compile libc because you
don't have compiler-rt because you don't have libc. Chicken or egg.

I.e. you need a cross-compiler to build a cross-compiler.

So, how do you build an LLVM-based cross-compiler?

I hope I am wrong in my assumptions and if it's so, please forgive me.
I also hope I made it clear.

Thanks for taking time and have a nice weekend!

--
caóc
_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

[Chris Bieneman via llvm-dev]

You’ve hit on one of the odd historical messes in LLVM. Compiler-RT can’t really be treated as a monolithic set of libraries when you are bootstrapping. In order to even configure the LLVM sanitizer runtimes you need libc and many other OS interfaces, so when bootstrapping you need to build the compiler-rt builtins library separately from the sanitizers.

This can be done either by using the `LLVM_ENABLE_RUNTIMES` CMake option (see DistributionExample-stage2.cmake in clang/cmake/caches), or by treating compiler-rt/lib/builtins as the top-level CMake file.

I would strongly suggest that you start with one of the CMake cache files under clang/cmake/caches as an example because setting up the CMake options to build a toolchain that can cross-compile is sadly quite complicated.

-Chris

> On Nov 6, 2020, at 4:13 PM, Cág via llvm-dev <llvm...@lists.llvm.org> wrote:
>
> Hi everyone,

[Martin Storsjö via llvm-dev]

On Fri, 6 Nov 2020, Cág via llvm-dev wrote:

> The process, in my opinion, should go like this:
> 1. Get the sources (llvm, lld, compiler-rt, libunwind, libcxx...).
> 2. Build an LLVM cross-compiler toolchain using native distribution's
> compiler (i.e. build an x86_64 clang executable that targets aarch64).
> 3. Cross-compile libc and other libraries/dependencies to run the
> userland.
> 4. Cross-compile the userland.
>

> It stops after that. To build compiler-rt you need C headers, libc
> runtimes, *and libclang_rt.a*. You can't cross-compile libc because you
> don't have compiler-rt because you don't have libc. Chicken or egg.

Yes, for such a full from-scratch setup, the procedure is a bit fiddly to
get right, and exactly how to do it is very dependent on how configurable
the lower level components of your target platform are for such bootstrap
setups.

I regularly build my toolchain that targets mingw, in exactly this way.
First I build the compiler, then I install the target "libc"'s headers,
then build the mingw equivalent of the libc.

Here, the "libc" consists of import libraries (that are generated from
symbol listings using llvm-dlltool) and static libraries, so libclang_rt
isn't needed yet. (The autoconf scripts for these bits define
AC_NO_EXECUTABLES, which avoids bailing out due to the compiler not being
able to link an executable.)

After this step, I build the compiler-rt builtins, and this is configured
with -DCMAKE_C_COMPILER_WORKS=1 to (iirc) make it not bail out even though
the toolchain still can't link anything. After building the compiler-rt
builtins, and installing them, the toolchain finally is complete enough to
link a working C program.

After this stage, higher level runtimes like libc++ can be built on top,
followed by other compiler-rt libraries.

See https://github.com/mstorsjo/llvm-mingw for the full set of scripts I
use to build my setup; built-all.sh is the toplevel script you can start
following if you want to dig in.

// Martin

[Cág via llvm-dev]

Hello Chris,

Chris Bieneman wrote:
> You’ve hit on one of the odd historical messes in LLVM. Compiler-RT
> can’t really be treated as a monolithic set of libraries when you are
> bootstrapping. In order to even configure the LLVM sanitizer runtimes
> you need libc and many other OS interfaces, so when bootstrapping you
> need to build the compiler-rt builtins library separately from the
> sanitizers.
> This can be done either by using the `LLVM_ENABLE_RUNTIMES` CMake option
> (see DistributionExample-stage2.cmake in clang/cmake/caches), or by
> treating compiler-rt/lib/builtins as the top-level CMake file.
>
> I would strongly suggest that you start with one of the CMake cache
> files under clang/cmake/caches as an example because setting up the
> CMake options to build a toolchain that can cross-compile is sadly
> quite complicated.

Thanks for the input. I will now be trying this cache file to see. At
the moment I am simply experimenting with different CMake options to see
if I can get it right. LLVM_ENABLE_RUNTIMES causes the freshly-built
Clang to be used as the compiler for the runtimes set, here it runs into
including host system headers (no libc exists yet).
Only BUILTINS are set to ON, no sanitizers or xrays.

It looks like a quite common request/question. Has a fix been proposed or
a reviews.llvm.org page exists? I'd follow it. Hacking CMake is fun
until you want to get the job done.

Cheers

[Cág via llvm-dev]

Hello Martin,

Martin Storsjö wrote:
>> 1. Get the sources (llvm, lld, compiler-rt, libunwind, libcxx...).
>> 2. Build an LLVM cross-compiler toolchain using native distribution's
>> compiler (i.e. build an x86_64 clang executable that targets aarch64).
>> 3. Cross-compile libc and other libraries/dependencies to run the
>> userland.
>> 4. Cross-compile the userland.

> Yes, for such a full from-scratch setup, the procedure is a bit fiddly
> to get right, and exactly how to do it is very dependent on how
> configurable the lower level components of your target platform are
> for such bootstrap setups.
>
> I regularly build my toolchain that targets mingw, in exactly this way.
> First I build the compiler, then I install the target "libc"'s headers,
> then build the mingw equivalent of the libc.

> After this stage, higher level runtimes like libc++ can be built on top,
> followed by other compiler-rt libraries.
> See https://github.com/mstorsjo/llvm-mingw for the full set of scripts I
> use to build my setup; built-all.sh is the toplevel script you can star

> following if you want to dig in.

I like this. I hope you don't mind if I borrow some of these ideas to
play with. C_COMPILER_WORKS is something I've never heard of.

Thank you!

[Cág via llvm-dev]

Hi everyone,

Just a quick update. Here's what worked for me here*:
1. Get the sources.
2. Build clang, llvm, lld.
3. Install libc headers to a sysroot.
4. Build compiler-rt builtins and crt with the freshly-built clang.
One need to set C_COMPILER_WORKS to skip the checks.
5. Build libc.a/libc.so

Now the freshly-built clang can compile a "Hello, World" program.

@Martin Storsjö
@Chris Bieneman
Thanks for the hints!

* - instead of previously supposed

1. Get the sources (llvm, lld, compiler-rt, libunwind, libcxx...).
2. Build an LLVM cross-compiler toolchain using native distribution's
compiler (i.e. build an x86_64 clang executable that targets aarch64).
3. Cross-compile libc and other libraries/dependencies to run the
userland.
4. Cross-compile the userland.

--

[Vlad Ivanov via llvm-dev]

Just a note about this:

>I like this. I hope you don't mind if I borrow some of these ideas to
>play with. C_COMPILER_WORKS is something I've never heard of.

I'm also working on building a clang-based toolchain from scratch.
To avoid CMake bailing out when a binary can't be linked, I use:

-DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY

Along with:

-DCMAKE_SYSTEM_NAME="Generic"
-DCMAKE_SYSTEM_PROCESSOR="powerpc"

Best Regards,

Vlad

[John Paul Adrian Glaubitz via llvm-dev]

Hello!

On 11/10/20 2:49 PM, Cág via llvm-dev wrote:
> Just a quick update. Here's what worked for me here*:
> 1. Get the sources.
> 2. Build clang, llvm, lld.
> 3. Install libc headers to a sysroot.

Alternatively, use a Debian-based system which allows co-installation of system
libraries for multiple architectures (Multi-Arch).

Never understood why other distributions aren't picking this mechanism up :-(.

Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - glau...@debian.org
`. `' Freie Universitaet Berlin - glau...@physik.fu-berlin.de
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

[Robinson, Paul via llvm-dev]

> -----Original Message-----
> From: llvm-dev <llvm-dev...@lists.llvm.org> On Behalf Of Cág via
> llvm-dev
> Sent: Tuesday, November 10, 2020 8:49 AM
> To: llvm...@lists.llvm.org
> Subject: Re: [llvm-dev] Building an LLVM cross-compiler
>
> Hi everyone,
>
> Just a quick update. Here's what worked for me here*:
> 1. Get the sources.
> 2. Build clang, llvm, lld.
> 3. Install libc headers to a sysroot.
> 4. Build compiler-rt builtins and crt with the freshly-built clang.
> One need to set C_COMPILER_WORKS to skip the checks.
> 5. Build libc.a/libc.so
>
> Now the freshly-built clang can compile a "Hello, World" program.

Can this recipe be documented somewhere in the building-llvm pages?
It's clearly not an obvious procedure.
Thanks,
--paulr

>
> @Martin Storsjö
> @Chris Bieneman
> Thanks for the hints!
>
>
> * - instead of previously supposed
> 1. Get the sources (llvm, lld, compiler-rt, libunwind, libcxx...).
> 2. Build an LLVM cross-compiler toolchain using native distribution's
> compiler (i.e. build an x86_64 clang executable that targets aarch64).
> 3. Cross-compile libc and other libraries/dependencies to run the
> userland.
> 4. Cross-compile the userland.
>
> --
> caóc
> _______________________________________________
> LLVM Developers mailing list
> llvm...@lists.llvm.org

> https://urldefense.com/v3/__https://lists.llvm.org/cgi-
> bin/mailman/listinfo/llvm-dev__;!!JmoZiZGBv3RvKRSx!tLCWq3aafmEQwZlJo-
> mAShbSmokQRv-2IgORiPv3w5i7k1kjD_E0jsNJEWO3Ug-xDw$

[Martin Storsjö via llvm-dev]

On Tue, 10 Nov 2020, Vlad Ivanov via llvm-dev wrote:

> Just a note about this:
>
>> I like this. I hope you don't mind if I borrow some of these ideas to
>> play with. C_COMPILER_WORKS is something I've never heard of.
>
> I'm also working on building a clang-based toolchain from scratch.
> To avoid CMake bailing out when a binary can't be linked, I use:
>
> -DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY

Oh, right, that's the one - I remembered that there was a more proper
cmake option to use, but I didn't find it when I sent the previous mail.

Yeah, that option probably is more correct than CMAKE_C_COMPILER_WORKS. We
set it already in libunwind/CMakeLists.txt, I guess we should set it in a
few more places, at least compiler-rt/lib/builtins/CMakeLists.txt.

// Martin

[Andrew Kelley via llvm-dev]

You could try using zig for this. Here's an example build script of zig
cross compiling llvm, lld, clang, and finally itself:

https://github.com/ziglang/zig-bootstrap

You can see the build script is not too complicated. The main trick is
using CC and CXX to use `zig cc` and `zig c++` as drop-in replacements
for a C and C++ compiler, respectively.

I wrote some more details about this here:
https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html

On 11/6/20 3:12 PM, Cág via llvm-dev wrote:
> Hi everyone,
>
> I recently sent a message that may or may not have made it to the list,
> on problems building an LLVM cross-compiler. Now the issue seems clear
> to me and is irrelevant at this point, I would like to discuss the general
> issue.
>
> Suppose you have a host system, a Linux distribution on an x86_64 machine,
> and want to build a cross-compiler for aarch64 (or any platform supported
> by LLVM, other than an AMD-/Intel-based machine) to cross compile a Linux
> distribution.
>

> The process, in my opinion, should go like this:

> 1. Get the sources (llvm, lld, compiler-rt, libunwind, libcxx...).
> 2. Build an LLVM cross-compiler toolchain using native distribution's
> compiler (i.e. build an x86_64 clang executable that targets aarch64).
> 3. Cross-compile libc and other libraries/dependencies to run the
> userland.
> 4. Cross-compile the userland.
>

> With LLVM it doesn't work:
> 1. You got the sources.
> 2. Built clang targetting, among others or only, aarch64/mips/etc.
> 3. Clang requires compiler-rt but you need to cross-compile compiler-rt
> for the target platform. *You don't have a cross-compiler*.
>

> It stops after that. To build compiler-rt you need C headers, libc
> runtimes, *and libclang_rt.a*. You can't cross-compile libc because you
> don't have compiler-rt because you don't have libc. Chicken or egg.
>