Cross Compiling

[Colin Horne]

Hello all,

I'm trying to cross compile a C++ codebase using either clang or GCC to produce binaries for aarch64/Linux, building on an x86_64/Linux platform.

The spack docs have many mentions of including arch=... and target=... in specs, but I haven't been able to find a definite reference that gives instructions on building a system root for the target, etc. Indeed, looking at the package.py file for linux-headers, it doesn't look like there's currently logic for installing linux-headers with non-native targets.

My current intention is to follow Preshing's excellent guide for building a GCC cross-compiler toolchain, tailoring the package.py files in spack's repository as needed.

https://preshing.com/20141119/how-to-build-a-gcc-cross-compiler/

I'm very keen to know whether anyone has previous experience in doing this with spack, and/or if I'm missing any existing functionality. Any general guidance/suggestions for my travels will be greatly appreciated too!

Thanks,

  Colin

[Colin Horne]

Hi all,

Continuing from my previous message, my approach has been to build and configure the cross compiler toolchain entirely outside spack and exposing it by adding a new compiler in compilers.yaml that appropriately sets CFLAGS, etc.

However, I'm struggling to understand how spack incorporates cross-compilation into the build process. 

In particular, taking GNU Autotools packages as an example, I need to pass additional parameters (--host) to ./configure. However, logic for achieving this doesn't seem to be encoded in the AutotoolsPackage base class and I'm unsure how I can otherwise communicate these parameters from compilers.yaml.

I was expecting that AutotoolsPackage::flags_to_build_system_args() might add "--host=tuple", where the tuple is appropriately set from spec.architecture when spec.architecture doesn't correspond to the system's architecture.

Any suggestions would be gratefully received! 

Many thanks!

  Colin

[Colin Horne]

Hi all,

In a rather desperate attempt to get cross compiling working, I tried directly modifying AutotoolsPackage::flags_to_build_system_args() to modify the arguments passed to "./configure" for autotools packages being built with an architecture different to the build system's architecture.

This worked beautifully for some packages, e.g., tar, but was immediately stumped by ncurses. Upon closer inspection, ncurses's package.py overrides the AutotoolsPackage::configure() method instead of overriding configure_args(). As such, it doesn't call flags_to_build_system_args(). The motivation behind doing this is presumably that ncurses executes "./configure" in two different directories ("build_ncurses" and "build_ncursesw") to simultaneously produce two builds (one with "--enable-widec" and one with "--disable-widec"), functionality that I don't believe is possible without overriding configure().

It feels like this functionality is unusual and possibly something that should be handled through variants instead of assuming the user wants both options available. However, the fact that the popular ncurses package.py absolutely precludes any means of passing additional parameters to "./configure" makes me wonder if I'm approaching this from the wrong angle.

If I try to cross-compile without telling "./configure" via "--host", then "./configure" outright fails with:

  >> 15    checking whether the C compiler works... configure: error: cannot run C compiled programs.

     16    If you meant to cross compile, use `--host'.

But yet, spack offers the "arch" and "target" flags, surely with the goal of enabling cross-compilation, and so I feel I'm missing something.

Is there a way to cross-compile autotools packages without passing additional arguments to "./configure" (e.g., by cunning use of CFLAGS)?

As always, any pointers would be very gratefully received!

Many thanks,

  Colin

[Elizabeth A. Fischer]

I think you are going down a road full of pain.

A long time ago, I was involved with programming an early ARM system (HP iPaq).  Everybody told me they had a "compile farm" of iPaqs ready to build stuff.  I thought that was inefficient / inelegant, and decided to build a cross compiler toolchain on my desktop instead.  Biggest headache ever.  Everything works in theory; but there are "gotchas" every step of the way.  If I were doing it again, I would just used the provided compiler farm.

ARM has come a long way since then, and 64-bit ARM systems are now available on the desktop.  If possible, it will almost certainly be cheaper to buy the required machines to build your stuff natively, than to pay for you to build / set up a cross compiler toolchain.

AFAIK, Spack has no support for cross compiling.  You would have to add the appropriate flags to `./configure`.  Probably the best way would be to modify AutotoolsPackage.  This would require a lot of thought.  And similar changes would have to be made to CMakePackage, etc. and everything that uses a hand-built makefile.  I'm not even sure if Python packages are POSSIBLE to cross-compile.  But if you do go down that direction, please submit a PR!

-- Elizabeth

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spack/d1f02a71-7b9e-4a17-8c80-defbb5c8e497%40googlegroups.com.

[Gamblin, Todd]

Colin, Elizabeth:

So, it’s not true that Spack has no support for cross-compiling. People do it frequently on systems like Trinity at LANL and Theta at ANL.  Also we’ve been cross compiling for a while on BG/Q.  Those are haswell/KNL and Power7/PowerPC A2 systems, though, so the match between the build environment and the run environment is much closer.  The way you compile things on those machines is:

spack install <package> target=mic_knl

And spack will inject the arch flags for the compiler onto the compile line.  This isn’t without issues — some packages that build executables to be run at build time need help (they typically need to be specially built with the front end target — we are working on a context manager for that). 

We do actually care about the x86_64/aarch64 case, as that is what a large upcoming ARM system will (unfortunately) look like.  Which system are you trying to build for?

I don’t think it would take much to get ncurses doing the right thing in configure(), but it would be good to look at your use case in more detail and see if there are options we can add to make it easier.  In particular, we can inject arguments directly on the command line, which circumvents the particulars of most build systems.  See here:

https://spack.readthedocs.io/en/latest/packaging_guide.html#compiler-flags

There is likely more we need to do though so your feedback is much appreciated.  Also note that there are people on Slack who care about this use case; i can put you in touch with them if you ping me there.

Todd

---
Sent from Workspace ONE Boxer

To view this discussion on the web visit https://groups.google.com/d/msgid/spack/CAC_jL3x7ZGOfV8OiptJTkrONs8Lrbqdt48DkhgG6Jcr9BGtT5A%40mail.gmail.com.

[Colin Horne]

Elizabeth, Todd--many thanks for your replies!

It's very good to receive your recommendations.

I'm going to continue in my quest to get my particular use case working and will report back if I succeed.

One interesting point to mention:

When cross-compiling cmake packages by setting "target=<target>", I note that the target gets propagated to the cmake build dependency itself, despite cmake only being needed by the build architecture. I override this (untested) by explicitly setting the target for cmake in the spec. However, at a more general level I would ask: is there ever a time that we want the target to propagate to build dependencies? As I understand it, build dependencies are by definition dependencies that are run/used at build time (i.e., with the build architecture), but which aren't linked to in the installed package and so will never be run on the target system.

From that point of view, I would suggest that the logic for consolidating specs might want to avoid propagating the target to build dependencies, but that's outwith my means to change: for my purposes, I will continue to directly override, hopefully if I can by directly setting the target architecture of known build dependencies in my package's package.py.

Thanks again for your inputs.

Cheers,

  Colin

[Rodrigo Amestica]

I wonder how far did Colin managed to get along his quest. I'm also trying to compile on x86_64 for aarch64 and this is not working. 

I want to compile libzmq for aarch64 with the gcc 9.2.0 compiler provided by a local petalinux installation; which I added manually to the compilers.yaml file. But trying to install the libzmq package with that compiler ends with the following error:

spack install lib...@4.3.4 %g...@9.2.0 os=xilinx target=aarch64
==> Error: Cannot find valid provider for virtual pkgconfig

 

xilinix is the operating_system I gave to the local compiler in the yaml file. Perhaps this was the wrong thing to do?

I'm new to spack and I cannot figure out how to interpret the error shown above. The present posting is the closest I have found about cross-compiling with spack.

Should I try something different or would it help if I provide more information about my environment?

Thanks,

 Rodrigo