Spack architecture targets behavior

mart...@gmail.com

unread,

Apr 15, 2021, 4:02:03 PM4/15/21

to Spack

Hi everyone,

I am building several different CPU architecture targets with the "targets=xxx" Spack install option, and, while examining the verbose build output I am not seeing the architecture specific compiler flags, like "-march=xxx". I was under impression that Spack does this automatically, i.e. it knows that if we e.g. use gcc and want to build for target=zen2, it'll add the "-march=znver2" flag.

If it does not, is it a responsibility of the package spec to define what build flags to use for a given architecture target by querying the self.spec.target and reacting accordingly?

Is there any documentation in this regard. I only found short snippets like https://spack.readthedocs.io/en/latest/packaging_guide.html?highlight=target#architecture-specifiers that show how a package spec can react to a target, but, does not say anything about the global behavior.

Thanks

Martin

Joseph Schoonover

unread,

Apr 15, 2021, 4:07:40 PM4/15/21

to mart...@gmail.com, Spack

Hey Martin,

I'm not a maintainer of the spack core, but have contributed a few packages. I think you're right, package maintainers would need to expose this to the application's build system through their package.py .

The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.

Dr. Joseph Schoonover

Chief Executive Officer

Senior Research Software Engineer

j...@fluidnumerics.com

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spack/24f6356e-5b31-4f58-ac85-252d09b7c097n%40googlegroups.com.

mart...@gmail.com

unread,

Apr 15, 2021, 6:16:41 PM4/15/21

to Spack

Thanks Joe, this correlates with my observations.

Now, what would be nice is if Spack could encapsulate the target architecture compiler flags e.g. in the spack.compiler module, https://spack.readthedocs.io/en/latest/spack.html?highlight=compiler.cc#module-spack.compiler. It appears that that's not the case now. Any plans to do this in the future? It would make the package.py logic simpler. Now I am thinking the approach would be something like

if self.spec.target >= 'sandybridge':
if self.compiler.cc == 'gcc':

my_cflags = '-march=sandybridge'

elif self.compiler.cc = 'icc':

my_cflags = '-mtune=sandybridge'

...

elif self.spec.target >= 'zen2':

if self.compiler.cc == 'gcc':

my_cflags = '-march=znver2'

elif self.compiler.cc = 'icc':

my_cflags = '-march=core-avx2'

....

If we had a self.compiler.target that would define the compiler-specific target flag pre-defined by Spack, or the compiler package, then we would not be the compiler-based conditions.

Or am I missing something? I feel like I do because Spack is e.g. aware that some gcc versions don't support some targets, but, I am not finding where that's coming from:

$ spack install hpl%g...@8.3.0 target=zen2
==> Error: cannot produce optimized binary for micro-architecture 'zen2' with g...@8.3.0 [supported compiler versions are 9.0:]

Thanks,

MC

mart...@gmail.com

unread,

Apr 15, 2021, 6:33:38 PM4/15/21

to Spack

Alright, sorry for the continuous spam - I think I found where Spack comes up with the knowledge of what compiler supports what architecture - it's in lib/spack/external/archspec/json/cpu/microarchitectures.json

There, for example:

...

"zen2": {
...

      "compilers": {
        "gcc": [
          {
            "name": "znver2",
            "versions": "9.0:",
            "flags": "-march={name} -mtune={name}"
          }
        ],
...

As can be seen, this json definition is also providing the -march and -mtune flags, which optimization_flags() method in lib/spack/external/archspec/cpu/microarchitecture.py seems to grab, but now how to pass it to the spec or the install line?

I'll keep on digging but it does look like Spack has the infrastructure built in.

MC

glenn-...@uiowa.edu

unread,

Apr 15, 2021, 6:36:10 PM4/15/21

to Spack

Spack injects the compiler flags via the compiler wrapper. I am not sure if it is documented, but if you want to see this, you need to turn on the '--debug' flag. When doing that, two files will be produced in the current directory:

spack-cc-{NAME}-{HASH}.in.log
spack-cc-{NAME}-{HASH}.out.log

The in.log files shows the compiler output before the wrapper injects flags. Likewise, the out.log file shows the compiler output after injecting the flags.

Gamblin, Todd

unread,

Apr 15, 2021, 6:44:08 PM4/15/21

to glenn-...@uiowa.edu, Spack

Hi all,

Glenn is right (thanks Glenn!) The target detection capability is coming from archspec, our library for detecting and reasoning about cpus and compilers:

http://github.com/archspec/archspec

You can learn more about that from this paper if you want the high level description:

https://tgamblin.github.io/pubs/archspec-canopie-hpc-2020.pdf

In Spack, archspec now lives in lib/spack/external:

https://github.com/spack/spack/tree/develop/lib/spack/external/archspec

And the information it’s using to determine flags is in this file:

https://github.com/spack/spack/blob/develop/lib/spack/external/archspec/json/cpu/microarchitectures.json

That will tell you all the microarchitecture targets that spack knows about, as well as the compilers and compiler flags it associates with them.

Note that at the moment, we are only adding options that have to do with the instruction set (e.g., -march) and not specific optimization flags (those are left to the packages, the build systems, and user configuration). The targets are designed to tell us where we can use optimized binaries — they’re currently aimed mainly at ensuring compatibility.

We should probably document this better, as we do get this question a lot.

Thanks!

-Todd

To view this discussion on the web visit https://groups.google.com/d/msgid/spack/1c1c8089-7066-465d-abde-6ecbb6576959n%40googlegroups.com.

mart...@gmail.com

unread,

Apr 16, 2021, 12:47:40 AM4/16/21

to Spack

Thanks Glenn and Todd, almost all clear now.

The only thing that remains to understand is what is actually Spack doing when the target is being used.

For example, when I run:

$ spack --debug install -v hpl@2.2%g...@10.2.0 target=zen2

The (verbose) terminal output is displaying the compilation lines like:

/uufs/chpc.utah.edu/sys/spack/linux-centos7-zen2/gcc-10.2.0/intel-mpi-2019.8.254-myqt2sr3rpud2yuagvnvacshjo45na2d/compilers_and_libraries_2020.2.254/linux/mpi/intel64/bin/mpicc -o HPL_dlacpy.o -c -DAdd_ -DF77_INTEGER=int -DStringSunStyle -DHPL_DETAILED_TIMING -DHPL_PROGRESS_REPORT -I/uufs/chpc.utah.edu/common/home/hpcapps/.spack/stage/spack-stage-hpl-2.2-ukyinlin2zcgbcyxike2lx4ij6ipfivq/spack-src/include -I/uufs/chpc.utah.edu/common/home/hpcapps/.spack/stage/spack-stage-hpl-2.2-ukyinlin2zcgbcyxike2lx4ij6ipfivq/spack-src/include/Linux-x86_64 -I/uufs/chpc.utah.edu/sys/spack/linux-centos7-zen2/gcc-10.2.0/intel-mkl-2020.3.279-uzs2iv2buhobuzfty5zvme5ppodm6tlu/include -I/uufs/chpc.utah.edu/sys/spack/linux-centos7-zen2/gcc-10.2.0/intel-mpi-2019.8.254-myqt2sr3rpud2yuagvnvacshjo45na2d/include -O3 ../HPL_dlacpy.c

That is, no arch opt flags ("-march=znver2 -mtune=znver2" in this case).

There are two files produced in the current directory that Glenn mentions:

spack-cc-hpl-ukyinli.in.log
spack-cc-hpl-ukyinli.out.log

The "in" file has the "uninjected" flags:

[cc] /uufs/chpc.utah.edu/sys/installdir/gcc/10.2.0/bin/gcc -o HPL_dlacpy.o -c -DAdd_ -DF77_INTEGER=int -DStringSunStyle -DHPL_DETAILED_TIMING -DHPL_PROGRESS_REPORT -I/uufs/chpc.utah.edu/common/home/hpcapps/.spack/stage/spack-stage-hpl-2.2-ukyinlin2zcgbcyxike2lx4ij6ipfivq/spack-src/include -I/uufs/chpc.utah.edu/common/home/hpcapps/.spack/stage/spack-stage-hpl-2.2-ukyinlin2zcgbcyxike2lx4ij6ipfivq/spack-src/include/Linux-x86_64 -I/uufs/chpc.utah.edu/sys/spack/linux-centos7-zen2/gcc-10.2.0/intel-mkl-2020.3.279-uzs2iv2buhobuzfty5zvme5ppodm6tlu/include -I/uufs/chpc.utah.edu/sys/spack/linux-centos7-zen2/gcc-10.2.0/intel-mpi-2019.8.254-myqt2sr3rpud2yuagvnvacshjo45na2d/include -O3 ../HPL_dlacpy.c -I/uufs/chpc.utah.edu/sys/spack/linux-centos7-zen2/gcc-10.2.0/intel-mpi-2019.8.254-myqt2sr3rpud2yuagvnvacshjo45na2d/compilers_and_libraries_2020.2.254/linux/mpi/intel64/include

while the "out" file has the "injected" flags:

Now, which one of those three lines Spack actually uses? If it is what's in the "out" file, why is the "install --verbose" displaying the first line (w/o the arch flags)? That's what made me believe that the arch flags are not being used.

My guess is that Spack takes and displays the line #1, massages it to line #2 to replace the MPI compiler wrapper with the direct compiler call, and then moves to line #3 that adds the Spack specific options.

And, yes, it would be very helpful to have this documented.

Thanks,

MC

Gamblin, Todd

unread,

Apr 16, 2021, 12:59:38 AM4/16/21

to mart...@gmail.com, Spack

Take a look at the MPI section of the packaging guide for the finer points of how MPI wrappers are handled:

https://spack.readthedocs.io/en/latest/packaging_guide.html#mpi-support-in-spack

#1 is just what the build system outputs. In the build environment we set MPICH_CC, and it points to the Spack compiler wrappers, which are all the symlinks in subdirectories of lib/spack/env in your spack installation. All those symlinks point to lib/spack/env/cc, which is the wrapper:

https://github.com/spack/spack/blob/develop/lib/spack/env/cc

The wrapper looks at environment variables to see that the real compiler wrapper is /uufs/chpc.utah.edu/sys/installdir/gcc/10.2.0/bin/gcc. It’s these lines that write the “in” and “out” logs:

https://github.com/spack/spack/blob/develop/lib/spack/env/cc#L579-L587

And it looks like we write the real compiler out for both the input and the output, which isn’t quite right. In the input, what’s really being invoked is the compiler wrapper. The files are really designed so that you can diff them and see what the wrappers are adding.

Anyway, short story is that your mpicc ends up invoking our wrappers, which invoke the real compiler with some flags added for the uarch and RPATHs. It allows us to inject flags into many builds, and it makes it much harder for a build system to accidentally include things from outside of the package’s dependencies. It will also hopefully in the future allow us to more easily do t-san, a-san, etc. builds, and to inject tools into builds.

I agree it’s confusing that the way we do this doesn’t end up printing the uarch flags on the compile line output by the build, but it’s a tradeoff for generality. If we sent the flags through the build system, packagers would need to do more work to ensure that happens. You *can* pass flags that way:

https://spack.readthedocs.io/en/latest/packaging_guide.html#compiler-flags

But this is package-specific, and we don’t currently pass the uarch flags that way — they’re still done by the compiler wrappers.

-Todd

To view this discussion on the web visit https://groups.google.com/d/msgid/spack/882decc4-23f5-4589-80eb-c0635709d060n%40googlegroups.com.

Joseph Schoonover

unread,

Apr 16, 2021, 8:27:48 AM4/16/21

to mart...@gmail.com, Spack

Todd, Glenn, and Martin,

This has been an interesting thread. Todd and Glenn, do you have a recommended strategy for contributing this kind of documentation on this?

There's enough here to get this started and I'm happy to help.

To view this discussion on the web visit https://groups.google.com/d/msgid/spack/882decc4-23f5-4589-80eb-c0635709d060n%40googlegroups.com.

mart...@gmail.com

unread,

Apr 16, 2021, 11:30:06 AM4/16/21

to Spack

Ditto to what Joe said. I'll be happy to help with the docs too, I'll also document this on our internal information that's much less organized that the Spack docs.

Reply all

Reply to author

Forward