Deploying MAGMA 2.7.0 with CCE 15.0.0 with Spack

[Wil Phan]

Hi,

I'm trying to deploy MAGMA 2.7.0 on Crusher using Spack. I want it to be compiled against 4 ROCm versions (ROCm 5.1.0, 5.2.0, 5.3.0, 5.4.0) using Cray CCE 14.0.2 and 15.0.0. I can't use GCC to compile because I need For some reason the builds using CCE 14.0.2 succeed, but those using CCE 15.0.0 fail. The error log seems to indicate hipcc trying to build a "fat" binary at link time targeting several different gfx* architectures, even though I specifically asked only to build for gfx90a.

  >> 6283     ld.lld: error: undefined reference due to --no-allow-shlib-undefined: .omp_offloading.img_start.cray_amdgcn-amd-amdhs
              a
     6284     >>> referenced by lib/libmagma.so
     6285    
  >> 6286     ld.lld: error: undefined reference due to --no-allow-shlib-undefined: .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
     6287     >>> referenced by lib/libmagma.so
     6288    
  >> 6289     ld.lld: error: undefined reference due to --no-allow-shlib-undefined: .omp_offloading.img_cache.cray_amdgcn-amd-amdhs
              a
     6290     >>> referenced by lib/libmagma.so
  >> 6291     clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
  >> 6292     make[2]: *** [CMakeFiles/testing_zaxpy.dir/build.make:108: testing/testing_zaxpy] Error 1
     6293     make[2]: Leaving directory '/gpfs/alpine/csc322/scratch/wyphan/spack/stage/spack-stage-magma-2.7.0-4exz4oqrjbgmueervz
              4nz2kdvzstenht/spack-build-4exz4oq'
  >> 6294     make[1]: *** [CMakeFiles/Makefile2:1485: CMakeFiles/testing_zaxpy.dir/all] Error 2

Attached is my Spack environment spack.yaml file and the complete Spack build log for the following Spack installation line:

spack install ma...@2.7.0%c...@15.0.0+rocm amdgpu_target=gfx90a ^h...@5.4.0 ^hip...@5.4.0 ^hips...@5.4.0 ^hsa-ro...@5.4.0 ^llvm-...@5.4.0

Please advise,

Wil

[Wil Phan]

Hi,

I'd like to follow up with this issue Any ideas on how to fix this?

In the meantime, Dossay (BNL/SOLLVE) has filed a bug report to HPE, but we have yet to hear anything back from them.

The use case for MAGMA here is to offload batched ZGETRI and ZGETRS in the GEM code, part of WDMApp. GEM is written in Fortran, so I need to build with CCE to get the Fortran modules generated. Unless the MAGMA Fortran module doesn't have interfaces for batched ZGETRI and ZGETRS yet?

I don't really want to write interface modules for hipSOLVER and cuSOLVER directly, since MAGMA already takes care of choosing the correct backend for a given GPU platform... Plus, it's a pain to maintain mixed C+Fortran code if that's the route that the GEM team should take.

Please advise,

Wil