GPU compilation on Ubuntu desktop

[Matej Ditte]

Hello,

I am trying to compile the GPU version of QMCPack 4.1.0 on my desktop (Intel i7-10700K, GeForce RTX 2070 SUPER sm_75) running Ubuntu 24.04.3.

My cmake script contains:

    -D QMC_GPU="openmp;cuda"              \

    -D QMC_GPU_ARCHS=sm_75                \

I have installed all the CUDA stuff from the default repo (sudo apt install nvidia-cuda-*)

     nvcc --version:
     Cuda compilation tools, release 12.0, V12.0.140

I have tried the compilation with Intel oneAPI 2025.2.1, GNU 13.3.0 + OpenMPI, and Clang 18/20 + OpenMPI compilers:

With Intel, cmake gives me the following error:

    CMake Error at CMake/IntelCompilers.cmake:31 (message):
      Invalid Intel GPU architecture "sm_75"! Did you miss "intel_gpu_" prefix?

With GNU, cmake gives me the following warning:

       CMake Warning at CMake/GNUCompilers.cmake:12 (message):
         QMCPACK OpenMP offload is not ready for GCC compiler.

and make gives me the following error:

        [ 23%] Linking CXX executable test_omptarget
        lto-wrapper: fatal error: could not find accel/nvptx-none/mkoffload in /usr/libexec/gcc/x86_64-linux-gnu/13/:/usr/libexec/gcc/x86_64-linux-gnu/13/:/usr/libexec/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/13/:/usr/lib/gcc/x86_64-linux-gnu/ (consider using ‘-B’)
        compilation terminated.
        /usr/bin/ld: error: lto-wrapper failed

Clang 18 from the default repo does not include clang-offload-packager, clang-linker-wrapper and probably some other stuff, so I got an error. I changed the repo to install version 20 with all the missing binaries, but this time cmake gives me this error:

      clang++-20: error: no library 'libomptarget-nvptx.bc' found in the default
      clang lib directory or in LIBRARY_PATH; use '--libomptarget-nvptx-bc-path'
      to specify nvptx bitcode library

I also tried compiling Clang 22 from source, following several step-by-step guides from different websites, but I haven’t succeeded. Has anyone managed to compile the GPU version in this trivial case (Ubuntu desktop with root access) and could help me?

Thank you very much!

Best,
Matej

[Paul R. C. Kent]

Dear Matej

Some quick comments. Hopefully someone else has a proven/known working/up to date install route for the compiler on Ubuntu24 and will comment. I'll follow up if needed - I don't have the exact combination immediately to hand. If you can make it, the office hours today might help, but I realize it is likely late evening where you are.

* Pursue the clang route. Only this compiler has good enough offload to NVIDIA GPUs. clang18 would be enough but it is preferable to use the latest version you can since there have been improvements each release.

* Your build should "just work" on Ubuntu, once the required software is available and configured appropriately. I do have this working 100% on RHEL9.6 for NVIDIA GPUs using the spack package manager for all the dependencies, but this is not likely to be the simplest working route. And on Ubuntu24, at least for AMD GPUs, I have this working entirely with packages from the Ubuntu repositories except for the AMD ROCm compilers (amdclang) and associated GPU libraries.

* To build the GPU version with the default optimal configuration, you only need to give the GPU architecture:

cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DQMC_MPI=0 -DQMC_GPU_ARCHS=sm_75 ..

* If you have not built the CPU version on your system yet, this is worth doing to verify all the other needed dependencies are there.

* In these cases, it is generally easiest to try installing without MPI first, and then add that later.

* We should put the shortest Ubuntu-NVIDIA route in the manual since it is likely very common; we only have the CPU route currently https://qmcpack.readthedocs.io/en/develop/installation.html#installing-on-ubuntu-linux-or-other-apt-get-based-distributions 

-- Paul

[Matej Ditte]

Hi Paul,

Thank you for your reply, and apologies for my delayed response. I went on vacation immediately after posting here, so I was also unable to attend the office hours.

I have a fully working Intel oneAPI CPU version. Following your suggestions, I successfully compiled the CPU serial Clang 20 and CPU OpenMPI Clang 20 versions. The parallel version failed unit test #44 and some of the -R short -LE unstable tests, but overall it runs. The failures might be related to using MKL with Clang and HDF5 compiled with Intel compilers.

After that, I tried to add GPU support on top of the serial Clang 20 version:

     cmake -DCMAKE_C_COMPILER=clang-20 -DCMAKE_CXX_COMPILER=clang++-20 -DQMC_MPI=0  -DQMC_GPU_ARCHS=sm_75  ..

unfortunately, I got the same error as before:

     clang++-20: error: no library 'libomptarget-nvptx.bc' found in the default
     clang lib directory or in LIBRARY_PATH; use '--libomptarget-nvptx-bc-path'
     to specify nvptx bitcode library

I managed to find the 'libomptarget-nvptx.bc' file in '/usr/lib/llvm-20/lib/' (it is not in the default '/usr/lib/clang/20.1.8/lib/linux/') and by using the flag or by adding the path into LIBRARY_PATH I moved forward, this time with less readable cmake error (in the attached cmake output).

Maybe some mess with two versions of clang? I first installed v18 without offloading from the default repo and then v20 with offloading from a different repo. I also tried to change the compilers to '/usr/lib/llvm-20/bin/clang' (because the missing lib is in llvm, not in the clang directory), but the result is the same. 

Best,

M