Using hpmc.pair.user.CPPPotential with gpu

73 views
Skip to first unread message

Irem

unread,
Apr 6, 2023, 9:18:21 PMApr 6
to hoomd-users
Hi all,

I am following the tutorial here. How do I get it to work with GPU? Once I switch to gpu, I get    a runtime error:

    self._cpp_obj = _jit.PatchEnergyJITGPU(
RuntimeError: Runtime compilation failed

What am I missing? I'm attaching the script below.

Best,
Irem
kern_frenkel_gpu.py

Joshua Anderson

unread,
Apr 7, 2023, 9:21:33 AMApr 7
to hoomd...@googlegroups.com
Irem,

The script runs without error on my workstation. To obtain the full error message from NVRTC, recompile with the CMake option ENABLE_DEBUG_JIT=ON and run the script again.
------
Joshua A. Anderson, Ph.D.
Research Area Specialist, Chemical Engineering, University of Michigan

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/5709c047-21ff-4a8c-9488-e7576aa05a51n%40googlegroups.com.
<kern_frenkel_gpu.py>

Message has been deleted

Tim Moore

unread,
Apr 11, 2023, 7:01:40 AMApr 11
to hoomd...@googlegroups.com
Irem,

As noted in the installation documentation, you also need the compilers package from conda-forge to use runtime compilation. You may also run into issues if you‘re using a full Anaconda distribution instead of mambaforge, miniforge, or miniconda. 

Tim

On Tue, Apr 11, 2023 at 6:45 AM Irem <irem....@gmail.com> wrote:
Interesting. Thank you for checking. I'm running HOOMD in a conda environment that I have set up with 

conda create -n hoomdgpu -c conda-forge hoomd=*=*gpu* python=3.8.6 gsd numpy

Is there any way to make that happen with a conda install, by any chance?

Best,
Irem
--
Tim Moore

Joshua Anderson

unread,
Apr 11, 2023, 8:11:11 AMApr 11
to hoomd...@googlegroups.com
Irem, 

No, you must build from source to enable the debug messages. You may find that building from source fixes the problem. I have seen this behavior before on the conda-forge builds but I have not been able to find a solution. In my test case, there was a cuda runtime / driver mismatch that causes PTX errors in NVRTC.

Tim,

No additional packages should be needed. We account for all required dependencies in the conda forge package.
------
Joshua A. Anderson, Ph.D.
Research Area Specialist, Chemical Engineering, University of Michigan

Irem Altan

unread,
Apr 18, 2023, 7:24:38 AMApr 18
to hoomd...@googlegroups.com
Thank you both. Yes, adding the compilers package makes no difference. Our HPC people compiled the debug version, and I’m getting the following output on a GPU node:

(hoomd) [ia253@p09r07n24 kern_frenkel]$ python kern_frenkel_gpu.py
Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
Traceback (most recent call last):
  File "kern_frenkel_gpu.py", line 99, in <module>
    sim.run(0)
  File "/gpfs/loomis/apps/avx/software/HOOMD-blue/3.10.0-foss-2020b-CUDA-11.3.1-Debug/hoomd/simulation.py", line 455, in run
    self.operations._schedule()
  File "/gpfs/loomis/apps/avx/software/HOOMD-blue/3.10.0-foss-2020b-CUDA-11.3.1-Debug/hoomd/operations.py", line 179, in _schedule
    self.integrator._attach(sim)
  File "/gpfs/loomis/apps/avx/software/HOOMD-blue/3.10.0-foss-2020b-CUDA-11.3.1-Debug/hoomd/operation.py", line 297, in _attach
    self._attach_hook()
  File "/gpfs/loomis/apps/avx/software/HOOMD-blue/3.10.0-foss-2020b-CUDA-11.3.1-Debug/hoomd/hpmc/integrate.py", line 420, in _attach_hook
    self._pair_potential._attach(self._simulation)
  File "/gpfs/loomis/apps/avx/software/HOOMD-blue/3.10.0-foss-2020b-CUDA-11.3.1-Debug/hoomd/operation.py", line 297, in _attach
    self._attach_hook()
  File "/gpfs/loomis/apps/avx/software/HOOMD-blue/3.10.0-foss-2020b-CUDA-11.3.1-Debug/hoomd/hpmc/pair/user.py", line 269, in _attach_hook
    self._cpp_obj = _jit.PatchEnergyJITGPU(
NameError: name '_jit' is not defined

I guess the proper error is the ‘_jit’ is not defined message. I’m not sure what the first 16bit vs. 32bit error is.

Some potentially helpful information:

- GPU node has 3090Ti, with NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8
- I can run MD simulations with HOOMD on this very GPU node with the same setup without a problem (can monitor gpu usage with nvidia-smi)
- The 16/32bit error occurs with the debug version only, and not on the conda installation, which also works for MD without a problem on the same node.
- HOOMD 3.10.0 compiled with cuda 11.3.1, with ENABLE_DEBUG_JIT=ON.
- python 3.8.6

Please let me know if more information is needed.

Best,
Irem

Joshua Anderson

unread,
Apr 18, 2023, 7:31:33 AMApr 18
to hoomd...@googlegroups.com
Irem,

`'_jit' is not defined` is a sign that HOOMD-blue was compiled with ENABLE_LLVM=off. Rebuild with ENABLE_LLVM=on. 

I would recommend CMAKE_BUILD_TYPE=Release, not Debug for this. ENABLE_DEBUG_JIT=ON adds the debugging information to the NVRTC compilation step needed to troubleshoot this issue.

HOOMD-blue does not produce the error message `Ignoring PCI device with non-16bit domain`. Check your MPI library.
------
Joshua A. Anderson, Ph.D.
Research Area Specialist, Chemical Engineering, University of Michigan

Irem Altan

unread,
Apr 21, 2023, 7:19:37 AMApr 21
to hoomd...@googlegroups.com
We have that flag enabled, but it’s still not working. We are having a weird issue where the proper LLVM libraries are not detected. I am trying to do this on an HPC setting, with the HPC staff helping me, but we are running into a problem where cmake looks in the wrong place for the LLVM and clang libraries, and we are not sure which specific library to point it to.

We have the following modules loaded (with versions):


OpenMPI/4.0.5-GCC-10.2.0

Clang/11.0.1-gcccuda-2020b

Python/3.8.6-GCCcore-10.2.0

LLVM/11.0.0-GCCcore-10.2.0

tbb/2020.3-GCCcore-10.2.0

Eigen/3.4.0-GCCcore-10.2.0

Cereal/1.3.2-GCCcore-10.2.0

CUDA/11.1.1-GCC-10.2.0

pybind11/2.6.2-GCCcore-10.2.0

CMake/3.22.1-GCCcore-10.2.0


I am on a GPU node with a 3090Ti card. When I try to configure cmake with:


cmake -B build/hoomd -S hoomd-blue  -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DENABLE_GPU=ON -DENABLE_MPI=ON -DENABLE_MPI_CUDA=ON -DENABLE_TBB=ON -DENABLE_LLVM=ON


I’m getting the following:


-- The C compiler identification is GNU 10.2.0

-- The CXX compiler identification is GNU 10.2.0

-- Detecting C compiler ABI info

-- Detecting C compiler ABI info - done

-- Check for working C compiler: /gpfs/loomis/apps/avx/software/GCCcore/10.2.0/bin/cc - skipped

-- Detecting C compile features

-- Detecting C compile features - done

-- Detecting CXX compiler ABI info

-- Detecting CXX compiler ABI info - done

-- Check for working CXX compiler: /gpfs/loomis/apps/avx/software/GCCcore/10.2.0/bin/c++ - skipped

-- Detecting CXX compile features

-- Detecting CXX compile features - done

-- Configuring HOOMD v3.11.0

-- The CUDA compiler identification is NVIDIA 11.1.105

-- Detecting CUDA compiler ABI info

-- Detecting CUDA compiler ABI info - done

-- Check for working CUDA compiler: /gpfs/loomis/apps/avx/software/CUDAcore/11.1.1/bin/nvcc - skipped

-- Detecting CUDA compile features

-- Detecting CUDA compile features - done

-- Found CUDALibs: /gpfs/loomis/apps/avx/software/CUDAcore/11.1.1/lib/libcudart.so

-- Found MPI_C: /gpfs/loomis/apps/avx/software/OpenMPI/4.0.5-GCC-10.2.0/lib/libmpi.so (found version "3.1")

-- Found MPI_CXX: /gpfs/loomis/apps/avx/software/OpenMPI/4.0.5-GCC-10.2.0/lib/libmpi_cxx.so (found version "3.1")

-- Found MPI: TRUE (found version "3.1")

-- Found cereal: /gpfs/loomis/apps/avx/software/Cereal/1.3.2-GCCcore-10.2.0/lib64/cmake/cereal

-- Found PythonInterp: /home/ia253/project/venv/hoomd/bin/python (found suitable version "3.8.6", minimum required is "3")

-- Found PythonLibs: /gpfs/loomis/apps/avx/software/Python/3.8.6-GCCcore-10.2.0/lib/libpython3.8.so

-- Performing Test HAS_FLTO

-- Performing Test HAS_FLTO - Success

-- Found pybind11: /gpfs/loomis/apps/avx/software/pybind11/2.6.2-GCCcore-10.2.0/include (found version "2.6.2" )

-- Found pybind11: /gpfs/loomis/apps/avx/software/pybind11/2.6.2-GCCcore-10.2.0/share/cmake/pybind11 /gpfs/loomis/apps/avx/software/pybind11/2.6.2-GCCcore-10.2.0/include (version 2.6.2)

-- Installing hoomd python module to: /home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd

-- Found eigen: /gpfs/loomis/apps/avx/software/Eigen/3.4.0-GCCcore-10.2.0/share/eigen3/cmake /gpfs/loomis/apps/avx/software/Eigen/3.4.0-GCCcore-10.2.0/include (version 3.4.0)

CMake Warning (dev) at CMake/hoomd/hoomd-macros.cmake:55 (find_package):

  Policy CMP0074 is not set: find_package uses <PackageName>_ROOT variables.

  Run "cmake --help-policy CMP0074" for policy details.  Use the cmake_policy

  command to set the policy and suppress this warning.


  Environment variable TBB_ROOT is set to:


    /gpfs/loomis/apps/avx/software/tbb/2020.3-GCCcore-10.2.0/


  For compatibility, CMake is ignoring the variable.

Call Stack (most recent call first):

  hoomd/CMakeLists.txt:308 (find_package_config_first)

This warning is for project developers.  Use -Wno-dev to suppress it.


-- Found TBB: /gpfs/loomis/apps/avx/software/tbb/2020.3-GCCcore-10.2.0/lib64/cmake/TBB /gpfs/loomis/apps/avx/software/tbb/2020.3-GCCcore-10.2.0/lib/libtbb.so.2 /vast/palmer/apps/avx.grace/software/tbb/2020.3-GCCcore-10.2.0/include

-- Found LLVM: /gpfs/loomis/apps/avx/software/LLVM/11.0.0-GCCcore-10.2.0/lib64/cmake/llvm llvm_library-NOTFOUND clang_library-NOTFOUND /gpfs/loomis/apps/avx/software/LLVM/11.0.0-GCCcore-10.2.0/include -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS (version 11.0.0)

-- Found plugin: example_plugins/pair_plugin

-- Found plugin: example_plugins/updater_plugin

-- Configuring done

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.

Please set them or make sure they are set and tested correctly in the CMake files:

clang_library

    linked by target "_jit" in directory /home/ia253/hoomd_install_test/hoomd-blue/hoomd/hpmc

llvm_library

    linked by target "_jit" in directory /home/ia253/hoomd_install_test/hoomd-blue/hoomd/hpmc


-- Generating done

CMake Generate step failed.  Build files cannot be regenerated correctly.


We are not sure why it says llvm_library-NOTFOUND and clang_library-NOTFOUND. We tried pointing to the correct libraries manually, by adding the following options:


-Dllvm_library=/gpfs/loomis/apps/avx/software/LLVM/11.0.0-GCCcore-10.2.0/lib/libLLVMCore.so -Dclang_library=/gpfs/loomis/apps/avx/software/Clang/11.0.1-gcccuda-2020b/lib/libclang.so


It builds and installs fine, but when I try to import the hoomd module, I get a jit error again:


(hoomd) [ia253@p09r07n24 lib]$ python

Python 3.8.6 (default, Feb 17 2021, 19:45:30)

[GCC 10.2.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import hoomd

Ignoring PCI device with non-16bit domain.

Pass --enable-32bits-pci-domain to configure to support such devices

(warning: it would break the library ABI, don't enable unless really needed).

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/__init__.py", line 73, in <module>

    from hoomd import hpmc

  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/hpmc/__init__.py", line 32, in <module>

    from hoomd.hpmc import pair

  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/hpmc/pair/__init__.py", line 11, in <module>

    from . import user

  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/hpmc/pair/user.py", line 18, in <module>

    from hoomd.hpmc import _jit

ImportError: /home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/hpmc/_jit.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZTVN4llvm3opt12InputArgListE


We suspect that we are not pointing to the correct *.so files for either clang or llvm (or both). There are many .so files in either directory:

(clang):
(hoomd) [ia253@p09r07n24 lib]$ ls *so
libarcher.so  libclang-cpp.so  libc++.so   libGPURuntime.so  libLTO.so  libomptarget.rtl.cuda.so    libomptarget.so  LLVMPolly.so
libc++abi.so  libclang.so      libgomp.so  libiomp5.so       libomp.so  libomptarget.rtl.x86_64.so  libRemarks.so

(llvm):
(hoomd) [ia253@p09r07n24 lib]$ ls *so
libLLVMAggressiveInstCombine.so  libLLVMCoverage.so           libLLVMFrontendOpenMP.so   libLLVMLTO.so             libLLVMOrcJIT.so        libLLVMTransformUtils.so
libLLVMAnalysis.so               libLLVMDebugInfoCodeView.so  libLLVMFuzzMutate.so       libLLVMMCA.so             libLLVMPasses.so        libLLVMVectorize.so
libLLVMAsmParser.so              libLLVMDebugInfoDWARF.so     libLLVMGlobalISel.so       libLLVMMCDisassembler.so  libLLVMProfileData.so   libLLVMWindowsManifest.so
libLLVMAsmPrinter.so             libLLVMDebugInfoGSYM.so      libLLVMInstCombine.so      libLLVMMCJIT.so           libLLVMRemarks.so       libLLVMX86AsmParser.so
libLLVMBinaryFormat.so           libLLVMDebugInfoMSF.so       libLLVMInstrumentation.so  libLLVMMCParser.so        libLLVMRuntimeDyld.so   libLLVMX86CodeGen.so
libLLVMBitReader.so              libLLVMDebugInfoPDB.so       libLLVMInterpreter.so      libLLVMMC.so              libLLVMScalarOpts.so    libLLVMX86Desc.so
libLLVMBitstreamReader.so        libLLVMDemangle.so           libLLVMipo.so              libLLVMMIRParser.so       libLLVMSelectionDAG.so  libLLVMX86Disassembler.so
libLLVMBitWriter.so              libLLVMDlltoolDriver.so      libLLVMIRReader.so         libLLVMObjCARCOpts.so     libLLVMSupport.so       libLLVMX86Info.so
libLLVMCFGuard.so                libLLVMDWARFLinker.so        libLLVMJITLink.so          libLLVMObject.so          libLLVMSymbolize.so     libLLVMXRay.so
libLLVMCodeGen.so                libLLVMExecutionEngine.so    libLLVMLibDriver.so        libLLVMObjectYAML.so      libLLVMTableGen.so      libLTO.so
libLLVMCore.so                   libLLVMExtensions.so         libLLVMLineEditor.so       libLLVMOption.so          libLLVMTarget.so        libRemarks.so
libLLVMCoroutines.so             libLLVMFrontendOpenACC.so    libLLVMLinker.so           libLLVMOrcError.so        libLLVMTextAPI.so

Is there an issue with the cmake config? The versions of thins we are using? Or maybe a bug in cmake that causes for it to look in the wrong places? Or a combination thereof? Please let me know if more information is needed. Sorry for the long e-mail — I tried to format all the info as best as possible.

Best,
Irem

Joshua Anderson

unread,
Apr 21, 2023, 9:50:16 AMApr 21
to hoomd...@googlegroups.com
Irem,

Thanks for the details, the CMake command and output are very helpful to answer your question.

Here is example output from CMake where it successfully finds the LLVM libraries:
-- Found LLVM: /usr/lib/llvm-13/cmake /usr/lib/llvm-13/lib/libLLVM.so /usr/lib/llvm-13/lib/libclang-cpp.so /usr/lib/llvm-13/include -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS (version 13.0.1)

Your output states that it found the LLVM CMake file in `/gpfs/loomis/apps/avx/software/LLVM/11.0.0-GCCcore-10.2.0/lib64/cmake/llvm` (which appears to be the correct place), but that it did not find libLLVM.so or libclang-cpp.so in the library directories given by that cmake file. It would appear that your build of LLVM was configured to not provide these files. HOOMD needs them. Note that libclang-cpp from a `clang` installation is not sufficient as HOOMD also requires the matching header files. When I build LLVM from source, I use the options:

    cmake -S llvm -B build \
        -D CMAKE_INSTALL_PREFIX=$ROOT -DLLVM_ENABLE_PROJECTS=clang \
        -DCLANG_LINK_CLANG_DYLIB=ON \
        -DLLVM_BUILD_LLVM_DYLIB=ON \
        -DLLVM_LINK_LLVM_DYLIB=ON \
        -DCMAKE_BUILD_TYPE=Release

Most LLVM installs, such as those on Ubuntu and conda-forge, provide these headers and libraries. Many HPC cluster modules do not, yours included. You will need to compile a compatible build of LLVM to use it with HOOMD-blue.

Also, I recommend you remove some of the CMake flags you set:
* `-DCMAKE_POSITION_INDEPENDENT_CODE=ON` is not necessary. CMake will automatically add -fPIC to those files that need it and leave it off for those that do not.
* `-DENABLE_MPI_CUDA=ON` will result in incorrect simulations when you run with more than one MPI rank on GPUs.
------
Joshua A. Anderson, Ph.D.
Research Area Specialist, Chemical Engineering, University of Michigan

Irem Altan

unread,
Apr 25, 2023, 7:28:33 AMApr 25
to hoomd...@googlegroups.com
Thank you for the information. I built llvm with the flags you provided:

cd llvm-project
cmake -S llvm -B build -DCMAKE_INSTALL_PREFIX=/home/ia253/software/llvm -DLLVM_ENABLE_PROJECTS=clang -DCLANG_LINK_CLANG_DYLIB=ON -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON -DCMAKE_BUILD_TYPE=Release


Then I try to do a fresh build of hoomd with:

cmake -B build/hoomd -S hoomd-blue  -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DENABLE_GPU=ON -DENABLE_MPI=ON -DENABLE_TBB=ON -DENABLE_LLVM=ON -DLLVM_DIR=/home/ia253/software/llvm/lib/cmake/llvm

Runs with two warnings but no errors (see attached — output_of_cmake_config.txt). Then, I try to build with:

cmake --build build/hoomd -j16

It gets until 84% but then fails with an error with no message (see build_error_1.txt). I then re-run the command to build, which fails at 88% (see build_error_2.txt). Running the command again, it gets to 95% (see build_error_3.txt). Eventually it gets stuck at 100%, but can’t get past a bunch of jit errors (see the other attached files).

What am I missing still?

Also note that I am loading the same modules, but not the LLVM and Clang modules, and instead pointing cmake towards the one I compiled.
build_error_5.txt
output_of_cmake_config.txt
build_error_5.txt
build_error_4.txt
build_error_3.txt
build_error_2.txt
build_error_1.txt
output_of_cmake_config.txt
build_error_4.txt
build_error_3.txt
build_error_2.txt
build_error_1.txt

Joshua Anderson

unread,
Apr 25, 2023, 7:41:16 AMApr 25
to hoomd...@googlegroups.com
Irem,

It would appear that LLVM has changed their API again (they do this regularly). You will need to checkout and build a previous version of LLVM. I currently test with versions 10, 11, 12, 13, and 14. I will make a new release of HOOMD-blue that also supports LLVM 15 at a later time.

Here is a tip: When you encounter an error during make, the relevant error is the *first* error in the output, not the last. "gmake: *** [all] Error 2" only indicates that there was an error previously which you did not include in your build_error files 1, 2, 3, or 4. You only need to run the build step once and report the first error (or batch of errors).

------
Joshua A. Anderson, Ph.D.
Research Area Specialist, Chemical Engineering, University of Michigan
--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/A8EDD5F8-3D2B-49FE-B18C-9F3E2F6E405F%40gmail.com.
<build_error_5.txt>

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/A8EDD5F8-3D2B-49FE-B18C-9F3E2F6E405F%40gmail.com.
<build_error_4.txt>

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/A8EDD5F8-3D2B-49FE-B18C-9F3E2F6E405F%40gmail.com.
<build_error_3.txt>

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/A8EDD5F8-3D2B-49FE-B18C-9F3E2F6E405F%40gmail.com.
<build_error_2.txt>

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/A8EDD5F8-3D2B-49FE-B18C-9F3E2F6E405F%40gmail.com.
<build_error_1.txt>

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/A8EDD5F8-3D2B-49FE-B18C-9F3E2F6E405F%40gmail.com.
<output_of_cmake_config.txt>


On Apr 21, 2023, at 9:50 AM, Joshua Anderson <joaa...@umich.edu> wrote:

Irem,

Thanks for the details, the CMake command and output are very helpful to answer your question.

Here is example output from CMake where it successfully finds the LLVM libraries:
-- Found LLVM: /usr/lib/llvm-13/cmake /usr/lib/llvm-13/lib/libLLVM.so /usr/lib/llvm-13/lib/libclang-cpp.so /usr/lib/llvm-13/include -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS (version 13.0.1)

Your output states that it found the LLVM CMake file in `/gpfs/loomis/apps/avx/software/LLVM/11.0.0-GCCcore-10.2.0/lib64/cmake/llvm` (which appears to be the correct place), but that it did not find libLLVM.so or libclang-cpp.so in the library directories given by that cmake file. It would appear that your build of LLVM was configured to not provide these files. HOOMD needs them. Note that libclang-cpp from a `clang` installation is not sufficient as HOOMD also requires the matching header files. When I build LLVM from source, I use the options:

    cmake -S llvm -B build \
        -D CMAKE_INSTALL_PREFIX=$ROOT -DLLVM_ENABLE_PROJECTS=clang \
        -DCLANG_LINK_CLANG_DYLIB=ON \
        -DLLVM_BUILD_LLVM_DYLIB=ON \
        -DLLVM_LINK_LLVM_DYLIB=ON \
        -DCMAKE_BUILD_TYPE=Release

Please let me know if more information is needed.

Best,
Irem

/home/ia253/software/llvm -DLLVM_ENABLE_PROJECTS=clang \
        -DCLANG_LINK_CLANG_DYLIB=ON \
        -DLLVM_BUILD_LLVM_DYLIB=ON \
        -DLLVM_LINK_LLVM_DYLIB=ON \
        -DCMAKE_BUILD_TYPE=Release
cmake -S llvm -B build -DCMAKE_INSTALL_PREFIX=cmake -S llvm -B build -DCMAKE_INSTALL_PREFIX=
Then I try to do a fresh build of hoomd with:

cmake -B build/hoomd -S hoomd-blue  -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DENABLE_GPU=ON -DENABLE_MPI=ON -DENABLE_TBB=ON -DENABLE_LLVM=ON -DLLVM_DIR=/home/ia253/software/llvm/lib/cmake/llvm

Runs with two warnings but no errors (see attached — output_of_cmake_config.txt). Then, I try to build with:

cmake --build build/hoomd -j16

It gets until 84% but then fails with an error with no message (see build_error_1.txt). I then re-run the command to build, which fails at 88% (see build_error_2.txt). Running the command again, it gets to 95% (see build_error_3.txt). Eventually it gets stuck at 100%, but can’t get past a bunch of jit errors (see the other attached files).

What am I missing still?

Also note that I am loading the same modules, but not the LLVM and Clang modules, and instead pointing cmake towards the one I compiled.

Please let me know if more information is needed.

Best,
Irem

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/A8EDD5F8-3D2B-49FE-B18C-9F3E2F6E405F%40gmail.com.
<build_error_5.txt>

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/A8EDD5F8-3D2B-49FE-B18C-9F3E2F6E405F%40gmail.com.
<build_error_4.txt>

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/A8EDD5F8-3D2B-49FE-B18C-9F3E2F6E405F%40gmail.com.
<build_error_3.txt>

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/A8EDD5F8-3D2B-49FE-B18C-9F3E2F6E405F%40gmail.com.
<build_error_2.txt>

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/A8EDD5F8-3D2B-49FE-B18C-9F3E2F6E405F%40gmail.com.
<build_error_1.txt>

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/A8EDD5F8-3D2B-49FE-B18C-9F3E2F6E405F%40gmail.com.
<output_of_cmake_config.txt>


On Apr 21, 2023, at 9:50 AM, Joshua Anderson <joaa...@umich.edu> wrote:

Irem,

Thanks for the details, the CMake command and output are very helpful to answer your question.

Here is example output from CMake where it successfully finds the LLVM libraries:
-- Found LLVM: /usr/lib/llvm-13/cmake /usr/lib/llvm-13/lib/libLLVM.so /usr/lib/llvm-13/lib/libclang-cpp.so /usr/lib/llvm-13/include -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS (version 13.0.1)

Your output states that it found the LLVM CMake file in `/gpfs/loomis/apps/avx/software/LLVM/11.0.0-GCCcore-10.2.0/lib64/cmake/llvm` (which appears to be the correct place), but that it did not find libLLVM.so or libclang-cpp.so in the library directories given by that cmake file. It would appear that your build of LLVM was configured to not provide these files. HOOMD needs them. Note that libclang-cpp from a `clang` installation is not sufficient as HOOMD also requires the matching header files. When I build LLVM from source, I use the options:

    cmake -S llvm -B build \
        -D CMAKE_INSTALL_PREFIX=$ROOT -DLLVM_ENABLE_PROJECTS=clang \
        -DCLANG_LINK_CLANG_DYLIB=ON \
        -DLLVM_BUILD_LLVM_DYLIB=ON \
        -DLLVM_LINK_LLVM_DYLIB=ON \
        -DCMAKE_BUILD_TYPE=Release

Please let me know if more information is needed.

Best,
Irem

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.

Irem Altan

unread,
Apr 25, 2023, 1:10:34 PMApr 25
to hoomd...@googlegroups.com
Hi,

I built version 14:

git clone --branch release/14.x https://github.com/llvm/llvm-project.git

cd llvm

cmake -S llvm -B build -DCMAKE_INSTALL_PREFIX=/home/ia253/software/llvm -DLLVM_ENABLE_PROJECTS=clang -DCLANG_LINK_CLANG_DYLIB=ON -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON -DCMAKE_BUILD_TYPE=Release

cmake --build build/

cmake --install build/


and then built HOOMD:

git clone --recursive https://github.com/glotzerlab/hoomd-blue


module load OpenMPI/4.0.5-GCC-10.2.0 Python/3.8.6-GCCcore-10.2.0  tbb/2020.3-GCCcore-10.2.0 Eigen/3.4.0-GCCcore-10.2.0 Cereal/1.3.2-GCCcore-10.2.0 CUDA/11.1.1-GCC-10.2.0 pybind11/2.6.2-GCCcore-10.2.0 CMake/3.22.1-GCCcore-10.2.0


source ~/project/venv/hoomd/bin/activate


cmake -B build/hoomd -S hoomd-blue  -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DENABLE_GPU=ON -DENABLE_MPI=ON -DENABLE_TBB=ON -DENABLE_LLVM=ON -DLLVM_DIR=/home/ia253/software/llvm/lib/cmake/llvm


The output of the cmake config is attached. There is one, probably insignificant warning, and both the llvm and clang libraries are found. Then, I build it without errors (see build_output.dat), but a bunch of warnings. However, I still get the jit error when trying to import the module:


(hoomd) [ia253@p09r07n28 ~]$ python

Python 3.8.6 (default, Feb 17 2021, 19:45:30)

[GCC 10.2.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import hoomd

Ignoring PCI device with non-16bit domain.

Pass --enable-32bits-pci-domain to configure to support such devices

(warning: it would break the library ABI, don't enable unless really needed).

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/__init__.py", line 73, in <module>

    from hoomd import hpmc

  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/hpmc/__init__.py", line 32, in <module>

    from hoomd.hpmc import pair

  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/hpmc/pair/__init__.py", line 11, in <module>

    from . import user

  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/hpmc/pair/user.py", line 18, in <module>

    from hoomd.hpmc import _jit

ImportError: libLLVM-14.so: cannot open shared object file: No such file or directory


I can verify that libLLVM-14.so is indeed produced after my compilation:


(hoomd) [ia253@p09r07n28 lib]$ pwd

/home/ia253/software/llvm/lib

(hoomd) [ia253@p09r07n28 lib]$ ls libLLVM*so

libLLVM-14.0.6.so  libLLVM-14.so  libLLVM.so


Is it possible that python doesn’t know where to find these libraries? But that should have taken care of when the HOOMD module was being installed, no? I’m also worried that it’s searching for libLLVM-14.so but not libLLVM.so. Am I somehow building two versions simultaneously, and the HOOMD cmake finds the wrong version?


Should I try an even older version of LLVM? I’ve included the command I run to import the other necessary modules above — maybe some of them have the wrong version, or are clashing with the LLVM libraries I built?


I should also note that, every time I try building and installing HOOMD, I delete the cloned repo, build directory, and, if I managed to install, all the files listed in build/hoomd/install_manifest.txt, and start from scratch.


Best,

Irem


output_cmake_config.dat
build_output.dat

Joshua Anderson

unread,
Apr 25, 2023, 2:45:56 PMApr 25
to hoomd...@googlegroups.com
Irem,

If HOOMD-blue compiled without errors, then you do not need to try a different version of LLVM.

Look at the output you provide, it gives the error why `_jit` cannot be imported: ImportError: libLLVM-14.so: cannot open shared object file: No such file or directory. It would seem that you do not have the location you installed LLVM to on your library search path. Add it, then _jit will import.
------
Joshua A. Anderson, Ph.D.
Research Area Specialist, Chemical Engineering, University of Michigan
--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/75A41D6E-7AA8-49CF-8BD8-E5EFBC45FA5B%40gmail.com.
<output_cmake_config.dat>


--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
On Apr 25, 2023, at 7:40 AM, Joshua Anderson <joaa...@umich.edu> wrote:

Irem,

It would appear that LLVM has changed their API again (they do this regularly). You will need to checkout and build a previous version of LLVM. I currently test with versions 10, 11, 12, 13, and 14. I will make a new release of HOOMD-blue that also supports LLVM 15 at a later time.

Here is a tip: When you encounter an error during make, the relevant error is the *first* error in the output, not the last. "gmake: *** [all] Error 2" only indicates that there was an error previously which you did not include in your build_error files 1, 2, 3, or 4. You only need to run the build step once and report the first error (or batch of errors).
------
Joshua A. Anderson, Ph.D.
Research Area Specialist, Chemical Engineering, University of Michigan
On Apr 24, 2023, at 8:17 PM, Irem Altan <irem....@gmail.com> wrote:

Thank you for the information. I built llvm with the flags you provided:

cd llvm-project
cmake -S llvm -B build -DCMAKE_INSTALL_PREFIX=/home/ia253/software/llvm -DLLVM_ENABLE_PROJECTS=clang -DCLANG_LINK_CLANG_DYLIB=ON -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON -DCMAKE_BUILD_TYPE=Release


Then I try to do a fresh build of hoomd with:

cmake -B build/hoomd -S hoomd-blue  -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DENABLE_GPU=ON -DENABLE_MPI=ON -DENABLE_TBB=ON -DENABLE_LLVM=ON -DLLVM_DIR=/home/ia253/software/llvm/lib/cmake/llvm

    cmake -S llvm -B build \
        -D CMAKE_INSTALL_PREFIX=$ROOT -DLLVM_ENABLE_PROJECTS=clang \
        -DCLANG_LINK_CLANG_DYLIB=ON \
        -DLLVM_BUILD_LLVM_DYLIB=ON \
        -DLLVM_LINK_LLVM_DYLIB=ON \
        -DCMAKE_BUILD_TYPE=Release

Most LLVM installs, such as those on Ubuntu and conda-forge, provide these headers and libraries. Many HPC cluster modules do not, yours included. You will need to compile a compatible build of LLVM to use it with HOOMD-blue.

Also, I recommend you remove some of the CMake flags you set:
* `-DCMAKE_POSITION_INDEPENDENT_CODE=ON` is not necessary. CMake will automatically add -fPIC to those files that need it and leave it off for those that do not.
* `-DENABLE_MPI_CUDA=ON` will result in incorrect simulations when you run with more than one MPI rank on GPUs.
------
Joshua A. Anderson, Ph.D.
Research Area Specialist, Chemical Engineering, University of Michigan

On Apr 20, 2023, at 9:52 AM, Irem Altan <irem....@gmail.com> wrote:

We have that flag enabled, but it’s still not working. We are having a weird issue where the proper LLVM libraries are not detected. I am trying to do this on an HPC setting, with the HPC staff helping me, but we are running into a problem where cmake looks in the wrong place for the LLVM and clang libraries, and we are not sure which specific library to point it to.

We have the following modules loaded (with versions):

OpenMPI/4.0.5-GCC-10.2.0
Clang/11.0.1-gcccuda-2020b
Python/3.8.6-GCCcore-10.2.0
LLVM/11.0.0-GCCcore-10.2.0
tbb/2020.3-GCCcore-10.2.0
Eigen/3.4.0-GCCcore-10.2.0
Cereal/1.3.2-GCCcore-10.2.0
CUDA/11.1.1-GCC-10.2.0
pybind11/2.6.2-GCCcore-10.2.0
CMake/3.22.1-GCCcore-10.2.0

(hoomd) [ia253@p09r07n24 lib]$ python

Python 3.8.6 (default, Feb 17 2021, 19:45:30)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hoomd
Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/__init__.py", line 73, in <module>
    from hoomd import hpmc
  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/hpmc/__init__.py", line 32, in <module>
    from hoomd.hpmc import pair
  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/hpmc/pair/__init__.py", line 11, in <module>
    from . import user
  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/hpmc/pair/user.py", line 18, in <module>
    from hoomd.hpmc import _jit
Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
Traceback (most recent call last):
/home/ia253/software/llvm -DLLVM_ENABLE_PROJECTS=clang \
        -DCLANG_LINK_CLANG_DYLIB=ON \
        -DLLVM_BUILD_LLVM_DYLIB=ON \
        -DLLVM_LINK_LLVM_DYLIB=ON \
        -DCMAKE_BUILD_TYPE=Release
cmake -S llvm -B build -DCMAKE_INSTALL_PREFIX=cmake -S llvm -B build -DCMAKE_INSTALL_PREFIX=
Then I try to do a fresh build of hoomd with:

cmake -B build/hoomd -S hoomd-blue  -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DENABLE_GPU=ON -DENABLE_MPI=ON -DENABLE_TBB=ON -DENABLE_LLVM=ON -DLLVM_DIR=/home/ia253/software/llvm/lib/cmake/llvm

    cmake -S llvm -B build \
        -D CMAKE_INSTALL_PREFIX=$ROOT -DLLVM_ENABLE_PROJECTS=clang \
        -DCLANG_LINK_CLANG_DYLIB=ON \
        -DLLVM_BUILD_LLVM_DYLIB=ON \
        -DLLVM_LINK_LLVM_DYLIB=ON \
        -DCMAKE_BUILD_TYPE=Release

Most LLVM installs, such as those on Ubuntu and conda-forge, provide these headers and libraries. Many HPC cluster modules do not, yours included. You will need to compile a compatible build of LLVM to use it with HOOMD-blue.

Also, I recommend you remove some of the CMake flags you set:
* `-DCMAKE_POSITION_INDEPENDENT_CODE=ON` is not necessary. CMake will automatically add -fPIC to those files that need it and leave it off for those that do not.
* `-DENABLE_MPI_CUDA=ON` will result in incorrect simulations when you run with more than one MPI rank on GPUs.
------
Joshua A. Anderson, Ph.D.
Research Area Specialist, Chemical Engineering, University of Michigan
On Apr 20, 2023, at 9:52 AM, Irem Altan <irem....@gmail.com> wrote:

We have that flag enabled, but it’s still not working. We are having a weird issue where the proper LLVM libraries are not detected. I am trying to do this on an HPC setting, with the HPC staff helping me, but we are running into a problem where cmake looks in the wrong place for the LLVM and clang libraries, and we are not sure which specific library to point it to.

We have the following modules loaded (with versions):

OpenMPI/4.0.5-GCC-10.2.0
Clang/11.0.1-gcccuda-2020b
Python/3.8.6-GCCcore-10.2.0
LLVM/11.0.0-GCCcore-10.2.0
tbb/2020.3-GCCcore-10.2.0
Eigen/3.4.0-GCCcore-10.2.0
Cereal/1.3.2-GCCcore-10.2.0
CUDA/11.1.1-GCC-10.2.0
pybind11/2.6.2-GCCcore-10.2.0
CMake/3.22.1-GCCcore-10.2.0

(hoomd) [ia253@p09r07n24 lib]$ python
Python 3.8.6 (default, Feb 17 2021, 19:45:30)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hoomd
Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/__init__.py", line 73, in <module>
    from hoomd import hpmc
  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/hpmc/__init__.py", line 32, in <module>
    from hoomd.hpmc import pair
  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/hpmc/pair/__init__.py", line 11, in <module>
    from . import user
  File "/home/ia253/project/venv/hoomd/lib/python3.8/site-packages/hoomd/hpmc/pair/user.py", line 18, in <module>
    from hoomd.hpmc import _jit
Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
Traceback (most recent call last):
-- 
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.

Irem Altan

unread,
Apr 26, 2023, 9:10:26 AMApr 26
to hoomd...@googlegroups.com
Ok, thank you! In case anyone else has a similar problem and wants to follow along, I did 

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/ia253/software/llvm/lib:/home/ia253/software/llvm/bin

Now, getting back to the original JIT compilation error on the GPU… I initially had tested this with v3.10, but now we are on v3.11. I’m getting an error message on both the CPU and GPU versions that I had not gotten before:

code.cc:39:29: error: invalid operands to binary expression ('const vec3<float>' and 'double')
vec3<float> r_hat_ij = r_ij / sqrt(dot(r_ij, r_ij));

Presumably the tutorial on the docs now need to be updated as well. I am getting the same error with the exact same script I had sent earlier in this chain of e-mails and that you had tested as well (albeit with v3.10).

The weird thing is, checking VectorMath.h, this operation should be valid, no? Not that this file has been updated between v3.10 and v3.11 anyway. I am attaching the full error message below. Also re-attaching the script for convenience.

Best,
Irem

jit_error.dat