QMCPACK on Summit - deterministic tests failing

[Benjamin Avramidis]

Hello QMCPACK development team,

We have recently been able to use the summit build script found within the qmcpack/config folder within the 'develop' branch to build qmcpack on summit. I believe this build script runs fine as I do not get any major build errors. 

One thing to note is that any python submodules including pyscf, numpy, etc., are not available through summit to be loaded using 'module load' . 

I have tried building qmcpack on both my /home directory and /project directory within summit. While the build seems fine, even though python submodules are not found, attempts at running the deterministic tests fail. Only 2% of the deterministic tests pass. 

I have tried running the tests a few different ways. 1) Within my /home directory, 2) within my /project directory, and 3) using a conda environment where I have installed the python submodules (pyscf, numpy, etc).

In all three cases I still get that the deterministic tests fail. I should note that I have tried running the deterministic tests on the login node (no batch script), as well as using a batch script (using bsub and requesting # of nodes, etc). In all cases I still get that all the deterministic tests fail. 

Please provide any information that would be helpful in building and running qmcpack on summit. In particular, successfully running the deterministic tests after building qmcpack.

Thanks,

Ben

[Paul R. C. Kent]

Please report the error messages you get when trying to run from within a job. On Summit this is required. Other people have reported success so there is either something simple going on or a recent change on the machine. For testing, having numpy (<2.0) and h5py will enable the majority of them. If you are happy to use GitHub, please post an issue on the QMCPACK repository. It would also be helpful to know what you are trying to do with QMCPACK. e.g. Do you actually need PySCF?

[Ye Luo]

This message was hold up for a few days and Ben contacted me and we had a few email exchanges.

I'm able to narrow down the issue.

One of the necessary modules loaded for compilation llvm/18.1.6-latest

had a line

prepend_path("LD_LIBRARY_PATH","/sw/summit/ums/stf010/llvm/18.1.6-20240528/lib:/sw/summit/ums/stf010/llvm/18.1.6-20240528/lib/powerpc64le-unknown-linux-gnu:/sw/summit/ums/stf010/llvm/18.1.6-20240528/lib/clang/18.1.6/lib/linux:/usr/lib64:/sw/summit/cuda/10.1.243/lib64")

The added `/usr/lib64` causing loading the system libstdc++ which is out-dated.

The solution is the following. Only load minimal modules and set the path for the needed LLVM libraries manually. Here is my ctest script

```

#!/bin/bash
# Begin LSF directives
#BSUB -P med123
#BSUB -J test
#BSUB -o tst.o%J
#BSUB -W 30
#BSUB -nnodes 1
#BSUB -alloc_flags smt1
# End LSF directives and begin shell commands

module load cuda/12.2.0 gcc/12.2.0
LLVM_ROOT=/sw/summit/ums/stf010/llvm/18.1.6-20240528

export LD_LIBRARY_PATH=$LLVM_ROOT/lib:$LLVM_ROOT/lib/powerpc64le-unknown-linux-gnu:$LD_LIBRARY_PATH

export LIBOMP_USE_HIDDEN_HELPER_TASK=0
export LIBOMPTARGET_MAP_FORCE_ATOMIC=FALSE

ldd ./bin/qmcpack
ctest --output-on-failure -R deter -j32

```

Most tests should pass. There are a few tests complaining about CPU oversubscribing that you can ignore.

[Benjamin Avramidis]

Hello Ye,

Using the provided script I am able to get %5 of the deterministic tests passed, compared to %2 previously. 

Most notably, /bin/qmcpack cannot be found using 'ldd ./bin/qmcpack' in the batch script. I do not see a /qmcpack in /bin. 

I've attached the script used as-well as the output. You'll notice I run the tests within one of the build directories provided the Summit build script in QMCPack GitHub. 

Thank you,

Ben

[Ye Luo]

In your case when building with complex, the executable is named qmcpack_complex. Anyway, the LD_LIBRARY_PATH issue has been resolved based on the output you provided.

Most remaining errors are I/O related. You were running on /ccs/proj/chm209/Source/qmcpack/build_summit_Clang18_cpu_cplx

/ccs/proj is read-only on summit

https://docs.olcf.ornl.gov/data/index.html#summary-of-storage-areas

So use a "Work" space.

Ye