Optimizing NWChem 7.0.0 build

[Kevin Basemann]

I am working with a installation of nwchem that does not seem to be running as efficiently as I would normally expect given the hardware I am working with. 

The cluster I am working with is mostly using Xeon E5-2669A v4 2.40 GHz nodes dual socket 22 core/socket, with 512GB RAM.

The build made by the local research computing team uses the following setup for building nwchem. I've already built out my own version of this using the ARMCI_NETWORK=MPI-PR rather than OPENIB and includes additional variables like BLAS_SIZE and SCALAPACK. 

export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export ARMCI_NETWORK=OPENIB
export NWCHEM_TARGET=LINUX64
export MPI_LOC=/usr/mpi/intel/mvapich2-2.3a
export MPI_LIB=$MPI_LOC/lib
export MPI_INCLUDE=$MPI_LOC/include
export LIBMPI="-lmpifort -lmpi"
export MKL64=/nas/longleaf/apps/intel/17.2/intel/compilers_and_libraries_2017/linux/mkl/lib/intel64
export BLASOPT="-L${MKL64} -Wl,--start-group -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core -Wl,--end-group"
export LAPACK_LIB=-lmkl_lapack95_lp64

cd $NWCHEM_TOP/src

make nwchem_config

make FC=ifort CC=icc >& make.log &

[Edoardo Aprà]

Could you quantify with a detailed example of the slow-downs you are observing? What is the baseline you are comparing with?

I must confess I rarely use mvapich. I would suggest you to use OpenMPI, instead

PS Using Scalapack might improve DFT calculations. However, MKL Scalapack does not provide mvapich binding (another reason for switching to OpenMPI)

[jeff.science]

The MKL ScaLAPACK library built against Intel MPI will work for all MPICH derivatives.  MVAPICH doesn't officially claim to be part of https://www.mpich.org/abi/ but one can diff mpi.h to see that MPICH and MVAPICH2 have the same ABI, and MPICH is ABI compatible with Intel MPI since at least 2014.  Intel also verified that the MKL ScaLAPACK built against Intel MPI works with Cray MPI on NERSC machines when I was there.

It doesn't matter a huge amount with Broadwell, but I recommend downloading the latest version of MKL, which is freely available as part of oneAPI, and can be installed from the command-line in $HOME without administrative privileges (https://software.intel.com/content/www/us/en/develop/documentation/installation-guide-for-intel-oneapi-toolkits-linux/top/installation/install-with-command-line.html). 

In my experience, with MPI-PR, there are small differences between Intel MPI, MVAPICH2 and Open-MPI on InfiniBand, for the types of communication NWChem does.  There are larger differences for RMA, but that isn't relevant unless one uses a different ARMCI configuration, and those are known to be similar to MPI-PR in most cases (and much worse in others).

Jeff

[Kevin Basemann]

Thank you for the advice. After getting my own local version compiled with MPI-PR and taking your advice and updating to the latest MKL and swapping mvapich for openmpi and I think my issue is resolved. 

Dr. Apra to your question, I did not have a good comparison off hand just experience with the time for calculations of similar scope and size on somewhat similar hardware.  I did a comparison between my install and the one using the parameters in the my original post. It was done on a DFT optimization and Hessian calculation of a closed shell molecule using B3LYP, 345 basis functions. With the install provided by the computing team here it had a wall time of 13597.3 and with the changes suggested its down to 1088.1s.