Compile with IBM Power 9 processors with XL compilers and ESSL library

Nathan Keilbart

unread,

Jun 30, 2021, 1:12:05 PM6/30/21

to NWChem Forum

Hello everyone,

I have been attempting to install NWChem version 7.0.2 on the Lassen server at LLNL which is similar to Summit at ORNL. My current configuration that I have working uses these settings:

export NWCHEM_TOP=/usr/workspace/keilbart/Lassen/nwchem-7.0.2

module load xl/2021.03.11
module load essl/6.3.0
module load spectrum-mpi/rolling-release

export NWCHEM_TOP=/usr/workspace/keilbart/Lassen/nwchem-7.0.2
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all"

export LARGE_FILES=TRUE
export USE_NOFSCHECK=TRUE
export ARMCI_NETWORK=MPI-PR
export USE_MPI=y
export USE_MPIF=y

export USE_INTERNALBLAS=y

export CC=xlc_r
export FC=xlf90_r

cd ${NWCHEM_TOP}/src
make nwchem_config -j
make all -j

This compilation does work fine using the internal blas libraries from NWChem but I am hoping to use XL and ESSL to compile. When I use these settings I get a compilation that finalizes:

module load xl/2021.03.11
module load essl/6.3.0
module load spectrum-mpi/rolling-release

export NWCHEM_TOP=/usr/workspace/keilbart/Lassen/nwchem-7.0.2_essl
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all"

export LARGE_FILES=TRUE
export USE_NOFSCHECK=TRUE
export ARMCI_NETWORK=MPI-PR
export USE_MPI=y
export USE_MPIF=y

export XLLIBDIR="/usr/tce/packages/xl/xl-2021.03.11/alllibs"
export BLASOPT="-L$ESSLLIBDIR64 -R$ESSLLIBDIR64 -lessl -llapackforessl -L$XLLIBDIR -R${XLLIBDIR} -lxlfmath -lxlf90_r -lm"
export LAPACK_LIB=$BLASOPT

export CC=xlc_r
export FC=xlf90_r

cd ${NWCHEM_TOP}/src
make nwchem_config -j
make all -j

When I attempt to run a calculation with the binary I get this message:

util_blas not ready 0
------------------------------------------------------------------------
------------------------------------------------------------------------
current input line :
0:
------------------------------------------------------------------------
------------------------------------------------------------------------
This error has not yet been assigned to a category
------------------------------------------------------------------------
For more information see the NWChem manual at https://github.com/nwchemgit/nwchem/wiki

For further details see manual section: No section for this category

From what I can tell, this is an issue that has been discussed https://github.com/nwchemgit/nwchem/issues/381 and resolved for intel compilers. Has anyone been able to successfully compile with these settings or should I open an issue on Github to discuss there?

Thank you for your time and consideration.

Nathan

Edoardo Aprà

unread,

Jun 30, 2021, 1:14:46 PM6/30/21

to NWChem Forum

Could you specify the origin of the NWChem source code you are using?

Was it one of the NWChem 7.0.2 tarballs from https://github.com/nwchemgit/nwchem/releases/tag/v7.0.2-release

or did you clone one of the branches of the github repository?

Nathan Keilbart

unread,

Jun 30, 2021, 1:17:53 PM6/30/21

to NWChem Forum

I cloned the master branch of 7.0.2 from github.

Edoardo Aprà

unread,

Jun 30, 2021, 1:21:14 PM6/30/21

to NWChem Forum

Thanks. That means you are using the master branch. I am assuming your source is in sync with the current status of the master branch.

Could you please send me the output of the following command

cd $NWCHEM_TOP/src/util

touch util_blasthreads.F

make FC=xlf_r V=1

Nathan Keilbart

unread,

Jun 30, 2021, 1:27:37 PM6/30/21

to NWChem Forum

I made sure to set the BLAS and LAPACK environmental variables as mentioned above and after executing the command got this:

xlf_r -c -q64 -qextname -qfixed -NQ40000 -NT80000 -qmaxmem=8192 -qxlf77=leadzero -qintsize=8 -O2 -g -I. -I/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/src/include -I/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/src/tools/install/include -DEXT_INT -DLINUX -DLINUX64 -DXLFLINUX -DCHKUNDFLW -DPARALLEL_DIAG -DNEED_LOC -DJOBTIMEPATH=\"/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/bin/LINUX64\" -DNWCHEM_LONG_PATHS -DNWCHEM_SRCDIR="\"/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/src\"" -DBYTE_SHORT_PACKING   -DBYTE_SHORT_PACKING    util_blasthreads.F
** util_blas_set_num_threads   === End of Compilation 1 ===
** util_blas_get_num_threads   === End of Compilation 2 ===
1501-510 Compilation successful for file util_blasthreads.F.
make[1]: Entering directory '/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/src/util'
make[1]: '/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/lib/LINUX64/libnwcutil.a' is up to date.
make[1]: Leaving directory '/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/src/util'
Making all in md5
make[1]: Entering directory '/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/src/util/md5'
make[2]: Entering directory '/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/src/util/md5'
make[2]: Leaving directory '/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/src/util/md5'
make[1]: Leaving directory '/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/src/util/md5'
make[1]: Entering directory '/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/src/util'
Got lock on /usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/lib/LINUX64/libnwcutil.lock
ar rU /usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/lib/LINUX64/libnwcutil.a util_blasthreads.o
echo /usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/lib/LINUX64/libnwcutil.a
/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/lib/LINUX64/libnwcutil.a
make[1]: Leaving directory '/usr/WS1/keilbart/Lassen/nwchem-7.0.2_essl/src/util'

I have everything in a script that I run so none of the other NWChem variables were set.

Edoardo Aprà

unread,

Jun 30, 2021, 2:08:18 PM6/30/21

to NWChem Forum

Could you report the output of the following command

env | grep BLAS

Nathan Keilbart

unread,

Jun 30, 2021, 2:15:47 PM6/30/21

to NWChem Forum

It only output my BLASOPT variable which is:

BLASOPT=-L/usr/tcetmp/packages/essl/essl-6.3.0/lib64 -R/usr/tcetmp/packages/essl/essl-6.3.0/lib64 -lessl -llapackforessl -L/usr/tce/packages/xl/xl-2021.03.11/alllibs -R/usr/tce/packages/xl/xl-2021.03.11/alllibs -lxlfmath -lxlf90_r -lm

Edoardo Aprà

unread,

Jun 30, 2021, 2:44:42 PM6/30/21

to NWChem Forum

I have just committed a fix for this issue

Please update you tree and recompile

cd $NWCHEM_TOP/src

git pull

cd util

make FC=xlf_r

cd ..

make FC=xlf_r link

Thanks for reporting this issue

Nathan Keilbart

unread,

Jun 30, 2021, 4:26:14 PM6/30/21

to NWChem Forum

Thank you Edoardo for helping me through this. I was able to recompile successfully with the settings I had and now simulations are starting. I am however receiving a new message during the dft iterations which states:

Singularity in Pulay matrix. Error and Fock matrices removed.

After looking around it's suggested it has to do with something with the compilation. When I run the exact same structure with the internal BLAS enabled it runs through the exact input file with no issues so I imagine it's something with my BLAS or LAPACK settings. Once again, those are:

export XLLIBDIR="/usr/tce/packages/xl/xl-2021.03.11/alllibs"
export BLASOPT="-L$ESSLLIBDIR64 -R$ESSLLIBDIR64 -lessl -llapackforessl -L$XLLIBDIR -R${XLLIBDIR} -lxlfmath -lxlf90_r -lm"
export LAPACK_LIB=$BLASOPT

I will attempt to recompile without some of the flags but the xl flags were suggested by the documentation for Lassen. Would you have any general suggestions for which libraries to use with ESSL? Thanks.

Edoardo Aprà

unread,

Jun 30, 2021, 5:57:32 PM6/30/21

to NWChem Forum

You need to set the following env. variables

export BLAS_SIZE=4

export USE_64TO32=y

and perform the following compilations steps

cd $NWCHEM_TOP/src

make 64_to_32

rm -rf tools/build tools/install

make FC=xlf_r

https://nwchemgit.github.io/Compiling-NWChem.html#how-to-deal-with-integer-size-of-linear-algebra-libraries

Nathan Keilbart

unread,

Jun 30, 2021, 7:23:01 PM6/30/21

to NWChem Forum

Once again, thank you for the assistance Edoardo. These settings allowed me to correctly compile the binary and the simulation appears to give similar responses to an intel version we have compiled.

In case this will help others in the future, these are the settings I have:

module load xl/2021.03.11
module load essl/6.3.0
module load spectrum-mpi/rolling-release

export NWCHEM_TOP=`pwd`

export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all"
export LARGE_FILES=TRUE
export USE_NOFSCHECK=TRUE
export ARMCI_NETWORK=MPI-PR
export USE_MPI=y
export USE_MPIF=y
export XLLIBDIR="/usr/tce/packages/xl/xl-2021.03.11/alllibs"
export BLASOPT="-L$ESSLLIBDIR64 -R$ESSLLIBDIR64 -lessl -llapackforessl -L$XLLIBDIR -R${XLLIBDIR} -lxlfmath -lxlf90_r -lm"
export LAPACK_LIB=$BLASOPT

export BLAS_SIZE=4
export USE_64TO32=y

export CC=xlc_r
export FC=xlf90_r

cd ${NWCHEM_TOP}/src

make 64_to_32

make nwchem_config -j
make all -j

Reply all

Reply to author

Forward