NWChem compilation issue for version 7.0.2

Zhiyong Zhang

unread,

Apr 15, 2021, 2:19:06 AM4/15/21

to NWChem Forum

I am compiling NWChem with the most recent git repo. I’ve made a clean clone of the current repo as of today. When I compiled the code I got the following error at the link stage. I am using The intel/2017xxx and I haven’t noticed any other compilation errors.

ifort -i8 -align -fpp -qopenmp -fimf-arch-consistency=true -align array64byte -O2 -g -fp-model source -L/work/02726/zhiyongz/stampede2/zhiyongz/nwchem/lib/LINUX64 -L/work/02726/zhiyongz/stampede2/zhiyongz/nwchem/src/tools/install/lib -o /work/02726/zhiyongz/stampede2/zhiyongz/nwchem/bin/LINUX64/nwchem nwchem.o stubs.o -lnwctask -lccsd -lmcscf -lselci -lmp2 -lmoints -lstepper -ldriver -loptim -lnwdft -lgradients -lcphf -lesp -lddscf -ldangchang -lguess -lhessian -lvib -lnwcutil -lrimp2 -lproperty -lsolvation -lnwints -lprepar -lnwmd -lnwpw -lofpw -lpaw -lpspw -lband -lnwpwlib -lcafe -lspace -lanalyze -lqhop -lpfft -ldplot -lnwpython -ldrdy -lvscf -lqmmm -lqmd -letrans -lpspw -ltce -lbq -lmm -lcons -lperfm -ldntmc -lccca -ldimqm -lfcidump -lnwcutil -lga -larmci -lpeigs -lperfm -lcons -lbq -lnwcutil -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -lpthread -lm -ldl -l64to32 -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -lpthread -lm -ldl -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -lpthread -lm -ldl -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/lib/release_mt -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/lib -lmpifort -lmpi -lmpigi -ldl -lrt -lpthread -lcomex -lmpifort -lmpi -lmpigi -ldl -lrt -lpthread -lrt -lpthread -lpthread -lnwcutil -lpython2.7 -ldl -lpthread -limf -lirc -lssp -lutil -limf -Xlinker -export-dynamic

/work/02726/zhiyongz/stampede2/zhiyongz/nwchem/src/tools/install/lib/libga.a(capi.o): In function `GA_Symmetrize':

capi.c:(.text+0x12394): undefined reference to `pnga_symmetrize'

/work/02726/zhiyongz/stampede2/zhiyongz/nwchem/src/tools/install/lib/libga.a(capi.o): In function `NGA_Symmetrize':

capi.c:(.text+0x123a4): undefined reference to `pnga_symmetrize'

/work/02726/zhiyongz/stampede2/zhiyongz/nwchem/src/tools/install/lib/libga.a(fapi.o): In function `ga_symmetrize_':

fapi.c:(.text+0x5194): undefined reference to `pnga_symmetrize'

/work/02726/zhiyongz/stampede2/zhiyongz/nwchem/src/tools/install/lib/libga.a(fapi.o): In function `nga_symmetrize_':

fapi.c:(.text+0x51a4): undefined reference to `pnga_symmetrize'

make: *** [all] Error 1

The following is how I compiled the code.

997 export NWCHEM_TOP=$PWD/nwchem

998 export NWCHEM_MODULES="all python"

999 export NWCHEM_TARGET="LINUX64"

1000 export TARGET="LINUX64"

1001 export ARMCI_NETWORK="MPI-PR"

1002 export USE_MPI="y"

1003 export LARGE_FILES="TRUE"

1004 export LIB_DEFINES="-DDFLT_TOT_MEM=16777216"

1005 export USE_NOIO="TRUE"

1006 export USE_NOFSCHECK="TRUE"

1007 export MA_USE_ARMCI_MEM="1"

1008 export USE_OPENMP="1"

1010 export MRCC_METHODS="TRUE"

1011 export PYTHONHOME=/opt/apps/intel17/python2/2.7.14/

1012 export PYTHONVERSION=2.7

1013 export USE_PYTHON="y"

1014 export USE_PYTHONCONFIG=Y

1015 export USE_64TO32=y

1016 export USE_SCALAPACK=y

1017 export BLASOPT=" -L${MKLROOT}/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -lpthread -lm -ldl"

1018 export LAPACK_LIB=" -L${MKLROOT}/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -lpthread -lm -ldl"

1019 export SCALAPACK=" -L${MKLROOT}/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -lpthread -lm -ldl"

1020 export BLAS_SIZE=8

1021 export SCALAPACK_SIZE=8

1022 export USE_KNL=y

1029 sed -i 's/-xMIC-AVX512/-xCORE-AVX2 -axCORE-AVX512,MIC-AVX512/g' $NWCHEM_TOP/src/config/makefile.h

1030 cd $NWCHEM_TOP/src

1031 make nwchem_config

1032 make FC=ifort CC=icc >& make.log &

Edoardo Aprà

unread,

Apr 15, 2021, 2:26:31 AM4/15/21

to NWChem Forum

What is the output of

ls -lrt $NWCHEM_TOP/src/tools

nm $NWCHEM_TOP/src/tools/install/lib/libga.a | grep -i pnga_symme

nm $NWCHEM_TOP/src/tools/build/global/src/ga_symmetr.o | grep -i ga_symme

Zhiyong Zhang

unread,

Apr 15, 2021, 12:05:39 PM4/15/21

to NWChem Forum

login1.stampede2(1056)$ ls -lrt $NWCHEM_TOP/src/tools

total 8340

drwx------ 22 zhiyongz G-813768 4096 Nov 5 22:15 ga-5.8

-rw------- 1 zhiyongz G-813768 162 Apr 14 19:12 README

-rwx------ 1 zhiyongz G-813768 1095 Apr 14 19:12 nxtval.sh

-rwx------ 1 zhiyongz G-813768 494 Apr 14 19:12 msmpi_patch.sh

-rwx------ 1 zhiyongz G-813768 22229 Apr 14 19:12 mingw64_patch.sh

-rwx------ 1 zhiyongz G-813768 280 Apr 14 19:12 makewin98.bat

-rw------- 1 zhiyongz G-813768 343 Apr 14 19:12 MakeFile

-rwx------ 1 zhiyongz G-813768 2995 Apr 14 19:12 install-armci-mpi

-rwx------ 1 zhiyongz G-813768 6453 Apr 14 19:12 guess-mpidefs

-rw------- 1 zhiyongz G-813768 20812 Apr 14 19:12 GNUmakefile

-rwx------ 1 zhiyongz G-813768 2082 Apr 14 19:12 get-tools-remote-expect

-rwx------ 1 zhiyongz G-813768 5240 Apr 14 19:12 get-tools-github

-rwx------ 1 zhiyongz G-813768 1273 Apr 14 19:12 get-tools-expect

-rwx------ 1 zhiyongz G-813768 108 Apr 14 19:12 gettools.bat

-rwx------ 1 zhiyongz G-813768 2689 Apr 14 19:12 get-tools

-rwx------ 1 zhiyongz G-813768 532 Apr 14 19:12 cvs-tools

-rw------- 1 zhiyongz G-813768 8403547 Apr 14 19:20 ga-5.8.tar.gz

-rw------- 1 zhiyongz G-813768 1 Apr 14 19:20 ifort.FC.stamp

-rw------- 1 zhiyongz G-813768 1 Apr 14 19:20 icc.CC.stamp

-rw------- 1 zhiyongz G-813768 1 Apr 14 19:20 ga-5.8.MPI-PR.DIR.stamp

drwx------ 15 zhiyongz G-813768 4096 Apr 14 19:47 build

drwx------ 5 zhiyongz G-813768 4096 Apr 14 19:47 install

login1.stampede2(1057)$ nm $NWCHEM_TOP/src/tools/install/lib/libga.a | grep -i pnga_symme

U pnga_symmetrize

login1.stampede2(1058)$ nm $NWCHEM_TOP/src/tools/build/global/src/ga_symmetr.o | grep -i ga_symme

From: nwchem...@googlegroups.com <nwchem...@googlegroups.com> on behalf of Edoardo Aprà <edoard...@gmail.com>
Sent: Wednesday, April 14, 2021 11:26 PM
To: NWChem Forum <nwchem...@googlegroups.com>
Subject: [nwchem-forum] Re: NWChem compilation issue for version 7.0.2

--
You received this message because you are subscribed to a topic in the Google Groups "NWChem Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nwchem-forum/nO8mZ3BDj6Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nwchem-forum...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nwchem-forum/de5f71fd-b694-4bd9-a9dc-21f92e73f580n%40googlegroups.com.

Zhiyong Zhang

unread,

Apr 15, 2021, 12:05:43 PM4/15/21

to NWChem Forum

login1.stampede2(1058)$ nm $NWCHEM_TOP/src/tools/build/global/src/ga_symmetr.o | grep -i ga_symme

Edoardo Aprà

unread,

Apr 15, 2021, 12:08:02 PM4/15/21

to NWChem Forum

Please post $NWCHEM_TOP/src/tools/build/config.log after renaming it to config.log.txt

What is the output of

cd $NWCHEM_TOP/src/tools/build

touch ../ga-*/global/src/ga_symmetr.c

make

nm global/src/ga_symmetr.o

Zhiyong Zhang

unread,

Apr 15, 2021, 5:44:52 PM4/15/21

to NWChem Forum

1075 cd $NWCHEM_TOP/src/tools/build

1076 touch ../ga-*/global/src/ga_symmetr.c

1077 make CC=icc FC=ifort

login3.stampede2(1081)$ nm global/src/ga_symmetr.o

login3.stampede2(1082)$ ls -l global/src/ga_symmetr.o

-rw------- 1 zhiyongz G-813768 720 Apr 15 15:32 global/src/ga_symmetr.o

ga_symmetric.o exists but nm shows nothing. Should it be compiled with gcc instead of icc. note that I have been setting the c compiler to icc.

Please see the attached log file for config.

config.log.txt

Edoardo Aprà

unread,

Apr 15, 2021, 5:47:33 PM4/15/21

to NWChem Forum

Please report the complete output of the commands (especially the output of the make command)

cd $NWCHEM_TOP/src/tools/build

touch ../ga-*/global/src/ga_symmetr.c

make

Zhiyong Zhang

unread,

Apr 15, 2021, 8:13:08 PM4/15/21

to NWChem Forum

Please see attached

make.log.src.tools.build.txt

Edoardo Aprà

unread,

Apr 15, 2021, 8:59:14 PM4/15/21

to NWChem Forum

Please report the output of the following two sequences of commands

1.

cd $NWCHEM_TOP/src/tools/build

icc -DHAVE_CONFIG_H -I. -I../ga-5.8 -Ima -I../ga-5.8/ma -I../ga-5.8/LinAlg/lapack+blas -Iglobal/src -I../ga-5.8/global/src -I../ga-5.8/global/testing -I../ga-5.8/pario/dra -I../ga-5.8/pario/eaf -I../ga-5.8/pario/elio -I../ga-5.8/pario/sf -Igaf2c -I../ga-5.8/gaf2c -I../ga-5.8/tcgmsg -I../ga-5.8/tcgmsg/tcgmsg-mpi -I../ga-5.8/comex/src-armci -no-vec -MT global/src/ga_symmetr.lo -MD -MP -MF global/src/.deps/ga_symmetr.Tpo -c ../ga-5.8/global/src/ga_symmetr.c -o global/src/ga_symmetr.o

nm global/src/ga_symmetr.o

2.

cd $NWCHEM_TOP/src/tools/build

gcc -DHAVE_CONFIG_H -I. -I../ga-5.8 -Ima -I../ga-5.8/ma -I../ga-5.8/LinAlg/lapack+blas -Iglobal/src -I../ga-5.8/global/src -I../ga-5.8/global/testing -I../ga-5.8/pario/dra -I../ga-5.8/pario/eaf -I../ga-5.8/pario/elio -I../ga-5.8/pario/sf -Igaf2c -I../ga-5.8/gaf2c -I../ga-5.8/tcgmsg -I../ga-5.8/tcgmsg/tcgmsg-mpi -I../ga-5.8/comex/src-armci -MT global/src/ga_symmetr.lo -MD -MP -MF global/src/.deps/ga_symmetr.Tpo -c ../ga-5.8/global/src/ga_symmetr.c -o global/src/ga_symmetr.o

nm global/src/ga_symmetr.o

Zhiyong Zhang

unread,

Apr 15, 2021, 9:19:18 PM4/15/21

to NWChem Forum

Both returned nothing.

Edoardo Aprà

unread,

Apr 15, 2021, 9:38:05 PM4/15/21

to NWChem Forum

Did you check the content of ga_symmetr.c?

Is it an empty file?

ls -l $NWCHEM_TOP/src/tools/ga-5.8/global/src/ga_symmetr.c

cat $NWCHEM_TOP/src/tools/ga-5.8/global/src/ga_symmetr.c

Zhiyong Zhang

unread,

Apr 16, 2021, 12:29:45 AM4/16/21

to NWChem Forum

It is empty! I was checking the ga_symmetr.o file and that was not empty but a small file. It did seem a little strange that it is a small file. It never occurred to me that ga_symmetr.c is empty. I didn't know it is possible to compile an empty file to an object file without error. Thanks Edo! Very much appreciate it.

Zhiyong Zhang

unread,

Apr 16, 2021, 12:32:32 AM4/16/21

to NWChem Forum

How should I update the ga_symmetr.c file from the git repo? Is it a file in the build directory or is the build directory generated during compilations?

Zhiyong Zhang

unread,

Apr 16, 2021, 12:39:16 AM4/16/21

to NWChem Forum

git diff src/tools/ga-5.8/global/src/ga_symmetr.c is empty. Is that file in the repo also empty?

Zhiyong Zhang

unread,

Apr 16, 2021, 1:00:09 AM4/16/21

to NWChem Forum

This is the part of the make log for get-tools for GA:

No Global Arrays found: Attempting running ./get-tools

./get-tools-github

configure present. no autogen.sh required

touch ga-5.8/armci/Makefile.in

cd build/armci && /bin/sh ./config.status

config.status: creating Makefile

config.status: creating tools/armci-config

config.status: creating config.h

config.status: config.h is unchanged

config.status: executing depfiles commands

config.status: executing libtool commands

Doesn't look like there is anything unusual. Perhaps there is some issue with the GA gitbub? Or could it be that there are data losses during the ./get-tools-github process? How do I check that?

Edoardo Aprà

unread,

Apr 16, 2021, 2:06:18 AM4/16/21

to nwchem...@googlegroups.com

It is not supposed to be empty.

--

You received this message because you are subscribed to a topic in the Google Groups "NWChem Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nwchem-forum/nO8mZ3BDj6Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nwchem-forum...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nwchem-forum/262e3f80-5648-4c6c-88ec-b5c4d3650abdn%40googlegroups.com.

Edoardo Aprà

unread,

Apr 16, 2021, 2:10:11 AM4/16/21

to nwchem...@googlegroups.com

cd $NWCHEM_TOP/src/tools

rm -rf ga* build install

make FC=ifort

cd ..

--

You received this message because you are subscribed to a topic in the Google Groups "NWChem Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nwchem-forum/nO8mZ3BDj6Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nwchem-forum...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nwchem-forum/bfdc8c98-f6c5-45e0-a58f-a26cfb80d43an%40googlegroups.com.

Zhiyong Zhang

unread,

Apr 16, 2021, 2:34:11 AM4/16/21

to NWChem Forum

It is no longer empty. I'll see if it works this time.

ls -l ga-5.8/global/src/ga_symmetr.c

-rw------- 1 zhiyongz G-813768 3318 Nov 2 12:36 ga-5.8/global/src/ga_symmetr.c

Zhiyong Zhang

unread,

Apr 17, 2021, 1:39:26 AM4/17/21

to NWChem Forum

The compilation was successful but I am still getting this error for QA/tests/autosym

NWChem CPHF Module

------------------

scftype = RHF

nclosed = 5

nopen = 0

variables = 20

# of vectors = 15

tolerance = 0.10D-03

level shift = 0.00D+00

max iterations = 50

max subspace = 150

c407-084.stampede2.tacc.utexas.edu.131974Received eager message(s) ptype=0x1 opcode=0xcc from an unknown process (err=49)

SCF residual: 9.338965333721499E-011

Iterative solution of linear equations

No. of variables 20

No. of equations 15

Maximum subspace 150

Iterations 50

Convergence 1.0D-04

Start time 4.3

iter nsub residual time

---- ------ -------- ---------

1 15 7.32D-02 4.4

------------------------------------------------------------------------

gasvd:d: failed 47283294961664

------------------------------------------------------------------------

current input line :

1463: task scf freq

------------------------------------------------------------------------

For more information see the NWChem manual at

https://github.com/nwchemgit/nwchem/wiki

For further details see manual section:

Zhiyong Zhang

unread,

Apr 17, 2021, 2:36:10 AM4/17/21

to NWChem Forum

The QA/tests/h2o_opt also failed:

Non-variational initial energy

------------------------------

Total energy = -75.502009

1-e energy = -118.566474

2-e energy = 35.736228

HOMO = -0.415525

LUMO = 0.027881

Starting SCF solution at 1.1s

----------------------------------------------

Quadratically convergent ROHF

Convergence threshold : 1.000E-04

Maximum no. of iterations : 30

Final Fock-matrix accuracy: 1.000E-07

----------------------------------------------

iter energy gnorm gmax time

----- ------------------- --------- --------- --------

1 -75.8791657860 8.72D-01 3.99D-01 0.5

c465-013.stampede2.tacc.utexas.edu.158233Received eager message(s) ptype=0x1 opcode=0xcc from an unknown process (err=49)

c443-063.stampede2.tacc.utexas.edu.8313Received eager message(s) ptype=0x1 opcode=0xcc from an unknown process (err=49)

------------------------------------------------------------------------

ga_iter_lsolve: dgesv failed 47918950121472

------------------------------------------------------------------------

current input line :

66: task scf optimize

------------------------------------------------------------------------

For more information see the NWChem manual at

https://github.com/nwchemgit/nwchem/wiki

For further details see manual section:

ga_iter_lsolve: dgesv failed 47949014892544

------------------------------------------------------------------------

current input line :

0:

[0] Received an Error in Communication: (0) 0:ga_iter_lsolve: dgesv failed:

------------------------------------------------------------------------

For more information see the NWChem manual at

https://github.com/nwchemgit/nwchem/wiki

For further details see manual section:

Edoardo Apra

unread,

Apr 17, 2021, 1:45:43 PM4/17/21

to nwchem...@googlegroups.com

Please go back to the two github issues you have filed and read carefully what you were suggested to do.

It does not make sense to repeat the same story for a third time .

In brief

ILP64=BLAS(SCALAPACK)_SIZE=8

LP64= BLAS(SCALAPACK)_SIZE=4

Please look carefully at your settings. You have been using MKL ILP64 and LP64 at the same time.

If you rather want to try a more automated way to build blas/lapack/scalapack, there is a mechanism now available in the master branch

You received this message because you are subscribed to the Google Groups "NWChem Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nwchem-forum...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nwchem-forum/6306a1a1-a42c-455f-8787-9f1402058354n%40googlegroups.com.

Zhiyong Zhang

unread,

Apr 17, 2021, 4:36:54 PM4/17/21

to NWChem Forum

I think I copied your suggestions from the github issue:

export BLASOPT=" -L${MKLROOT}/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -lpthread -lm -ldl"

export LAPACK_LIB=" -L${MKLROOT}/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -lpthread -lm -ldl"

export SCALAPACK=" -L${MKLROOT}/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -lpthread -lm -ldl"

Looks like there was a typo there. I've corrected that and linked against the correct libraries and looks like it is working now.

Thanks Edo for the help. I appreciate it.

Zhiyong Zhang

unread,

Apr 18, 2021, 2:03:59 AM4/18/21

to NWChem Forum

There are a number of failed QA tests from the pspw packages. Output of one of the failed calculation is attached. After a couple of iterations, there is an out of stack memory error. Also there are bad grid message at some point. Not sure if that is an error associated with the calculation though.

============ Grassmann lmbfgs iteration ============

>>> ITERATION STARTED AT Sat Apr 17 22:24:39 2021 <<<

iter. Energy DeltaE DeltaRho

------------------------------------------------------

------------------------------------------------------------------------

out of stack memory 0

------------------------------------------------------------------------

current input line :

29: task pspw energy ignore

------------------------------------------------------------------------

The output shows nproc = 2 but the calculation is involked with mpirun -n 4 -ppn 2 -print-rank-map -host $SLURM_NODELIST and the -print-rank-map does produce the expected binding of processes to the two nodes.. Is nproc actually the number of nodes in the output file?

I compiled it from the git clone of the most recent nwchem repo. According to the release note, it is 7.0.2 but nwchem output shows 7.0.1 though. The release notes shows ga is v5.7.2 but looks like the clone has ga 5.8 instead. Could that be an issue?

Thanks!

Zhiyong

Zhiyong Zhang

unread,

Apr 18, 2021, 2:10:19 AM4/18/21

to NWChem Forum

Please see the attached output file

pspw_becke97_h2o.out.txt

Reply all

Reply to author

Forward