Error in compiling nwchem-7.0.2 with Intel oneapi

[Vijay Mocherla]

Hi,

I am trying to compile a  tuned copy of nwchem-7.0.2 using intel's oneapi to run production level calculations on a workstation(a machine two intel xenon 22 core chips and 0.5 Tb memory).

I've cloned nwchem's github repository and use the following compile script :

```

# bash script to compile NWChem with intel compilers and MKL
export NWCHEM_TOP=/home/vijay/nwchem  
export NWCHEM_TARGET=LINUX64  
export NWCHEM_MODULES="all python"
export ARMCI_NETWORK=MPI-PR
# python variables
export PYTHONVERSION=3.9
export PYTHONHOME=/home/lib/intel/oneapi/intelpython
#export PYTHONLIBTYPE=so
#export USE_PYTHON64=y

# MPI variables
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/home/lib/intel/oneapi/mpi/latest
export MPI_LIB=$MPI_LOC/lib
export MPI_INCLUDE=$MPI_LOC/include
export LIBMPI="-lmpifort -lmpi"
# intel compiler and MKL variables
export CC=icc
export FC=ifort
export CXX=icpx
export LD_LIBRARY_PATH=/home/lib/intel/oneapi/compiler/2022.0.1/linux/lib
export MKLROOT=/home/lib/intel/oneapi/mkl/latest
export USE_SCALAPACK=y
export SCALAPACK="-L$MKLROOT/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_blacs_intelmpi_ilp64 -lpthread -lm -L/home/lib/intel/oneapi/clck/latest/lib/intel64"
export SCALAPACK_SIZE=8
export SCALAPACK_LIB="$SCALAPACK"
export BLAS_SIZE=8
export BLASOPT="-L$MKLROOT/lib/intel64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_core -liomp5 -lpthread -lm -L/home/lib/intel/oneapi/clck/latest/lib/intel64"
export LAPACK_LIB=$BLASOPT
export USE_NOIO=y

cd $NWCHEM_TOP/src
echo $PWD
make realclean
make nwchem_config NWCHEM_MODULES="all python"
make V=1 -j11 &> make.log

```

I try to compile  the above as :
```

nohup bash compile.sh &> err.log &

```

I have encountered some issues but, I could deal with most of them by creating symbolic links to relevant libraries within the oneapi folder , where the installer expects to find them. But, I have not been able to get around with the following error:
```

ld: /home/lib/intel/oneapi/compiler/2022.0.1/linux/bin/intel64/../../bin/intel64/../../lib/icx-lto.so: error loading plugin: libimf.so: cannot open shared object file: No such file or directory
make: *** [all] Error 1

```
I am including the detailed outputs in the attachments as err.log and make.log.

Can someone please help me with this error and also, maybe with better compile configurations? Would really appreciate any help and suggestions.

Thanks,

Vijay

[Edoardo Aprà]

It's not a good idea to set the MPI variables https://nwchemgit.github.io/Compiling-NWChem.html#automatic-detection-of-mpi-variables-with-mpif90

How do you set the oneapi configuration?

Are you using the following command?

source /opt/intel/oneapi/setvars.sh

[Daniel Mejia Rodriguez]

Vijay,

I do see another potential issue. Even when you source the oneapi setvars.sh file, you will still override the LD_LIBRARY_PATH with your current script (and it seems to point to the wrong directory)

I recommend setting oneapi configuration as Edo mentioned AND removing the line exporting the LD_LIBRARY_PATH. 

Best,

Daniel

--
You received this message because you are subscribed to the Google Groups "NWChem Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nwchem-forum...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nwchem-forum/81261cdf-61b5-4084-8650-1cbe71bde389n%40googlegroups.com.

[Vijay Mocherla]

Yes. I am using "setvars.sh". I am specifying MPI variables only to make sure that intel's MPI is used and not openMPI or mpich that are also installed.

I have had issues with them when I tried to use separate compile a copy with them and also when I was trying to use a copy of nwchem I installed using conda. (see for example: https://github.com/nwchemgit/nwchem/issues/463)

Without the LD_LIBRARY_PATH variable but with the MPI variables specified, I end up with following errors.(see for the make.log file attached in this email)

```

/apps/intel/oneapi/clck/latest/lib/intel64/libutil.so: undefined reference to `clck::fs::abspath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/apps/intel/oneapi/clck/latest/lib/intel64/libutil.so: undefined reference to `clck::find_cluster_definition(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<clck::ClusterDefinition>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<clck::ClusterDefinition> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
/apps/intel/oneapi/clck/latest/lib/intel64/libutil.so: undefined reference to `VTT for std::__cxx11::basic_stringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4.21'
/apps/intel/oneapi/clck/latest/lib/intel64/libutil.so: undefined reference to `GetEvaluationError(void*)'
/apps/intel/oneapi/clck/latest/lib/intel64/libutil.so: undefined reference to `clck::str::lower(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)'
/apps/intel/oneapi/clck/latest/lib/intel64/libutil.so: undefined reference to `clck::str::strip(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)'
/apps/intel/oneapi/clck/latest/lib/intel64/libutil.so: undefined reference to `clck::fs::file_exists(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/apps/intel/oneapi/clck/latest/lib/intel64/libutil.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)@GLIBCXX_3.4.21'
/apps/intel/oneapi/clck/latest/lib/intel64/libutil.so: undefined reference to `operator delete(void*, unsigned long)@CXXABI_1.3.9'
/apps/intel/oneapi/clck/latest/lib/intel64/libutil.so: undefined reference to `EnvGetInstanceName(void*, void*)'

make: *** [all] Error 1

```

Vijay

[Edoardo Aprà]

You environment does not seem ready for a NWChem compilation.

Could you sent the output of the following commands:

1.

mpif90 -show

2.

wget https://gist.githubusercontent.com/jcwright77/a5e1d66886bc17b0f7936466739cc287/raw/76cc1e095910d1733f260d953da6494e16e29b87/fpi_parallel.f

mpif90 fpi_parallel.f -o fpi.x

mpirun -np 2 fpi.x

[Vijay Mocherla]

Here's the output

1. $ mpif90 -show

gfortran -I"/apps/intel/oneapi/mpi/2021.4.0/include/gfortran/4.8.0" -I"/apps/intel/oneapi/mpi/2021.4.0/include" -L"/apps/intel/oneapi/mpi/2021.4.0/lib/release" -L"/apps/intel/oneapi/mpi/2021.4.0/lib" -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker "/apps/intel/oneapi/mpi/2021.4.0/lib/release" -Xlinker -rpath -Xlinker "/apps/intel/oneapi/mpi/2021.4.0/lib" -lmpifort -lmpi -lrt -lpthread -Wl,-z,now -Wl,-z,relro -Wl,-z,noexecstack -Xlinker --enable-new-dtags -ldl

2.  Output from  $ mpirun -np 2 ./fpi.x

 Process            1  of            2  is alive
 Process            0  of            2  is alive
Enter the number of intervals: (0 quits)
10
  pi is approximately: 3.1424259850010983  Error is: 0.0008333314113051
Enter the number of intervals: (0 quits)
100
  pi is approximately: 3.1416009869231241  Error is: 0.0000083333333309
Enter the number of intervals: (0 quits)

[Vijay Mocherla]

I tried to compile it WITHOUT specifying MPI variables and WITHOUT overriding LD_LIBRARY_PATH.

But, I am still not able to compile it. Please find the make.log file attached.

Thanks,

Vijay

On Thursday, March 10, 2022 at 6:38:34 AM UTC+5:30 Edoardo Aprà wrote:

[Daniel Mejia Rodriguez]

Is there a reason to set CXX=icpx ? Why not use icpc instead?

[Edoardo Aprà]

What Linux version have you been using? I am inclined to think that Intel OneAPI might not have been tested on all possible linux  platforms.

[Edoardo Aprà]

Continuing on the path spotted by Daniel Mejia .. why did you set /home/lib/intel/oneapi/clck/latest/lib/intel64 in the SCALAPACK settings? This seems to be causing the linking problem?

You can check the right settings for MKL with the Intel webpage https://www.intel.com/content/www/us/en/developer/tools/oneapi/onemkl-link-line-advisor.html

[Vijay Mocherla]

@Daniel Thanks for pointing that out. I've changed that to CXX=icpc. I just ended up using icpx out of a habit. Lately, I've been using icx and icpx to compile most C/C++ code. But, I guess icpx doesn't ensure backwards compatibility. So, ideally if one uses CC=icx and CXX=icpx instead of CC=icc and CXX=icpc, I'd expect to encounter a ton of errors.

@Edoardo The workstations on which I am trying install nwchem are running CentOS 7.

[Edoardo Aprà]

icpc  as C++ compiler on Centos 7 does not see a safe choice according to https://www.intel.com/content/www/us/en/developer/articles/system-requirements/intel-oneapi-base-toolkit-system-requirements.html

Since most of  the computing intensive part of NWChem is in Fortran, I would recommend that, even for the C compiler, to stick to gcc. In other words

CC=gcc

FC=ifort

Once you have a stable ifort installation, if you really want to see the effect of swapping icc for gcc, you might try that in a later stage.

[Vijay Mocherla]

I am still not able to compile a stable copy even after using CC=gcc and FC=ifort.

I get the same error as I got previously.

```
/home/lib/intel/oneapi/clck/2021.5.0/lib/intel64/libutil.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)@GLIBCXX_3.4.21'
/home/lib/intel/oneapi/clck/2021.5.0/lib/intel64/libutil.so: undefined reference to `operator delete(void*, unsigned long)@CXXABI_1.3.9'
/home/lib/intel/oneapi/clck/2021.5.0/lib/intel64/libutil.so: undefined reference to `EnvGetInstanceName(void*, void*)'

make: *** [all] Error 1

```

[Edoardo Aprà]

Did you change the settings for Scalapack?

[Vijay Mocherla]

Yes I did. Here's the new configuration that I have set up in compile script:

# bash script to compile NWChem with intel compilers and MKL
export NWCHEM_TOP=/home/vijay/nwchem
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all python"
export ARMCI_NETWORK=MPI-PR
# python variables
export PYTHONVERSION=3.9
export PYTHONHOME=/home/lib/intel/oneapi/intelpython
#export PYTHONLIBTYPE=so
#export USE_PYTHON64=y

# MPI variables
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y

# intel compiler and MKL variables

export CC=gcc
export FC=gfortran

export MKLROOT=/home/lib/intel/oneapi/mkl/latest
export USE_SCALAPACK=y
export SCALAPACK="-L$MKLROOT/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_blacs_intelmpi_ilp64 -lpthread -lm"

export SCALAPACK_SIZE=8
export SCALAPACK_LIB="$SCALAPACK"
export BLAS_SIZE=8
export BLASOPT="-L$MKLROOT/lib/intel64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_core -liomp5 -lpthread -lm"

export LAPACK_LIB=$BLASOPT
export USE_NOIO=y

cd $NWCHEM_TOP/src
echo $PWD
make realclean
make nwchem_config NWCHEM_MODULES="all python"
make V=1 -j11 &> make.log

~                           

To view this discussion on the web visit https://groups.google.com/d/msgid/nwchem-forum/ea4dedfc-390f-4a9c-891a-2b89e8726065n%40googlegroups.com.

[Edoardo Aprà]

Then, the only other fix I can think of is to avoid python (I have never managed to use the one shipped with OneAPI).

cd $NWCHEM_TOP/src

make nwchem_config NWCHEM_MODULES=all

make FC=ifort link

Could you post the output of these three commands?

[Vijay Mocherla]

I could finally compile a copy by excluding python with CC=icc and FC=ifort. Though I am able to run jobs, the build seems to fail almost all jobs in doqmtests.mpi. Please find the make.log and doqmtests.log in the attachments below.


Vijay

[Edoardo Aprà]

Not a single job in doqmtests.log has even started.

make.log shows that you set ARMCI_NETWORK=MPI-PR, therefore you need to set the -np option of mpirun to value greater than 1

https://nwchemgit.github.io/ARMCI.html

Before doing the full suite of tests from doqmtests.mpi (that you should be running with as doqmtests.mpi 2 to start 2 processes),

have you tried a simple test, say

cd $NWCHEM_TOP/src

mpirun  -np 2 ../bin/LINUX64/nwchem

[Vijay Mocherla]

Yes, I did and some of the calculations(scf, dft etc) went through, just fine. My bad, the wrong log file, apart from some of tests that need python, there are a couple of tests(for example see below) that fail, though I am completely sure why that's happening.

Running tests/h2o-b3lyp-disp/h2o-b3lyp-disp

     cleaning scratch
     copying input and verified output files
     running nwchem (/home/vijay/nwchem/bin/LINUX64/nwchem)  with 4 processors

     NWChem execution failed
    23: task dft freq
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------

 For more information see the NWChem manual at https://nwchemgit.github.io

 For further details see manual section:


[0] Received an Error in Communication: (-1) 0: ptsalloc: increase memory in input line:
Abort(-1) on node 0 (rank 0 in comm 496): application called MPI_Abort(comm=0x84000003, -1) - process 0
  ptsalloc: increase memory in input line                   0
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line :
     0:
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 For more information see the NWChem manual at https://nwchemgit.github.io


 For further details see manual section:

To view this discussion on the web visit https://groups.google.com/d/msgid/nwchem-forum/7b346cfe-fc2c-4ff5-8301-d54f68ed8d1an%40googlegroups.com.

[Edoardo Aprà]

You can try the following to fix this memory issue (this will generate a new NWChem executable):

cd $NWCHEM_TOP/src

../contrib/getmem.nwchem FC=ifort

[Vijay Mocherla]

That actually didn't work, and returns the following error :

  clustrfix.c:(.text+0x43): undefined reference to `_intel_fast_memset'
  /home/vijay/nwchem/lib/LINUX64/libpeigs.a(clustrxx5.o): In function `clustrinv5_':
  clustrxx5.c:(.text+0x43c): undefined reference to `_intel_fast_memset'
  clustrxx5.c:(.text+0x4b1): undefined reference to `_intel_fast_memset'
  /home/vijay/nwchem/lib/LINUX64/libpeigs.a(clustrxx5.o):clustrxx5.c:(.text+0xb37): more undefined references to `_intel_fast_memset' follow
  collect2: error: ld returned 1 exit status
  make: *** [link] Error 1

I've found a similar error on the intel forum (see: https://community.intel.com/t5/Intel-Fortran-Compiler/undefined-reference-to-intel-fast-memcpy/td-p/758815) and apparently recompiling with FC=ifort CXX=icc CC=icc  using following commands solves the problem:

  cd $NWCHEM_TOP/src
  echo $PWD
  make realclean
  make nwchem_config NWCHEM_MODULES="all"

  make V=1 FC=ifort CC=icc CXX=icc -j11 &> make.log

Most of the qm tests went through this, and the more lengthy ones are still running.

@Edoardo Aprà Thanks! a lot for all the help, really appreciate it.

Vijay

You received this message because you are subscribed to a topic in the Google Groups "NWChem Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nwchem-forum/dJDTUw-yIpE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nwchem-forum...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nwchem-forum/3fd51c21-84c2-41e9-aeb4-aeeb0fb47a0an%40googlegroups.com.