error with gnu toolchain , gnu compilers, Thread number has changed

[Ronald Cohen]

This is for the release 6.1 tarball .The error occurs in  dbcsr/dist/dbcsr_dist_method. I have rebuilt many times using different option but always get this

with the local.psmp arch. The code riuns fine with

export OMP_NUM_THREADS=1

but fails for >1 with:

*******************************************************************************
 *   ___                                                                       *
 *  /   \                                                                      *
 * [ABORT]                                                                     *
 *  \___/                        Thread number has changed                     *
 *    |                                                                        *
 *  O/|                                                                        *
 * /| |                                                                        *
 * / \                                     dbcsr/dist/dbcsr_dist_methods.F:515 *
 *******************************************************************************


 ===== Routine Calling Stack =====

           10 dbcsr_create_new
            9 build_overlap_matrix
            8 build_core_hamiltonian_matrix
            7 qs_energies_init_hamiltonians
            6 qs_energies
            5 qs_forces
            4 cp_eval_at
            3 geoopt_bfgs
            2 cp_geo_opt
            1 CP2K

It fails whether I use elpa or not.

My toolchain command is:

./install_cp2k_toolchain.sh --with-cmake=install --with-libint=install  --with-libxc=install --with-elpa=/home/rcohen/gnu --enable-cuda=no --enable-omp=yes --with-gcc=install --with-binutils=install

and local.psmp is:

C          = /home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/gcc-7.3.0/bin/gcc
CXX         = /home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/gcc-7.3.0/bin/g++
CPP         =
AR          = ar -r
FC          = mpif90
LD          = mpif90
#
DFLAGS      = -D__LIBXSMM  -D__parallel  -D__FFTW3  -D__LIBINT -D__LIBINT_MAX_AM=6 -D__LIBDERIV_MAX_AM1=5 -D__LIBXC -D__SCALAPACK -D__ELPA=201705 
#
WFLAGS      = -Werror=aliasing -Werror=ampersand -Werror=c-binding-type -Werror=intrinsic-shadow -Werror=intrinsics-std -Werror=line-truncation -Werror=tabs -Werror=realloc-lhs-all -Werror=target-lifetime -Werror=underflow -Werror=unused-but-set-variable -Werror=unused-variable -Werror=unused-dummy-argument -Werror=conversion -Werror=zerotrip -Werror=uninitialized -Wno-maybe-uninitialized -Wuse-without-only
#
FCDEBFLAGS  = -ffree-form -std=f2003 -fimplicit-none
CFLAGS      = -march=knl -fno-omit-frame-pointer -g  -O3 -funroll-loops -ffast-math $(PROFOPT) -fopenmp -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/mpich-3.2.1/include'  -m64 -I/mnt/beegfs/intel/compilers_and_libraries_2019.4.243/linux/mkl/include -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/fftw-3.3.7/include' -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libint-1.1.6/include' -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libxc-4.0.3/include' -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libxsmm-1.9.0/include' -I'/home/rcohen/gnu/include/elpa_openmp-2018.11.001/modules' -I'/home/rcohen/gnu/include/elpa_openmp-2018.11.001/elpa' $(DFLAGS)
FCFLAGS     = -march=knl -fno-omit-frame-pointer -g  -O3 -funroll-loops -ffast-math $(PROFOPT) -fopenmp -I${MKLROOT}/include/intel64/lp64 -m64 -I${MKLROOT}/include -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/mpich-3.2.1/include'  -m64 -I/mnt/beegfs/intel/compilers_and_libraries_2019.4.243/linux/mkl/include -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/fftw-3.3.7/include' -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libint-1.1.6/include' -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libxc-4.0.3/include' -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libxsmm-1.9.0/include' -I'/home/rcohen/gnu/include/elpa_openmp2018.11.001/elpa''

FCDEBFLAGS  = -ffree-form -std=f2003 -fimplicit-none
CFLAGS      = -march=knl -fno-omit-frame-pointer -g  -O3 -funroll-loops -ffast-math $(PROFOPT) -fopenmp -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/mpich-3.2.1/include'  -m64 -I/mnt/beegfs/intel/compilers_and_libraries_2019.4.243/linux/mkl/include -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/fftw-3.3.7/include' -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libint-1.1.6/include' -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libxc-4.0.3/include' -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libxsmm-1.9.0/include' -I'/home/rcohen/gnu/include/elpa_openmp-2018.11.001/modules' -I'/home/rcohen/gnu/include/elpa_openmp-2018.11.001/elpa' $(DFLAGS)
FCFLAGS     = -march=knl -fno-omit-frame-pointer -g  -O3 -funroll-loops -ffast-math $(PROFOPT) -fopenmp -I${MKLROOT}/include/intel64/lp64 -m64 -I${MKLROOT}/include -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/mpich-3.2.1/include'  -m64 -I/mnt/beegfs/intel/compilers_and_libraries_2019.4.243/linux/mkl/include -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/fftw-3.3.7/include' -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libint-1.1.6/include' -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libxc-4.0.3/include' -I'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libxsmm-1.9.0/include' -I'/home/rcohen/gnu/include/elpa_openmp-2018.11.001/modules' -I'/home/rcohen/gnu/include/elpa_openmp-2018.11.001/elpa' $(FCDEBFLAGS) $(WFLAGS) $(DFLAGS)
#
LDFLAGS     = $(FCFLAGS) -Wl,--enable-new-dtags -L'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/mpich-3.2.1/lib' -Wl,-rpath='/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/mpich-3.2.1/lib'  -L'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/fftw-3.3.7/lib' -Wl,-rpath='/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/fftw-3.3.7/lib' -L'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libint-1.1.6/lib' -Wl,-rpath='/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libint-1.1.6/lib' -L'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libxc-4.0.3/lib' -Wl,-rpath='/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libxc-4.0.3/lib' -L'/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libxsmm-1.9.0/lib' -Wl,-rpath='/home/rcohen/CP2K/cp2k-6.1.0/tools/toolchain/install/libxsmm-1.9.0/lib' -Wl,-rpath='/home/rcohen/gnu/lib'
LIBS        = -lelpa_openmp  -lxsmmf -lxsmm -ldl -lpthread -lxcf03 -lxc -lderiv -lint -lfftw3 -lfftw3_omp  -lmpi -lmpicxx  ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_gf_lp64.a ${MKLROOT}/lib/intel64/libmkl_gnu_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a -Wl,--end-group -lgomp -lpthread -lm -ldl -lstdc++

I had cp2k built fine but my system migrated to centos and I had to rebuild. I had a similar problem in the past trying to use the intel toolchain.

Thanks for help!

Sincerely,

Ron

[Ronald Cohen]

I got it working —the problem was elpa after all. Doing enable-omp in the configure was not enough. I also needed to add -fopenmp to the FLAGS. Now psmp works

fine with threads.

Ron

 

---

Ronald Cohen

Extreme Materials Initiative
Geophysical Laboratory
Carnegie Institution
5251 Broad Branch Rd., N.W.
Washington, D.C. 20015
rco...@carnegiescience.edu
office: 202-478-8937
skype: ronaldcohen

twitter: @recohen3

--
You received this message because you are subscribed to a topic in the Google Groups "cp2k" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cp2k/vrgNXE8L0F4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cp2k+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/cd7bcd1f-4250-4528-aab7-ff5dd656417d%40googlegroups.com.

[Hans Pabst]

Thank you for sharing the root cause! I had a similar issue recently with POPT (Thread number has changed), which was root-caused to an issue with GFortran (see https://github.com/cp2k/dbcsr/issues/217#issuecomment-519066844).

 Hans

To unsubscribe from this group and all its topics, send an email to cp...@googlegroups.com.

[Ole Schütt]

Hi Ron,

thanks for investigating! A couple days ago we switch the cp2k developer
branch to ELPA 2019.05.001. In our nightly tests everything works fine
[1]. Hence, I'm wondering if the problem got resolved or our tests are
missing it?

-Ole

[1] https://dashboard.cp2k.org/archive/current-psmp/index.html

>> cp...@googlegroups.com.

>> To view this discussion on the web visit
>>
> https://groups.google.com/d/msgid/cp2k/cd7bcd1f-4250-4528-aab7-ff5dd656417d%40googlegroups.com

>> [1].
>
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/cp2k/cd7bcd1f-4250-4528-aab7-ff5dd656417d%40googlegroups.com?utm_medium=email&utm_source=footer
> _______________________________________________
> CP2K-user mailing list
> CP2K...@lists.cp2k.org
> https://lists.cp2k.org/listinfo/cp2k-user

[Ronald Cohen]

I had also tried the latest development version with  ELPA 2019.05.001, and had the same problem. I rolled back to 6.2 since I did not know the source of the problem 

at the time. I think it is key to add -fopenmp to the flags evens if some flags are already present.

Thanks,

Ron

---
Ron Cohen
reco...@gmail.com
skypename: ronaldcohen

twitter: @recohen3

To unsubscribe from this group and all its topics, send an email to cp2k+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/8adac8e11acb4da4dd764763acdc813f%40schuett.name.