Compiling on Argonne XC40 Theta

[Gani Annaberdiyev]

Hi users and developers,

I've been trying to compile QWalk on Theta for a significant amount of time with no success.

The makefile I use is:

######################################################################
CXX:= CC
CXX_SERIAL:=g++
CXXFLAGS := -g -O2 -mkl
CXXFLAGS +=  -O3 -fno-alias -funroll-loops -ffast-math -fomit-frame-pointer
CXXFLAGS += -DUSE_MPI -DUSE_RESTRICT -DUSE_BLAS -DUSE_LAPACK $(INCLUDEPATH)


LAPACK_LIBS := -L$(MKLROOT)/lib/intel64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -Wl,--end-group -lpthread -lm -ldl
LAPACK_INCLUDE := -I${MKLROOT}/include
BLAS_LIBS :=  $(LAPACK_LIBS)
BLAS_INCLUDE := $(LAPACK_INCLUDE)
DEBUG:= -DNO_RANGE_CHECKING   -DNDEBUG  


######################################################################
# This is the invokation to generate dependencies
DEPENDMAKER:=CC -MM $(INCLUDEPATH)

The loaded modules are:

Currently Loaded Modulefiles:
  1) modules/3.2.11.1
  2) intel/18.0.0.128
  3) craype-network-aries
  4) craype/2.5.15
  5) udreg/2.3.2-6.0.7.1_5.13__g5196236.ari
  6) ugni/6.0.14.0-6.0.7.1_3.13__gea11d3d.ari
  7) pmi/5.0.14
  8) dmapp/7.1.1-6.0.7.1_6.2__g45d1b37.ari
  9) gni-headers/5.0.12.0-6.0.7.1_3.11__g3b1768f.ari
 10) xpmem/2.2.15-6.0.7.1_5.11__g7549d06.ari
 11) job/2.2.3-6.0.7.1_5.44__g6c4e934.ari
 12) dvs/2.7_2.2.120-6.0.7.1_12.1__g74cb2cc4
 13) alps/6.6.43-6.0.7.1_5.46__ga796da32.ari
 14) rca/2.2.18-6.0.7.1_5.48__g2aa4f39.ari
 15) atp/2.1.3
 16) perftools-base/7.0.4
 17) PrgEnv-intel/6.0.4
 18) craype-mic-knl
 19) cray-mpich/7.7.3
 20) nompirun/nompirun
 21) darshan/3.1.5
 22) trackdeps
 23) xalt
 24) gcc/7.3.0

I attached the compile log here. I get the following message on both compute and login nodes:

Illegal instruction (core dumped)

Any suggestions will be very helpful since this has been a time sink for me.

[Gani Annaberdiyev]

The log files

[Huihuo Zheng]

Hi Gani, 
When you ran QWalk, did you use “aprun -n … -N … qwalk-XXX”? If you did not launch job using aprun, it actually did not run on KNL compute nodes, but on mom nodes which are not KNL nodes. 

When you compile it using CC, by default, it turns on AVX512 which is only available on KNL nodes. If you run the executable on non-AVX512 architecture, you will get "Illegal instruction (core dumped)”. The login nodes on Theta are not KNL nodes either. 

Please try to run with “aprun -n “ command. 

Below is some useful information on how to run jobs on Theta: https://www.alcf.anl.gov/files/Knight_Software_JobSubmission2018.pdf. Feel free to send question to sup...@alcf.anl.gov if you encounter any issue.

Best, 

Huihuo

--
You received this message because you are subscribed to the Google Groups "QWalk user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qwalk-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qwalk-users/0bc689a4-0706-4035-a67c-aa9b024862b5%40googlegroups.com.
<compile.err><compile.log>

[Gani Annaberdiyev]

Thank you Huihuo, this was indeed the problem!

Best, 

Huihuo

To unsubscribe from this group and stop receiving emails from it, send an email to qwalk...@googlegroups.com.