Hi Everyone,
I have been trying to install deal.ii on NERSC Cori cluster for use on the KNL compute nodes using the following procedure :
Clone and setup :
git clone --single-branch --branch dealii-9.1 https://github.com/dealii/dealii.git
mkdir build
mkdir dealii_trial
cd build
Build and install on an interactive node :
cmake -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn -DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_PETSC=ON -DDEAL_II_WITH_P4EST=ON -DDEAL_II_WITH_LAPACK=ON -DCMAKE_INSTALL_PREFIX=/global/project/projectdirs/m2360/packagesCPFE/dealii_trial/ ../dealii
which completes without any issues after which
make -j 32 install
eventually yields the following error :
--------------------------------------------------------------------------
[ 27%] Building CXX object bundled/umfpack/AMD/Source/CMakeFiles/obj_amd_long_debug.dir/amd_defaults.cc.o
/bin/sh: expand_instantiations: command not found
make[2]: *** [source/numerics/CMakeFiles/obj_numerics_inst.dir/build.make:91: source/numerics/data_out_dof_data.inst] Error 127
make[2]: *** Waiting for unfinished jobs....
---------------------------------------------------------------------------
Details on procedure before cloning deal.ii are included below :
(i) I first loaded the necessary modules so that module list displayed
Currently Loaded Modulefiles:
1) modules/3.2.11.4 16) rca/2.2.20-7.0.1.1_4.46__g8e3fb5b.ari
2) altd/2.0 17) atp/2.1.3
3) darshan/3.1.7 18) PrgEnv-intel/6.0.5
4) intel/19.0.3.199 19) craype-mic-knl
5) craype-network-aries 20) cray-mpich/7.7.14
6) craype/2.6.2 21) craype-hugepages2M
7) udreg/2.3.2-7.0.1.1_3.31__g8175d3d.ari 22) valgrind/3.15.0
8) ugni/6.0.14.0-7.0.1.1_7.33__ge78e5b0.ari 23) zlib/1.2.11
9) pmi/5.0.14 24) metis/5.1.0
10) dmapp/7.1.1-7.0.1.1_4.48__g38cf134.ari 25) parmetis/4.0.3
11) gni-headers/5.0.12.0-7.0.1.1_6.28__g3b1768f.ari 26) superlu-dist/6.1.1
12) xpmem/2.2.20-7.0.1.1_4.10__g0475745.ari 27) hypre/2.15.1
13) job/2.2.4-7.0.1.1_3.36__g36b56f4.ari 28) cmake/3.14.4
14) dvs/2.12_2.2.156-7.0.1.1_8.9__g5aab709e 29) cray-hdf5/1.10.2.0
15) alps/6.6.58-7.0.1.1_6.4__g437d88db.ari 30) petsc-real-64/3.11.2
(ii) Set environment variables for dynamic linking
export XTPE_LINK_TYPE=dynamic
export CRAYPE_LINK_TYPE=dynamic
(iii) p4est installation
./configure --prefix=/global/project/projectdirs/m2360/packagesCPFE/p4est --host=x86_64 --enable-vtk-zlib=yes --enable-mpi=yes F77=ftn FC=ftn CC=cc CXX=CC
make -j 32
make -j 32 install
export P4EST_DIR=/global/project/projectdirs/m2360/packagesCPFE/p4est
I did initially submit a ticket on NERSC support for assistance with the installation process and after a few iterations, the person(https://github.com/JBlaschke) helping me out asked me to contact the developers. I would really appreciate your help on this.
Thank you.
--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/aaff8931-185f-46d0-8f2c-eb535ae9428en%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/0b6d6ab9-f630-4517-a2a0-7106604aa391n%40googlegroups.com.
Cross compiling for compute nodes in the Cray Linux Environment can be done without needing a separate toolchain file. Specifying -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment on the CMake command line will ensure that the appropriate build settings and search paths are configured. The platform will pull its configuration from the current environment variables and will configure a project to use the compiler wrappers from the Cray Programming Environment’s PrgEnv-* modules if present and loaded.
The default configuration of the Cray Programming Environment is to only support static libraries. This can be overridden and shared libraries enabled by setting the CRAYPE_LINK_TYPE environment variable to dynamic.
Running CMake without specifying CMAKE_SYSTEM_NAME will run the configure step in host mode assuming a standard Linux environment. If not overridden, the PrgEnv-* compiler wrappers will end up getting used, which if targeting the either the login node or compute node, is likely not the desired behavior. The exception to this would be if you are building directly on a NID instead of cross-compiling from a login node. If trying to build software for a login node, you will need to either first unload the currently loaded PrgEnv-* module or explicitly tell CMake to use the system compilers in /usr/bin instead of the Cray wrappers. If instead targeting a compute node is desired, just specify the CMAKE_SYSTEM_NAME as mentioned above.
In my case I checked that PrgEnv-intel is one of the modules loaded by default. I did specify CMAKE_SYSTEM_NAME earlier while running cmake, when attempting the installation process on the login node and on the interactive node. I will now try the following :
1) Compiling with PrgEnv-intel on the compute node with CMAKE_SYSTEM_NAME=CrayLinuxEnvironment.
2) Unloading PrgEnv-intel and compiling on the login node.
Please let me know if you have any other suggestions. Thank you.
Best,
Aaditya