deal.ii installation on NERSC Cori
Yanjun Lyu
Hello,
I am trying to install the latest version of deal.II on NERSC Cori in order to be able to run the PRISMS-PF framework, which is deal.II-based. This framework only needs deal.II to be configured the p4est and the MPI options. I would like to be able to run on KNL nodes.
The main questions I have are:
1) I am not sure what are the recommended environment variables and modules I need to load beforehand building deal.II on Cori
2) I do not know whether I should do the installation in a KNL compute mode, rather than in a login node (which is Haswell).
Below are the set of steps which we used for a previous installation. Are these correct? If not, could you point out the problems? (we are observing poor scalability beyond 1 node. Any insight for this is appreciated. Thank you!
1. Building and configuring deal.II within a KNL node requires an interactive job
$ salloc -N 1 -n 68 —account=xxxx -C knl -q interactive -t 4:00:00
2. Load/unload modules
module unload cray-libsci/19.06.1
module load cmake/3.14.4
module swap craype-haswell craype-mic-knl
Currently Loaded Modulefiles:
1) modules/3.2.11.4
2) altd/2.0
3) darshan/3.1.7
4) craype-network-aries
5) intel/19.0.3.199
6) craype/2.6.2
7) udreg/2.3.2-7.0.1.1_3.41__g8175d3d.ari
8) ugni/6.0.14.0-7.0.1.1_7.43__ge78e5b0.ari
9) pmi/5.0.14
10) dmapp/7.1.1-7.0.1.1_4.56__g38cf134.ari
11) gni-headers/5.0.12.0-7.0.1.1_6.33__g3b1768f.ari
12) xpmem/2.2.20-7.0.1.1_4.16__g0475745.ari
13) job/2.2.4-7.0.1.1_3.43__g36b56f4.ari
14) dvs/2.12_2.2.164-7.0.1.1_13.3__g354a5276
15) alps/6.6.58-7.0.1.1_6.13__g437d88db.ari
16) rca/2.2.20-7.0.1.1_4.56__g8e3fb5b.ari
17) atp/2.1.3
18) PrgEnv-intel/6.0.5
19) craype-mic-knl
20) cray-mpich/7.7.10
21) craype-hugepages2M
22) nano/2.6.3
23) cmake/3.14.4
24) Base-opts/2.4.139-7.0.1.1_4.78__gbb799dd.ari
3. Set environment variables
export XTPE_LINK_TYPE=dynamic
export CRAYPE_LINK_TYPE=dynamic
4. Before installing dealii, we need to install the p4est dependency. To install p4est do the following:
$ cd $HOME
$ mkdir p4est_files
$ mkdir p4est_install
$ cd p4est_files
5. Download p4est tarball and setup script
$ wget http://p4est.github.io/release/p4est-2.2.tar.gz
$ wget https://www.dealii.org/9.2.0/external-libs/p4est-setup.sh
6. Install p4est using the setup script
$ chmod u+x p4est-setup.sh
$ ./p4est-setup.sh p4est-2.2.tar.gz $HOME/p4est_install
7. Once installed without errors, set the environment variable pointing to the p4est installation directory.
$ export P4EST_DIR=$HOME/p4est_install
8. Now ready to compile and install deal.ii with the following commands:
$ cd $HOME
$ wget https://dealii.43-1.org/downloads/dealii-9.2.0.tar.gz
$ tar xvzf dealii-9.2.0.tar.gz
$ mkdir dealii_install
$ mkdir build
$ cd build
9. type the following command
$ cmake -DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_P4EST=ON-DCMAKE_INSTALL_PREFIX=$HOME/dealii_install $HOME/dealii-9.2.0
10 install and test deal.ii
$ make install
$ make test
11. Once this has completed successfully, deal.ii has been installed, ensure that the directories where p4est and deal.ii are installed are set as environment variables. To do this add the following lines to the file .bashrc (or .bash_profile)
$ export P4EST_DIR=$HOME/p4est_install
$ export DEAL_II_DIR=$HOME/dealii_install
