[Alexander Morozov]

Dear All

Recently, we have successfully installed Dedalus v2.2006 on Archer 2 UK National Supercomputing Service. The process of installing against the native Cray libraries was not entirely straightforward and the following instructions might be useful for future Archer 2 users.

We have not performed any 'standard' benchmarking but our own code scales almost perfectly up to 32768 cores (256 nodes).

N.B. The install was performed on Archer 2 4-cabinet system but the procedure is expected to stay the same when the full system becomes available before the end of the year(?).

Alexander

WORKDIR=/work/[project code]/[project code]/[username]/
cd $WORKDIR

export PYTHONUSERBASE=$WORKDIR/.local  # Recommended in ARCHER2 docs for user-level Python package installation

module load cray-python
module load cray-fftw

# Build & install h5py, linked to Cray HDF5 libraries, compiled using Cray compiler (CC=cc)
module load cray-hdf5-parallel
CC=cc pip install --user --no-binary=h5py h5py

# Now we'll do a manual build & install of Dedalus
# First create temporary build directory
mkdir -p /tmp/$USER
cd /tmp/$USER
# Download and extract the latest (v2.2006) Dedalus source release
wget https://github.com/DedalusProject/dedalus/archive/v2.2006.tar.gz
tar xf v2.2006.tar.gz
cd dedalus-2.2006

# Patch Dedalus' setup.py to add explicit MPI library dependency
sed -i -e "/^libraries = /s/]/, 'mpi']/" setup.py

# Now build Dedalus extension libraries using GCC, linking to Cray FFTW & correct MPI libraries
export FFTW_PATH=$CRAY_FFTW_PREFIX
export MPI_PATH=$CRAY_MPICH_BASEDIR/gnu/$PE_MPICH_GENCOMPILERS_CRAY
CC=gcc python setup.py install --user

[Keaton Burns]

Hi Alexander,

Thanks for sending this along! Would you mind if we added these instructions to the online documentation as well?

I'd also be happy to chat more if you end up benchmarking with Dedalus on Archer 2.  There might be some config options / problem formulation decisions that are best for scaling on that specific hardware.

Best,

-Keaton

--
You received this message because you are subscribed to the Google Groups "Dedalus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/CA%2B%2Bqik80xFCRP6Xqb_3geCQ0LRp1FyE7MKzo1h-Fk_-pPxt%3DuA%40mail.gmail.com.

[Alexander Morozov]

Hi Keaton

Not at all, please feel free to use it as you see fit. 

I would be happy to discuss how to benchmark Archer in a way that would be useful to this community. If you have any suggestions, please let me know.

Best wishes,

Alexander