Problems with installation Dedalus v3

[Sadokat]

Dear All, 

I installed dedalus v3 on the Topola HPC cluster (Huawei E9000).
• Architecture: Intel(R) Xeon(R) CPU E5–2650 v3 (Haswell), x86 64. 

However, I noticed that my jobs are running significantly slower than expected. I am run jobs with srun and SLURM_MPI_TYPE=pmi_v4.

Here is what I have done so far:

1. Switched to a compute node and created a Python virtual environment:

/bin/python -m venv ~/dedalus source ~/dedalus/bin/activate

2. The FFTW module provided by the system lacks MPI support, so I built my own FFTW with MPI enabled:

./configure \  --prefix=$HOME/fftw \  CC=mpicc \  CXX=mpicxx \  FC=mpif90 \  CFLAGS="-O3 -march=haswell" \  --enable-shared \  --enable-mpi \  --enable-threads \  --enable-openmp \  --enable-avx2 make -j make install

3. Loaded recommended modules( Administrator recommend to us openmpi with SLURM_MPI_TYPE=pmi_v4 and run jobs with srun):

module load common/mpi/openmpi module load common/hdf5/3.3.10

4. Set environment variables and installed Python packages:

export SLURM_MPI_TYPE=pmi_v4 export CC=mpicc export CXX=mpicxx export FC=mpif90 export F77=mpif90 pip3 install --no-cache-dir numpy scipy pip3 install --no-binary=mpi4py --no-cache-dir mpi4py pip3 install --no-binary=:all: --no-cache-dir pyfftw pip3 install --no-cache-dir --no-build-isolation \    http://github.com/dedalusproject/dedalus/zipball/master/

5. I also attempted installing NumPy with MKL support, but this did not work as mkl demands another mpi 

here the list of available modules:

common/mpi/mpich/4.2.3

common/bedtools2/2.31.1
 common/compilers/gcc/8.5.0
 common/compilers/gcc/13.2.0 ### with loading this not works 
 common/compilers/intel-oneapi-compilers/2024

 common/fftw/3.3.10

common/mpi/openmpi/5.0.3
 common/intel-oneapi-mkl/2024.0.0

common/netcdf-c/4.9.2
apps/anaconda/2024-10

common/netlib-scalapack/2.2.0
common/openblas/0.3.26. 

I would greatly appreciate any advice on proper installation.

with best regards,

Sadokat

[Keaton Burns]

Hi Sadokat,

To help narrow down the issue, can you test a simple script (e.g. the 2D RBC example) on a single node with increasing numbers of cores and report the performance?

To double check, were you using openmpi when you built FFTW? Also are you disabling threading with OMP_NUM_THREADS=1?

Recently we noticed that the conda builds against mpich were showing normal performance up to several cores, but largely degraded performance at higher core counts (4 and above, in my local tests). I haven’t figured out why, but we have disabled the mpich conda builds in favor of openmpi in the meantime. And the openmpi builds seem to be working fine.

Best,

-Keaton

--
You received this message because you are subscribed to the Google Groups "Dedalus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dedalus-users/28518614-f9eb-44c1-a37f-ab0d4e42c68an%40googlegroups.com.

[Sadokat]

Dear Keaton,

Thank you very much for your fast reply.

I built FFTW with OpenMPI and disable threading during execution by setting OMP_NUM_THREADS=1 in my job scripts.

I’ve attached the performance report.

with best regards,

Sadokat

[Sadokat]

Dear Keaton,

As you mentioned, the openmpi build is the fastest. I verified that everything is installed correctly with openmpi.

The hpc system appears older compared to the other Intel-based supercomputers I use (roughly ~50% difference), and it is about 10–20% slower than my local pc, likely due to lower clock speeds and older hardware. I'm a little confused by that. Still, it remains a useful resource.

Thank you also for confirming the mpich conda build behavior. I had observed it before but assumed it was an installation issue.

Thank you again, your response helped clarify several points.

Best regards,

Sadokat