Does Dedalus only run on 1-core?

[Kevin Modica]

Hi all,

I am relatively new to using Dedalus, and have mainly used it to solve some simple problems in the past. I've recently started running Dedalus job's on my university's cluster, where you submit the job using "sbatch MyJob.job" and can select multiple numbers of cores / nodes. For now I think I should be able to run it in 1 node as long as I have multiple cores, but I'm getting the same out of memory errors (not enough memory to perform factorization) whether I run with 1 core or 40. I tested it by finding a number of spectral modes that works on 1 core, and then doubling it so it runs out of memory. Then when I upped it to 40 cores, I received the same out of memory errors. 

I've included my job code below, my Dedalus code is pretty involved, and I'm working to try to come up with a minimal example, but I thought I'd ask this if people know.

Is there a flag somewhere that lets Dedalus access the memory across multiple cores (within the same node, so I shouldn't need openmpi or pympi right?) How do I use all that  memory that is available?

Thanks,

Kevin

-------------------Code Below----------

#!/bin/bash

#SBATCH -N 1

#SBATCH --ntasks-per-node=40

#SBATCH --time=99:00:00

#SBATCH --output=logfile

cd $SLURM_SUBMIT_DIR

/bin/hostname

module load conda

eval "$(conda shell.bash hook)"

conda activate dedalus3

python3 -m Script_Runner_multiRun.py

[Jeffrey S. Oishi]

Hi Kevin,

This is completely dependent on your local system. If your scheduler allows one node to take all of the local memory (which most do), then Dedalus will use all available memory. Unfortunately, this is not something we can provide support for. You'll need to ask your local system administrators.

Jeff

--
You received this message because you are subscribed to the Google Groups "Dedalus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dedalus-users/5c93b6e2-3d25-49af-aa22-ba4e96e7bcf8n%40googlegroups.com.

[Jeffrey S. Oishi]

Sorry, that should read "one core on a node to take all of the local memory"

[Kevin Modica]

Gotcha. Thanks for letting me know!

[Yves Dubief]

Kevin,

How did you compile dedalus? If your compilation did not identify the existing slurm+mpi libraries and created its own mpi library, you would be able to run parallel on one core, but mpi could not know how to run on multiple cores. If you believe that the case, let me know I have instructions to create a dedalus from miniconda on our local cluster.

Yves

[Kevin Modica]

Hi Yves,

I compiled it using miniconda on my local cluster. I followed the full stack conda installation instructions on the documentation. How would I check if it identified the existing libraries? I'd appreciate any instructions you can send me to check that/ create a Dedalus again.

Thanks,

Kevin

[Yves Dubief]

Here is my procedure (for d3). The key is to compile mpi4py with the slurm-compiled mpi library for your infiniband or similar system available on your cluster. Here I have made my mpicc default in my .bashrc, otherwise you need to specify the full path in the "export CC=mpicc", "MPICC=mpicc" "CC-mpicc", etc. ">>>" are the instructions for the installation of miniconda. Since I have 2 minicondas I make sure that python3 and pip3 are linked to the right one with "which" command. Note that the versions for numpy and mpi4py are specified to avoid issues well documented in this forum.

Let me know if you have any question.

Yves


mkdir -p opt-d3/local

bash Miniconda3-latest-Linux-x86_64.sh

>>> yes

>>> ~/opt-d3/miniconda3

>>> no

source ~/opt-d3/miniconda3/bin/activate

which python3

which pip3

pip3 install --upgrade cython numpy setuptools wheel

pip3 install numpy==1.26.4

cd ~/opt-d3

wget http://www.fftw.org/fftw-3.3.10.tar.gz

tar xzvf fftw-3.3.10.tar.gz

cd fftw-3.3.10

./configure --prefix=$HOME/opt-d3/local CC=mpicc CXX=mpicxx F77=mpif90 MPICC=mpicc MPICXX=mpicxx --enable-shared --enable-mpi --enable-openmp --enable-threads

make

make install

cd ..

wget --content-disposition --no-check-certificate https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.1/src/hdf5-1.12.1.tar.bz2

tar xvf hdf5-1.12.1.tar.bz2

cd hdf5-1.12.1/

./configure --prefix=$HOME/opt-d3/local CC=mpicc CXX=mpicxx F77=mpif90 MPICC=mpicc MPICXX=mpicxx --enable-shared --enable-parallel

make

make install

cd ..

git clone https://github.com/h5py/h5py.git

cd h5py

export CC=mpicc

export HDF5_MPI="ON"

export HDF5_DIR=$HOME/opt-d3/local

export LD_LIBRARY_PATH=$HOME/opt-d3/local/:$LD_LIBRARY_PATH

export FFTW_PATH=$HOME/opt-d3/local

export MPI_PATH=/gpfs1/arch/x86_64-rhel7/openmpi-4.1.4-slurm-ib/

CC=mpicc pip3 install mpi4py==3.1.6

pip3 install --no-binary=h5py h5py

CC=mpicc pip3 install --no-cache --no-build-isolation http://github.com/dedalusproject/dedalus/zipball/master/