MPI scaling issue

[AmirHossein Ghaemi]

Dear Basilisk developers and users,

I am running a 3D Basilisk simulation with MPI parallelization. Everything runs correctly, but I’m observing severe performance degradation when using multiple nodes — even though MPI is enabled and the job runs without errors.

• When I use 1 node (64 MPI ranks) → performance is good.

• When I use 2 nodes (128 MPI ranks) → the simulation becomes about 10× slower.

• Increasing further makes it even slower.

---

🧰 Compilation and run setup

I compile with MPI enabled as follows:

qcc -source -D_MPI=1 -D_ADAPT=1 main.c -disable-dimensions

mpicc -Wall -O3 -std=c99 \
  -D_XOPEN_SOURCE=700 -D_GNU_SOURCE=1 \
  _main.c -o main_mpi \
  -I"$BASILISK" -I"$BASILISK/gl" \
  -L"$BASILISK/gl" -lglutils -lfb_tiny -lm

and Run script using:

cd "$PBS_O_WORKDIR"/AHG_VR

module purge
module load intel-compilers/2022.2.1
module load impi/2021.7.1

mpirun -v6 ./main_mpi 11 4 0.008 1750 200

Questions

1. Are my MPI compilation and run commands correct for a Basilisk simulation across multiple nodes?

2. Why might performance drop drastically when moving from 1 node to 2+ nodes?
   Could this be related to:

   • Domain decomposition and inter-node communication,

   • AMR load imbalance,

   • or Intel MPI fabric settings?

Any feedback or tips on improving multi-node scaling would be greatly appreciated.

Thank you for your time and expertise.