Parallel Compiling Syntax and D_MPI flag

[Louis Steytler]

Hi All,

What is the correct syntax for compiling a Basilisk problem to run in parallel with MPI?

To run the 3D bubble example on 92 MPI processes, would the following syntax be OK?:

CC99='mpicc -std=c99 -D_MPI=92' qcc -Wall -O2 -D_MPI=92 -grid=octree bubble.c -o bubble -lm

What is the purpose of the "D_MPI" flag? Does this specify the number of pieces the domain will be decomposed to and should this be equal to the number of MPI processes?

I did not get any error using the above syntax, but the computation is slower than expected, so I am not sure if I am doing this right.

Any advice would be very much appreciated.

Thanks very much,

Louis

[Stephane Popinet]

> What is the correct syntax for compiling a Basilisk problem to run in
> parallel with MPI

See:

http://basilisk.fr/src/Tips#parallel-runs-with-mpi
http://basilisk.fr/src/Tips#running-on-supercomputers

> What is the purpose of the "D_MPI" flag? Does this specify the number of
> pieces the domain will be decomposed to and should this be equal to the
> number of MPI processes?

-D_MPI=1 is a standard C compiler option and is the command line
equivalent of adding

#define _MPI 1

in the source code. When this macro is defined to something different
from zero, MPI parallelisation is turned on in the Basilisk source code.
Otherwise the code is serial, or parallelised with OpenMP if the proper
compiler option is used (this is compiler-dependent but is often
-fopenmp). Note that for the moment using both OpenMP and MPI is not
possible (i.e. -D_MPI=1 -fopenmp will not work).

A value of _MPI larger than one is only meaningful when using the
pre-defined Basilisk Makefile, see:

http://basilisk.fr/src/Tips#using-makefiles

cheers

Stephane

[Louis Steytler]

Hi Stephane,

>> A value of _MPI larger than one is only meaningful when using the
>> pre-defined Basilisk Makefile, see:

When using the pre-defined Basilisk Makefile, what does it mean when D_MPI > 1?

Would there be any advantage to using D_MPI > 1 vs D_MPI = 1 in this case?

I tried using the predefined Basilisk Makefile on a Blue Gene/Q cluster, but that produced these errors:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[steytler@cetuslac1 bubble]$ CC='mpicc -D_MPI=92' make bubble.tst
/home/steytler/basilisk/src/Makefile.defs:8: Makefile.tests: No such file or directory
/home/steytler/basilisk/src/Makefile.defs:147: Makefile.deps: No such file or directory
sh /home/steytler/basilisk/src/tests.sh
updating Makefile.tests
updating Makefile.deps
/home/steytler/basilisk/src/qcc -MD -o bubble.s.d bubble.c
updating Makefile.deps
qcc -O2 -g -Wall -pipe -D_FORTIFY_SOURCE=2 -o bubble/bubble bubble.c -lm
[bubble.tst]
ls -rtl
/home/steytler/basilisk/src/grid/fpe.h: In function 'gdb':
/home/steytler/basilisk/src/grid/fpe.h:16: warning: ignoring return value of 'system', declared with attribute warn_unused_result
/home/steytler/basilisk/src/output.h: In function 'output_matrix':
/home/steytler/basilisk/src/output.h:142: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
/home/steytler/basilisk/src/output.h:145: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
/home/steytler/basilisk/src/output.h:149: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
/home/steytler/basilisk/src/output.h:159: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
/home/steytler/basilisk/src/output.h: In function 'output_ppm':
/home/steytler/basilisk/src/output.h:433: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
/home/steytler/basilisk/src/output.h: In function 'output_gfs':
/home/steytler/basilisk/src/output.h:722: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
/home/steytler/basilisk/src/output.h:724: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
/home/steytler/basilisk/src/output.h:749: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
/home/steytler/basilisk/src/output.h:782: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
/home/steytler/basilisk/src/output.h: In function 'dump':
/home/steytler/basilisk/src/output.h:909: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
/home/steytler/basilisk/src/output.h:922: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
/home/steytler/basilisk/src/output.h:924: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
At top level:
cc1: warning: unrecognized command line option "-Wno-unused-result"
/home/steytler/basilisk/src/runtest: line 34: mpirun: command not found
make: *** [bubble.tst] Error 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[Stephane Popinet]

It seems you are confusing two separate things:

1) compiling the code
2) running the code

1) When you compile the code using -D_MPI=(not zero) you get an
executable e.g. 'example' which can be run using any number of MPI
processes. In this case the (not zero) value does not matter.

2) You can then run this code using (usually, but apparently not on Blue
Gene which does things its own way) something like:

mpirun -np (any number) ./example

where (any number) is the number of processes (and of course also the
number of partitions).

Now if you use the default Basilisk makefile (but only in this case),
these two operations (compiling and running the code) are combined into
one i.e.

CC='mpicc -D_MPI=8' make example.tst"

will first compile the code with -D_MPI=8 and then run it using 'mpirun
-np 8'. So only in this case the value of _MPI matters.

Note that on supercomputers the way to start jobs is quite specific (you
need to use a queuing system etc..) so that the Makefile recipe will
probably not work (as you found out). Which is why I recommended the
other option.

cheers

Stephane