running multiple simulations at the same time with mpi processes

[arl1990]

Hello,

I couldn't find an answer here until now. But maybe there is a solution now. 

For example, if I have 24 cores ( 48 virtual ) and i run a single simulation on windows with "fds_local -p 24 - o 1 XXX.fds" it works fine. 
Defined in the .fds file:  "&MESH ID='1' IJK=...   MPI_PROCESS=0( .. 1 .. 23) /" 

so, 24 differnet meshes and each assigned 1 core. ( works faster than with - o 2 ).

But if i want to run for example 2 simulations at the same time, each with 12 meshes and 12 cores i cant get a satisfying result as the computation time will increase (1 Simulation, 24 Mpi processes will finish in 3h and 2 simulations running  at the same time, each of them needs 7h40min.)

--- starting the first simulation with "fds_local -p 12 - o 1 XXX.fds"

--- Defined in the .fds file:  "&MESH ID='1' IJK=...   MPI_PROCESS=0( .. 1 .. 11) /" 

--- checking the Windoes Task manager / resource manager there is action in almost all 48 cores. 

--- starting the second simulation with "fds_local -p 12 - o 1 XXX.fds"

--- Defined in the .fds file:  "&MESH ID='1' IJK=...   MPI_PROCESS=0( .. 1 .. 11) /" 

(As the definition of the MPI processes needs to start at 0 again)

--- checking the Windoes Task manager / resource manager there is action in almost all 48 cores. but 2 of them running at about 100% (CPU 26 and 30) and another 2 running at almost 0% (CPU 27 and 31) all the other CPUs vary over time between 100% and 0% 
(The cells for each mesh are almost the same) 

Maybe the 2 running at 100% do slow down the simulation time. 

Is there any solution to fix this on windows ? 

if not, is there a way to make this work on linux ? Or should i keep going with 24 meshes / cores in 1 simulation. 

Thanks

[Kevin McGrattan]

You need a job scheduler. Windows will not automatically assign unique cores to your meshes. There is a high performance computing functionality that works for windows, but it is something that you must configure, like you would a domain network. In Linux, this type of job scheduling is common and part of the default OS.

[arl1990]

Thanks for the fast reply. I guess I will try to figure it out for Linux (should be easier) . As I am new to Linux, could someone give me an example how to run 2 simulations at the same time on a single computer (Linux) with the PBS/Torque job scheduler mentioned in the Userguide ? 

As this should be the Input file for a single simulation with MPI processes:  
From the Userguide:

"#!/bin/bash
#PBS -N job_name
#PBS -e <pwd>/job_name.err
#PBS -o <pwd>/job_name.log
#PBS -l nodes=4:ppn=2
#PBS -l walltime=24:0:0
export OMP_NUM_THREADS=2
export I_MPI_PIN_DOMAIN=omp
cd <pwd>
mpiexec -n 8 <full_path>/fds job_name.fds"

How would it look like for 2 Simulations with 12 MPI processes each at the same time, like mentioned above ? 

Thanks very much 

[Randy McDermott]

We do this with bash shell scripting.  I suggest you first get a single job running using qfds.sh from the repo.  Clone the repository to space in your account on the linux machine.

https://github.com/firemodels/fds

The bash script looks something like this...

#!/bin/bash

# This script runs a set of Validation Cases on a Linux machine with a batch queuing system.

export QFDS=<path to fds/Utilities/Scripts/qfds.sh>

$QFDS -p 12 jobname_1.fds
$QFDS -p 12 jobname_2.fds

# end of bash script

You put this script in the working directory and run it with

$ bash my_script.sh

Once you get the hang of it, you can also manipulate which directories the cases run in, etc.  It's all scriptable.  But is starts with being able to run one case with qfds.sh

--
You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/42f2a3ec-6ed4-465d-8f10-c21a829ba874n%40googlegroups.com.

[Kevin McGrattan]

https://github.com/firemodels/fds/wiki/Installing-and-Running-FDS-on-a-Linux-Cluster