OpenMP option in Linux cluster

[Marina Fragkopoulou]

Hello all,

I want to run FDS in a Linux cluster in which version FDS 6.0.0 is currently installed. The cluster has multiple machines but I would like to use one machine with 16 cores.

I created the FDS model with 16 meshes of similar size.

Since I do not have experience with parallel processing in a Linux cluster, I am very confused. Any help would be appreciated!

In order to initiate the simulation in the cluster, i am using a qsub file:

#PBS -l nodes=1:nameofmachine:ppn=16

#PBS -N testrun

#############################################################

## PBS (DO NOT CHANGE!)                                    ##

#############################################################

#PBS -l cput=20:00:00

#PBS -W umask=002

#PBS -j oe

#PBS -c c=1

#PBS -V

export OMP_NUM_THREADS=16

cd $PBS_O_WORKDIR

umask 002

cp $PBS_NODEFILE $PBS_O_WORKDIR/hostfile.gen

chmod 664 hostfile.gen

NPROC=`wc -l <$PBS_NODEFILE`

source /wb_apps/FDS/shortcuts/bashrc_fds6 intel64

echo $LD_LIBRARY_PATH

ulimit -s unlimited

#############################################################

## END OF PBS (DO NOT CHANGE!)                             ##

#############################################################

runid=testrun

However, my .out file shows that OpenMP is Disabled:

 Fire Dynamics Simulator

 Compilation Date : Sun, 03 Nov 2013

 Version          : FDS 6.0.0 Parallel

 OpenMP Disabled

 SVN Revision No. : 17279

 Job TITLE        : 150MW_fire

 Job ID string    : GN_testrun

and the error message I receive gives the following:

Fire Dynamics Simulator

 Compilation Date : Sun, 03 Nov 2013

 Current Date     : February  6, 2017  16:01:18

 Version: FDS 6.0.0; MPI Enabled; OpenMP Disabled

 SVN Revision No. : 17279

 Job TITLE        : 150MW_fire

 Job ID string    : testrun

 Time Step:      1,    Simulation Time:      0.06 s

 Time Step:      2,    Simulation Time:      0.10 s

 Time Step:      3,    Simulation Time:      0.13 s

 Time Step:      4,    Simulation Time:      0.17 s

 Time Step:      5,    Simulation Time:      0.20 s

 Time Step:      6,    Simulation Time:      0.23 s

 Time Step:      7,    Simulation Time:      0.26 s

mpirun: killing job...

--------------------------------------------------------------------------

mpirun noticed that process rank 0 with PID 12044 on node galatea.cluster.intern exited on signal 0 (Unknown signal 0).

--------------------------------------------------------------------------

mpirun: clean termination accomplished

forrtl: error (78): process killed (SIGTERM)

Image              PC                Routine            Line        Source             

Stack trace terminated abnormally.

forrtl: error (78): process killed (SIGTERM)

Image              PC                Routine            Line        Source             

Stack trace terminated abnormally.

[Glenn Forney]

fds 6.0.0 is a very old version.  I don't know whether it was built with openmp enabled.  Try installing the latest fds.  This is what it gives me .  Also, the fds repo comes with a script named qfds.sh which you may find useful for running fds on a linux cluster .  It creates and runs the script file used by qsub .

 Fire Dynamics Simulator

 Current Date     : February  6, 2017  10:52:54

 Version          : FDS 6.5.3

 Revision         : FDS6.5.3-598-geb56ed1

 Revision Date    : Thu Jan 19 16:12:59 2017 -0500

 Compilation Date : Jan 22, 2017  18:04:30

 MPI Enabled; Number of MPI Processes:          1

 OpenMP Enabled; Number of OpenMP Threads:   4

--
You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+unsubscribe@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/b39628b9-9532-4059-b65b-7d40b900428d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Glenn Forney

[Kevin]

Install the latest version of FDS (6.5.3). I do not recall if OpenMP was working back then.

[Randy McDermott]

Yes, install the latest, but I think if you are using 12 meshes, you want to run you case with MPI and a single OMP thread on each.  This will run much faster than utilizing your cores OMP.

--
You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+unsubscribe@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/a2749999-f1ba-4ac5-806b-120fb4e7958e%40googlegroups.com.

[Marina Fragkopoulou]

Kevin and Glenn,

Thank you for your immediate reaction! 

I will contact the cluster administrator and request for the installation of FDS 6.5.3. After that, I will come back to you and inform you if it worked.

Best regards,

Marina

[Glenn Forney]

fyi: there is nothing special about the installation.  you could install it in your home directory if you want.  You only need permission to write the directory where it is being installed.

--
You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+unsubscribe@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/8a1f6b89-d13d-4904-8705-a4561ebf3dc0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Glenn Forney

[Marina Fragkopoulou]

Randy,

I will check this option and run it this way, then. I mistakenly understood that the MPI option is used only in case you want to use multiple machines in a cluster. 

Thank you for your advice!

Best regards,

Marina

To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+u...@googlegroups.com.

To post to this group, send email to fds...@googlegroups.com.

[Randy McDermott]

In general, you should use up all your cores mapping to MESH with MPI before you start adding OMP.

For example, if you have 4 meshes and 4 cores, then set OMP_NUM_THREADS=1 and -np=4 for MPI.  If you have, say, 8 cores but still only 4 meshes, then -np=4 and OMP_NUM_THREADS=2.  But if you had 8 meshes, you would again use -np=8 and OMP_NUM_THREADS=1.  Hope this is clear.

To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+unsubscribe@googlegroups.com.

To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/42cdbe7d-2ba6-41d5-9a27-3d75c5dc76f5%40googlegroups.com.

[Marina Fragkopoulou]

Randy,

Thank you for the clarification. So, in order to make it work this way I have to:

1. Install latest FDS version (6.5.3)

2. Type the following commands in the qsub file (in order for each of my 16 meshes to be assigned to one core):

    export OMP_NUM_THREADS=1

    export mpirun -np 16 fds -hostfile job_name.fds

3. Change the command that refers to the FDS directory:

    source /wb_apps/FDS/shortcuts/bashrc_fds6 intel64

Is that correct? 

Marina

[Randy McDermott]

Not sure if you need to do #3.  I think the installer will handle your .bashrc_fds.

To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+unsubscribe@googlegroups.com.

To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/3dd7fc74-b5f8-40aa-b02d-000606e9965d%40googlegroups.com.

[Marina Fragkopoulou]

Hello again,

I have installed version 6.5 of FDS and changed the options in the qsub file.

However, I have another issue. Although the simulation seems to be running, I get no output files. Could you suggest what the problem would be?

In the fds I have defined my output:

&HEAD CHID='GN_testrun'

      TITLE='150MW_fire'

      /

 

&TIME T_END=900.0

      /

&DUMP DT_ISOF    = 5 

      DT_PL3D    = 5 

 DT_RESTART = 30 

      WRITE_XYZ  = .TRUE.

      PLOT3D_QUANTITY(1:5)='TEMPERATURE', 'U-VELOCITY', 'V-VELOCITY', 'W-VELOCITY' /

Thank you in advance for your help.

Marina

[Kevin]

How do you know the job is running? What do you see if you do this:

tail -n 20 GN_testrun.out

[Marina Fragkopoulou]

Hi Kevin,

The problem had to do with permissions to write in the cluster and it was solved by the IT. Thank you for your reaction.

So, I followed the recommended steps and the mpi worked for this case with FDS 6.5.3 version:

- A simulation with 8 meshes with cells of the same size (0.5*0.5*0.5) in a machine with 16 processors and -np=8 and OMP_NUM_THREADS=1 (Linux)

- However, I altered the same case and I created 16 meshes (8 meshes with a grid size of 0.25*0.25*0.25 m and 8 meshes with 2.0*1.0*1.0 m in two rows the one on top of the other). Then in a machine with 16 processors (and 100GB memory) I tried to run it with -np=16 and OMP_NUM_THREADS=1 (Linux).

In the qsub file that I use to start the simulation in the cluster, I added the commands (based on previous posts):

ulimit -s unlimited

ulimit -v unlimited

Then, I always receive the following error:

 Mesh      1 is assigned to MPI Process      0

 Mesh      2 is assigned to MPI Process      1

 Mesh      3 is assigned to MPI Process      2

 Mesh      4 is assigned to MPI Process      3

 Mesh      5 is assigned to MPI Process      4

 Mesh      6 is assigned to MPI Process      5

 Mesh      7 is assigned to MPI Process      6

 Mesh      8 is assigned to MPI Process      7

 Mesh      9 is assigned to MPI Process      8

 Mesh     10 is assigned to MPI Process      9

 Mesh     11 is assigned to MPI Process     10

 Mesh     12 is assigned to MPI Process     11

 Mesh     13 is assigned to MPI Process     12

 Mesh     14 is assigned to MPI Process     13

 Mesh     15 is assigned to MPI Process     14

 Mesh     16 is assigned to MPI Process     15

 Completed Initialization Step  1

 Completed Initialization Step  2

forrtl: severe (174): SIGSEGV, segmentation fault occurred

Image              PC                Routine            Line        Source             

fds                00000000008042A5  Unknown               Unknown  Unknown

fds                0000000000801EC7  Unknown               Unknown  Unknown

fds                00000000007B31D4  Unknown               Unknown  Unknown

fds                00000000007B2FE6  Unknown               Unknown  Unknown

fds                000000000072D969  Unknown               Unknown  Unknown

fds                0000000000734850  Unknown               Unknown  Unknown

libpthread.so.0    000000303980F7E0  Unknown               Unknown  Unknown

fds                0000000000E9E9EA  Unknown               Unknown  Unknown

fds                0000000000E95B08  Unknown               Unknown  Unknown

fds                0000000000818174  MAIN__                    167  main.f90

fds                000000000040981E  Unknown               Unknown  Unknown

libc.so.6          000000303901ED1D  Unknown               Unknown  Unknown

fds                0000000000409729  Unknown               Unknown  Unknown

Can you maybe give me a clue of what the problem is about?

Thank you once again!

Marina

[Kevin]

Create an issue on the Issue Tracker and submit your input file.

[Marina Fragkopoulou]

Done. The issue is:

https://github.com/firemodels/fds/issues/4718

Thank you!

[Marina Fragkopoulou]

Hello,

After following your advice, I am successfully running with OpenMP and MPI enabled.

However, I did not notice any significant improvement in the running time for my specific model. So, I would like to ask you which option is recommended for the model I am running. 

My model consists of 6 fine meshes (0.25*0.25*0.25m) and 6 coarse meshes (2.00*1.00*1.00m) on top of the fine ones. The fine meshes have the same number of cells between them and the coarse as well. However, the number of cells of a coarse mesh (29400 cells) differs significantly from the fine meshes (3200000 cells). In total the model consists of 19376400 cells.

The machine I am using has 100GB of memory and 16 CPU's.

I am running now the model with the following:

- Using 12 CPU's for my 12 meshes and thus setting OMP_NUM_THREADS=1 and -np 12. 

My questions are:

- Is the option of MPI suitable for a model where this comfiguration of meshes?

- If I divided the model into 16 meshes so that each mesh would have less cells and set -np 16, would I get lower simulation time?

- Is there another suggestion for running such a model?

Thank you in advance for your help!

[Randy McDermott]

Yes, MPI is suitable.  But you do not want to go over the number of cores you have available---the meshes (processes) would just start competing for cores.  You do want to load balance as best you can, though. I would divide your fine meshes in two and map all fine meshes evenly to the 12 cores, then divide your coarse meshes in two and map each to a core.  You will have one fine mesh and one coarse mesh per core.  Your coarse mesh is only 1 % of your fine mesh, it will not slow things down.

To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+unsubscribe@googlegroups.com.

To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/27c3de39-1b5e-4fda-b319-51db553e0da8%40googlegroups.com.

[Marina Fragkopoulou]

Dear Randy,

Thank you very much for your help. I already changed the meshes according to your suggestion and I will update regarding the simulation time reduction as soon as I will have the model running. 

Regards,

Marina F.