[FDS-SMV Developer Blog] FDS and OpenMP

[Kevin McGrattan]

Christian Rogsch of the University of Wuppertal has implemented OpenMP directives that enable FDS to make use of multiple processors on a single computer. This is a second option for running FDS in parallel. In fact, although it still needs a bit of testing, OpenMP and MPI should work together. OpenMP (Open Multi-Processing) will run a single mesh FDS job faster by making use of all available processors on a single computer. MPI (Message Passing Interface) requires that you break up the single mesh into multiple meshes, each of which runs on a separate computer. Ideally, we should be able to break up a case over multiple meshes on multiple computers using MPI, and allow OpenMP to run the individual mesh calculations faster.

As a test, I have posted to the Downloads page two executables:

fds5_openmp.exe
fds5_openmp_mpi.exe

These are for 32 bit Windows machines. If this test is successful, we will compile and release executables for other platforms as well. Note that there is currently no guidance in the User's Guide about OpenMP. These executables should work just like fds5.exe and fds5_mpi.exe. That is, there should be nothing you need to install, and you should not need extra libraries. At least, that is what we hope and why we need to test these before doing a more general release. What you should notice is that when you run fds5_openmp.exe on a job, all of your available processors or cores should work on the case. You can check your system performance to make sure.

Please let us know if these executables work for you, or more importantly if they don't. Of course, you need to have a computer, or computers, with multiple processors. At the start of the run, FDS should tell you how many processors (or "threads") it thinks it has access to.

Good luck and thanks again to Christian for all his efforts!

--
Posted By Kevin McGrattan to FDS-SMV Developer Blog at 9/04/2009 05:49:00 PM

[DanielBak]

Kevin, Christian, All

I wish I could report that this great option works for me...I did
download and tried to run the fds5_openmp.exe file with a simple input
file. I got the following error:

<?xml version="1.0" encoding="UTF-16"?>
<DATABASE>
<EXE NAME="fds5_openmp.exe" FILTER="GRABMI_FILTER_PRIVACY">
<MATCHING_FILE NAME="fds5_openmp.exe" SIZE="5419008"
CHECKSUM="0xB98D0E3C" MODULE_TYPE="WIN32" PE_CHECKSUM="0x0"
LINKER_VERSION="0x0" LINK_DATE="09/04/2009 21:18:04"
UPTO_LINK_DATE="09/04/2009 21:18:04" />
</EXE>
<EXE NAME="kernel32.dll" FILTER="GRABMI_FILTER_THISFILEONLY">
<MATCHING_FILE NAME="kernel32.dll" SIZE="989696"
CHECKSUM="0x2D998938" BIN_FILE_VERSION="5.1.2600.5781"
BIN_PRODUCT_VERSION="5.1.2600.5781" PRODUCT_VERSION="5.1.2600.5781"
FILE_DESCRIPTION="Windows NT BASE API Client DLL"
COMPANY_NAME="Microsoft Corporation" PRODUCT_NAME="Microsoft® Windows®
Operating System" FILE_VERSION="5.1.2600.5781 (xpsp_sp3_gdr.
090321-1317)" ORIGINAL_FILENAME="kernel32" INTERNAL_NAME="kernel32"
LEGAL_COPYRIGHT="© Microsoft Corporation. All rights reserved."
VERFILEDATEHI="0x0" VERFILEDATELO="0x0" VERFILEOS="0x40004"
VERFILETYPE="0x2" MODULE_TYPE="WIN32" PE_CHECKSUM="0xFE572"
LINKER_VERSION="0x50001" UPTO_BIN_FILE_VERSION="5.1.2600.5781"
UPTO_BIN_PRODUCT_VERSION="5.1.2600.5781" LINK_DATE="03/21/2009
14:06:58" UPTO_LINK_DATE="03/21/2009 14:06:58" VER_LANGUAGE="English
(United States) [0x409]" />
</EXE>
</DATABASE>

I really appreciate all the great work and improvements!!! Cheers,
Daniel

[Deepak]

Dear Kevin,

Seems to be its starting, but unable to perform the calculations, without starting the calculations its terminating the program.

Please see the message here in below:

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Program Files\newFDS\test_openmp>fds5_openmp cp_fire_actual_3_jet.fds

 Fire Dynamics Simulator

 Compilation Date : Fri, 04 Sep 2009

Version: 5.4.0 MPI Disabled; OpenMP Enabled
 Number of available threads:   2

 SVN Revision No. : 4675

 Job TITLE        : cp fire actual3 jet
 Job ID string    : cp_fire_actual3_jet


C:\Program Files\newFDS\test_openmp>

Please help me.

Best Regards
Deepak

[Jonna]

Hi all!

Just thought I`d share my little first test-run with the open-mp
option. Tried it on two different cases, one personal inputfile, and
one from the test-cases. Both cases were run under Windows Vista
(64bit - I know the released version was for 32bit but i wanted to
test anyway, the "normal" 32bit FDS version works under 64bit without
problems).

The personal input file made fds_openmp crash (the usual windows
program error msg).

Then I tried it with room_fire example. It gave me the error-msg "
ERROR: cannot use both finite rate REAC and mixture fraction REAC".

I then tried the room_fire example on a 32bit machine too, just to
check if it was a problem with 64bit . Got the same errormsg there.

Thanks for the openmp version, im guessing this will seriously cut
down simulationtime on processors with multiple cores!

- Jonna

[PeteM]

I have tried the FDS5_openmp on a few very simple single mesh cases
(Windws 7 dual core P4 laptop 2GB RAM) and they seem to run fine
although I have not yet compared the output with standard FDS5 run.

I have monitored both cores of the machine and they are indeed both
utilised with a noticeable increase in overall computation speed.

Trying to run a multi-mesh file gives the same errors others are
reporting - "ERROR: cannot use bot finite rate REAC nd mixture
fraction REAC"

Will test run the MPI version on our cluster once the LINUX verson is
released.

Cheers

Pete

> > 05:49:00 PM- Hide quoted text -
>
> - Show quoted text -

[PeteM]

OK, seems like the multi-mesh involvement is a bit of a red herring -
The simple single mesh files I ran successfully have had their meshes
split and re-run as multi-mesh jobs.

All ran OK with FDS5_openmp and used both processor cores.

Pete

> > - Show quoted text -- Hide quoted text -

[rogsch]

Dear Deepak,

thanks for your report. The problem is, that you have not enough
"stacksize" to run the simulation. The solution is as follows:

You have to create a new system variable in your system properties.
You will find the menu on System Properties -> Advanced -> Environment
Variables. Then in the system variables menu you have to add a new
variable called OMP_STACKSIZE and the value should be 16M . I have
attached a screenshot in the files section called OMP_STACKSIZE.bmp.
The new variable is the one which is blue. If the problem still
occurs, enlarge this value like 32M but normally 16M are enough.

Maybe you need to restart your system. But then it should work.

If you have still problems, please let us know.

Regards,
Christian

On 5 Sep., 09:35, Deepak <deepak.gan...@gmail.com> wrote:
> Dear Kevin,
>
> Seems to be its starting, but unable to perform the calculations, without
> starting the calculations its terminating the program.
>
> Please see the message here in below:
>
> Microsoft Windows XP [Version 5.1.2600]
> (C) Copyright 1985-2001 Microsoft Corp.
>
> C:\Program Files\newFDS\test_openmp>fds5_openmp cp_fire_actual_3_jet.fds
>
>  Fire Dynamics Simulator
>
>  Compilation Date : Fri, 04 Sep 2009
>
> Version: 5.4.0 MPI Disabled; OpenMP Enabled
>  Number of available threads:   2
>
>  SVN Revision No. : 4675
>
>  Job TITLE        : cp fire actual3 jet
>  Job ID string    : cp_fire_actual3_jet
>
> C:\Program Files\newFDS\test_openmp>
>
> Please help me.
>
> Best Regards
> Deepak
>

[Christian Rogsch]

I am able to reproduce the error "ERROR: cannot use both finite rate
REAC and mixture fraction REAC" but I do not know where it comes
from :-(

I will debug this case and try to fix this error.

Regards,
Christian

Jonna schrieb:

[Deepak]

Hello Christian,

Thanks for your help & efforts.

I have tried with both 16M & 32M stack size, but unfortunately its not working.

Regards
Deepak

--
Sent from my mobile device

[rogsch]

Deepak,

that's strange. Do you get any other message, or is it the same
situation as before (start and stop without any message). If so,
please post your input file, thus I can test it. If you do not want to
post it open for everybody, you can send me an email.

Regards
Christian

[Deepak]

No, its the same as previous(start and stop without error). I will
send you the input file tomorrow.

[rogsch]

The problem with REAC in the room_fire.fds file seems to be compiler
based. I compiled a new windows fds version with a compiler I have
access to, and the file run without this error-message. I have to
check if I'm allowed to post files (executables) created with this
compiler, because it's not "my own" compiler, it's hosted at a
supercomputing center...

Regards,
Christian

[Jonna]

K, nice work, i guess we`ll wait for a new compile from the developers
then if you cant upload.
Thanks for the effort!

- Jonna

[Deepak]

Chiristain,

the problem solved, openmp is working for only when you solve the flow equations (without introducing any fire).

Actually what I did was to avoid the REAC problem, I have removed some parameters related to REAC line and testing that, then its terminating without starting the simulation.

But when you try to solve the same with fire, then the same REAC problem I am facing.

Thank you very much christian.

Best Regards
Deepak

[rogsch]

All,

I uploaded a new FDS OpenMP version to the download section (the new
file has a size of 1.9 mb). This should fix the problems with REAC.
Room_Fire.fds should now run with this new files.

If still any problems occure, please write a short report.

Regards,
Christian

[Kevin]

This might be a problem with GoogleCode. I deleted my old copies of
the openmp exe's and then downloaded the latest executables. The time
stamps on the files (5:13 and 5:18 PM, Sept 4) look like when I
compiled. How were you able to create a new bundle with the same name
as the old? Typically, GoogleCode does not allow this, although if you
delete the old file you may be able to use the same name again.

[rogsch]

I uploaded a new one, it is now called
fds_5.4.0_openmp_win32_update.exe

I created the archive on my computer and named it exactly like the one
which is on GoogleCode. Then I uploaded it by using "new file" and
entered the description and the rows. That's all. I think, GoogleCode
just overwrites your file, because the number of downloads are the
same on my upload as on your upload. This is a little bit strange. To
solve the problem, I renamed the installer.

[Deepak]

I have downloaded the new version, now its working fine without any error messages.

But I have noticed one thing that, I have compared both serial & openmp parallel calculations, its strange that openmp calculation is taking more time than serial calculations, any idea why it is taking more time than serial calculations??

BR
Deepak

[rogsch]

Aha! Very good to hear that it's working fine!

If you have small test_cases like room_fire you will not benefit from
OpenMP. All kind of parallelization has an overhead. This is the time
the computers "needs to say": "calculate this in parallel". If you
have a small testcase like room_fire, which has ca. 67.000cells, the
parallel overhead is larger than the time you "win" by using
parallelization. If you have a testcase with ca. 1.000.000 cells, you
will see, that the OpenMP version will run faster, because in such a
case the parallel overhead is very small comparing to computational
time. In such a case "you will win the game".

Regards,
Christian

[Kevin]

Christian -- you were right. I was downloading an older version of the
"bundle." Your new executables work properly. Now I need to figure out
why my compilation failed. Thanks.

> > > > > - Show quoted text -- Hide quoted text -

[Jonna]

The new compiled version worked for me too, great work rogsch :)

- Jonna

[Deepak]

Christian,

Thank you very much for your efforts & help.

Yes, here the message passing communication from one thread to another thread is taking time. I have noticed this at initial few time steps the parallel openmp calculations are slower but after some time steps the openmp calculations are little bit faster than serial.

Best Regards
Deepak

[rogsch]

Deepak,

you are right. I tested room_fire.fds with double number of cells in
each direction, and for a 1 second run I got the following results:
1 Thread : 233 sec.
2 Threads: 226 sec.
4 Threads: 211 sec.

In this case it is a very poor kind of parallelization.

I tested also jet_fan.fds, 2 seconds, and double number of cells in
each direction. The results are:
1 Thread : 43.7 sec
2 Threads: 30.4 sec
4 Threads: 24.1 sec

In this case the parallelization is better...

If you look only at the velo subroutine time (for jet_fan), you will
see that parallelization with OpenMP here works excellent:
1 Thread : 12,48 sec
2 Threads: 6,81 sec
4 Threads: 3,86 sec

This is a parallelization of ca. 90%, which is very good for a code.

The actual time/parallelization differences are based on the code. To
get an optimal performance there have to be some changes in the code,
which are not done for the 5.4.0 release. In this case the
parallelization is just based on the "original" code without
modifications. Actually the velo and divg routines benefit a lot from
the OpenMP parallelization, PART, FIRE and other routines do not
benefit. This is what we try to solve for future releases. The goal is
to get a "like velo performance" for the whole code. Actually we want
to know from users: Does the new OpenMP parallelization work you, do
you have any problems, and how much faster is the calculation. I hope
that future releases will have a better "total" parallelization. This
kind of work will be continued, it was just the first step...

Regards,
Christian

[Deepak]

Christain,

Yes, you were right.

Now I am testing the new case with around 1,455,000 cells case, for this case I am going to test both OpenMP & MPI version of FDS in our Quadcore machines. I will post the updates for both in couple of weeks, so that we can get to know which one is better. This might help to improve the parallization.

Thank you
Deepak

[Kevin]

I recompiled the OpenMP version with Intel 11.1.035 and the error in
room_fire has been solved. However, I now get access violations in,
for example, the Verification case radiation_plane_layer_4_4.fds.
These are very small, short test cases. Is there a problem when the
case is too small? I have 2 threads available on my XP laptop.

> > > > > > > > > > > > > These are for 32 bit- Hide quoted text -
>
> - Show quoted text -...
>
> read more »

[charlie.thornton]

I'm getting different answers between the serial version and the
openmp version on the fires/cable_tray problem. Here's an image
comparing the temperature slice:

http://www.thunderheadeng.net/images/cable_tray_compare.png

Input file:

http://www.thunderheadeng.net/images/cable_tray_pyrosim.fds

It looks like the openmp version is running quite a bit colder.

On the bright side, I haven't experienced any errors/crashes and the
openmp version has been much faster in my tests. The executables I
used were fds5.exe from the latest 5.4.0 release and fds5_openmp.exe
from the "fds_5.4.0_openmp_win32_update.exe" download.

- Charlie

> > > > > > > > > > > > > > mesh FDS job faster by making use of all available...
>
> read more »

[Emanuele Gissi]

I am ready to test an eventual Linux version of the OpenMP executable
on a dual quad core.
Will you publish a binary? Or should I compile it myself? Or should I
just wait a moment?
Emanuele

> ...
>
> read more »

[David Mcgill]

Hello Emanuele,

If you do compile a LINUX version can you make it available? I would
like to try it on our cluster.

Thanks

Dave

Emanuele Gissi wrote:
> I am ready to test an eventual Linux version of the OpenMP executable
> on a dual quad core.
> Will you publish a binary? Or should I compile it myself? Or should I
> just wait a moment?
> Emanuele
>
>
>

--
Dave McGill
Professor
School of Fire Protection
Seneca College of Applied Arts and Technology
1750 Finch Ave. E
Toronto, ON, M2J 2X5, Canada
416-491-5050, ext. 6186

[Christian Rogsch]

I'm actually looking on an error reported by Charlie Thornton. If I
found the problem I think I can post a linux version.

Regards,
Christian

David Mcgill schrieb:

[Kevin]

I suggest we delay a release for Linux until we solve the Windows
problems and then run the entire Verification suite on both Linux and
Windows. I'll do a blog post this morning.

> > 416-491-5050, ext. 6186- Hide quoted text -

[rogsch]

Charlie,

thanks for your report. It is a bug in the OpenMP version of the
radiation solver. I have to check another reported problem, if I
hopefully find the solution for the other problem I think I will post
a new OpenMP test version.

Regards,
Christian

[charlie.thornton]

I'm happy to help and glad you figured it out so quickly. I think the
OpenMP version will be very popular - thanks for all your hard work!

- Charlie

[jcutonilli]

I noticed a difference in behavior between a static compile and a
dynamic compile. The static compile spawned multiple processes, while
the dynamic compile only had one. I have not checked the results to
see if they were different.

The intel compiler manual does warn against static compiles. "The use
of static OpenMP libraries is not recommended, because they might
cause multiple libraries to be linked in an application. The condition
is not supported and could lead to unpredictable results."

John

On Sep 9, 12:40 pm, "charlie.thornton" <charlie.thorn...@gmail.com>
wrote:

[Kevin]

Thanks, John. Yes, I've seen that warning, but I could not find a
clearly defined alternative to static linking other than somehow
distributing libraries, which could be a real mess. Static linking has
been a problem for us over the years, and I've always wondered why the
compiler developers make it so hard. Don't they expect their customers
to develop and distribute executables to others who may not have the
necessary run-time libs?

> > > > > - Show quoted text -- Hide quoted text -

[rogsch]

All,

I have uploaded a new FDS 5.4.1 OpenMP test version. This version
should fix reported problems with radiation solver and not starting
the radiation_plane_layer_4_4.fds file. I did some tests on my
computer and it runs.

For those who want to compile FDS on their own computer please note
that I uploaded a new makefile (makefile_openmp) for the Intel
Compiler 11.1. This is done to solve some compiler problems. And do
not forget: Compiling and Debugging OpenMP programs is one of the last
adventures of mankind ;-)

Regards,
Christian

[Boehmer]

Thanks Rogsch for the great work on this. This feature looks promising
and I hope it will continue to improve.

To give my brief experience, I tried the openMP executable, v 5.4.1,
on a vista 64 bit machine and after I set the stack size to 32M it
worked smoothly. I tried with a simple model with 512,000 cells
containing a fan, a sprinkler and a radiant panel. There was no
combustion in the model and it was run for 3 seconds.

Using 8 threads (4 physical and 4 virtual cores) on a quad core with
Hyper-threading enabled gave savings of 41 % over the single core.
Turning off Hyper-threading and using only 4 cores increased the time
savings to 46 %. Since FDS does not support Hyper-threading having
this enabled appears to have a negative effect.

Below is the time savings in percent from the single to the quad core
simulations based on the .out file.

1 core 4 cores Savings
MAIN 1037.5 1945.5 47%
DIVG 166.5 426.6 61%
MASS 147.8 327.3 55%
VELO 186.9 588.5 68%
PRES 94.8 96.2 1%
WALL 51.0 56.9 10%
DUMP 12.7 12.9 1%
PART 75.3 117.6 36%
RADI 294.8 311.8 5%
FIRE 0.0 0.0 0%
COMM 0.0 0.0 0%
SubTot 1029.8 1937.8 47%


- Havard.

> ...
>
> read more »

[charlie.thornton]

I've run 4 test simulations both with the serial and openmp versions
of FDS 5.4.1 32-bit on two different computers (I'm gathering
benchmark data for a web page). Using the updated 5.4.1 version, I'm
now getting similar answers out of the openmp and serial versions.
The openmp answers don't seem to be "exactly" the same, I assume this
is because the cells are being iterated over in a different order or
some similar detail. The openmp version is averaging a little better
than 30% faster (e.g. serial = 10 minutes, openmp = 7 minutes) on my
quad core computer, but I've only run it on relatively tiny problems
so far (5400, 19200, 67392, 144000 cells).

I'm excited about what I've seen in the 5.4.1 version, thanks
Christian!

- Charlie

> > > > > > > > > > > > > > The personal input file made...
>
> read more »

[kok leong]

Hi Boehmer, What is Hyper Threading? How to turn it on?

> ...
>
> read more »

[charlie.thornton]

Hyper-threading is a feature available on some Intel processors (P4,
Core i7, Atom). It helps the chip context-switch more efficiently and
it "tricks" the operating system into thinking you have twice as many
available processors - reducing latency when you have lots of jobs
running. I'm guessing it actually reduced Boehmer's speedup (very
slightly) because OpenMP was running 8 jobs on 4 processors.

- Charlie

> > > > > > > > > > > > > > > Hi all!...
>
> read more »

[Boehmer]

Charlie described about as much as I know on the subject of Hyper-
threading (HT) too. I ran it on a core i7 CPU. I believe it was state
here before that FDS does not support HT and from what I saw it
slightly increased the run time versus using only the 4 physical
cores. It took ~100 s longer on a 1000 s run time.

If you have one of the Intel CPUs Charlie mentioned (P4, i7, Atom) you
can enable HT in the BIOS.

- Havard

On Sep 17, 9:45 am, "charlie.thornton" <charlie.thorn...@gmail.com>
wrote:

> ...
>
> read more »

[F-Sim.de]

> here before that FDS does not support HT and from what I saw it
> slightly increased the run time versus using only the 4 physical
> cores. It took ~100 s longer on a 1000 s run time.

Every multithreaded program makes use of HT. The Problem is, that it
only makes sense if the threads have to wait for information in memory
because of cache-misses or if the threads make use of floating-point
and integer operations ~50 : 50.
FDS does none of both, so HT is of no use here.

[rogsch]

All,

thanks for your speedup report.
@Charlie: could you explain where you found differences between serial
and OpenMP version? Maybe there is an error in OpenMP programming,
because normally the results should be the same. If it is possible,
submit a file where I can see the difference easily.

@all: The OpenMP version does not benefit from Hyperthreading, some
points are already discussed here. Thus we have only Floating-Point
calculations in FDS, there is no positive effect because for each
thread we have to use the same parts of the processor, other part are
"sleeping". If we would have some "Integer-only" operations, these
"sleeping parts" could be used and we will benefit. Another point is
the OpenMP or Parallelization overhead. If we have more cores, it's a
little bit more difficult or time consuming to split work to all
threads. If we have now threads which cannot be used (becauwe we have
only floating point operations) this threads have to wait until the
running threads are finished. This means, our for "real" threads (the
cores) have to compute the same amount of work, but we have this
additional OpenMP/Parallelization splitting to do. If we add this time
to our computational time, it should be a little bit slower. But why
is it sometimes a little bit faster? This could be becuase if we split
the work into 8 Threads (Hyperthr.) instead of 4 threads(cores) the
memory usage per thread is only half size, but in total the sum is the
same. Now it could be that some of our smaller threads are so small
that they can use the cache of the prozessor, which is faster than the
RAM of your computer. Thus we benefit from the "movement time" cache-
CPU because it is faster then RAM-CPU. That's why we sometimes see
this small benefit in Hyperthreading. But I think we will find several
other explanations for this effect.

Some of the routines FDS uses are not parallelized yet, because that
needs some changes in the code (or sometimes the compiler is not able
to compile the OpenMP code...). I am still working on this thus future
versions should run faster with OpenMP parallelization, so please be
patient...

Regards,
Christian

[charlie.thornton]

I think the differences I'm seeing can be explained by floating point
precision and are not changing the results in a meaningful way.
Here's the serial and openmp results side-by-side for my simple burner
fire problem. Input file:

http://www.thunderheadeng.net/images/burner.fds

Temperature slice for selected frames:

http://www.thunderheadeng.net/images/frames.jpg

The serial run is on the left and the OpenMP run on the right. In
frame 42, there is no visible difference between slices for the two
runs. In frame 43, the OpenMP run has an extra hot spot in the lower
right. In frames 44,50 we can see that the two runs are starting to
visibly diverge. Frames 75,100 show completely different-looking
temperature slices.

Diff for the CSV data gathered by the temperature sensor at z=1.5m:

http://www.thunderheadeng.net/images/compare.png

When comparing the two CSV output files, it appears that the least
significant digits of the time and temperature begin to vary early
on. This appears to carry over into subsequent iterations and cause
increasing variance.

Graph for the CSV temperature sensor data:

http://www.thunderheadeng.net/images/temp_graph.png

This is a scatter plot of the data in the two temperature device CSV
files.

- Charlie

[cuub]

Hello All,
I have a quastion. Now I'm trying to check new OpenMP release on my
cluster (four duo-core computers on Windows System). My question is
how to start simulation with fds5_openmp_mpi.exe?
With an old version of FDS (without OpenMp) I used config.txt file:
c:\mpiexec -file config.txt
my config txt is:
exe \\_path to folder with fds file and exe file
dir \\_path to a folder where the resuls are hold
hosts
xx1 2 (the number of threadds)
xx2 2
xx3 2
xx4 2

With an old version I need to have at least 8 meshes to use all my 8
threads.
How to prepare fds file now? How many meshes should I use (al least).
Shoul I use config file?
If so how to prepare it?
I would be gratefull for any help,
Best wishes,
Jacob.

[Kevin]

You have choices. OpenMP will automatically use as many cores as there
are on a single machine. So you could break your domain into 4 meshes
and assign one mesh to each machine. The OpenMP will then help to run
each mesh faster on the individual machines.

[charlie.thornton]

I have a model which runs under the serial FDS executable, but crashes
with the OpenMP FDS executable. Here is the input file:

http://www.thunderheadeng.net/images/atf_nist_multifloor_hi_omp.fds

I've run two versions of this problem. The original, smaller version
(3 grids, 228k cells) ran under both simulators just fine. The hi-res
version is modified to have 4 grids, 1026k cells. It seems likely
that my problem is memory-related, but the serial run used only a
little over 1 GB.

The simulation generates this output:

***********
Version: 5.4.1 MPI Disabled; OpenMP Enabled
Number of available threads: 4

SVN Revision No. : 4713

Job TITLE : ATF NIST Experiments with corridors on two floors
connected by a single stairwell.
Job ID string : atf_nist_multifloor_hi_omp

***********

Then crashes - Vista x64 raises a generic "fds5_openmp.exe has stopped
working" error dialog.

- Charlie

[cuub]

Thank You Sir for your answer,
I thought so. Bur this option doesn't work for me. I have a fds file
with four meshes.
My config.txt file looks like:

exe \\_path to folder with fds file and exe file
dir \\_path to a folder where the resuls are hold
hosts

xx1 1 (the number of threads, I also tried without "1")
xx2 1
xx3 1
xx4 1

When I use fds5_openmp_mpi.exe file my simulation "die" with "Fatal
error in MPI_Waitall: Other MPI error, error stack:...".
The same fds file run by fds5_mpi.exe wors fine.
I tried to run very small simulation with OpenMp_MPI FDS which is
divided only to two meshes and it works fine.
Greetings,
Jacob

[Kevin]

Please start an Issue Tracker for this problem. Remember to include
the input file.

[Kevin]

Start an Issue Tracker -- it's easier to work the problem in that
forum. Thanks.

On Sep 21, 10:24 am, "charlie.thornton" <charlie.thorn...@gmail.com>
wrote:

> > > Jacob.- Hide quoted text -

[charlie.thornton]

Should we start sending OpenMP-specific issues through the issue
tracker? I had been keeping them out here since I didn't know if
Christian was an issue tracker...tracker.

- Charlie

[drjfloyd]

Christian can be assigned bug reports through the tracker, so that
would be a good place for them.

On Sep 21, 3:45 pm, "charlie.thornton" <charlie.thorn...@gmail.com>
wrote:

[rogsch]

Jason is right.

I found your issue and wrote a comment. Just for information: it's a
memory problem, it works with a 64bit version, which I compiled to
test the case...

[dji...@gmail.com]

Just a note to all of those involved with or contemplating an OpenMPI
implementation with FDS.

Our work here at the Univ. of Tennessee involves high performance
computing (HPC) using FDS. We recompile the FDS code under
OpenSolaris, optimizing the code for specific quad processor CPU
chipsets.

Using FDS with Sun's HPC ClusterTools 8.2, which is based on the
OpenMPI 1.3.3 release, we have experienced significant improvements
with shortening the execution times for large, multi-mesh models.

Dave Icove

On Sep 4, 6:08 pm, Kevin McGrattan <mcgra...@gmail.com> wrote:
> Christian Rogsch of the University of Wuppertal has implemented OpenMP
> directives that enable FDS to make use of multiple processors on a
> single computer. This is a second option for running FDS in parallel.
> In fact, although it still needs a bit of testing, OpenMP and MPI
> should work together. OpenMP (Open Multi-Processing) will run a single

> mesh FDS job faster by making use of all available processors on a
> single computer. MPI (Message Passing Interface) requires that you
> break up the single mesh into multiple meshes, each of which runs on a
> separate computer. Ideally, we should be able to break up a case over
> multiple meshes on multiple computers using MPI, and allow OpenMP to
> run the individual mesh calculations faster.
>
> As a test, I have posted to the Downloads page two executables:
>
> fds5_openmp.exe
> fds5_openmp_mpi.exe
>
> These are for 32 bit Windows machines. If this test is successful, we
> will compile and release executables for other platforms as well. Note
> that there is currently no guidance in the User's Guide about OpenMP.
> These executables should work just like fds5.exe and fds5_mpi.exe. That
> is, there should be nothing you need to install, and you should not
> need extra libraries. At least, that is what we hope and why we need to
> test these before doing a more general release. What you should notice
> is that when you run fds5_openmp.exe on a job, all of your available
> processors or cores should work on the case. You can check your system
> performance to make sure.
>
> Please let us know if these executables work for you, or more
> importantly if they don't. Of course, you need to have a computer, or
> computers, with multiple processors. At the start of the run, FDS
> should tell you how many processors (or "threads") it thinks it has
> access to.
>
> Good luck and thanks again to Christian for all his efforts!
>
> --
> Posted By Kevin McGrattan to FDS-SMV Developer Blog at 9/04/2009
> 05:49:00 PM

[nonlinear]

i thought i would report on my experience trying to run fds5_openmp
for 32-bit in 64-bit vista. I initially had the same results as Jonna
- i.e. i would get a windows application error when running the
executable. I then tried defining OMP_STACKSIZE = 32M, which Havard
reported worked on his machine, and after initially rebooting vista
immediately BSODed. I was able to reboot normally, but when I try to
run fds5_openmp.exe, I no longer get the windows application error,
but instead fds gives "ERROR: Memory allocation failed for QQ in the
routine DUMP." I have to admidt i'm not quite sure what this
OMP_STACKSIZE variable is, so i removed it from the system.

i'm not sure if this is useful to anyone.

really looking forward to seeing fds5_openmp compiled in 64-bit
win!!! like really, really. has anyone compiled this, or can anyone
give an estimate of when this will be available?

[nonlinear]

does anyone have an estimate for when the 64-bit version will be
released?

thanks!

[Kevin]

We first want to work out the bugs in the 32 bit version before making
it more complicated with 64 bit. If you want to compile it yourself,
buy the Intel Fortran compiler and use the makefile in the FDS
Repository.

[Apollo]

Have installed the latest version of fds5_openmp.exe and tried running
jetfan-fds (example folder) on my PC. Found an error as per attached
jpg file. Unable to understand. I have removed the line OMP = 16M as
it also return another error that 16M was too large. System reboot
thereafter without OMP listed in the Environment Variable menu.

PC specs: Intel Core2 1.66GHz + 2GB DDRAM
WinXP SP3.

only fds_openmp.exe, fds_openmp_mpi.exe and jetfan.fds in the folder.

Pls advise.

[Christian Rogsch]

Seems to be a compiler problem, that a OpenMP library is not linked.
Have you tried it on other computers? Normally, it should work, but I
will look at some forums, maybe one has the same problem.

Regards
Christian

[Apollo]

thanks for your feedback.

I've installed the OpenMPI version over to another Core2Duo system
which has a clean winXP and no other program.

Yes, it is working perfectly when I tested out it with another
example.

> > Pls advise.- Hide quoted text -

[Paul Hart]

I compiled an OpenMP version of Linux x64. A simple two mesh case
shows a reduction in run time of about 50% which is consistent with
info presented earlier in this thread. I did have a segmentation fault
occur initially but this was solved by establishing the environmental
variable KMP_STACKSIZE equal to 20000000 using the command $export
KMP_STACKSIZE=20M

On Sep 21, 1:45 pm, "charlie.thornton" <charlie.thorn...@gmail.com>
wrote:
> Should we start sendingOpenMP-specific issues through the issue

> tracker?  I had been keeping them out here since I didn't know if
> Christian was an issue tracker...tracker.
>
> - Charlie
>
> On Sep 21, 9:35 am, Kevin <mcgra...@gmail.com> wrote:
>
> > Start an Issue Tracker -- it's easier to work the problem in that
> > forum. Thanks.
>
> > On Sep 21, 10:24 am, "charlie.thornton" <charlie.thorn...@gmail.com>
> > wrote:
>
> > > I have a model which runs under the serial FDS executable, but crashes

> > > with theOpenMPFDS executable.  Here is the input file:

>
> > >http://www.thunderheadeng.net/images/atf_nist_multifloor_hi_omp.fds
>
> > > I've run two versions of this problem.  The original, smaller version
> > > (3 grids, 228k cells) ran under both simulators just fine.  The hi-res
> > > version is modified to have 4 grids, 1026k cells.  It seems likely
> > > that my problem is memory-related, but the serial run used only a
> > > little over 1 GB.
>
> > > The simulation generates this output:
>
> > > ***********
> > > Version: 5.4.1 MPI Disabled;OpenMPEnabled
> > >  Number of available threads:   4
>
> > >  SVN Revision No. : 4713
>
> > >  Job TITLE        : ATF NIST Experiments with corridors on two floors
> > > connected by a single stairwell.
> > >  Job ID string    : atf_nist_multifloor_hi_omp
>
> > > ***********
>
> > > Then crashes - Vista x64 raises a generic "fds5_openmp.exe has stopped
> > > working" error dialog.
>
> > > - Charlie
>
> > > On Sep 21, 7:28 am, Kevin <mcgra...@gmail.com> wrote:
>

> > > > You have choices.OpenMPwill automatically use as many cores as there

> > > > are on a single machine. So you could break your domain into 4 meshes

> > > > and assign one mesh to each machine. TheOpenMPwill then help to run

> > > > each mesh faster on the individual machines.
>
> > > > On Sep 21, 6:51 am, cuub <deyv...@gmail.com> wrote:
>
> > > > > Hello All,

> > > > > I have a quastion. Now I'm trying to check newOpenMPrelease on my

> > > > > cluster (four duo-core computers on Windows System). My question is
> > > > > how to start simulation with fds5_openmp_mpi.exe?

> > > > > With an old version of FDS (withoutOpenMp) I used config.txt file:

[Paul Hart]

Using the compiler flag instructing the compiler to link to static
OpenMP runtime libraries results in in further reduction in runtime.

[SimonFrey]

Hello

When experimenting with the actual openmp-version of FDS (FDS 5.4.2,
RV 4959), the program crashes completely if a variable gravity vector
is defined.

The same case runs fine in the not-openmp version.

Cheers
Simon

[dr_jfloyd]

Please post bug reports to the issue tracker along with the simplest
possible version of your input file that reproduces the error.

[Kristopher Overholt]

Hi Laurent:

I mentioned in another discussion thread that the MPI versions of FDS should work on Mac OS X. However, just like on Linux or Windows, there is some initial setup required that can be somewhat involved.

On OS X, you will likely need to obtain the Developer Tools (in Xcode), possibly install a Fortran/C compiler, and then compile/setup/install an MPI library such as LAM/MPI or Open MPI. Then, ideally, you should be able to run the MPI versions of FDS for OS X. Refer to the User Guide section on "Starting an FDS Calculation (Multiple Processor Version)". However, note that the MPI versions of FDS are more commonly used on Linux and Windows.

Kris

On Wed, Aug 8, 2012 at 9:33 PM, richer....@googlemail.com <richer....@gmail.com> wrote:

Hi Kevin, hi all,

I am writing my message quite a few months after yours was sent to the world. however, i have a question:

I purchased a 12-core machine from Apple and was intending to use it to get back into using FDS for analyses. My question is as follows: Are there versions of the FDS files available for running analyses on multiple core processor APPLE/MAC machines?

Although i have used FDS when it came out around 2000 and some time later (2006), i am still a beginner as a lot of time passes before get back into the FDS world. I may need some hand-holding and i thought i'd go to the source for information.

Thanks for any information you may be able to provide me.

Regards,

Laurent

On Saturday, 5 September 2009 02:08:43 UTC+4, Kevin wrote:

Christian Rogsch of the University of Wuppertal has implemented OpenMP directives that enable FDS to make use of multiple processors on a single computer. This is a second option for running FDS in parallel. In fact, although it still needs a bit of testing, OpenMP and MPI should work together. OpenMP (Open Multi-Processing) will run a single mesh FDS job faster by making use of all available processors on a single computer. MPI (Message Passing Interface) requires that you break up the single mesh into multiple meshes, each of which runs on a separate computer. Ideally, we should be able to break up a case over multiple meshes on multiple computers using MPI, and allow OpenMP to run the individual mesh calculations faster.

As a test, I have posted to the Downloads page two executables:

fds5_openmp.exe
fds5_openmp_mpi.exe

These are for 32 bit Windows machines. If this test is successful, we will compile and release executables for other platforms as well. Note that there is currently no guidance in the User's Guide about OpenMP. These executables should work just like fds5.exe and fds5_mpi.exe. That is, there should be nothing you need to install, and you should not need extra libraries. At least, that is what we hope and why we need to test these before doing a more general release. What you should notice is that when you run fds5_openmp.exe on a job, all of your available processors or cores should work on the case. You can check your system performance to make sure.

Please let us know if these executables work for you, or more importantly if they don't. Of course, you need to have a computer, or computers, with multiple processors. At the start of the run, FDS should tell you how many processors (or "threads") it thinks it has access to.

Good luck and thanks again to Christian for all his efforts!

--

Posted By Kevin McGrattan to FDS-SMV Developer Blog at 9/04/2009 05:49:00 PM

--

You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.

To post to this group, send email to fds...@googlegroups.com.
To unsubscribe from this group, send email to fds-smv+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/fds-smv/-/vje6OVGj6MgJ.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Kristopher Overholt]

I should also give details on the procedural setup on my Mac (Macbook Pro running OS X 10.8) that allows the MPI version of FDS to work for me.

1) Install Xcode (free from the Mac App Store), which includes the "Developer Tools". Also, install "Command Line Tools" in Xcode.

2) I have the latest OS X Intel C, C++, and Fortran compilers installed, version 12.1.XX.

3) Compile and install Open MPI using the Intel compiler suite (download Open MPI and configure using ./configure CC=icc CXX=icpc F77=ifort FC=ifort)

4) Build FDS MPI OS X version from the repository (cd to FDS_Compilation/mpi_intel_osx_64, ./make_fds.sh).

I can then run FDS MPI cases using the standard command "mpirun -np X <fds_mpi_executable> <fds_case.fds>", where X is the number of processors to use, <fds_mpi_execuatable> is the FDS MPI executable that I built in Step 4, and <fds_case.fds> is the FDS case that I want to run.

There are likely other ways to get the MPI version of FDS working on OS X using alternative compilers and MPI libraries. But the setup that I described above is very similar to the setup that the FDS development team uses on Linux (Intel compiler suite and Open MPI).

Kris

--
Kristopher J. Overholt
Ph.D. Candidate | Fire Research
The University of Texas at Austin
http://www.koverholt.com
Mobile: (832) 736-3473