Running FDS on Amazon EC2

[leenkne...@gmail.com]

Dear FDS-users,

We are on a schedule to finish a number of simulations and are running out of local calculating resources to run them. I have some experience with running Openfoam on Amazon EC2 and was wondering if it makes sense to run our FDS-cases on Amazon.

I have not found a AMI with FDS ready installed? I should manage to do this myself, but would appreciate some feedback on which types of processors to use and the potential speed-up with parallel calculations. In my limited experience, running an FDS-case on 2 or 4 nodes results in nearly equal calculation times.

Thanks!

Sarah

[Dave McGill]

Hi Sarah,

Have you considered using Sabalcore? (http://www.sabalcore.com/vertical-markets/engineering/fds-smv/). FDS, and all of the associated MPI libraries are already installed. I started using them last year, and have been quite happy with the service.

Dave

[Randy McDermott]

Sarah,

We have discussed setting up an Amazon Machine Image (AMI) for FDS and that is in our future plans, but one is not currently available.  We would be interested in hearing about your experience if you set one up.  

Thank you!

Randy

--
You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+unsubscribe@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/f8ed3bac-1396-4c93-bb9d-09814d398c90%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[leenkne...@gmail.com]

Thanks for the feedback. I will take a look at Sabalcore and test the free trial. If we do end up at Amazon, I can give some feedback about setting up an AMI with FDS. 

Some feedback regarding the best strategy for mesh assignment to multiple cores is also appreciated, ex. does it make sense to run a case on more than 4 cores?

I have a case with a total of 36 meshes, divided over 4 floors (9 meshes per floor, each mesh represents a room or corridor, these chunks are very different in size, but each core has roughly the same number of cells assigned). The total mesh size is appr. 2.3 million cells. So far I am running it on 4 cores (one floor per core), but the speed-up from 2 to 4 cores is only 20-30 %. It takes me about 10 minutes to simulate 1 second, using i7-6700k cores. Is this expected or are we losing efficiency somewhere? It seems extremely slow to me, but I am rather new to FDS and find it hard to assess. 

My bottom line question is: if we turn to cloud computing, the big theoretical advantage is to run a case with lots of parallel cores. However, that seems less relevant for FDS. I can run several cases in parallel, but the parallellization of one single case is not really better than what we are doing now locally, correct? I am just trying to assess if cloud computing is worth the cost for FDS.

Thanks!

Sarah

[Kevin]

If you only have 4 cores, it doesn't make much sense to break the case into 36 meshes. Break it into 4 meshes, one per floor. However, don't be surprised if it runs slowly. 10 min per s of simulated time is far too slow for a realistic application that could require tens of simulated minutes. You should check the time step size in the .out file. Roughly speaking, the time step size should be the size of a grid cell divided by the maximum value of the velocity. Sometimes, people set up cases in which gases flow quickly through a tiny hole, slowing everything down. Or very powerful sprinklers might slow things down. Try to figure out if your time step is appropriate for your case -- if it's not, you might want to investigate before trying to run big jobs on big computers. I find that most people just create an elaborate geometry and fire scenario, and then just launch it without studying whether the basic output diagnostics make sense.

If you find that your simulation is "healthy"; that is, behaving as you would expect both physically and numerically, then explore getting more cores and running bigger domains. I am amazed that even with all the cheap computing resources available these days, that people still try to cram big FDS jobs on a single Windows PC.

[dr_jfloyd]

It often makes sense to divide into more meshes than cores. In Sarah's case, if each floor has some complex serpentine corridor and she is only looking at smoke movement in the corridor and perhaps a small number of rooms on each floor, then 1 mesh per floor will waste a significant amount of computational time by meshing cells that don't play any role in the results. 

[Drew Martin]

Sarah,

I have been using FDS on windows instances on EC2 for about 6 months and I have had a few issues with it, but overall it works really well (and is worth the cost) for two use cases:

• The number of simulations for the project exceeds our internal cluster
• The deadline for completing the FDS runs is faster than our internal cluster is able to complete.

It sounds like you are running into both of these issues, so as long as you have a healthy budget on this project AWS sounds like good resource for you. I have found that it takes a bit of work to set it up though and it is important that you think about the full fds run before you start (i.e. building an AMI, running the FDS run with appropriate storage attached to the instance, and downloading the data after the run is finished). 

1. Because there were no AMI's available I have created several corresponding to our internal needs. I would recommend building an AMI on one of the cheaper instances and then using it for your compute work load. This might include the installation of FDS, any scripts or other software you will be using to run the FDS files, any software or tools you are using to move your data around.

2. Make sure when you start you attached enough storage to your instance so that it does not fill up (this leads to a long process of expanding the drive)

3. One of the bigger challenges we found once the runs are done is what to do with the files. As you know a moderate FDS run can produce anywhere from 20GB to 200GB of data and moving that from AWS to your personal computer quickly is challenging. 

FDS RUN

It sounds like you have one fds file that has 36 meshes. Ideally if you have the budget you would select the c4.8xlarge  instance as you can run each mesh on a separate core. That is where you will find the greatest speedup, but it is expensive to do this. I would recommend that you re-mesh your model to match the number of 'vCPUs' that are in the instance type you are using.

Improvements

I am working on a number of improvements for our process. 

1. We are using windows instances, which can be time consuming and difficult to set up and I end up using one instance per simulation, which can be difficult to track. Ideally I will switch over to building a linux HPC, however I have no experience with building one.

2. Cost control is important when using the instances and managing this better is critical.

3. Storage is a huge cost when scaling up and needs to be managed 

Drew

[dr_jfloyd]

You don't have to remesh if you have more meshes than MPI processes; you can use MPI_PROCESS on MESH to assign meshes to specific processes. If you have meshes on the same process where combining them doesn't increase the cell count, then it would be good to combine them.  You should also try and balance the cell counts on each process as best you can.

[leenkne...@gmail.com]

Thanks everyone for all your feedback.

Thank you Kevin for your remark, I was indeed reducing my time step by letting too much air pass through a too small outlet. This is fixed and we have a speed-up of roughly factor 5, which is very nice. 

I will try the free trial at Sabalcore, see how much time that could save us. 

Will also give Amazon a go, if we need to. 

I have indeed used the mesh strategy described by dr_jfloyd: defining meshes only for the corridors and rooms that I need, and assigning them to the processes with MPI_PROCESS. 

Sarah

[leenkne...@gmail.com]

We are testing Sabalcore, and I ran into a questions. A quick google search did not turn up anything useful.

I tried to run a case on 4 cores for 200 s, then pauze it and then restart the case on 8 cores with an adapted FDS-script? The only changes to the script are the assignment of the meshes to the MPI_PROCESS, taking into account that they have to be ordered counting upwards monotonically.

The case runs from t=0s for 4 and 8 cores. The restart on 4 cores also runs without problems, but I haven't been able to get the restart function to work when switching to 8 cores.

I get the following error message:
forrtl: severe (67): input statement requires too much data

Thanks!

Sarah

[dr_jfloyd]

It may be that you cannot do this.  There are some variable arrays (used for doing the MPI information exchanges) that are dimensioned with the number of MPI processes being used.  If we are writing one of those to the restart file, then an array written with a dimension of 4 would lead to your error message if you tried to read it with a dimension of 8.

[Kevin]

I'm fairly certain that you cannot do this. It was never our intention to restart FDS with a different number of MPI processes.

[Michael]

Hello,

some weeks ago I configured a single c4.8xlarge machine with 36 cores and AMI Linux to run FDS MPI projects.

I'm not a linux pro and so it was a little bit tricky but after a while it worked.

I ran the same test case (36 meshes)  on EC2 and Sabalcore. The simulation at Sabalcore needed about 70% of the time compared to EC2 and so I have not continued to pursue this matter.

Perhaps with another EC2 instance or another Linux or MPI version it will be faster.

Best regards 

Michael

 

Am Dienstag, 3. Januar 2017 12:42:49 UTC+1 schrieb leenkne...@gmail.com:

[leenkne...@gmail.com]

Just to inform people: I did manage on a local machine to run a case on 2 processors and later on make a restort on 4 processors (using MPI_PROCESS to assign meshes).

If you try this though, check the heat output in the HRR file, if the fire curve is correct. I haven't had the time yet to check it here, but it looks like the heat output may take a dip at the restart time.

[Marina Simó Martí]

Hello,

I'm also using an AWS EC2 instance to run FDS in a Linux shell. It is a c6i.24xlarge instance that has 24 processors and 96 vCPU (I'm not using any HPC cluster). The program works when running simple cases and just using the command "fds" + "input.file". However, I need to specify the MPI processes and how many CPUs I want per MPI task. I haven't been able to figure out how to do it, if someone could tell me the lines I should type that would be awesome.

I'm using this line:

mpiexec -np 5 /home/ec2-user/FDS/FDS6/bin/fds 2_Baixa_cat.fds

-np specifies the 5 MPI processes, however I don't know how to assign CPUs. And the "export OMP_NUM_THREADS="

 doesn't work either, I can't activate the OpenMP 

1. On the other hand, the simulation starts but it gives me the following error:

[ec2-user@ip-X 2_Baixa]$ bash fds_input.sh

 Starting FDS ...

 MPI Process      3 started on ip-Xeu-west-1.compute.internal
 MPI Process      0 started on ip.Xeu-west-1.compute.internal
 MPI Process      1 started on ip.Xeu-west-1.compute.internal
 MPI Process      2 started on ip-Xeu-west-1.compute.internal
 MPI Process      4 started on ip-Xeu-west-1.compute.internal

 Reading FDS input file ...

WARNING: SPEC FUEL VAPOR is not in the table of pre-defined species. Any unassigned SPEC variables in the input were assigned the properties of nitrogen.

 Fire Dynamics Simulator

 Current Date     : February  2, 2024  12:50:54
 Revision         : FDS-6.8.0-0-g886e009-release
 Revision Date    : Tue Apr 18 07:06:40 2023 -0400
 Compiler         : ifort version 2021.7.1
 Compilation Date : Apr 18, 2023 15:20:17

 MPI Enabled;    Number of MPI Processes:       5
 OpenMP Disabled

 MPI version: 3.1
 MPI library version: Intel(R) MPI Library 2021.6 for Linux* OS


 Job TITLE        : (Concatenated) Simulaci� AET article: continuitat baixa, FCC < 50%, density = 0.2
 Job ID string    : 2_Baixa_cat

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
libc.so.6          00007F9B40C54DD0  Unknown               Unknown  Unknown
fds                0000000006D848A6  Unknown               Unknown  Unknown
fds                0000000006D70077  Unknown               Unknown  Unknown
fds                0000000006AB7C99  Unknown               Unknown  Unknown
fds                000000000040A71D  Unknown               Unknown  Unknown
libc.so.6          00007F9B40C3FEB0  Unknown               Unknown  Unknown
libc.so.6          00007F9B40C3FF60  __libc_start_main     Unknown  Unknown
fds                000000000040A636  Unknown               Unknown  Unknown

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 0 PID 8749 RUNNING AT ip-172-31-6-71.eu-west-1.compute.internal
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 1 PID 8750 RUNNING AT ip-172-31-6-71.eu-west-1.compute.internal
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 2 PID 8751 RUNNING AT ip-172-31-6-71.eu-west-1.compute.internal
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 4 PID 8753 RUNNING AT ip-172-31-6-71.eu-west-1.compute.internal
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

2. If I only execute, without specifying "-np":

mpiexec /home/ec2-user/FDS/FDS6/bin/fds 2_Baixa_cat.fds

for some reason I don't understand it enables 47 MPI Processess, even though I only have 9 meshes.

===================================================================================
[ec2-user@ip-X 2_Baixa]$ bash prova.sh

 Starting FDS ...

 MPI Process     11 started on ip.X.eu-west-1.compute.internal
 MPI Process     26 started on ipX.eu-west-1.compute.internal
 MPI Process     28 started on ip-X.eu-west-1.compute.internal
 MPI Process     42 started on ip-X.eu-west-1.compute.internal
 MPI Process      0 started on ip-X.eu-west-1.compute.internal
 MPI Process      3 started on ip.Xeu-west-1.compute.internal
 MPI Process      4 started on ip-X.eu-west-1.compute.internal
 MPI Process     12 started on ip-X.eu-west-1.compute.internal
 MPI Process     13 started on ip-X.eu-west-1.compute.internal
 MPI Process     15 started on ip-X.eu-west-1.compute.internal
 MPI Process     16 started on ip-X.eu-west-1.compute.internal
 MPI Process     33 started on ip-X.eu-west-1.compute.internal
 MPI Process     35 started on ip-X.eu-west-1.compute.internal
 MPI Process     46 started on ip-X.eu-west-1.compute.internal
 MPI Process      2 started on ip-X.eu-west-1.compute.internal
 MPI Process      5 started on ip-X.eu-west-1.compute.internal
 MPI Process      6 started on ip-X.eu-west-1.compute.internal
 MPI Process      9 started on ip-X.eu-west-1.compute.internal
 MPI Process     10 started on ip-X.eu-west-1.compute.internal
 MPI Process     14 started on ip-X.eu-west-1.compute.internal
 MPI Process     19 started on ip-X.eu-west-1.compute.internal
 MPI Process     20 started on ip-X.eu-west-1.compute.internal
 MPI Process     23 started on ip-X.eu-west-1.compute.internal
 MPI Process     25 started on ip-X.eu-west-1.compute.internal
 MPI Process     34 started on ip-X.eu-west-1.compute.internal
 MPI Process     40 started on ip-X.eu-west-1.compute.internal
 MPI Process      7 started on ip-X.eu-west-1.compute.internal
 MPI Process     17 started on ip-X.eu-west-1.compute.internal
 MPI Process     18 started on ip-X.eu-west-1.compute.internal
 MPI Process     22 started on ip-X.eu-west-1.compute.internal
 MPI Process     24 started on ip-X.eu-west-1.compute.internal
 MPI Process     27 started on ip-X.eu-west-1.compute.internal
 MPI Process     29 started on ip-X.eu-west-1.compute.internal
 MPI Process     30 started on ip-X.eu-west-1.compute.internal
 MPI Process     31 started on ip-X.eu-west-1.compute.internal
 MPI Process     32 started on ip-X.eu-west-1.compute.internal
 MPI Process     36 started on ip-X.eu-west-1.compute.internal
 MPI Process     38 started on ip-X.eu-west-1.compute.internal
 MPI Process     45 started on ip-X.eu-west-1.compute.internal
 MPI Process      1 started on ip-X.eu-west-1.compute.internal
 MPI Process      8 started on ip-X.eu-west-1.compute.internal
 MPI Process     21 started on ip-X.eu-west-1.compute.internal
 MPI Process     37 started on ip-X.eu-west-1.compute.internal
 MPI Process     39 started on ip-X.eu-west-1.compute.internal
 MPI Process     41 started on ip-X.eu-west-1.compute.internal
 MPI Process     43 started on ip-X.eu-west-1.compute.internal
 MPI Process     44 started on ip-X.eu-west-1.compute.internal
 MPI Process     47 started on ip-X.eu-west-1.compute.internal

 Reading FDS input file ...


ERROR: The number of MPI processes, 48, exceeds the number of meshes, 9 (CHID: 2_Baixa_cat)

ERROR: FDS was improperly set-up - FDS stopped (CHID: 2_Baixa_cat)