performance versus processor count

[David Sheppard]

Are there any recent studies about the scalability of FDS with different numbers of processors? The last time I checked the performance versus processor count started to really drop off when the CPU count got above about 400 CPUs.  Is this still the case?

I have an upcoming project with a very large domain and I am trying to determine how many processors to use.

[Randy McDermott]

Dave,

We only have access to 432 cores, so that is how high we go to test.  But that number is not the limit of scaling.  The strong scaling *should* be able to go much higher, it depends on where you start.  The communication will start to become expensive once you hit about 16^3 \approx 4000 cells per mesh.

On Wed, Sep 27, 2023 at 12:22 PM 'David Sheppard' via FDS and Smokeview Discussions <fds...@googlegroups.com> wrote:

Are there any recent studies about the scalability of FDS with different numbers of processors? The last time I checked the performance versus processor count started to really drop off when the CPU count got above about 400 CPUs.  Is this still the case?

I have an upcoming project with a very large domain and I am trying to determine how many processors to use.

--
You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/ae660a74-1d59-4151-8869-2462074cdfabn%40googlegroups.com.

[David Sheppard]

The last time that I did a scaling study for my typical types of FDS models, I found that I got about a 40% increase in speed each time I doubled the number of processors.  For example, if a mode ran 100s in a given time period, doubling the number of processors would allow the model to run 140s in the same time. This relationship broke down when I got to the 300 to 400 processor count range, when the speed increase was less as I added processors.  FDS never failed in my scaling tests up to the maximum number of processors that I tested, 864.

There have been a lot of FDS updates since I last ran scaling tests. I was just wondering if anyone had done a more recent study.

FYI: I typically consider using more processors for two reason: 

(1) for a set grid size when I want to use a bigger computational domain, and 

(2) when the domain size is fixed, and I want to use a smaller grid size.

[Kevin McGrattan]

Dave, we used the attached plot to assess scalability. It is in the User's Guide. In a nutshell, we take a very large mesh and run 100 time steps with one process. Then we split the mesh in half, quarters, etc. Perfect scaling would be where the solid black line follows the dashed line. After 432 processes (meshes), we see that the job is taking about twice as long as ideal. That's 50% efficiency. This case is very simple, so a more realistic case might be a bit worse. Here we are looking at the various subroutines and looking to see what is not scaling.

[dr_jfloyd]

This is a plot a made for our cluster at the FSRI (Currently 6 blades with 2 x 32 cores for 384 cores) while we were playing with UEFI settings. Runs are powers of 2 up to 64 cores then +32 core intervals to 384. COMP is the sum over modules with no MPI calls, COMP_COMM are computational modules with MPI calls, and COMM is main end of timestep MPI exchanges.  COMP stays linear up to 8 cores at which point we start hitting memory bus throughput limits for a single CPU. That penalty increases until the blade fills at 64 cores and then goes back to being linear.

[David Sheppard]

Thank you for the feedback. I got similar results, about 5 years ago, during my last speed test. I just wanted to see if anything has changed before I started a new project with a very large domain.