FDS on Windows HPC Server

[Kevin]

Has anyone run FDS on a Windows HPC (High Performance Computing) Server?

https://en.wikipedia.org/wiki/Windows_HPC_Server_2008

I am curious to know if the current FDS-SMV download for Windows will work under this framework. If you do use it, tell us what version of FDS you are running.

Thanks

[Andrew]

Hi

We are currently running FDS on a Windows 2012 HPC server.  FDS version 6.5.2 and intel mpi v5.1.3.  The only difference is the mpi runtimes were installed seperately rather than using the bundled installer.

[Kevin]

Thanks. Are you using the binary (fds.exe) that is released with the FDS download, or do you compile yourself? Did you have any trouble installing or running cases initially?

[Andrew]

fds binary.

Yes it took a bit of work to get it running initially and there is still room for improvement in the way I've implemented it.

Is this something that you are investigating for a release version?

[Kevin]

Sort of. I was curious if the binary that we compile would run under HPC. Had you run any other MPI programs on your cluster? What were some of the difficulties, in general?

[Andrew]

Yes we do run aother MPI programmes on the cluster sequentially and multiple FDS cases.

One of the main issues was working out where to point to the different files and the exact commands to use.  Also it required some work to set variables for password authentication.  OpenMP doesnt work that well as the schedular struggles to work out what cores to assign to it.

[Kevin]

On our linux cluster, I run combined Open MPI and OpenMP jobs using a PBS script like this:

#!/bin/bash
#PBS -N job_name(MPI_IB)
#PBS -e /home4/mcgratta/job_name.err
#PBS -o /home4/mcgratta/job_name.log
#PBS -l nodes=4:ppn=12
#PBS -l walltime=999:0:0
export OMP_NUM_THREADS=6
mpirun --report-bindings --bind-to core --map-by socket:PE=6 -np 8 fds job_name.fds

Our cluster has two sockets per node, and 6 cores per socket. The mpirun command options direct the scheduler to assign 1 MPI process to each socket, and this process then utilizes the 6 cores associated with that socket. For OpenMP to work well, there must be dedicated cores for it, and you don't want those cores to be spread across multiple processors/sockets and you don't want those cores to be dedicated to anything else.

You should see if there is a way to do this on your HPC cluster. The tricky part is that you need to know how your computers or nodes are configured. It is very confusing because the computer vendors sometimes inflate the number of "processing units" to include things like hyper-threading.

But if you get MPI running well, you will want to use that most of the time. I only use combined MPI and OpenMP when our cluster is empty and there is an abundance of free cores.

[JD]

Andrew,

Is the operating system of all the nodes in your cluster Windows HPC?

I have a cluster that consists of a Windows Server 2008 machine that is used as data storage and where I start my jobs, and my "compute nodes" are FDS-dedicated, multi-core, multi-socket Windows 10 machines. Ideally, i would like to only transition the Server machine to HPC.

[Andrew]

Yes

Everything is running Windows Server with HPC pack.  I think you can use Windows 10 worker nodes on a windows HPC cluster but have no experience of doing this.