Compiling FDS 5.5.3 with OpenMPI 1.8.4 on Linux

[Sam J]

Hi,

I have successfully setup a Linux cluster with FDS 6.5.1 and OpenMPI 1.8.4 on all nodes. From this point onwards, my team expects to have all new case runs done with this FDS 6.5.1 or newer, but we also want a working version of FDS 5.5.3 around because it was used for running hundreds of cases on old projects. All our past FDS5 runs were done on Windows and we only got on board with Linux recently in order to resolve MPI clustering issues on Windows. Needless to say, we are still climbing the Linux learning curve. While we only want FDS 5.5.3 for sanity sake, installing it alongside FDS 6.5.1 is appearing to be quite the challenge.

I had downloaded the precompiled version of FDS 5.5.3 for Linux-64 from bintray but this was compiled with LAM/MPI. LAM-MPI is nowhwhere to be found anymore(links are dead) and in some discussions, users noted that it shares the same mpirun name as OpenMPI. This would require the user to be establish a method to run the right mpi program for the right fds even if LAM-MPI can be found. The only plausible option seems to be to compile fds5.5.3 with openmpi 1.8.4.

As a Linux newbie, here is the part where I need the expertise of someone who has compiled fds5.5.3 with OpenMPI 1.8.4 recently or has the skills to do so:

What I have tried/done so far:

• Downloaded github commits from circa Nov 1 2010 which is when FDS release 5.5.3 was compiled. This was on page 554 for me when browsing the commits (a517fdb).
  
• Installed gfortran by typing sudo apt-get install gfortran since getting the intel fortran compiler seems to require jumping through a few hoops.
  

Where I am lost:

• -The downloaded commits seem to have a file called make_fds.csh in the mpi_gnu_linux_64 folder. I have examined this file which points to the set_gnufort.csh file in the scripts folder. The set_gnufort.csh file has these lines:
     echo Using GNU Fortran compilers
     if ("$platform" == "ia32") then
           set path = (/usr/local/lam7g/bin $path)
      endif
     if ("$platform" == "intel64") then
             set path = (/usr/local/lam7g_64/bin $path)
      endif
  Since my platform is intel 64 I have changed the path to where OpenMPI resides on my cluster nodes: (/home/$USER/.openmpi/bin $path)
  Is this the right idea?
  
  
• After the previous change I am simply invoking the make_fds.csh file by navigating to the proper directory in Ubuntu terminal window and typing: "csh make_fds.csh"
  I believe I have to type csh because by default, ubuntu terminal uses bash shell language. After typing the above, I immpediately get a bunch of errors mentioning .f90 files.

Apologies for not being specific with the compilation errors in this post. I can update this thread with the exact error tomorrow. However, by outlining my steps, I am looking to find out what additional steps I am missing. Are there additional changes that I need to make to compile with OpenMPI-1.8 4? Is the GNU Fortran compiler problematic in general? Should I be looking into getting the Intel Fortran compiler? I am just a clueless newbie at this point.

I would really appreciate if a anyone who has recently compiled FDS 5.5.3 with OpenMPI or has the required skillset could help me out here. I am hoping to not have to dual boot windows just to have a working copy of FDS 5.5.3 on these machines.

Thanks

Sam

[Sam J]

Attached are the errors are getting when attempting to use GNU Fortran compiler:

[Sam J]

Ok I finally had success compliling FDS 5.5.3 with openMPI 1.8.4 on linux. Here are the steps if it is helps anyone:

First I extracted the precompiled open-mpi 1.8.4 tar uploaded on bintray by fds-smv team and changed all my paths and library paths to this rather than a self compiled version of openmpi 1.8.4 since I figured that this configuration of openmpi was probably used to compile the last few versions of fds6.

Then, I forked and cloned the git
then did a checkout of the SHA associated with the Nov1,2010 edit. In my past attempts I was manually downloading zip fron github website for the files associated with the old commit. Not sure if there was anything wrong with that approach but thought I might as well install git.

I installed the intel fortran compiler as part of the parallel studio xe 2016 suite. Got a serial id for non commercial use emailed to me within a couple of business days and didnt bother downloading most of the develepor tools as long as the core conpilers were getting installed (few hundred megabytes). I used the path of the installed compiler to set the IFORT_COMPILER variable in my .bashrc file.

I used the same make_fds.sh and set_mpidist.sh scripts for new fds6 version as opposed to .csh scripts included in the past. I also scrolled down to mpi_intel_linux_64 in the makefile and changed fortran compiler flags to mpifort instead of mpif77 and c compiler to mpicc instead of icc with the same fortran compiler flag options used in the current fds6 makefile.

After the above, I could get thr compiling to start but got the following errors that were causing it to fail:
LD: cannot find ltorque
LD: cannot find lnuma

This was easily fixed by installing the missing libraries. I simply typed
sudo apt-get update
sudo apt-get install libtorque-dev
sudo apt-get install libnuma-dev

This finally allowed the compilation to finish with just a few warnings. Still not sure if I should have used any different compiler options but I have tested the compiled fds5_mpi_intel_linux_64 and it works.

[LOH AI CHOONG]

Hi Sam,

A side topic. You mentioned about the shift to Linux to resolve MPI clustering issues on Windows. Are you saying that you was using cluster in Window before in fds5.5.3a? Is the clusters are in window domain? Asking so because I am kind of like trying to do a cluster run in window 7 , which is workgroup based, of cause failed. 

And for Linux, hows the things go? I mean are you facing any special or remarkable problem on shifting from window to linux? I am considering a shift also for my current employer which have 14 of the workstations, all in windows 7. Reason is I cant run more than 1 mpi run in window machine for fds 6.5. 

Regards,

Lacer (CFD Engineer in a small fire firm)  

[Cian Davis]

Hi Lacer,

The only difficulty with moving to Linux is your familiarity with it. Our cluster (used mainly for FDS) is completely Linux. It has a low powered head node, with a lot of disk space (7TB), which is mounted on each of our 8 compute nodes (using NFS). There is no graphical user interface running on any machine and users login to the nodes using SSH and run their FDS jobs command-based (usually in screen so they can disconnect but leave the job running). You will also need to install and configure other software, such as Samba so that people can access the files from a windows machine.

In my case, Linux was the obvious option as I have a lot of experience administering Linux machines. If you only have experience with Windows machines, it will take you time to learn how to install and maintain Linux machines. As a Linux advocate, I would say the time will probably be worth it (I have seen on this list that the same job will run faster on the same machine with Linux rather than Windows, but I have no hard data I can point to) but it will be a choice you will have to make yourself.

On a side note, my approach has been to build compute nodes using high-end i7 desktop processors, rather than go for Xeons, even though these give extra cores. The reason is that while the overall performance of the processor is better, the compute performance per core drops like a stone. Also, the cost per unit of performance is much higher (more expensive CPU, motherboard, RAM...). Generally, we run with a minimum of 150k cells per core so my logic was that anything below this, the MPI overhead would mean the gains would be small - though posts I have seen here suggest that there are improvements down to 40^3 cells (I think that was the number). I have meant to do tests, but just haven't found the time.

Regards,

Cian

--
You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+u...@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/928f4733-ca22-457a-870e-cbe740d5959e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[LOH AI CHOONG]

Hi Cian,

Yea, I am kind of completely new in Linux, just installed the Ubuntu desktop version in personal computer and played around for few days before only(I still doing most of the things in windows).  Alot of the terminology u have shared are quite new to me, such as those NFS, SSH, Samba. Where would you suggest to read/learn all these up in minimum time? 

About the nodes, I will be working to link a 14 to 20 machines, most of them with Xeon and in windows. I have been trying to do that since a year ago, which I was using fds5.5.3a, to link them in workgroup. End up not successful, yet. This shift to fds6.5 is kind of like a necessary as most of the fire engineer are using fds6 already (right?), although fds5 solving speed still preferred , i.e. I can solve 25 mil mesh number case with 35 meshes in a single machine yet still load another smaller case of 2mil mesh together in the same machine, FDS6 seems like dont have such luxury anymore as understand from your post. 

Regards,

Lacer

[Sam J]

Just to provide an update for anyone venturing into this territory (compiling FDS 5.5.3 for use on linux cluster), compiling with OpenMPI 1.8.4 was a bad idea for me. The run times were super slow (as much as 4 times) compared to running FDS 5.5.3 running on same spec WIndows machine with mpich 1.0.3. I don't have the expertise to figure out why and it could also be some mistake in compilation. However, I decided to just use the same mpich version as windows. One can still find MPICH2 v1.0.3 for linux. Newer mpich versions for linux didn't work for me but MPICH2 v1.0.3 worked flawlessly with FDS 5.5.3 on the linux cluster with same performance as Windows. 

I tried a few different configurations when compiling mpich 1.0.3 on Linux such as different process managers (mpd/smpd) and ch3:device(ssm/shm/sock). While smpd worked great for single node multicore runs, I could not get it working on a cluster of nodes. Finally, configuring mpich with pm=mpd and ch3:device=ssm worked best for me. Setting up mpdrings across the cluster using ssh took some trial and error but I finally found out this was being caused by a problem in /etc/hosts file. 

Just had to recompile FDS5.5.3 after pointing the scripts to the mpich installation as I had previously mentioned. I also had to change compiler flags in the makefile to the following to compile successfully without errors: