Running WRF-Chem with multiple nodes on HPC cluster

[Lin Lu]

Hi all, 

I have a few questions about running the WRF-Chem model with MOSAIC-MOZART 4 bin chemistry, particularly about reducing the elapsed runtime and how to best run the model using multiple nodes and memory specifications. Following are some of my queries. I hope any experienced user(s) can help me.  

1. Is it possible to use multiple nodes for running ./wrf.exe ? If yes, can anyone tell me how to do that?

I have complied using dmpar intel option and currently issuing the following command o submit a job in the HPC cluster-

# Grid Engine options
# -pe specifies the parallel environment to use
#$ -pe sharedmem 32
#$ -l h_vmem=8G
# Initialise the modules framework and load required modules
. /etc/profile.d/modules.sh
# Load openmpi module. Must load MPI compilers and libraries before  running MPI code
module load openmpi/1.10.1
# Run the program
mpirun -np 32 ./wrf.exe

Could you please tell me if and how I can I use multiple nodes to run the model?

2. Do I need to specify the larger memory requirements?? 

3. i am currently getting one day output in 2 days runtime. I am using a 120*170 domain grid and 20km  resolution.

Can you sugggest how can i reduce the very high runtime as I need to do a 3 month simulation.

I will be very thankful for the suggestions.

Regards, Lin

[Alexander Frie]

Hello Lin,

Yes, you can run wrf chem with multiple nodes.  The recommended node numbers for WRF (not wrf chem, but a good reference point) based on domain size can be found here.  

https://forum.mmm.ucar.edu/phpBB3/viewtopic.php?f=73&t=5082

For your smallest-sized domain:
((e_we)/25) * ((e_sn)/25) = most amount of processors you should use

For your largest-sized domain:
((e_we)/100) * ((e_sn)/100) = least amount of processors you should use  

Your execution command looks right. One way to check to see if WRF's running on multiple nodes is to look at rsl.xxx files as there should be 1 for every processor.  

You also need to make sure you did compiled a parallel version of WRF.  

I know this is late, but I hope this helps you or someone in the future!

Alex