WRF-Hydro Simulation Issue with Higher Processor Counts

[msvramana]

Hello everyone,

I am currently running the WRF-Hydro model on an HPC system that has 40 processors available. For a one-month simulation, I attempted to run the model using 12 processors. However, the model failed to run and terminated with an "MPI aborted" error. Since the error does not appear to be related to the model setup itself, I reran the same configuration using 4 processors, and the simulation is running successfully.

I am trying to understand the reason for this behavior and would appreciate your guidance on the following questions:

1. Are there any restrictions or recommendations in WRF-Hydro regarding the maximum number of processors that can be used? For example, is there a limitation that would prevent efficient execution with more than 4 processors?
2. In the WRF atmospheric model, I am familiar with the CFL stability criterion and how to manage it appropriately. However, I have not come across any similar CFL-related criteria or processor-related constraints in WRF-Hydro. Am I overlooking any important considerations in this regard?

For reference, I have attached the two WRF-Hydro namelist files used in these simulations.

Thanks and regrds,

Sai.

[Soren Rasmussen]

Hi Sai,

 

1. Are there any restrictions or recommendations in WRF-Hydro regarding the maximum number of processors that can be used? For example, is there a limitation that would prevent efficient execution with more than 4 processors?

There are not necessarily restrictions on the number of processors, though we have seen issues when users are running with reach based channel routing configurations and with a higher number of processors. 

Could you build and run the model in Debug mode to see where the model crashes with the higher core count? I have a reproductor but want to make sure we are getting the same bug.

I build in Debug mode with the following commands from the top source directory:

```

$ cmake -B build -DCMAKE_BUILD_TYPE=Debug

$ make -C build -j 4

```

Cheers,

Soren

--
You received this message because you are subscribed to the Google Groups "wrf-hydro_users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wrf-hydro_use...@ucar.edu.
To view this discussion visit https://groups.google.com/a/ucar.edu/d/msgid/wrf-hydro_users/CAGgRQ_9HcZcT%3D-6-Rzy0mVZLAyAxk_hter%3DPfS4uLUftJZUBsg%40mail.gmail.com.

--

Soren Rasmussen, Ph.D.

Water Cycle Applications Program

Research Applications Laboratory

NSF National Center for Atmospheric Research (NCAR)

[Soren Rasmussen]

Hi Sai,

In WRF-Hydro PR#898 I have a fix to the reproducer I have. You could try testing with that PR to see if it fixes your issue?

Cheers,

Soren

[msvramana]

Hi Soren,

Thank you for your response and for your suggestions regarding the MPI abort issue.

Following your recommendation, I checked my WRF-Hydro build configuration and found that the model had not previously been compiled in Debug mode. I have now rebuilt WRF-Hydro in Debug mode using the following commands from the top source directory:

$cmake -B build -DCMAKE_BUILD_TYPE=Debug

$make -C build -j 4

I also verified that the build was successful and confirmed that the build type is now set to:

CMAKE_BUILD_TYPE=Debug

My next step is to rerun the same one-month simulation using 12 processors, which previously failed with the MPI aborted error. Once the run completes (or crashes), I will collect the debug output and share the relevant log files and error messages with you for further investigation.

Thank you again for your help and for looking into this issue. I appreciate your support.

Best regards,

Sai

To view this discussion visit https://groups.google.com/a/ucar.edu/d/msgid/wrf-hydro_users/CADwawewCAZhtZ%3D6yZjT%3DkBv-ektivv46BUB6RuGqJ3Tik3RPOA%40mail.gmail.com.

--

.....MSVRAMANA.....