Parallel Execution Limitations in WRF-Hydro

32 views
Skip to first unread message

Aref Farhangmehr

unread,
Jun 12, 2025, 3:53:15 AMJun 12
to wrf-hydro_users
Hi WRF-Hydro Team;

I hope this message finds you well.

I'm currently attempting to run WRF-Hydro in parallel using mpirun, but I'm encountering a limitation: I cannot run the model with more than 2 cores when creating DOMAIN files using ArcCatalog, and no more than 4 cores when generating DOMAIN files using Python scripts. The error messages:
================================================================================                                                                                                             ===
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 57294 RUNNING AT localhost.localdomain
=   EXIT CODE: 134
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
================================================================================                                                                                                             ===
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
================================================================================                                                                                                             ===


Here are a few details about my setup:

WRF-Hydro version: 5.1.1 and 5.3.0 compiled by pgf90
OS: Rocky Linux version: 8.10
Command used: mpirun -np 8 ./wrf_hydro_NoahMP.exe


Could you please advise me how to fix it?

Thanks in advance

Best Regards,
Aref

aubrey

unread,
Jun 13, 2025, 2:44:51 PMJun 13
to wrf-hydro_users, Aref Farhangmehr
Hi Aref:
Do you have access to another Fortran compiler? I don't think we support the PGI compiler anymore. And I recommend testing with the latest v5.4 code. v5.1.1 is quite old and may not behave as expected.

Thanks!
Aubrey

Aref Farhangmehr

unread,
Aug 6, 2025, 1:53:41 AMAug 6
to aubrey, wrf-hydro_users
Dear Aubrey;

Thank you for your email.

I have the same problem with the version of 5.3 and have not compiled V5.4!

Regards
Aref

Aref Farhangmehr

unread,
Aug 12, 2025, 9:44:42 AMAug 12
to aubrey, wrf-hydro_users
Hi Aubrey;

We have checked v5.4, the same problem!.

When I run crotona example I do not have any limitations for all versions.

Regards
Aref

Aubrey Dugger

unread,
Aug 12, 2025, 1:23:38 PMAug 12
to Aref Farhangmehr, wrf-hydro_users
Hi Aref:
Can you provide more details on your setup? Are you running a gridded channel routing or reach-based configuration? Have you tested (as we always recommend) running with just the land surface model (with idealized forcings), then adding the routing options one by one to see where it fails?

I have seen cases where, if a large part of the domain is water or does not include any reaches, the code may fail if the tile passed to a processor does not include any valid routing geometries. Could this be happening in your domain?

I would also make sure you are testing the latest versions of the GIS pre-processing workflows, in case there have been bug fixes.

Let us know how the testing goes.

Thanks!
Aubrey


--
-----------------------------------------------------------
Aubrey Dugger
NCAR Research Applications Laboratory
Office: 303-497-8418, Cell: 310-663-5115

Aref Farhangmehr

unread,
Aug 13, 2025, 2:33:40 AMAug 13
to Aubrey Dugger, wrf-hydro_users
Hi Aubrey;

Thank you for your kind response.

I ran the Reach-based example case (Crotona) and re-generated the DOMAIN in ArcCatalog. Interestingly, the original domain (downloaded directly from GitHub) runs without any limitations, while the re-generated  DOMAIN only runs with maximum of 4 cores!. (please kindly find the attached re-generated DOMAIN).


I am not sure I am using the latest version of  GIS pre-processing toolbox. I use the version which was created on 24/09/2019 by Kevin Sampson (https://github.com/NCAR/wrf_hydro_gis_preprocessor/blob/master/wrfhydro_gis/Build_Routing_Stack.py).

Thank you in advance for your helps.

Regards;
Aref 
regenerated_DOMAIN.tar.gz

Aref Farhangmehr

unread,
Sep 3, 2025, 8:07:36 AMSep 3
to Aubrey Dugger, wrf-hydro_users
Dear Aubrey;

Hope this email finds you well.

I wanted to follow up to see if you had a chance to look at the regenerated domain file I attached previously. Were you able to identify the specific issue that causes it to run only with 4 cores?

Again thank you for your support.

Regards;
Aref

Arezoo RafieeiNasab

unread,
Sep 3, 2025, 12:34:16 PMSep 3
to wrf-hyd...@ucar.edu, Aubrey Dugger
Hi Aref, 

I was going to do the checks, and my apologies for the delay. I will test it this week and will get back to you. 

Thanks!
Arezoo

--
You received this message because you are subscribed to the Google Groups "wrf-hydro_users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wrf-hydro_use...@ucar.edu.
To view this discussion visit https://groups.google.com/a/ucar.edu/d/msgid/wrf-hydro_users/CAJ%3DJChTKAoVkHR2sT89P4FJ%2BBH%3DfNpOWP5n7-19woQ%3DEETNikg%40mail.gmail.com.


--
Arezoo Rafieei Nasab, Ph.D.
NCAR/RAL Project Scientist II

Aref Farhangmehr

unread,
Sep 3, 2025, 12:55:17 PMSep 3
to wrf-hyd...@ucar.edu, Aubrey Dugger
Hi Arezoo;

Many thanks for the response and update.

You received this message because you are subscribed to a topic in the Google Groups "wrf-hydro_users" group.
To unsubscribe from this topic, visit https://groups.google.com/a/ucar.edu/d/topic/wrf-hydro_users/PDQK3AwkijE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wrf-hydro_use...@ucar.edu.
To view this discussion visit https://groups.google.com/a/ucar.edu/d/msgid/wrf-hydro_users/CALWrc66AKHmmNQeSHD7SNdq-h6eWE7A-XGKnGKUBAmLpcXrsFA%40mail.gmail.com.

Arezoo RafieeiNasab

unread,
Sep 3, 2025, 12:56:02 PMSep 3
to wrf-hyd...@ucar.edu, Aubrey Dugger
Hi Aref, 

I downloaded the domain that you shared in the earlier email, and tried it on our system and it is compiling with more than 4 cores. I tried 1, 4, 9 and 25 and all worked fine on my end. There are some differences in the domain files, but it worked fine for me. I ran it with the namelists/restart files that are distributed with the example case. I attached the namelists here, just changed the FORC_TYP to 4. Are you using the same namelists?

Thanks!
Arezoo
hydro.namelist
namelist.hrldas

Aubrey Dugger

unread,
Sep 3, 2025, 12:59:59 PMSep 3
to Arezoo RafieeiNasab, wrf-hyd...@ucar.edu
Hi Aref:
The issue may be specific to the compiler you are using. Are you still testing with PGI? As I mentioned before, there may be issues with this older compiler. Can you try with intel or gfort?

Thanks!
Aubrey

Aref Farhangmehr

unread,
Sep 3, 2025, 1:39:23 PMSep 3
to wrf-hyd...@ucar.edu, Aubrey Dugger
Thank  you Arezoo!

I am using the namelists which were downloaded directly from github.
I tested with the default example test case forcing type (FORC_TYP = 1) and have just tested FORC_TYP = 4, again with the same problem.

Regards;
Aref

You received this message because you are subscribed to a topic in the Google Groups "wrf-hydro_users" group.
To unsubscribe from this topic, visit https://groups.google.com/a/ucar.edu/d/topic/wrf-hydro_users/PDQK3AwkijE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wrf-hydro_use...@ucar.edu.
To view this discussion visit https://groups.google.com/a/ucar.edu/d/msgid/wrf-hydro_users/CALWrc66YHfgcSqJkK7uEkPQyEQWbRRqbZCceAN6xV8SA9verSA%40mail.gmail.com.

Aref Farhangmehr

unread,
Sep 3, 2025, 1:48:51 PMSep 3
to wrf-hyd...@ucar.edu, Arezoo RafieeiNasab
Hi Aubrey;

We are using gfortran 8.5.0 which is shipped with Rocky 8.

Which version would you recommend for best compatibility?

Regards;
Are

You received this message because you are subscribed to a topic in the Google Groups "wrf-hydro_users" group.
To unsubscribe from this topic, visit https://groups.google.com/a/ucar.edu/d/topic/wrf-hydro_users/PDQK3AwkijE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wrf-hydro_use...@ucar.edu.
To view this discussion visit https://groups.google.com/a/ucar.edu/d/msgid/wrf-hydro_users/CA%2BOs1Rcgoi407kerwDne5q27Rnx0hfQLpsoiBf_cdsACRddYHA%40mail.gmail.com.

Arezoo RafieeiNasab

unread,
Sep 3, 2025, 2:00:55 PMSep 3
to Aref Farhangmehr, wrf-hyd...@ucar.edu
Hi Aref, 

If you share the exact namelists that you used, I could give it a try. Just as a confirmation that the physics options are not the cause of problems. 

Thanks!
Arezoo

Aref Farhangmehr

unread,
Sep 3, 2025, 2:06:34 PMSep 3
to Arezoo RafieeiNasab, wrf-hyd...@ucar.edu
Thank you again Arezoo;

Please kindly find the attached namelists.

Regards;
Aref
hydro.namelist
namelist.hrldas

Arezoo RafieeiNasab

unread,
Sep 3, 2025, 3:20:36 PMSep 3
to Aref Farhangmehr, wrf-hyd...@ucar.edu
Ok, confirming that the namelists of yours is not a problem. As Aubrey mentioned, this is most probably a compiler issue. She will get back to you. 
Thanks!
Arezoo

Aubrey Dugger

unread,
Sep 3, 2025, 3:46:53 PMSep 3
to wrf-hyd...@ucar.edu, Aref Farhangmehr
Hi Aref:
Just checking with our software engineers, they suggest you update to a more recent compiler. We currently only support gfortran 11 or later since those are the versions that are being actively supported by GNU. Let us know if you still have an issue using a more recent compiler.


Thanks!
Aubrey

Reply all
Reply to author
Forward
0 new messages