Parallel Execution Limitations in WRF-Hydro

[Aref Farhangmehr]

Hi WRF-Hydro Team;

I hope this message finds you well.

I'm currently attempting to run WRF-Hydro in parallel using mpirun, but I'm encountering a limitation: I cannot run the model with more than 2 cores when creating DOMAIN files using ArcCatalog, and no more than 4 cores when generating DOMAIN files using Python scripts. The error messages:
================================================================================                                                                                                             ===
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 57294 RUNNING AT localhost.localdomain
=   EXIT CODE: 134
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
================================================================================                                                                                                             ===
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
================================================================================                                                                                                             ===


Here are a few details about my setup:

WRF-Hydro version: 5.1.1 and 5.3.0 compiled by pgf90
OS: Rocky Linux version: 8.10

Command used: mpirun -np 8 ./wrf_hydro_NoahMP.exe


Could you please advise me how to fix it?

Thanks in advance

Best Regards,
Aref

[aubrey]

Hi Aref:

Do you have access to another Fortran compiler? I don't think we support the PGI compiler anymore. And I recommend testing with the latest v5.4 code. v5.1.1 is quite old and may not behave as expected.

Thanks!

Aubrey

[Aref Farhangmehr]

Dear Aubrey;

Thank you for your email.

I have the same problem with the version of 5.3 and have not compiled V5.4!

Regards

Aref

[Aref Farhangmehr]

Hi Aubrey;

We have checked v5.4, the same problem!.

When I run crotona example I do not have any limitations for all versions.

Regards

Aref

[Aubrey Dugger]

Hi Aref:

Can you provide more details on your setup? Are you running a gridded channel routing or reach-based configuration? Have you tested (as we always recommend) running with just the land surface model (with idealized forcings), then adding the routing options one by one to see where it fails?

I have seen cases where, if a large part of the domain is water or does not include any reaches, the code may fail if the tile passed to a processor does not include any valid routing geometries. Could this be happening in your domain?

I would also make sure you are testing the latest versions of the GIS pre-processing workflows, in case there have been bug fixes.

Let us know how the testing goes.

Thanks!

Aubrey

--

-----------------------------------------------------------

Aubrey Dugger

NCAR Research Applications Laboratory

Office: 303-497-8418, Cell: 310-663-5115

adu...@ucar.edu

[Aref Farhangmehr]

Hi Aubrey;

Thank you for your kind response.

I ran the Reach-based example case (Crotona) and re-generated the DOMAIN in ArcCatalog. Interestingly, the original domain (downloaded directly from GitHub) runs without any limitations, while the re-generated  DOMAIN only runs with maximum of 4 cores!. (please kindly find the attached re-generated DOMAIN).

I am not sure I am using the latest version of  GIS pre-processing toolbox. I use the version which was created on 24/09/2019 by Kevin Sampson (https://github.com/NCAR/wrf_hydro_gis_preprocessor/blob/master/wrfhydro_gis/Build_Routing_Stack.py).

Thank you in advance for your helps.

Regards;

Aref 

[Aref Farhangmehr]

Dear Aubrey;

Hope this email finds you well.

I wanted to follow up to see if you had a chance to look at the regenerated domain file I attached previously. Were you able to identify the specific issue that causes it to run only with 4 cores?

Again thank you for your support.

Regards;

Aref

[Arezoo RafieeiNasab]

Hi Aref, 

I was going to do the checks, and my apologies for the delay. I will test it this week and will get back to you. 

Thanks!

Arezoo

--
You received this message because you are subscribed to the Google Groups "wrf-hydro_users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wrf-hydro_use...@ucar.edu.
To view this discussion visit https://groups.google.com/a/ucar.edu/d/msgid/wrf-hydro_users/CAJ%3DJChTKAoVkHR2sT89P4FJ%2BBH%3DfNpOWP5n7-19woQ%3DEETNikg%40mail.gmail.com.

--

Arezoo Rafieei Nasab, Ph.D.

NCAR/RAL Project Scientist II

[Aref Farhangmehr]

Hi Arezoo;

Many thanks for the response and update.

You received this message because you are subscribed to a topic in the Google Groups "wrf-hydro_users" group.
To unsubscribe from this topic, visit https://groups.google.com/a/ucar.edu/d/topic/wrf-hydro_users/PDQK3AwkijE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wrf-hydro_use...@ucar.edu.
To view this discussion visit https://groups.google.com/a/ucar.edu/d/msgid/wrf-hydro_users/CALWrc66AKHmmNQeSHD7SNdq-h6eWE7A-XGKnGKUBAmLpcXrsFA%40mail.gmail.com.

[Arezoo RafieeiNasab]

Hi Aref, 

I downloaded the domain that you shared in the earlier email, and tried it on our system and it is compiling with more than 4 cores. I tried 1, 4, 9 and 25 and all worked fine on my end. There are some differences in the domain files, but it worked fine for me. I ran it with the namelists/restart files that are distributed with the example case. I attached the namelists here, just changed the FORC_TYP to 4. Are you using the same namelists?

Thanks!

Arezoo

[Aubrey Dugger]

Hi Aref:

The issue may be specific to the compiler you are using. Are you still testing with PGI? As I mentioned before, there may be issues with this older compiler. Can you try with intel or gfort?

Thanks!

Aubrey

[Aref Farhangmehr]

Thank  you Arezoo!

I am using the namelists which were downloaded directly from github.

I tested with the default example test case forcing type (FORC_TYP = 1) and have just tested FORC_TYP = 4, again with the same problem.

Regards;

Aref

You received this message because you are subscribed to a topic in the Google Groups "wrf-hydro_users" group.
To unsubscribe from this topic, visit https://groups.google.com/a/ucar.edu/d/topic/wrf-hydro_users/PDQK3AwkijE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wrf-hydro_use...@ucar.edu.
To view this discussion visit https://groups.google.com/a/ucar.edu/d/msgid/wrf-hydro_users/CALWrc66YHfgcSqJkK7uEkPQyEQWbRRqbZCceAN6xV8SA9verSA%40mail.gmail.com.

[Aref Farhangmehr]

Hi Aubrey;

We are using gfortran 8.5.0 which is shipped with Rocky 8.

Which version would you recommend for best compatibility?

Regards;

Are

You received this message because you are subscribed to a topic in the Google Groups "wrf-hydro_users" group.
To unsubscribe from this topic, visit https://groups.google.com/a/ucar.edu/d/topic/wrf-hydro_users/PDQK3AwkijE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wrf-hydro_use...@ucar.edu.
To view this discussion visit https://groups.google.com/a/ucar.edu/d/msgid/wrf-hydro_users/CA%2BOs1Rcgoi407kerwDne5q27Rnx0hfQLpsoiBf_cdsACRddYHA%40mail.gmail.com.

[Arezoo RafieeiNasab]

Hi Aref, 

If you share the exact namelists that you used, I could give it a try. Just as a confirmation that the physics options are not the cause of problems. 

Thanks!

Arezoo

[Aref Farhangmehr]

Thank you again Arezoo;

Please kindly find the attached namelists.

Regards;

Aref

[Arezoo RafieeiNasab]

Ok, confirming that the namelists of yours is not a problem. As Aubrey mentioned, this is most probably a compiler issue. She will get back to you. 

Thanks!

Arezoo

[Aubrey Dugger]

Hi Aref:

Just checking with our software engineers, they suggest you update to a more recent compiler. We currently only support gfortran 11 or later since those are the versions that are being actively supported by GNU. Let us know if you still have an issue using a more recent compiler.

See: https://wrf-hydro.readthedocs.io/en/latest/model-code-config.html#requirements

Thanks!

Aubrey

To view this discussion visit https://groups.google.com/a/ucar.edu/d/msgid/wrf-hydro_users/CALWrc67anSGKREPvckY%3DwPgFxR4EJkJ%3DMXBmG0GxAiFpCn1uSw%40mail.gmail.com.