MPI crash when using channel_option = 2 (diffusive wave routing) even with compound_channel = .FALSE.

[Hai Dong]

Hi all,

I am encountering a reproducible MPI crash when running WRF-Hydro with
channel_option = 2 (diffusive wave routing). The same configuration runs correctly in single-core mode, but fails under MPI, regardless of io_form_outputs.

The error consistently indicates missing CHAN_DEPTH, followed by a memory corruption abort.

I have attached files  two namelist for WRFHydro.

Hope anyone could help me out! Thanks a lot.

Best wishes!

Hai

[aubrey]

Hi Hai:

A couple of follow-up questions to help us diagnose the issue:

1) channel_option = 2 is the Muskigum-Cunge routing method, not the diffusive wave method. This method requires a route link file (which I see you are providing) since it operates on vector reaches. The diffusive wave method operates on a gridded channel network. When you run the GIS pre-processor, you have to choose one or the other of these setups and you cannot mix and match. So first, I just want to confirm that you are intending to run the reach-based Muskigum Cunge method and that you specified a reach-based setup when you ran the GIS pre-processor.

2) You are specifying a 5500 meter routing grid resolution and an aggregation factor of 1. So this means both your land model and the routing models are running at 5.5 km resolution. This is definitely too coarse for overland routing and may also be too coarse for the subsurface routing (conceptually this might be OK for deeper subsurface routing, but I have not tested at these coarse resolutions myself and likely not suitable for 2-m soil columns). The lateral terrain routing modules are really intended for much finer resolutions (meters to hundreds of meters).

3) As always, for debugging we suggest you start with the simplest configuration (idealized forcings FORC_TYP=4) and land model only (SUBRTSWCRT=0, OVRTSWCRT=0, CHANRTSWCRT=0, GWBASESWCRT=0), then gradually add complexity (see the slides here: https://ral.ucar.edu/sites/default/files/public/projects/wrf-hydro/training-materials/wrf-hydro-implementation-bestpractices-nov-2020-updated.pdf). Have you tried this? Does the model still crash when running over multiple processors in this simplest configuration? When you start adding routing physics one-by-one, at what point does it fail in parallel?

4) We have seen an issue with reach-based configurations when running in parallel where, in the domain decomposition, one of the processors receives a portion of the domain that does not contain any reaches. This causes a crash. We see this when people are running domains with large sections of ocean coverage, for example. The domain may run with up to a certain number of cores, but when you increase the core count over a certain number it fails. We are looking into this issue but have not yet implemented a fix.

Let us know how it goes.

Thanks!

Aubrey