MPI/PSM2 Error on Some Stations When Running PyWrfHydroCalib

[Fatemeh Shirkhanloo]

Hi everyone,

I was running a group of WRF-Hydro simulations using PyWrfHydroCalib. This workflow has worked fine for me before, but this time some of the stations are failing with an MPI/PSM2 error on some nodes, while others run normally with the same setup.

Here is an example of the error I am seeing (from node compute040):

PSM2 can't open hfi unit: -1 (err=23) PSM2 was unable to open an endpoint. MPI_INIT failed PML add procs failed An error occurred in MPI_Init_thread

I am not sure why this is happening now, since previous PyWrfHydroCalib group submissions were successful and I haven’t changed anything in the configuration.

Has anyone any suggestions on what might be causing it?

Thank you,
Fatemeh