NaNs appearing early in HYCOM simulation — input files checked

[Alfredo Terrazas]

Dear HYCOM team,

I’m currently running a HYCOM simulation with grid dimensions 541x385 (expt_01.1; the base configuration in experiment expt_01.0 (no forcing, no nesting) runs without issues), and I’ve encountered an issue where NaNs appear after first time step. The model crashes with a segmentation fault.

Here is part of the message from the log file:

 51463201 mean basin temp, saln, dens        NaN       NaN       NaN
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source            
libc.so.6          00001555516E3050  Unknown               Unknown  Unknown
hycom              0000000000622628  Unknown               Unknown  Unknown
hycom              0000000000663420  Unknown               Unknown  Unknown
hycom              0000000000646967  Unknown               Unknown  Unknown
hycom              0000000000568395  Unknown               Unknown  Unknown
hycom              00000000004036D0  Unknown               Unknown  Unknown
hycom              000000000040366D  Unknown               Unknown  Unknown
libc.so.6          00001555516CE24A  Unknown               Unknown  Unknown
libc.so.6          00001555516CE305  __libc_start_main     Unknown  Unknown
hycom              0000000000403581  Unknown               Unknown  Unknown

To troubleshoot, I ran the hycom_NaN tool on all input .a files and confirmed that none of them contain NaNs or Infs. I’m including below a summary of the check I performed using the following script:

bash

CopyEdit

#!/bin/bash NX=541 NY=385 for file in *.a; do echo "Checking file: $file" hycom_NaN "$file" $NX $NY done

Results summary:
All .a input files processed (forcing, regional, relax, restart, etc.) showed zero NaNs/Infs across hundreds of fields each. Example output:

Revisando archivo: relax.sssrmx.a

           1  FIELDS PROCESSED,            0  CONTAINED NaNs and/or Infs

Revisando archivo: relax.temp.a

         432  FIELDS PROCESSED,            0  CONTAINED NaNs and/or Infs

Revisando archivo: restart_118f.a

         374  FIELDS PROCESSED,            0  CONTAINED NaNs and/or Infs

I've also reviewed my blkdat.input configuration, including values like batrop and baclin, and they seem reasonable.

I attached these files:

• blkdat.input

• Full model log (run.log)

Could you help me trying to figure out what might be causing NaNs to appear so early in the simulation if all input files are clean?

I would appreciate any suggestions for what else I can check or adjust to identify the source of the instability.

Thank you in advance for your help!

Best regards,
Alfredo Terrazas

[Alan Wallcraft]

This might be a land-sea mismatch between the restart file and the input files.

The nesting archive files need to have sea behind any open boundaries on the last row or column, but the archive file you use to generate the initial restart file needs to have the same bathymetry as the model run with the last row and column all land.

If this is OK, you can turn on PIPE_NAN (like PIPE_DEBUG that is commented out in the standard model script):

C
C --- turn on NaN debugging.
C
touch PIPE_NAN

touch      PIPE_NAN
/bin/rm -f PIPE_NAN
C
C --- archive output in a separate tar directory
C

This will tell you where the 1st NaN is encountered.

Alan.

[Alfredo Terrazas]

Dear Alan,

Thank you very much for your quick and helpful response.

I checked the bathymetry, and I can confirm that the last row and column in my model domain are set as land. To verify the consistency between land-sea masks in my input and restart files, I used the hycom_sea_ok tool on all relevant .a files in the SCRATCH directory.

From this check, I found that the relaxation files (relax.*.a) were the ones reporting land-sea inconsistencies. I’ve attached the output log from this verification for your reference.

The strange part is that all the input files, including the relaxation ones, were generated using the standard HYCOM tools and scripts, and with the same bathymetry used for the model run. That’s why I’m not sure what might have caused these inconsistencies in sea/land node alignment.

Thanks again for your guidance.

Best regards,
Alfredo Terrazas

[Mengli Zhang]

Dear  Alfredo,

May I ask how you solved this problem? I also encountered the same problem.

Best regards,

Mengli Zhang

[Alfredo Terrazas]

Thanks for checking in! I’ve actually managed to solve the problem.

It turned out that the issue was with the relaxation files — they were incorrectly generated due to a compatibility problem between the original scripts and the new operating system I’m working on. This caused inconsistencies in the land-sea mask, which triggered the NaNs early in the run.

To fix it, I regenerated the entire region from scratch, including all the relaxation and nesting files, making sure to use updated tools. I also validated everything using hycom_sea_ok, and now the simulation is running correctly.

As Alan suggested, I recommend checking the land-sea mask consistency carefully and considering reprocessing the input files if needed.

Best of luck, and feel free to reach out if I can help in any way!

Best regards,
Alfredo Terrazas