HYCOM 1/4° crash at 2011/161 (oneta after barotp)

[Efraime Daniel]

Hi Alan,

I am running the global HYCOM 1/4°. The simulation progresses normally for more than 3 years, but it crashes when reaching date 2011/161, at step 9681120 (day = 40338).

Right after the archiving step, the diagnostics of the atmospheric fields (wind, air temperature, radiation, etc.) are printed and all seem to be within reasonable ranges. Shortly after that, the model stops with the following message:
9681360 min of oneta after barotp: *********

As far as I understand, the asterisks indicate a numerical overflow in the output format, meaning that the minimum value of oneta after the barotropic step is outside the representable range. Right after this, messages such as the following appear:
min/max of s after advection: N/A (thin layer)
My questions are:

• Is this type of error more commonly associated with instability in the barotropic step (timestep, CFL condition, bathymetry), or with the atmospheric forcing?
• Does it make sense to specifically check the forcing files around days 40338–40339?
• Do you recommend any additional diagnostics to better identify the source of the problem (e.g., checking minimum layer thickness or limiting oneta)?

Thanks for your help.
Best regards,
Efraime

[Alan Wallcraft]

The usual way HYCOM dies is with a negative dp (layer thickness), which indicates a CFL violation (velocity too high).  For example:

 84213420 i,j,k=  540  240 13 neg. dp (m) in loop 15       -1.1     max
 84213420 i,j,k=  540  240 14 neg. dp (m) in loop 15       -1.2     max
 84213420 i,j,k=  542  240 15 neg. dp (m) in loop 15      -0.13E+04 max
 84213420 i,j,k=  542  240 15 neg. dp (m) in loop 15      -0.13E+04 fatal

This negative value is fatal if it is <= -10m.

This is only checked every 3 time steps, so it is possible some other bad thing happens before this check.  However, I would be very surprised if you got a segmentation fault with no "neg. dp" messages.  The advantage of these messages is that they tell you where the error occurred.  You can then use hycom_profile on a 3-D archive file to see what that water column looks like.

The usual approach is to halve the baclin time step, leaving the batrop time step as close to its existing value as possible, are rerrun the last model segment.

It would not hurt to check the forcing files.  Their .b files have min and max values, but a NaN would not show up.  So run hycom_NaN on the forcing files as a check.

Finally, you may get fewer neg. dp's if you set the MOMTUM_CFL macro at compile time:

# Miscellaneous CPP flags (-DSTOKES -DOCEANS2 etc...)
# -DSTOKES  : Stokes drift
# -DOCEANS2 : master and slave HYCOM in same executable
# -DMOMTUM_CFL     : include an explicit CFL limiter
# -DMOMTUM4_CFL    : include an explicit CFL limiter
# -DRDNEST_MASK    : mask velocity outliers
# -DLATBDT_NPLINE3 : update pline every 3 time steps
# -DMASSLESS_1MM   : lowest substantial mass-containing layer > 1mm thick
#setenv OCN_MISC ""
#setenv OCN_MISC "-DMASSLESS_1MM -DRDNEST_MASK -DLATBDT_NPLINE3"
setenv OCN_MISC "-DMASSLESS_1MM -DMOMTUM_CFL"

If it is already set, this might be why you don't see neg. dp's.  The downside of MOMTUM_CFL is that it non-physically clips large velocities and so will cause problems of its own if turned on too often.  It produces at report in the .log file, search for maxspd:

!
! ---   Only report new CFL maximums
!
        do k= 1,kk
          if     (cflclp(k).gt.cflmax(k)) then
            cflmax(k) = cflclp(k)  !save new maximum
            if (mnproc.eq.1) then
            write(lp, '(i9,a, &
                         &'' L '',i3,'' maxspd,clip (m/s):'',2f10.4)') &
                nstep,c_ydh, &
                k,cflspd(k),cflclp(k)
            call flush(lp)

Alan.