MPAS5 Error while running at 30 km

[Abhi Prasad]

I have tried to run MPAS5 with NCEP FNL data using different meshes.

The test cases worked for meshes at 480 km, 240 km, 120 km and 60 km, but it failed at 30 km (namelist attached).

I ensured that I have changed my config_dt and config_len_disp. I also increased the number of CPUs and Memory I used on the HPC accordingly.

I wonder if the problem is about model stability or computational resource requirements.

The end of error log shows the following:

--- subroutine MPAS_to_phys - pressure(1) < pressure(2):
 i      =        1689
 latCell=   13.9471762072369    
 lonCell=   80.6715623486968    
       1    1689       1  0.61214630E+02  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.18395633E-01
       1    1689       2  0.79846222E+02  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.18033204E-01
       1    1689       3  0.10741546E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.17608667E-01
       1    1689       4  0.14392234E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.16989025E-01
       1    1689       5  0.18936687E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.15433956E-01
       1    1689       6  0.24374904E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.13907315E-01
       1    1689       7  0.30706886E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.13041010E-01
       1    1689       8  0.37932640E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.12461770E-01
       1    1689       9  0.46052195E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.11635877E-01
       1    1689      10  0.55065686E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.10405131E-01
       1    1689      11  0.64973622E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.88077234E-02
       1    1689      12  0.75777545E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.67764189E-02
       1    1689      13  0.86958975E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.43497662E-02
       1    1689      14  0.90032806E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.30465216E-02
       1    1689      15  0.90065113E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.22425620E-02
       1    1689      16  0.90104914E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.15317804E-02
       1    1689      17  0.90139806E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.99655871E-03
       1    1689      18  0.90156573E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.57599699E-03
       1    1689      19  0.90139814E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.31453923E-03
       1    1689      20  0.90077477E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.16356227E-03
       1    1689      21  0.90019214E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.73013287E-04
       1    1689      22  0.89971587E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.27927321E-04
       1    1689      23  0.89936850E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.18283849E-04
       1    1689      24  0.89914978E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.97778865E-05
       1    1689      25  0.89904651E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.51838399E-05
       1    1689      26  0.89903824E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.26576006E-05
       1    1689      27  0.89910123E+03  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.00000000E+00  0.19745109E-05
forrtl: error (78): process killed (SIGTERM)

I would appreciate any suggestions from MPAS Users.

Thanks,
Abhi

[Dominikus Heinzeller]

Hi Abhi,

can you tell us at which model time step this problem occurred, right at the beginning or how far into the model run?

Cheers

Dom

--
You received this message because you are subscribed to the Google Groups "MPAS-Atmosphere Help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpas-atmosphere-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<namelist.atmosphere>

[Abhi Prasad]

Hi Dom,

The problem occurred right at the beginning at the first time step.

The error log file can be found here: https://drive.google.com/open?id=0B3l9KZx1e4AgSHp6X2MyQ3dVYXc

I look forward to some suggestions.

Thanks,
Abhi

To unsubscribe from this group and stop receiving emails from it, send an email to mpas-atmosphere-help+unsub...@googlegroups.com.

[Dominikus Heinzeller]

Abhi,

I suspect it has to do with the initial conditions, in particular the temperature extrapolation at levels close to the surface. 

We have a hotfix release coming up soon for MPAS 5.0, in which we will give the users the possibility to choose between different temperature extrapolation options at runtime (in namelist.init_atmosphere).

In the meanwhile, could you try the following and let me know if it makes any difference?

edit src/core_init_atmosphere/mpas_init_atm_cases.F

go to line 4409/4410 and change

            t(k,iCell) = vertical_interp(target_z, nfglevels_actual-1, &

                                  sorted_arr(:,1:nfglevels_actual-1), order=1, extrap=1)

to

            t(k,iCell) = vertical_interp(target_z, nfglevels_actual-1, &

                                  sorted_arr(:,1:nfglevels_actual-1), order=1, extrap=0)

Then, do a clean-compile of the init_atmosphere core and rerun the vertical interpolation step to regenerate init.nc (maybe keep the old init.nc with a different name for a later comparison).

Cheers,

Dom

To unsubscribe from this group and stop receiving emails from it, send an email to mpas-atmosphere-...@googlegroups.com.

[MPAS-Atmosphere Help]

Hi, Abhi.

I agree with Dom that this looks like a problem with the initial conditions. Can you take a look at the 'theta' field in your initial conditions file and see that it looks reasonable?

Best regards,

Michael

[Abhi Prasad]

Thanks Dom. I will try this and see how this turns out!