Program to create static.nc file

[huangw...@gmail.com]

hello there,

I tried running init_atmosphere_model, for grid 2562.
It run for a long time, and job got killed with message:

task 0 of 1 is running
=>> PBS: job killed: walltime 9066 exceeded limit 9000

Here is the tail of log.0000.err:

....
/lustre/whuang/data4wrf/geog/varsso/42001-42600.13801-14400
/lustre/whuang/data4wrf/geog/varsso/42601-43200.13801-14400
--- end interpolate VARSSO

BEGIN INTERPOLATION OF STATISTICAL FIELDS FOR GRAVITY WAVE DRAG OVER OROGRAPHY
min MeshD = 1.00000000000000
max MeshD = 1.00000000000000
min dcEdge = 412312.173690322
max dcEdge = 508603.608908459

The stream and namelist files are:

whuang@cy001:/lustre/whuang/mpas/init.2562> cat streams.init_atmosphere
<streams>
<immutable_stream name="input"
type="input"
filename_template="/lustre/whuang/mpas/data/x1.2562.grid.nc"
input_interval="initial_only" />

<immutable_stream name="output"
type="output"
filename_template="x1.2562.static.nc"
packages="initial_conds"
output_interval="initial_only" />
</streams>
whuang@cy001:/lustre/whuang/mpas/init.2562> cat namelist.init_atmosphere
&nhyd_model
config_init_case = 7
config_start_time = '2016-12-05_06:00:00'
config_stop_time = '2016-12-05_12:00:00'
config_theta_adv_order = 3
/
&dimensions
config_nvertlevels = 1
config_nsoillevels = 1
/
&data_sources
config_geog_data_path = '/lustre/whuang/data4wrf/geog/'
config_met_prefix = 'FILE'
config_sfc_prefix = 'SST'
config_fg_interval = 21600
config_landuse_data = 'USGS'
/
&vertical_grid
config_ztop = 30000.0
config_nsmterrain = 1
config_smooth_surfaces = true
config_dzmin = 0.3
config_nsm = 30
/
&preproc_stages
config_static_interp = true
config_vertical_grid = false
config_met_interp = false
config_input_sst = false
config_frac_seaice = false
/
&io
config_pio_num_iotasks = 0
config_pio_stride = 1
/
&decomposition
config_block_decomp_file_prefix = '/lustre/whuang/mpas/data/x1.2562.graph.info.part.'
/

ncdump of the static.nc file shows:

....
:config_explicit_proc_decomp = "NO" ;
:config_proc_decomp_file_prefix = "graph.info.part." ;
:file_id = "nelkvbfxiw" ;
data:

xtime =
"2016-12-05_06:00:00 " ;

latCell = 0.463647609448067, 0.463647610007084, 1.5707963267949,
-0.463647609448067, -0.463647610007084, -0.463647609448067,
0.463647609336263, -0.463647609336263, -1.5707963267949,
0.463647608106426, -0.463647608106426, 0.463647609448067,
0.553574358639438, -2.00000016548074e-10, 6.00000049644223e-10,
0.553574359815009, 1.0172219683695, 8.00000066192297e-10,
4.00000033096148e-10, 0.553574360050123, 1.01722196894013,
.....

Thanks for your help.

Wei

[Dominikus Heinzeller]

Hi Wei,

I think this problem has nothing to do with MPAS itself. From your log file:

> task 0 of 1 is running
> =>> PBS: job killed: walltime 9066 exceeded limit 9000

—> Your job is exceeding the walltime set for your job by the PBS scheduler. You need to set set a longer walltime if that is possible on your system.

Cheers

Dom

> --
> You received this message because you are subscribed to the Google Groups "MPAS-Atmosphere Help" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mpas-atmosphere-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Wei Huang]

Dom,

This is very coarse domain, and it took much less time on such domain on other machines. And, I have tried longer time, no success.

I won't say it is a true MPAS issue, but something related to the system, or combined. I am asking if someone had run into such situation, and how it is resolved.

Thanks,

Wei

> To unsubscribe from this group and stop receiving emails from it, send an email to mpas-atmosphere-help+unsub...@googlegroups.com.

[Laura Fowler]

Hi Wei:

Your error message comes from the fact that the directory dir_gwdo is
not defined when you use the 2562 mesh. I did not think that users
used a mesh as coarse of 2562 and I omitted that case. Since I do not
know which version of the MPAS sourcecode you are using, I suggest
that you edit your file mpas_init_atm_static.F in the directory
./src/core_init_atmosphere.

If you do not mind, can you please replace the lines:

dir_gwdo = ' '
if(minMeshD == 1.0_RKIND .and. maxMeshD == 1.0_RKIND) then
!... uniform 10242 mesh:
if(mindcEdge .ge. 200000._RKIND .and. maxdcEdge .lt. 260000._RKIND) then
dir_gwdo = 'orogwd_2deg'

with

dir_gwdo = ' '
if(minMeshD == 1.0_RKIND .and. maxMeshD == 1.0_RKIND) then
! ... uniform 2562 mesh:
if(mindcEdge .ge. 400000._RKIND .and. maxdcEdge .lt. 560000._RKIND) then
dir_gwdo = 'orogwd_2deg'
!... uniform 10242 mesh:
elseif(mindcEdge .ge. 200000._RKIND .and. maxdcEdge .lt. 260000._RKIND) then
dir_gwdo = 'orogwd_2deg'

This will help define the directory dir_gwdo for your mesh resolution.
Also, to ensure that MPAS will correctly abort if dir_gwdo is not
found, please uncomment the lines:

! write(mess,*) 'GWDO: Interpolation not available. The
initialization will abort'
! call physics_error_fatal(mess)

Can you let me know if this helps. Can you tell me the version of MPAS
that you are using so that I can try to reproduce that error as well.

Thanks,
Laura

>> > an email to mpas-atmosphere-...@googlegroups.com.

>> > For more options, visit https://groups.google.com/d/optout.
>>
>

> --
> You received this message because you are subscribed to the Google Groups
> "MPAS-Atmosphere Help" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mpas-atmosphere-...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
!-------------------------------------------------------------------------------------------------------------
Laura D. Fowler
Mesoscale and Microscale Meteorology Division (MMM)
National Center for Atmospheric Research
P.O. Box 3000, Boulder CO 80307-3000

e-mail: la...@ucar.edu
phone: 303-497-1628

!-------------------------------------------------------------------------------------------------------------

[Wei Huang]

Laura,

I'll make the changes you suggested, and recompile and rerun.

Actually, my 10242 mesh failed as well, with message:

task     0 of     1 is running

=>> PBS: job killed: walltime 16284 exceeded limit 16200

MPT ERROR: MPI_COMM_WORLD rank 0 has terminated without calling MPI_Finalize()

        aborting job

I'll re-run both meshes with the new executable, and see how it goes.

Thanks,

Wei

>> > an email to mpas-atmosphere-help+unsub...@googlegroups.com.

>> > For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "MPAS-Atmosphere Help" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mpas-atmosphere-help+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

[Laura Fowler]

Wei:

I build the static file using the MPAS-Release sourcecode (without any
of the modifications I suggested to you in a separate e-mail) using
the 2562 mesh. I build that file using ifort and gfortran. I could not
reproduce the errors you described in your e-mail and my 2 runs
finished successfully.

The end of the logfiles are:

BEGIN INTERPOLATION OF STATISTICAL FIELDS FOR GRAVITY WAVE DRAG OVER OROGRAPHY

min MeshD = 1.0000000000000000
max MeshD = 1.0000000000000000
min dcEdge = 412312.17369032156
max dcEdge = 508603.60890845890

Since I do not specify the directory dir_gwdo with the 2562 mesh, the
arrays vard2d, con, oa1, oa2, oa3, and oa4, and ol1, ol2, ol3, and ol4
are all zero in my static file, but MPAS did not crash and finished
successfully. This is what I expected.

So I am wondering if something bad happens when you write your netcdf
file or a typo in the streams.init_atmosphere. May be you should put
some print statements after the call to that subroutine?

Hope that helps,

>> >> > an email to mpas-atmosphere-...@googlegroups.com.

>> >> > For more options, visit https://groups.google.com/d/optout.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "MPAS-Atmosphere Help" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to mpas-atmosphere-...@googlegroups.com.

[Dominikus Heinzeller]

Hi Wei,

following up on Laura’s suggestion: you could look at

mpas_stream_manager.F:

in src/framework and activate the debug output, i.e. remove the “!” from the following line:

#define STREAM_DEBUG_WRITE(M) ! write(stderrUnit,*) 'DEBUG: '//M

so that it reads

#define STREAM_DEBUG_WRITE(M) write(stderrUnit,*) 'DEBUG: '//M

This will give you more information on what MPAS is trying to read and write at a certain stage.

Also, you could change the iotype for writing the static.nc file. By default it is set to pnetcdf. You could try

<immutable_stream name="output"

                type="output"

                io_type=“netcdf"

                filename_template="x1.2562.static.nc"

                packages="initial_conds"

                output_interval="initial_only" />

or “netcdf4” (see MPAS Atmosphere User’s Guide)

Cheers,

Dom

[Wei Huang]

Thanks to Dom and Laura's help, I get the static file generated, and then I am able to generate initial condition files and run for mesh 2562, 10242, 40962, and 163842. I appreciate the help from both of you.

I guess that for some reason, defualt io_type=“pnetcdf" did not work for me, but io_type=“netcdf" worked.

Now, I got a new problem, when I tried with mesh 655362, I got error message from log.0000.err:

....

 /lustre/whuang/data4wrf/geog/varsso/42601-43200.13801-14400

 --- end interpolate VARSSO

 BEGIN INTERPOLATION OF STATISTICAL FIELDS FOR GRAVITY WAVE DRAG OVER OROGRAPHY

 min MeshD  =   1.00000000000000

 max MeshD  =   1.00000000000000

 min dcEdge =   22846.6443128399

 max dcEdge =   31792.2024273164

 ------------------------------ FATAL CALLED ------------------------------

  GWDO: Interpolation not available. The initialization will abort

 ERROR: MPAS core_physics abort

The difference of namelist and streams between mesh 163842, and mesh 655362 is as below:

whuang@cy001:/lustre/whuang/mpas/init.655362> diff ../init.163842/namelist.init_atmosphere .

37c37

<     config_block_decomp_file_prefix = '/lustre/whuang/mpas/data/x1.163842.graph.info.part.'

---

>     config_block_decomp_file_prefix = '/lustre/whuang/mpas/data/x1.655362.graph.info.part.'

whuang@cy001:/lustre/whuang/mpas/init.655362> diff ../init.163842/streams.init_atmosphere .

4c4

<                   filename_template="/lustre/whuang/mpas/data/x1.163842.grid.nc"

---

>                   filename_template="/lustre/whuang/mpas/data/x1.655362.grid.nc"

10c10

<                   filename_template="x1.163842.static.nc"

---

>                   filename_template="x1.655362.static.nc"

Did I miss some geo-files, or something else?

Thanks,

Wei

an email to mpas-atmosphere-help+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups
"MPAS-Atmosphere Help" group.
To unsubscribe from this group and stop receiving emails from it, send
an

email to mpas-atmosphere-help+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

!-------------------------------------------------------------------------------------------------------------
Laura D. Fowler
Mesoscale and Microscale Meteorology Division (MMM)
National Center for Atmospheric Research
P.O. Box 3000, Boulder CO 80307-3000

e-mail: la...@ucar.edu
phone: 303-497-1628


!-------------------------------------------------------------------------------------------------------------

-- 
!-------------------------------------------------------------------------------------------------------------
Laura D. Fowler
Mesoscale and Microscale Meteorology Division (MMM)
National Center for Atmospheric Research
P.O. Box 3000, Boulder CO 80307-3000

e-mail: la...@ucar.edu
phone: 303-497-1628

!-------------------------------------------------------------------------------------------------------------

-- 
You received this message because you are subscribed to the Google Groups "MPAS-Atmosphere Help" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mpas-atmosphere-help+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "MPAS-Atmosphere Help" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mpas-atmosphere-help/86YQI20s-Lc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mpas-atmosphere-help+unsub...@googlegroups.com.

[Dominikus Heinzeller]

Hi Wei,

although I was not involved in the development of the gwdo scheme, I can tell from the source code that it has been implemented only for certain grid spacings on uniform meshes, and not at all for irregular meshes. The source code in release 5.0 is a bit different from yours, but you can see from src/core_init_atmosphere/mpas_init_atm_static.F:

 dir_gwdo = '   '

 if(minMeshD == 1.0_RKIND .and. maxMeshD == 1.0_RKIND) then

    !... uniform 10242 mesh:

    if(mindcEdge .ge. 200000._RKIND .and. maxdcEdge .lt. 260000._RKIND) then

       dir_gwdo = 'orogwd_2deg'

    elseif(mindcEdge .ge. 90000._RKIND .and. maxdcEdge .lt. 150000_RKIND) then

       dir_gwdo = 'orogwd_1deg'

    elseif(mindcEdge .ge. 40000._RKIND .and. maxdcEdge .lt. 70000._RKIND) then

       dir_gwdo = 'orogwd_30m'

    else

       write(0,*)

!      write(mess,*) 'GWDO: Interpolation not available. The initialization will abort'

!      call physics_error_fatal(mess)

       write(mess,*) 'GWDO: Interpolation not available. Set config_gwdo_scheme = .false.'

       return

    endif

 else

    write(0,*)

!   write(mess,*) 'GWDO: The input mesh must be a uniform mesh. The initialization will abort'

!   call physics_error_fatal(mess)

    write(mess,*) 'GWDO: The input mesh must be a uniform mesh. Set config_gwdo_scheme = .false.'

    return

 endif

 write(0,*) 'dir_gwdo   =    ', trim(dir_gwdo)

 write(0,*)

If you have higher-res gdwo data (I do have orogwd_10m, orogwd_20m, orogwd_30m, orogwd_1deg, orogwd_2deg on my system), you might try to add an entry for your 30km mesh using the _20m data. However, I do remember from conversations with others that it is not clear whether the gdwo will give you any benefit.

Alternatively, you could also simply set 

config_gwdo_scheme = .false.

in your namelist.init_atmosphere and run MPAS w/o GWDO. Maybe Michael, Laura or any of the other MPAS developers can give more details.

Cheers

Dom

[Wei Huang]

Dom,

Thanks for the help.

I guess the MPAS developers should change the "dir_gwdo" part of code to:

 dir_gwdo = '   '

 if(minMeshD == 1.0_RKIND .and. maxMeshD == 1.0_RKIND) then

   ! ... uniform 2562 mesh:

    if(mindcEdge .ge. 400000._RKIND .and. maxdcEdge .lt. 560000._RKIND) then

       dir_gwdo = 'orogwd_2deg'

    !... uniform 10242 mesh:

    elseif(mindcEdge .ge. 200000._RKIND .and. maxdcEdge .lt. 260000._RKIND) then

       dir_gwdo = 'orogwd_2deg'

    elseif(mindcEdge .ge. 90000._RKIND .and. maxdcEdge .lt. 150000_RKIND) then

       dir_gwdo = 'orogwd_1deg'

    elseif(mindcEdge .ge. 40000._RKIND .and. maxdcEdge .lt. 70000._RKIND) then

       dir_gwdo = 'orogwd_30m'

    elseif(mindcEdge .ge. 20000._RKIND .and. maxdcEdge .lt. 35000._RKIND) then

       dir_gwdo = 'orogwd_10m'

    elseif(mindcEdge .ge. 10000._RKIND .and. maxdcEdge .lt. 20000._RKIND) then

       dir_gwdo = 'orogwd_10m'

    else

       write(0,*)

       write(mess,*) 'GWDO: Interpolation not available. The initialization will abort'

       call physics_error_fatal(mess)

       write(mess,*) 'GWDO: Interpolation not available. Set config_gwdo_scheme = .false.'

       return

    endif

 else

    write(0,*)

!   write(mess,*) 'GWDO: The input mesh must be a uniform mesh. The initialization will abort'

!   call physics_error_fatal(mess)

    write(mess,*) 'GWDO: The input mesh must be a uniform mesh. Set config_gwdo_scheme = .false.'

    return

 endif

 write(0,*) 'dir_gwdo   =    ', trim(dir_gwdo)

 write(0,*)

Which should be able to handle mesh: 2562, 655362. and 2621442, or even 5898242.

Otherwise, users may continue to hit the issues I am dealing with in the future.

Again, thanks for the help.

Wei

an email to mpas-atmosphere-help+unsubscrib...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups
"MPAS-Atmosphere Help" group.
To unsubscribe from this group and stop receiving emails from it, send
an

email to mpas-atmosphere-help+unsubscrib...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

!-------------------------------------------------------------------------------------------------------------
Laura D. Fowler
Mesoscale and Microscale Meteorology Division (MMM)
National Center for Atmospheric Research
P.O. Box 3000, Boulder CO 80307-3000

e-mail: la...@ucar.edu
phone: 303-497-1628


!-------------------------------------------------------------------------------------------------------------

-- 
!-------------------------------------------------------------------------------------------------------------
Laura D. Fowler
Mesoscale and Microscale Meteorology Division (MMM)
National Center for Atmospheric Research
P.O. Box 3000, Boulder CO 80307-3000

e-mail: la...@ucar.edu
phone: 303-497-1628

!-------------------------------------------------------------------------------------------------------------

-- 
You received this message because you are subscribed to the Google Groups "MPAS-Atmosphere Help" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mpas-atmosphere-help+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "MPAS-Atmosphere Help" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mpas-atmosphere-help/86YQI20s-Lc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to mpas-atmosphere-help+unsubscrib...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[MPAS-Atmosphere Help]

Hi, Wei.

In MPAS v5.0, we have a new method for computing the GWDO static fields directly on the native MPAS mesh: all that's needed is to set

config_native_gwd_static = true when generating the static.nc file. For MPAS v4.0, I agree that we are missing coverage in the conditionals that decide which resolution of data to use for the GWDO static interpolation. I think the static files created by MPAS v5.0 are compatible with MPAS v4.0 (though the initial conditions are not), so if the use of the YSU gravity wave drag scheme is important for your application, you could create the static files with MPAS v5.0, then switch back to a previous version of MPAS to create the initial conditions and run the model.

Best regards,

Michael

an email to mpas-atmosphere-help+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups
"MPAS-Atmosphere Help" group.
To unsubscribe from this group and stop receiving emails from it, send
an

email to mpas-atmosphere-help+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

!-------------------------------------------------------------------------------------------------------------
Laura D. Fowler
Mesoscale and Microscale Meteorology Division (MMM)
National Center for Atmospheric Research
P.O. Box 3000, Boulder CO 80307-3000

e-mail: la...@ucar.edu
phone: 303-497-1628


!-------------------------------------------------------------------------------------------------------------

-- 
!-------------------------------------------------------------------------------------------------------------
Laura D. Fowler
Mesoscale and Microscale Meteorology Division (MMM)
National Center for Atmospheric Research
P.O. Box 3000, Boulder CO 80307-3000

e-mail: la...@ucar.edu
phone: 303-497-1628

!-------------------------------------------------------------------------------------------------------------

-- 
You received this message because you are subscribed to the Google Groups "MPAS-Atmosphere Help" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mpas-atmosphere-help+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "MPAS-Atmosphere Help" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mpas-atmosphere-help/86YQI20s-Lc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mpas-atmosphere-help+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.