Looking for x5.6488066.cvt.infor

[Anh The Vu]

Hi,

Can you give me file x5.6488066.cvt.infor?

I want to run MPAS with 15-km – 3-km mesh, and want to have a new "cvt" partition file when interpolating the static, geographical fields in parallel.

I already installed Metis successfully.

Thank you.

[MPAS-Atmosphere Help]

Hi, Anh.

Whereas the x5.6488066.graph.info file can be partitioned with Metis to give, e.g., x5.6488066.graph.info.part.1024, the x5.6488066.cvt.part.256 file that we provide with the 15-3 km mesh download is produced in a completely different way -- i.e., it isn't produced from an x5.6488066.cvt.info file using Metis. The reason for this is that Metis isn't able to guarantee any properties of the geometric shape of each partition, while the parallel interpolation of static fields currently requires that each partition be convex.

If at all possible, I'd recommend to simply use the x5.6488066.cvt.part.256 file when interpolating the static fields in parallel for the 15-3 km mesh; you'll need to have at least several hundred processors anyway to run the model itself (we typically use several thousand or more cores when simulating on this mesh). If there is a specific reason why using 256 MPI tasks to produce the static file is not possible, we could probably create another CVT partition file with a different number of partitions, though this would take some non-trivial effort. In this case, just let me know.

Best regards,
Michael

[Anh The Vu]

Hi Michael,

Thank you for your feedback.

However, I am only able to run 256 MPI tasks with the file x5.6488066.cvt.part.256, aren't I?. It takes very long time to process. My job has been running for more than 2 days and still not finished yet. So how much time does it regularly take to process static, geographical field interpolation step?

Best regards,

Anh V.
Vào 20:14:36 UTC-4 Thứ Ba, ngày 15 tháng 8 năm 2017, MPAS-Atmosphere Help đã viết:

[MPAS-Atmosphere Help]

Hi, Anh.

Generally, it should take less than an hour to process the static fields. Depending on your MPI library, it is possible that one of the MPI tasks is progressing through the static interpolation faster than the others and timing out when it reaches the next collective MPI call in the code. Does it look like all of the tiles of topography, land use, etc. have been processed before the code stalls? If you can post the log.0000.err file that is generated after running for an hour or so, I could take a look to see if there are any clues as to where the code is getting stuck.

Best regards,
Michael

[Anh The Vu]

This is the log.0000.err. I am running MPAS ver 5.1:

--
You received this message because you are subscribed to a topic in the Google Groups "MPAS-Atmosphere Help" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mpas-atmosphere-help/1Xl9Au8mxpE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mpas-atmosphere-help+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

==============================
Anh The Vu, Graduate Student,

Department of Earth and Atmospheric Science,

Indiana University, USA,

Phone: 812-606-8989,

Alternative email: an...@iu.edu

 

[Anh The Vu]

Using default double-precision reals

 Reading namelist from file namelist.init_atmosphere

 *** Encountered an issue while attempting to read namelist record interpolation

 _control

     The following values will be used for variables in this record:

         config_extrap_airtemp = linear

 Reading streams configuration from file streams.init_atmosphere

Found mesh stream with filename template x5.6488066.rotated.grid.nc

Using default io_type for mesh stream

  ** Attempting to bootstrap MPAS framework using stream: input

 Bootstrapping framework with mesh fields from input file 'x5.6488066.rotated.gr

 id.nc'

Parsing run-time I/O configuration from streams.init_atmosphere ...

 -----  found immutable stream "input" in streams.init_atmosphere  -----

        filename template:  x5.6488066.rotated.grid.nc

        filename interval:  none

        direction:          input

        reference time:     initial_time

        record interval:    -

        input alarm:        initial_only

 -----  found immutable stream "output" in streams.init_atmosphere  -----

        filename template:  x5.6488066.static.nc

        filename interval:  none

        I/O type:           Parallel-NetCDF

        direction:          output

        reference time:     initial_time

        record interval:    -

        output alarm:       initial_only

        package:            initial_conds

 -----  found immutable stream "surface" in streams.init_atmosphere  -----

        filename template:  x1.10242.sfc_update.nc

        filename interval:  none

        direction:          output

        reference time:     initial_time

        record interval:    -

        output alarm:       86400

        package:            sfc_update

 ----- done parsing run-time I/O from streams.init_atmosphere -----

Reading dimensions from input streams ...

 ----- reading dimensions from stream 'input' using file x5.6488066.rotated.grid

 .nc

                     nCells = 6488066

                     nEdges =19464192

                  nVertices =12976128

                        TWO =       2

                   maxEdges =       7

                  maxEdges2 =      14

               vertexDegree =       3

 ----- done reading dimensions from input streams -----

Processing decomposed dimensions ...

 ----- done processing decomposed dimensions -----

Assigning remaining dimensions from definitions in Registry.xml ...

                      THREE =       3

                    FIFTEEN =      15

                  TWENTYONE =      21

                         R3 =       3

                nVertLevels =       1 (config_nvertlevels)

                nSoilLevels =       1 (config_nsoillevels)

                  nFGLevels =      38 (config_nfglevels)

              nFGSoilLevels =       4 (config_nfgsoillevels)

              nVertLevelsP1 =       2

                    nMonths =      12 (config_months)

 ----- done assigning dimensions from Registry.xml -----

On 17 August 2017 at 16:42, Anh The Vu <thean...@gmail.com> wrote:

This is the log.0000.err. I am running MPAS ver 5.1:

To unsubscribe from this group and all its topics, send an email to mpas-atmosphere-help+unsubscrib...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

==============================
Anh The Vu, Graduate Student,

Department of Earth and Atmospheric Science,

Indiana University, USA,

Phone: 812-606-8989,

Alternative email: an...@iu.edu

 

[Anh The Vu]

This is my submit job script:

#!/bin/bash -l

#SBATCH -N 6

#SBATCH -n 288

#SBATCH -t 168:00:00

#SBATCH -J MPAS_geog

#SBTACH -e init.err

cd ../MPAS-Release-5.1

ulimit -u unlimited

srun -N 6 -n 256 ./init_atmosphere_model

On 17 August 2017 at 17:02, Anh The Vu <thean...@gmail.com> wrote:

This is my submit job script:

#!/bin/bash -l

#SBATCH -N 6

#SBATCH -n 288

#SBATCH -t 168:00:00

#SBATCH -J MPAS_geog_4.0

#SBTACH -e init.err

cd ../MPAS-Release-4.0

ulimit -u unlimited

srun -N 6 -n 256 ./init_atmosphere_model

[Anh The Vu]

This is my submit job script:

#!/bin/bash -l

#SBATCH -N 6

#SBATCH -n 288

#SBATCH -t 168:00:00

#SBATCH -J MPAS_geog_5.1

#SBTACH -e init.err

cd ../MPAS-Release-5.1

ulimit -u unlimited

srun -N 6 -n 256 ./init_atmosphere_model

Vào 16:39:41 UTC-4 Thứ Năm, ngày 17 tháng 8 năm 2017, MPAS-Atmosphere Help đã viết:

[Dominikus Heinzeller]

Hi Anh,

I think there are two possibilities, (a) MPAS is hanging while trying to read the mesh file, i.e. an I/O problem, or (b) insufficient memory.

How much memory does each of your 6 nodes have? For reference, on a cluster with 32GB memory per node (incl. OS etc; net about 24GB of memory available to MPAS), I use 64 nodes with only 4 MPI tasks on each of them, i.e. I have 1.5TB of memory available to MPAS. That may be way too much, though.

First thing to try therefore is to throw more memory at the problem. If this doesn’t help, you could try to compile the code in debug mode and also add additional debugging output from the MPAS stream manager. For this, you would have to change line 4 in src/framework/mpas_stream_manager.F from

#define STREAM_DEBUG_WRITE(M) ! write(stderrUnit,*) M

to

#define STREAM_DEBUG_WRITE(M) write(stderrUnit,*) M

before recompiling the code (in debug mode).

Cheers,

Dom

--
You received this message because you are subscribed to the Google Groups "MPAS-Atmosphere Help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpas-atmosphere-...@googlegroups.com.

[Anh The Vu]

Hi,

How to build the MPAS in debug mode?

Thank you.

To unsubscribe from this group and stop receiving emails from it, send an email to mpas-atmosphere-help+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dominikus Heinzeller]

Hi Anh,

could you please try to use the available documentation before asking questions to the forum?

MPAS-Atmosphere Users’ Guide

http://www2.mmm.ucar.edu/projects/mpas/mpas_atmosphere_users_guide_5.2.pdf

Section 3.3, page 10

answers your question.

Regards,

Dom

To unsubscribe from this group and stop receiving emails from it, send an email to mpas-atmosphere-...@googlegroups.com.

[Anh The Vu]

Hi,

I already change line 4 and reinstalled the model in debug mode but it still doesn't work. The attachment is file log.0000.err

Vào 02:58:15 UTC-4 Thứ Hai, ngày 21 tháng 8 năm 2017, Dominikus Heinzeller đã viết:

[du...@ucar.edu]

Hi, Anh.

This looks like it may be a memory issue. From the log.0000.err file, it looks like you have config_nfglevels=38 and config_nfgsoillevels=4 in your namelist.init_atmosphere file. Having these set to any value other than 1 will cause the init_atmosphere_model program to allocate significantly more memory than is required to interpolate just the static fields. Can you try setting

config_nvertlevels = 1
config_nsoillevels = 1
config_nfglevels = 1
config_nfgsoillevels = 1

following Section 7.2.1 of the Users' Guide? This should reduce to a minimum the memory that is allocated at runtime.

As Dom had inquired, how much memory do you have on each of the 6 nodes where the init_atmsophere_model program is being run? If the amount of memory per node is not enough, and if you have more nodes available, it may be necessary to "under-subscribe" each node, i.e., to assign fewer MPI tasks to each node to gain access to the larger aggregate memory of more nodes.

Kind regards,
Michael