hycom_parallel memory problem help !

[zxtgool]

Dear Sir,

 

I'm doing high-resolution simulation using HYCOM, so the grid is very large. I choose MPI parallelization type. When running, it takes infinite time before printing "now opening forcing field..." . and the CPU's memory remains little...  Some people suggested it maybe out of memory.  Can you help me, is memory lackness the reason and how can I fix it ?  Is it possible doing large grid simulation using HYCOM, if yes, what should I adjust ?

 

A far as i know, in MITgcm, every tile has its own result file, and user can post-process by combining the results into one complete file. According to this ,it saves a lot of memory and time.  How does hycom save its memory and time ?  How can I get the memory every tile needs ?

 

Thank you so much! Best wishes!

Xiaoting, Ocean University of China

 

2012-05-17

---

zxtgool

[Alan Wallcraft]

Using MPI only helps if you are running across multiple nodes. No
matter what approach you use (serial, OpenMP or MPI) it is very
important that the computer system as a whole has enough physical
memory to hold HYCOM entirely in memory (not using virtual memory at
all). If you are using a single Linux computer /proc/meminfo returns
memory stats, for example:

ozark 41> head -1 /proc/meminfo
MemTotal: 49411740 kB

In this case. ozark has 48GB of physical memory.

Most of HYCOM's arrays are in common, and so the size command gives a
good indication of how much memory is needed, for example:

eslogin3 147> size -d hycom
text data bss dec hex filename
4617132 427664 213145360 218190156 d01514c hycom

In this case each MPI task requires about 208 MB (1 MB = 2^10 B) of
memory, and since this system has 8 cores per node it only needs 1.7
GB per node. However, this case is setup for 48 MPI tasks so it would
use 6 nodes in total and its total memory requirement is 10 GB. This
is a relatively small test case. Note that from a memory point of
view, I could run this test case on ozark but ozark only has 8 cores -
so I would recompile HYCOM for 8 MPI tasks (needing more memory per
task) or for OpenMP.

The bottom line is that there is a limit to how big a region can be
run on any given computer resource. If you want to run large grids,
you almost certainly need a Linux Cluster with multiple nodes. Also,
Infiniband networking is about the minimum you need between nodes
(most variants of Ethernet introduce too much latency).

The long time before you got to the forcing files might have been due
to page faulting into virtual memory, as I said above, ocean models
have to fit in physical memory. It might instead have been due to
running more than one mpi task per CPU, which would also thrash the
system. It could also have been simply due to running a very big grid
on a small computer. Small is relative, the biggest grid with have
used with HYCOM is 9000x6595x32 and it requires about 1.8 GB of memory
per MPI task when running with 1,000 MPI tasks.