Weather forecasts using CoArray Fortran

[Beliavsky]

http://phys.org/news/2015-08-future-weather-agency-titan-advance.html
The future of forecasting: Leading weather agency turns to Titan to advance science of prediction
August 21, 2015 by Gregory Scott Jones And Jeremy Rumsey

...

Using Titan, the team was able to run the IFS model at the highest resolution ever (2.5-kilometer global resolution), using Coarray Fortran, an extension of Fortran, to optimize communication between nodes. This brought a major improvement in performance with only a modest introduction of new code.

The team also achieved the benchmark of simulating a 10-day forecast in under one hour using a 5-kilometer global resolution model on 80,000 Titan cores. Without the addition of coarrays, the simulations would have required up to 120,000 cores. This meant that, by reworking basic communication procedures, the team was able to achieve a 33 percent increase in performance--a vital improvement necessary for meeting the team's future scalability goals.

...

The IFS forecast model relies heavily on the conventional Message Passing Interface (MPI), the mechanism responsible for code communication in countless applications. As models are scaled up in resolution, there needs to be a corresponding reduction in the model time-step, and at some point MPI communications can become crippling and stall performance. In fact, these communications can account for up to 50 percent of the IFS's execution time at high core counts.

Although ECMWF has yet to harness the power of Titan's GPUs, the traditional CPUs were sufficient to identify scaling issues in their model.

To address the communication bottleneck, the team found a solution in using coarrays, which are part of the Fortran 2008 standard and fully supported by the Cray Fortran compiler on Titan.

[michael siehl]

Thank you very much for pointing to this article, which is the first one I am aware of that points out the real performance power behind Coarray Fortran. See the following excerpts:

"Applications such as ECMWF's IFS model use OpenMP (...) for computations on the nodes and MPI for communications across the nodes. Generally, this arrangement works well for many applications, but (...), it can bog down the application, wasting precious runtime--it simply does not easily allow for any overlap of computation and communication. The use of coarrays (also called PGAS, for Partitioned Global Address Space) in Fortran, however, allows a simple syntax for data to be communicated between tasks at the same time computations are being performed."
.
.
""We have overlapped the communication with computation instead of having just the OpenMP parallel region for the computation," Mozdzynski said. "Not everything was perfect when we started using coarrays, but over time each compiler release got better, so much so that at this time everything is working perfectly with our coarray implementation.""
.
.
"Essentially, by overlapping computation and communication, the team's processing times have decreased dramatically. (...)"


It is exactly that overlap between computation and communication which I experience in my own coarray programming and believe to be the real performance power behind Coarray Fortran and it's PGAS memory. I plan to implement a buffer in the PGAS memory of worker images shortly. This buffer will require some program logic code which can easily be executed on a separate image because the PGAS memory of the worker images is remotely accessible. Thus, the worker images themselves can be kept busily working on their tasks without being involved in much of the remote communication and, more importantly, without the need of waiting for any remote communication to be completed.

regards
Michael