OpenCoarrays 1.0.0 has been released

[Damian Rouson]

For those interested in free access to coarray Fortran (CAF) parallel
programming, OpenCoarrays 1.0.0 was posted online on Sunday at

https://github.com/sourceryinstitute/opencoarrays/releases

For an overview and links to additional documentation, please see

https://github.com/sourceryinstitute/opencoarrays

or read below.

OpenCoarrays 1.0.0 provides

1. A runtime library ("libcaf_mpi.a" by default),
2. A compiler wrapper ("caf"), and
3. A program launcher ("cafrun").

The library has been tested extensively on several platforms and should be
ready for use in production code. The compiler wrapper and program launcher
are very simple prototypes with limited capabilities that we hope will grow
over time. We welcome users to contribute issue reports at
https://github.com/sourceryinstitute/opencoarrays/issues.

OpenCoarrays 1.0.0 supports three categories of compilers:

1. OpenCoarrays-Aware (OCA) Coarray Fortran (CAF) Compilers.
2. Non-OCA CAF Compilers
3. Non-CAF Compilers

As of release 5.1.0, GNU Fortran is an OCA CAF compiler that uses
OpenCoarrays to support most Fortran 2008 coarray features and some features
proposed for Fortran 2015 in the draft technical specification TS 18508
Additional Parallel Features in Fortran [1]. Supported features include the
collective subroutines.

OpenCoarrays contains an "opencoarrays" module that extends the capabilities
of non-OCA CAF compilers and non-CAF compilers via use association, e.g.,
"use opencoarrays, only : co_sum."

For non-CAF compilers, OpenCoarrays provides CAF capabilities that don't
require the use of codimensions. Although this is a severe restriction, the
choice to offer this capability came from experience with several
pre-release users whose applications' algorithmic needs were completely
covered by a combination of collective subroutines, synchronization
statements, error termination, and the intrinsic functions this_image() and
num_images().

For additional information, including a list of contributors and related
publications, please visit http://www.opencoarrays.org. Many thanks go to
the pre-release users and the OpenCoarrays team, especially lead developer
Alessandro Fanfarillo and GFortran compiler developer Tobias Burnus for
their invaluable work.

[1] http://isotc.iso.org/livelink/livelink?func=ll&objId=17181227&objAction=Open

[michael siehl]

Hi Damian,
that's really great news. Congratulations to all of you for that amazing job. At this time, I am still using GFortran 5.0 together with an older OpenCoarrays version, but will install and use the new version shortly. As you may remember, to me the most important feature is the ability to use the same source code files together with at least two compilers, currently GFortran/OpenCoarrays and Intel ifort. This already worked perfectly with the older version and surely will do with the new version.
To understand the importance of your development, I think it's good to keep an eye on the huge pace of parallel hardware development (namely 'Knights Landing', see page 3 here for example: https://software.intel.com/sites/default/files/managed/d8/6a/MESH%20Group_ANSYS_Brief.pdf ), and also keep an eye on the requirements to (efficiently) develop (efficient) parallel software for these computers (namely 'multi-level parallelism' or 'Simultaneous Heterogeneous Computing': https://software.intel.com/sites/default/files/managed/d8/6a/MESH%20Group_ANSYS_Brief.pdf ). Although the paper does not mention Fortran, coarrays, or PGAS at all, and although PGAS may only be one among other required parallel programing techniques, we should be assured it will be one of the more powerfull techniques.
Let me give another example for this. As far as my current understanding goes, being on a certain coarray image, there is no guarantee that you can access a certain file system in your code when it is later executed on a distributed parallel machine. Therefore I am changing my code currently to avoid any file system access, except from image 1. This felt bad because, from my serial programing, I am used to maintain an error handling or tracing on the file system to get a means for accessing tracing or error information from a file in case something goes badly wrong in my code. This would've been even more important in my parallel programing, since I probably won't be able to debug the code (which may only work for SPMD code but not for my MPMD-like code). My currently expected solution for that lose of accessing the file system sounds PGAS memory (coarrays). I think (and partly know) that it is possible to access the PGAS memory of a remote image, even if that image is idle, and possibly even if it's code execution has crashed (ok, I am not sure about that currently). Otherwise, I could tell further reasons why PGAS memory (coarrays) are much more useful than only for simple image-to-image data transfer.

best regards
Michael

[Damian Rouson]

On Wednesday, July 29, 2015 at 9:31:23 AM UTC-7, michael siehl wrote:
>
>
> Hi Damian,
> that's really great news. Congratulations to all of you for that amazing job.

Thanks for your kind words and encouragement.

> To understand the importance of your development, I think it's good to keep an eye on the huge pace of parallel hardware development (namely 'Knights Landing', see page 3 here for example: https://software.intel.com/sites/default/files/managed/d8/6a/MESH%20Group_ANSYS_Brief.pdf ), and also keep an eye on the requirements to (efficiently) develop (efficient) parallel software for these computers (namely 'multi-level parallelism' or 'Simultaneous Heterogeneous Computing': https://software.intel.com/sites/default/files/managed/d8/6a/MESH%20Group_ANSYS_Brief.pdf ). Although the paper does not mention Fortran, coarrays, or PGAS at all, and although PGAS may only be one among other required parallel programing techniques, we should be assured it will be one of the more powerfull techniques.

We just submitted a paper to ParCo 2014 (http://www.parco.org) that demonstrates one approach to exploiting many-core architectures with coarrays. Alessandro will give a related tutorial at the Fortran standards committee meeting in London next week. If the paper is accepted

> Let me give another example for this. As far as my current understanding goes, being on a certain coarray image, there is no guarantee that you can access a certain file system in your code when it is later executed on a distributed parallel machine. Therefore I am changing my code currently to avoid any file system access, except from image 1. This felt bad because, from my serial programing, I am used to maintain an error handling or tracing on the file system to get a means for accessing tracing or error information from a file in case something goes badly wrong in my code. This would've been even more important in my parallel programing, since I probably won't be able to debug the code (which may only work for SPMD code but not for my MPMD-like code). My currently expected solution for that lose of accessing the file system sounds PGAS memory (coarrays). I think (and partly know) that it is possible to access the PGAS memory of a remote image, even if that image is idle, and possibly even if it's code execution has crashed (ok, I am not sure about that currently). Otherwise, I could tell further reasons why PGAS memory (coarrays) are much more useful than only for simple image-to-image data transfer.

Regarding fault-tolerance, you might be interested in the "failed image" section of the aforementioned draft technical standard TS 18508 (see the link in the original post on this thread). The TS is in the advanced stages of consideration for inclusion in Fortran 2015 and will be discussed at next week's meeting.

Regarding parallel file I/O, you raise valid points. I don't think the committee is considering new features for Fortran 2015, but for the time being it might be helpful to note that CAF and MPI can be mixed in the same code. That might be an attractive option if MPI's parallel file I/O capabilities provide what you need. I have no experience with this so I can't comment further.

Best Regards,
Damian

[michael siehl]

>Regarding parallel file I/O, you raise valid points. I don't think the >committee is considering new features for Fortran 2015, but for the time being >it might be helpful to note that CAF and MPI can be mixed in the same code. >That might be an attractive option if MPI's parallel file I/O capabilities >provide what you need.

Hi Damian,
thanks for your hints. But let me point out, that the current parallel file I/O of Coarray Fortran is not that bad. (The main reason for mentioning it was that I wanted to show another reason for the usefulness of PGAS memory in practice ;-). My current solution allows image 1 to access the file system and then, through a user supplied file, may give further images (in my case the Team Manager images) access to the file system. That way, it is very easy to change access to the file system for specific images in the future without any code changes. Therefore, I personally would not require any additional features to CAF's file access yet.
My intention was to point out, that the use of PGAS memory could be of advantage for certain tasks, especially if it does help avoiding the file system. Specifically, one might store some sort of information (e.g. a small amount of tracing data) in the PGAS memory of each image, which might be remotely accessed only in the rare case of an algorithm failure on a certain image. Of course, it's a little bit esoteric because I have not tried this yet.
On the other hand, using flags in PGAS memory to avoid synchronizations is something which works well in my development so far.

best regards
Michael

[Stefano Zaghi]

Dear Damian, great congratulations to all of you, OpenCoarrays is a wonderful, exciting project.

My best regards,
Stefano

[michael siehl]

just a brief feedback:
On Linux Ubuntu with OpenMPI 1.8.8 and GCC 6, OpenCoarrays 1.0.0 (libcaf_mpi.a) works very well. The code compiles and the executables run exactly as expected. Great job.

michael

[Damian Rouson]

Thanks for the great feedback and encouragement.

Damian

[Damian Rouson]

We're glad to hear it. We have posted a 1.0.1 release with some fixes and continue to post updates as they become available.

Damian

[michael siehl]

>We're glad to hear it. We have posted a 1.0.1 release with some fixes and >continue to post updates as they become available.
>
>Damian

BTW, in comparing ifort 15 executables with those from GFortran/OpenCoarrays directly, I currently do experience two main advantages of the GFortran/OpenCoarrays executables. First, the much faster run-time perfomance of the OpenCoarrays executables, and secondly there seems to be a better run-time error checking with OpenCoarrays. In a specific case, I used SYNC IMAGES(1) without the required SYNC IMAGES(*) statement. The ifort 15 executable crashes without any feedback, the terminal must be restarted. In comparision, the OpenCoarrays/GFortran executable gives an appropriate error message (see below) and does not crash the terminal:

Fortran runtime error on image 8: SYNC IMAGES failed
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was:

Process name: [[43077,1],7]
Exit code: 1
--------------------------------------------------------------------------
ms@ms-P6622:

best regards
Michael

[Damian Rouson]

On Sunday, August 16, 2015 at 3:13:42 PM UTC-4, michael siehl wrote:
>
> BTW, in comparing ifort 15 executables with those from GFortran/OpenCoarrays directly, I currently do experience two main advantages of the GFortran/OpenCoarrays executables. First, the much faster run-time perfomance of the OpenCoarrays executables,

Could you give some details of the scenarios in which you see a performance difference? One of the main issues of which I'm aware is that, in some situations in which a coarray appears on the left-hand side of an assignment, Intel transfers data element-by-element rather than sending the whole array at once even when the array is contiguous. An example is

module criminally_bad_performance
implicit none
contains
subroutine ifort_transfers_elementwise(original)
real, contiguous :: original(:,:)
real, allocatable :: copy(:,:)[:]
if (num_images()<2) error stop
if (this_image()==2) copy(:,:)[1] = original(:,:)
end subroutine
end module

Reviewing the assembly code produced by compiling with the flags "-S -coarray=shared -coarray-num-images=4 -c" shows a data transfer (a call to a procedure with "CAF_PUT" in the name) wrapped by a LOCK/UNLOCK pair, where are wrapped by two conditional jumps (assembly instruction "jle"). The two jumps presumably represent nested loops over the two array dimensions. Furthermore, every unit increase in the number of array dimensions in the above coarray assignment results in an additional jump, which corresponds to an additional layer of nesting of the data transfer loops.

Such compiler behavior was first reported to me by a now-retired Intel engineer in August 2012. Then a non-Intel engineer pointed me to the corresponding assembly code and I submitted a related bug report in March 2014. It really worries me that Intel hasn't fixed this yet. I think this performance issue hurts Fortran. For every person like you who is willing to discuss the problem publicly, I bet there are hundreds of people who silently walk away thinking that this represents a problem in CAF generally rather than a problem Intel's implementation of CAF.

:(

[michael siehl]

>Could you give some details of the scenarios in which you see a performance
>difference?

Hi Damian and thanks for your interest. My performance testing so far is very, very rudimentary. Occasionally I use the slowest processor I have available (an old dual core AMD Turion mobile) and just compare the executables from the different compilers visually. I use coarrays of derived type with static components only, and do image-to-image transfer only for selected coarray derived type components. My last test did not involve much image-to-image data transfer, just few INTEGER arrays and scalars, but also CHARACTER arrays containing just one element. (I made the experience that CHARACTER data can only be transferred as coarray arrays. CHARACTER coarray scalars did not work in my programing. I've tried this with ifort as well as with OpenCoarrays).

[michael siehl]

>It really worries me that Intel hasn't fixed this yet.

Ifort 15 (as well as 13 and 14) executables were really slow in my tests, but not those from ifort 16 (I tested the beta versions lately). I can't tell for sure yet, mainly because I've tested the ifort 16 beta on a different Linux system (but with the same hardware), but I'd bet they did improve coarray performance, maybe even dramatically. Someone else should try ifort 16 for coarray performance testing.

[michael siehl]

>> I bet there are hundreds of people who silently walk away thinking that this >> represents a problem in CAF generally rather than a problem Intel's
>> implementation of CAF.

Really, I fear that many (potential) users do mainly focus on that kind of mpi-like performance (one-sided communication and use of symmetric coarrays) when they judge Coarray Fortran's usefulness. Possibly a very short-sighted view and by far not enough to achieve anything for upcoming many-core computers. Let me give some example:

To keep the (worker) images busy I want to implement some sort of buffer or caching in the PGAS memory of these images. This will require some program logic that needs to interact with PGAS memory on remote images too.
With Coarray Fortran we have three obvious ways for remote data transfer. First, we can transfer data from the current (executing) image to a remote image. Second, we can transfer data from a remote image to the current image. And third, we can use the current image to transfer data from one remote image to another remote image.
The third alternative seems to be very unattractive because it involves three images and thus brings more data transfer than necessarily required. (Even if the main data transfer is only between the both remote images). But I currently see some main advantages for my programming using that third alternative, because it may allow to easily separate the program logic for that buffer or cache on the worker images into an separate object model that runs on a distinct image only for that purpose. That would allow me to leave the worker images just for what they are intended for, namely execute the program logic of there tasks using SIMD parallelism and possibly some further MIMD parallelism. At the same time, the distinct image could manage the buffer or cache of a certain number of worker images. Thus, Coarray Fortran and it's PGAS memory does already influence the design and interactions of my object models.

[Damian Rouson]

What version of OpenCoarrays are you using? I made some fixes to character data transfer with co_broadcast in OpenCoarrays 1.0.1 (https://github.com/sourceryinstitute/opencoarrays/releases/tag/1.0.1). Possibly I'll need to make similar changes for other character data transfers. If you don't mind writing a minimal example that reproduces the problem, please submit it via the OpenCoarrays Issues page (https://github.com/sourceryinstitute/opencoarrays/issues).

Damian

[Damian Rouson]

On Friday, August 21, 2015 at 4:21:42 AM UTC-4, michael siehl wrote:
> >It really worries me that Intel hasn't fixed this yet.
>
> Ifort 15 (as well as 13 and 14) executables were really slow in my tests, but not those from ifort 16 (I tested the beta versions lately). I can't tell for sure yet, mainly because I've tested the ifort 16 beta on a different Linux system (but with the same hardware), but I'd bet they did improve coarray performance, maybe even dramatically. Someone else should try ifort 16 for coarray performance testing.

I submitted the code posted earlier in a bug report to Intel and they reported that the issue with that specific example was fixed in an update that was released April 1 (my build was from March 26) so that probably explains the improvement you're seeing. Apparently there remain cases where ifort transfers coarray data element-by-element, but I don't know the specifics of what determines when such behavior happens.

At least I'm really glad to hear Intel addressing these issues. I know of at least one person who abandoned coarray Fortran based on experience with the Intel compiler. My guess is there are many others who have done the same.

Damian

[michael siehl]

>What version of OpenCoarrays are you using? I made some fixes to character >data transfer with co_broadcast in OpenCoarrays 1.0.1 (https://github.com >/sourceryinstitute/opencoarrays/releases/tag/1.0.1). Possibly I'll need to >make similar changes for other character data transfers. If you don't mind >writing a minimal example that reproduces the problem, please submit it via >the OpenCoarrays Issues page (https://github.com/sourceryinstitute/opencoarrays >/issues).

Indeed, I used an older version of OpenCoarrays (and ifort) when I tried scalar CHARACTER (as derived type component) with coarrays the last time. I will retry with the new OpenCoarrays version (and with ifort 16) shortly (a couple of days) and will inform you about the result. A small example program, in case it fails, should not be of any problem.

regards
michael

[michael siehl]

>>(I made the experience that CHARACTER data can only be transferred as coarray >>arrays. >CHARACTER coarray scalars did not work in my programing. I've tried >>this with ifort as well as >with OpenCoarrays).
>>

>>What version of OpenCoarrays are you using?  I made some fixes to character >>data transfer with >co_broadcast in OpenCoarrays 1.0.1 (https://github.com/). >> Possibly I'll need to make similar >changes for other character data >>transfers.  If you don't mind writing  a minimal example that >reproduces the >>problem, please submit it via the OpenCoarrays Issues page >>(https://github.com/).

Sorry Damian, this was my mistake. The original problem I had with scalar CHARACTER data transfer through coarrays was with ifort 14. Four month ago, I did new testing with ifort 15 and also with Gfortran/OpenCoarrays and thought it did not work again, but it did work. I just did not recognize, because the output of my test program was split. I just made new tests using the same setup as four month ago.
Therefore, scalar CHARACTER data transfer with coarrays DOES work with ifort 15 as well as with all versions of OpenCoarrays I have.

regards
michael

[Damian Rouson]

On Monday, August 24, 2015 at 1:14:42 PM UTC-7, michael siehl wrote:
> Therefore, scalar CHARACTER data transfer with coarrays DOES work with ifort 15 as well as with all versions of OpenCoarrays I have.
>

Bravo! Life is great when things work out better than expected.

:D