ComEx / Global Arrays question

[Brad Chamberlain]

Hi HPC Tools team at PNNL --

For the latest release of Global Arrays, what is the preferred ComEx
implementation when running on a Cray XC? From the documentation at
GitHub, the implication seems to be that the MPI implementation is, but
browsing the source, it looks like there may be MPI-3 and DMAPP
implementations (among others) so wanted to make sure the documentation
was up-to-date.

Thanks,
-Brad

[Palmer, Bruce J]

We are still recommending the MPI-PR implementation as the preferred runtime for the Crays. The MPI3 runtime is based on MPI RMA. We have shown that it passes all tests when running using MPICH and the CH4 devices, but this implementation is only as good as the underlying MPI-3 RMA implementation and this is still sketchy for a lot of MPI libraries. When it works, the MPI3 runtime is seems to be comparable to MPI-PR for puts and gets but lags on accumulates. I don't know much about the DMAPP runtime.

Bruce

[Jeff Hammond]

(adding my $0.02 as an experienced user not affiliated with PNNL)

I concur with Bruce on this.  I have had trouble with the DMAPP port in the past when running NWChem at NERSC, although I have not tried it in years.  Relative to the DMAPP back-end, not only is MPI-PR completely reliable on Cray machines but it's also faster (~30%) for the latency-sensitive case of NWChem SCF.

When I compared MPI-PR to ARMCI-MPI with Casper on NERSC Cray machines in 2015 or 2016, MPI-PR performed the same or slightly better.  Furthermore, MPI-PR avoids RMA bugs in Cray MPI that we are currently fighting when trying to use ARMCI-MPI with Casper.

Jeff

--
You received this message because you are subscribed to the Google Groups "hpctools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hpctools+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jeff Hammond
jeff.s...@gmail.com
http://jeffhammond.github.io/

[Palmer, Bruce J]

One other thing. The MPI-PR port has excellent asynchronous progress. The MPI RMA-based ports (definitely MPI3 and I assume ARMCI-MPI) rely on whatever the MPI-2/3 implementation is giving you. For some implementations, you will need to set environment variables (something like MPI_ASYNC_PROGRESS) to get good asynchronous progress.

 

Bruce

 

Jeff

 

--

To unsubscribe from this group and stop receiving emails from it, send an email to hpctools+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

 

--

Jeff Hammond
jeff.s...@gmail.com
http://jeffhammond.github.io/

--

You received this message because you are subscribed to the Google Groups "hpctools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hpctools+u...@googlegroups.com.

[Jeff Hammond]

Indeed, MPI-PR implements progress in an effective way (processes, not threads) by default.  If one has an application that does not require asynchronous progress, the MPI-TS implementation can be used, but that back-end exists primarily for certain usages of NWChem where one-sided communication isn't critical.

By default, ARMCI-MPI depends on MPI-3 RMA to make asynchronous progress, which is far from certain. I added an option to drive progress via a Pthread (ARMCI_PROGRESS_THREAD=1) but that is the wrong approach, although not unlike what MPICH_ASYNC_PROGRESS has done in the past.

The recommended way to get asynchronous progress with ARMCI-MPI (or any MPI-3 RMA library that uses MPI_Win_allocate) is to use Casper, which is doing the same thing as MPI-PR (MPI_Comm_split to divide MPI_COMM_WORLD into an application communicator and a "hidden" progress communicator), just underneath the MPI-3 RMA interface as part of a PMPI interposition library.

An important caveat of both MPI-PR and ARMCI-MPI + Casper is that they require a process for progress so launching the application with P processes per node (ppn) causes the application to run with P-1 ppn (or P-G in the case of Casper if one requests >1 "ghost" processes).  Please keep this in mind when comparing Global Arrays to other programming models.

Jeff

 

Jeff

 

--

To unsubscribe from this group and stop receiving emails from it, send an email to hpctools+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

 

--

Jeff Hammond
jeff.s...@gmail.com
http://jeffhammond.github.io/

--
You received this message because you are subscribed to the Google Groups "hpctools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hpctools+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "hpctools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hpctools+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brad Chamberlain]

Re-pinging on this as I don't believe I ever heard back. [for context,
I'm helping someone shepherd an SC18 submission who wants to make sure
they're using a reasonable basis of comparison if comparing to global
arrays].

Thanks,
-Brad

[Palmer, Bruce J]

There was a fairly long thread on this last week, with responses from both myself and Jeff Hammond. Bottom line, we are still recommending PR for large calculations.

Bruce

[Jeff Hammond]

Brad:

Are you not signed up the HPCtools list?  Bruce and I both sent two replies to your post.

Jeff

--
You received this message because you are subscribed to the Google Groups "hpctools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hpctools+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.