Using spack on Cray

[Kelly Thompson]

Is there a getting started guide for using spack on a Cray (XC30)?  I know the 'newarch' branch was merged recently, but I don't know how to use the new features. The docs at http://software.llnl.gov/spack/ don't seem to cover the newarch feature (at least it doesn't seem to cover my situation).

My Use Case is building parmetis on Trinity (http://www.lanl.gov/projects/trinity/specifications.php). The PrgEnv-intel programming environment is preloaded to point to in...@15.0.5 and cray-...@7.3.3.  I want to build parmetis using this compiler and MPI. I can edit packages.yaml to tell spack that mpich@3.2 should be used and point spack to the cray-mpich installation and mark it as not buildable. However, there are no compiler wrappers (e.g. mpicc) provided with the installation of cray-...@7.3.3. Instead Cray expects me to use it's PrgEnv compile wrappers (shell scripts CC, cc and ftn that live in a different directory are used no matter what compiler or MPI are selected). How can I tell spack to use these wrappers instead of mpicc? Or maybe there is another way to build on a Cray? I could edit compilers.yaml and customize the intel compilers to point to the cray compile wrappers. Is this the intended mechanism for Cray?

Any advice to get me started would be helpful.

Thank you,

-kt

[Guilherme Peretti Pezzi]

I cannot help you with Spack on Cray, but I can tell you how I would do with EasyBuild (where Cray support is already implemented to properly use the wrappers).

1 - Create a new toolchain version for CrayIntel by tailoring the module versions (cray-libsci, cray-mpich and intel) in order to fit to what you have available on your system. 

You can use this file as reference:

https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/c/CrayIntel/CrayIntel-2015.11.eb

2 - Install the new toolchain: 

$ eb CrayIntel-your_version.eb

3 - Ask easybuild to try using an existing parmetis recipe with your new toolchain. For example:

$ eb ParMETIS-4.0.3-intel-2016a.eb --try-toolchain=CrayIntel,your_version -r

This ParMETIS build completed on our XC30 at CSCS (Piz Daint) with CrayIntel/2015.11 (though I didn't test it).

Cheers,

G.

[Todd Gamblin]

Guilherme,

Spack uses the compiler wrappers out of the box — are you having trouble getting it to do this?

We're actually working on hashing out Cray details in another thread — it got fairly high traffic, but perhaps  it should be moved to the main list.  There is also a soon-to-be merged documentation update here:

https://github.com/LLNL/spack/pull/1125

-Todd

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at https://groups.google.com/group/spack.
For more options, visit https://groups.google.com/d/optout.

[Guilherme Peretti Pezzi]

Todd, 

I didn't try the latest version of Spack on Cray, but I'm curious to see how things have changed since the last time I've tried. 

Since this post didn't get any attention, I (wrongly) assumed that it was not yet working.

Cheers,

G.

[Todd Gamblin]

No worries!  The Cray stuff could use some documentation but I'm hesitant to put everything out there until we work out details with NERSC and ORNL folks. They're currently trying it out.

The compiler detection currently works on Cray and if you run `spack compilers` out of the box it should detect compilers based on modules.  We've also added architecture support that allows targeting the front-end or back-end (compute node) OS, and well as syntax to set the target (haswell, ivybridge, etc.) on demand.

-Todd

[Andrew Turner]

It worked for me out of the box on our XC for Gromacs. I am currently looking at creating a package for VASP and also have the following on my list to try:

* CP2K

* CASTEP

* LAMMPS

as these are high usage on ARCHER.

Cheers

Andy

My Use Case is building parmetis on Trinity (http://www.lanl.gov/projects/trinity/specifications.php). The PrgEnv-intel programming environment is preloaded to point to in...@15.0.5 and cray-...@7.3.3.  I want to build parmetis using this compiler and MPI. I can edit packages.yaml to tell spack that mp...@3.2 should be used and point spack to the cray-mpich installation and mark it as not buildable. However, there are no compiler wrappers (e.g. mpicc) provided with the installation of cray-...@7.3.3. Instead Cray expects me to use it's PrgEnv compile wrappers (shell scripts CC, cc and ftn that live in a different directory are used no matter what compiler or MPI are selected). How can I tell spack to use these wrappers instead of mpicc? Or maybe there is another way to build on a Cray? I could edit compilers.yaml and customize the intel compilers to point to the cray compile wrappers. Is this the intended mechanism for Cray?

Any advice to get me started would be helpful.

Thank you,

-kt

[Adam Stewart]

Andrew,

VASP and CP2K packages would be very useful for us at ANL. Once you have them ready I would be happy to test them.

Adam

[Verónica Vergara L.]

I second that! All the packages you mentioned are heavily used at ORNL as well. If you need a hand testing, please let me know.

Best,

Verónica