Add NAG Compiler Support

[ajstew...@gmail.com]

What is involved in adding support for the NAG Fortran compilers? Specifically, I have a user who would like MPI compiled with GCC for C and C++ and with NAG for Fortran. The compilers.yaml file for this would look like:

compilers:
  linux-x86_64:
    nag@6.0:
      cc: /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-4.4.7/gcc-5.3.0-hvus5iytlc7lrecmmywli3jdky74xf5l/bin/gcc
      cxx: /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-4.4.7/gcc-5.3.0-hvus5iytlc7lrecmmywli3jdky74xf5l/bin/g++
      f77: /blues/gpfs/home/software/nag/6.0/bin/nagfor
      fc: /blues/gpfs/home/software/nag/6.0/bin/nagfor

But when I try to install anything with it, Spack complains that the 'nag' compiler is not yet supported. I'm willing to do some work on this on my end, but someone may have to point me in the right direction to get me started.

[Todd Gamblin]

Hi,

Getting the support in there is pretty easy.  Look in lib/spack/spack/compilers and add a new file called nag.py.  It should contain a class called "Nag" that extends "Compiler" from compiler.py.  The class should look a lot like those in the other compiler files in the lib/spack/spack/compilers directory.

The compiler classes only require a few things:

1. Executable names to search for, for C, C++, F77, and FC (e.g. 'gcc', 'g++', 'gfortran', 'gfortran')

2. Potential suffixes & prefixes (the ones in gcc.py are for the various macports builds of gcc, e.g., *-mp-<version>)

3. Classmethods to get the version of the compiler.  Generally, you implement default_versoin(), which is called if none of the specific ones are defined, and then you implement cc_version, cxx_version, f77_version, and fc_version if there are special cases for the different compilers.

If there are special rpath args you can override that too, but I think NAG uses -Wl,-rpath like most things so you should be good there.

I think with NAG the challenge will be picking how you want "versions" to work.  Spack searches for compilers and tries to group quadruples of C/C++/F77/FC with similar versions.  It looks like you want the class to return '6.0' for NAG when it sees gcc 5.3.  You could think of other mappings if you want to handle other gcc versions.

The best example to look at, and the closest thing to this right now is in xl.py (provided thanks to François Bissey):

https://github.com/LLNL/spack/blob/develop/lib/spack/spack/compilers/xl.py

XL C compilers are always 2 versions behind the fortran version, so the fc_version and f77_version methods just subtract 2 from the compiler version.  I think NAG could be implemented with some slightly more complicated mapping from NAG versions to gcc versions.  

Let me know if this gets you going, and if you have other questions.

-Todd

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at http://groups.google.com/group/spack.
For more options, visit https://groups.google.com/d/optout.

[François Bissey]

On 12/11/15 13:27, Todd Gamblin wrote:
> The best example to look at, and the closest thing to this right now is
> in xl.py (provided thanks to François Bissey):
>
> https://github.com/LLNL/spack/blob/develop/lib/spack/spack/compilers/xl.py
>
> XL C compilers are always 2 versions behind the fortran version, so the
> fc_version and f77_version methods just subtract 2 from the compiler
> version. I think NAG could be implemented with some slightly more
> complicated mapping from NAG versions to gcc versions.
>

I actually answered and pointed him to the PR but I forgot that by
default we answer users and not the list here, so no one else saw it.

Francois

[Todd Gamblin]

On 12/10/15, 4:33 PM, "sp...@googlegroups.com on behalf of François
Bissey" <sp...@googlegroups.com on behalf of

Thanks! Anything interesting in that email worth posting to the list? :)


>
>Francois

[François Bissey]

On 12/11/15 13:35, Todd Gamblin wrote:
> On 12/10/15, 4:33 PM, "sp...@googlegroups.com on behalf of François
> Bissey" <sp...@googlegroups.com on behalf of
> francoi...@canterbury.ac.nz> wrote:
>
>> On 12/11/15 13:27, Todd Gamblin wrote:
>>> The best example to look at, and the closest thing to this right now is
>>> in xl.py (provided thanks to François Bissey):
>>>
>>>
>>> https://github.com/LLNL/spack/blob/develop/lib/spack/spack/compilers/xl.p
>>> y
>>>
>>> XL C compilers are always 2 versions behind the fortran version, so the
>>> fc_version and f77_version methods just subtract 2 from the compiler
>>> version. I think NAG could be implemented with some slightly more
>>> complicated mapping from NAG versions to gcc versions.
>>>
>> I actually answered and pointed him to the PR but I forgot that by
>> default we answer users and not the list here, so no one else saw it.
>
> Thanks! Anything interesting in that email worth posting to the list? :)
>

No. Very much the same as you with less details but I pointed him to

https://github.com/LLNL/spack/pull/39

as an example. Unfortunately it has collected some extra bits somehow.

Francois

[François Bissey]

https://github.com/kiwifb/spack/commit/5f77af6a8f8b328f066383e0786dc5f2373061b8
is actually more useful as it has only the bits I added and show
the other files that have to be touched for the compiler to be usable.

Francois

[Todd Gamblin]

On 12/10/15, 4:38 PM, "François Bissey" <francoi...@canterbury.ac.nz>

wrote:

>On 12/11/15 13:35, Todd Gamblin wrote:
>> On 12/10/15, 4:33 PM, "sp...@googlegroups.com on behalf of François
>> Bissey" <sp...@googlegroups.com on behalf of
>> francoi...@canterbury.ac.nz> wrote:
>>
>>> On 12/11/15 13:27, Todd Gamblin wrote:
>>>> The best example to look at, and the closest thing to this right now
>>>>is
>>>> in xl.py (provided thanks to François Bissey):
>>>>
>>>>
>>>>
>>>>https://github.com/LLNL/spack/blob/develop/lib/spack/spack/compilers/xl
>>>>.p
>>>> y
>>>>
>>>> XL C compilers are always 2 versions behind the fortran version, so
>>>>the
>>>> fc_version and f77_version methods just subtract 2 from the compiler
>>>> version. I think NAG could be implemented with some slightly more
>>>> complicated mapping from NAG versions to gcc versions.
>>>>
>>> I actually answered and pointed him to the PR but I forgot that by
>>> default we answer users and not the list here, so no one else saw it.
>>
>> Thanks! Anything interesting in that email worth posting to the list?
>>:)
>>
>No. Very much the same as you with less details but I pointed him to
>
>https://github.com/LLNL/spack/pull/39
>
>as an example. Unfortunately it has collected some extra bits somehow.

It looks like you pulled the latest develop branch in there at some point,
but that's odd... maybe something else happened in the mainline. Hm..
I'd say to reset the branch in your fork and force push but I'm not sure
that wouldn't clutter things worse. Probably not worth it for something
already merged.

Thanks again!

[Todd Gamblin]

On 12/10/15, 4:41 PM, "sp...@googlegroups.com on behalf of François

Bissey" <sp...@googlegroups.com on behalf of
francoi...@canterbury.ac.nz> wrote:
>

>https://github.com/kiwifb/spack/commit/5f77af6a8f8b328f066383e0786dc5f2373

>061b8
>is actually more useful as it has only the bits I added and show
>the other files that have to be touched for the compiler to be usable.

Ah nice. A couple comments:

The parts in lib/spack/env/cc are probably not necessary. Many of the
links in lib/spack/env are vestigial -- they are intended to catch
compiler calls for builds that somehow don't properly use the spack
wrappers (cc, c++, f77, f90). If your build respects the CC, CXX, FC, and
F77 env vars, it will use the right wrappers automatically. If it uses
Makefiles and doesn't patch them, it probably should patch them.

I think I will eventually remove the extra links and only leave the main
ones in lib/spack/env. I think I may also rename them to spack-cc,
spack-c++, etc., because that ends up being much clearer in the build
logs. It would also make it easy to analyze a build log and see if there
are spots where the wrapper compiler is not being used properly. We've
had users here get confused that the build logs say "cc" and they wonder
exactly which cc that was.

The extra bit in _default_order is probably not strictly necessary for
NAG, unless you want Spack to look for NAG as the default compiler on
systems where gcc and the others are unavailable. I think that's more
relevant for XL -- I can imagine a system with only XL, but I can't
imagine one with only NAG.

-Todd

[François Bissey]

I'll admit that it takes me back and looking at it just now I wonder if
I actually made a mistake in "__init__.py" that may have been a remnant
of my early experiment:
_default_order = ['gcc', 'intel', 'pgi', 'clang', 'xlc']
it should be 'xl' not 'xlc' since I named the file 'xl.py' I am sure.

(and yes I do have access to an AIX machine with XL as the only
available compiler, they indeed exist).

Francois

[ajstew...@gmail.com]

Thanks for the quick responses!

Is it kosher to mix compilers like this? Ideally I guess I would have preferred to be able to specify multiple compilers on the command line. Or to be able to use a Fortran only compiler like NAG and have Spack automatically search for alternative compilers for the C/C++ parts according to the _default_order.

If neither of those are possible, what if I write a nag.py file that only mentions NAG and just points to None for the C/C++ parts. Then I use the compilers.yaml snippet above that points to both. Could that work? Or would that cause unforeseen problems?

If the only possible way to get this done is to write a single nag.py file that includes NAG for Fortran and GCC for C/C++, maybe I can name it nag_gcc.py and figure out a way to have the versions work out to nag-6.0-gcc-5.3.0. Although I don't see an easy way to specify the version on the command line. Hmm. I'm just trying to think of the most extensible way to do this. I'm sure other people will also want to mix compilers, so we should figure out the best way to do things from a design standpoint.

[ajstew...@gmail.com]

Alright, I have a very simple nag.py built. I modeled it after clang.py since it is also missing some compilers. It looks like this:

class Nag(Compiler):
    # Subclasses use possible names of C compiler
    cc_names = []

    # Subclasses use possible names of C++ compiler
    cxx_names = []

    # Subclasses use possible names of Fortran 77 compiler
    f77_names = ['nagfor']

    # Subclasses use possible names of Fortran 90 compiler
    fc_names = ['nagfor']

    @classmethod
    def default_version(self, comp):
        """The '-V' option works for nag compilers.
           Output looks like this::

               NAG Fortran Compiler Release 6.0(Hibiya) Build 1037
               Product NPL6A60NA for x86-64 Linux
               Copyright 1990-2015 The Numerical Algorithms Group Ltd., Oxford, U.K.
        """
        return get_compiler_version(
            comp, '-V', r'NAG Fortran Compiler Release ([0-9.]+)')

See my message above. I'm assuming there's no way to specify multiple compilers on the command line or to let Spack search _default_order for the C/C++ parts. I'm hoping that my second idea will work, in which I have separate nag.py and gcc.py files but a yaml snippet like I first posted that will magically work. If not then I'll have to write a nag_gcc.py file I guess. That will be a bit more complicated.

As for the nag.py file, `spack compiler info nag` now works as expected. However, `spack compiler add /soft/nag/6.0` returns "==> Found no new compilers". How can I get this working?