Module load logic on Cray

[Andrew Turner]

Next question on the Cray support in Spack ;-)

I am writing a package for VASP on Cray XC and it seems pretty simple but the compile falls over with (full log below):

--snip--
gcc -cpp -E -P -C -o preclib.f90 preclib.F
ftn -O1 -ffree-form -ffree-line-length-none -c -o preclib.o preclib.f90
Error:  PE_LIBSCI is not available for the GNU compiler
--snip-- 

Consulting local Cray support they seem to think this is due to the order in which various modules have been loaded (the Cray module environment is great when it works but is fragile wrt order of module loads and even with loading modules that are already loaded, the EasyBuild people ran up against this issue too).

So, my actual question is: where in Spack are the various modules for the Cray programming environment loaded/unloaded? I have looked in the compiler/os/platform definitions and do not see anything there that correspond to module loads. How does Spack switch to, say, the gnu programming environment? By hand I would issue, for example:

module swap PrgEnv-cray PrgEnv-gnu

module swap gcc gcc/5.2.0

but Spack also seems to be doing stuff with:

craype-ivybridge

gcc

because their position in the module stack switches around compared to when I do stuff by hand. I suspect this is the root of my problems.

Any thoughts much appreciated.

Cheers

Andy

==> '/bin/cp' '/home/z01/z01/aturner/software/spack/var/spack/repos/builtin/packages/vasp/makefile.include.crayxc_gnu' 'makefile.include'
==> '/opt/modules/3.2.10.2/bin/modulecmd' 'python' 'list'
Currently Loaded Modulefiles:
  1) modules/3.2.10.2
  2) eswrap/1.3.2-1.020200.1274.0
  3) switch/1.0-1.0502.57058.1.58.ari
  4) craype-network-aries
  5) craype/2.4.2
  6) pbs/12.2.401.141761
  7) cray-mpich/7.2.6
  8) packages-archer
  9) bolt/0.6
 10) nano/2.2.6
 11) leave_time/1.0.0
 12) quickstart/1.0
 13) ack/2.14
 14) xalt/0.6.0
 15) epcc-tools/6.0
 16) cray-libsci/13.2.0
 17) udreg/2.3.2-1.0502.9889.2.20.ari
 18) ugni/6.0-1.0502.10245.9.9.ari
 19) pmi/5.0.7-1.0000.10678.155.25.ari
 20) dmapp/7.0.1-1.0502.10246.8.47.ari
 21) gni-headers/4.0-1.0502.10317.9.2.ari
 22) xpmem/0.1-2.0502.57015.1.15.ari
 23) dvs/2.5_0.9.0-1.0502.1958.2.55.ari
 24) alps/5.2.3-2.0502.9295.14.14.ari
 25) rca/1.0.0-2.0502.57212.2.56.ari
 26) atp/1.8.3
 27) PrgEnv-gnu/5.2.56
 28) gcc/5.1.0
 29) craype-ivybridge
 30) fftw/3.3.4.5
==> 'make' 'all'
mkdir build/std ; \
    cp src/makefile src/.objects makefile.include build/std ; \
    make -C build/std VERSION=std all
mkdir: cannot create directory `build/std': File exists
make[1]: Entering directory `/tmp/aturner/spack-stage/spack-stage-9zNrVI/vasp.5.4.1/build/std'
rsync -ru ../../src/lib .
cp makefile.include lib
make -C lib
make[2]: Entering directory `/tmp/aturner/spack-stage/spack-stage-9zNrVI/vasp.5.4.1/build/std/lib'
gcc -cpp -E -P -C -o preclib.f90 preclib.F
ftn -O1 -ffree-form -ffree-line-length-none -c -o preclib.o preclib.f90
Error:  PE_LIBSCI is not available for the GNU compiler
make[2]: *** [preclib.o] Error 1 

For reference, if I do the module changes by hand the stack looks like:

Currently Loaded Modulefiles:

  1) modules/3.2.10.2

  2) eswrap/1.3.2-1.020200.1274.0

  3) switch/1.0-1.0502.57058.1.58.ari

  4) craype-network-aries

  5) craype/2.4.2

  6) pbs/12.2.401.141761

  7) craype-ivybridge

  8) cray-mpich/7.2.6

  9) packages-archer

 10) bolt/0.6

 11) nano/2.2.6

 12) leave_time/1.0.0

 13) quickstart/1.0

 14) ack/2.14

 15) xalt/0.6.0

 16) epcc-tools/6.0

 17) gcc/5.1.0

 18) cray-libsci/13.2.0

 19) udreg/2.3.2-1.0502.9889.2.20.ari

 20) ugni/6.0-1.0502.10245.9.9.ari

 21) pmi/5.0.7-1.0000.10678.155.25.ari

 22) dmapp/7.0.1-1.0502.10246.8.47.ari

 23) gni-headers/4.0-1.0502.10317.9.2.ari

 24) xpmem/0.1-2.0502.57015.1.15.ari

 25) dvs/2.5_0.9.0-1.0502.1958.2.55.ari

 26) alps/5.2.3-2.0502.9295.14.14.ari

 27) rca/1.0.0-2.0502.57212.2.56.ari

 28) atp/1.8.3

 29) PrgEnv-gnu/5.2.56

 30) fftw/3.3.4.5

[Robert French]

I do not have an answer, but I do agree that PrgEnv-gnu is getting loaded too late. Out of curiosity, what paths is cray-libsci setting when it gets loaded before your PE does?

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at https://groups.google.com/group/spack.
For more options, visit https://groups.google.com/d/optout.

[Andrew Turner]

aturner@eslogin003:~> module show cray-libsci

-------------------------------------------------------------------

/opt/cray/modulefiles/cray-libsci/13.2.0:

conflict libsci 

conflict cray-libsci 

setenv CRAY_LIBSCI_DIR /opt/cray/libsci/13.2.0 

setenv CRAY_LIBSCI_BASE_DIR /opt/cray/libsci/13.2.0 

setenv CRAY_LIBSCI_VERSION 13.2.0 

setenv CRAY_LIBSCI_PREFIX_DIR /opt/cray/libsci/13.2.0/GNU/5.1/x86_64 

setenv LIBSCI_VERSION 13.2.0 

setenv LIBSCI_BASE_DIR /opt/cray/libsci/13.2.0 

prepend-path PE_PRODUCT_LIST CRAY_LIBSCI 

prepend-path CRAY_LD_LIBRARY_PATH /opt/cray/libsci/13.2.0/GNU/5.1/x86_64/lib 

prepend-path MANPATH /opt/cray/libsci/13.2.0/man:/opt/cray/man/csmlversion 

setenv PE_LIBSCI_MODULE_NAME cray-libsci/13.2.0 

setenv PE_LIBSCI_VOLATILE_PRGENV CRAY GNU INTEL 

setenv PE_LIBSCI_GENCOMPILERS_CRAY_x86_64 8.3 

setenv PE_LIBSCI_GENCOMPS_CRAY_x86_64 83 

setenv PE_LIBSCI_GENCOMPILERS_GNU_x86_64 5.1 4.9 

setenv PE_LIBSCI_GENCOMPS_GNU_x86_64 51 49 

setenv PE_LIBSCI_GENCOMPILERS_INTEL_x86_64 14.0 

setenv PE_LIBSCI_GENCOMPS_INTEL_x86_64 140 

prepend-path PE_PKGCONFIG_PRODUCTS PE_LIBSCI 

setenv PE_LIBSCI_VOLATILE_PKGCONFIG_PATH /opt/cray/libsci/13.2.0/@PRGENV@/@PE_LIBSCI_GENCOMPS@/@PE_LIBSCI_TARGET@/lib/pkgconfig 

prepend-path PE_LIBSCI_REQUIRED_PRODUCTS PE_MPICH 

setenv PE_LIBSCI_PKGCONFIG_VARIABLES PE_LIBSCI_OMP_REQUIRES_@openmp@ 

setenv PE_LIBSCI_OMP_REQUIRES  

setenv PE_LIBSCI_OMP_REQUIRES_openmp _mp 

prepend-path PE_PKGCONFIG_LIBS libsci_mpi:libsci 

setenv PE_LIBSCI_PKGCONFIG_LIBS libsci_mpi:libsci 

module-whatis Cray Scientific Libraries 

-------------------------------------------------------------------

[Andrew Turner]

I should also say that I am able to replicate the module order produced by Spack by hand with:

module swap PrgEnv-cray PrgEnv-gnu

module remove craype-ivybridge

module remove gcc

module load gcc

module load craype-ivybridge

module load fftw

but I still cannot replicate the error I see when compiling using Spack as VASP compiles fine if I do this. If I could find out the exact module commands Spack is using and when then I could investigate further but I am struggling to identify what it is doing.

It would be useful if the "-v" option to Spack showed what modulecmd commands are being issued to help debug this.

Cheers

Andy

[Robert French]

Yes, sorry, I was not clear on that. If you do "module display cray-libsci" with PrgEnv-gnu loaded, modulecmd will re-evaluate cray-libsci as though it were getting loaded after PrgEnc-gnu which is, I think, what we want Spack to do.

It would be interesting to see what the value of CRAY_LIBSCI_PREFIX_DIR is from the context of your Vasp#install method.

If you have a minute to mess with it (I'm AFCray right now), could you put something like this at the top of the install method?

sh("echo","$CRAY_LIBSCI_PREFIX_DIR")

I would bet all my flops that it doesn't point to the gnu version of libsci, and if that's true, it is a good indication that Spack is loading the Programming Environments too late.

-Robert

[Andrew Turner]

Hi Robert,

The command you listed just threw an error as "sh" was not defined. I got the value instead using os.environ["CRAY_LIBSCI_PREFIX_DIR"] and checked there isn't some sort of subshell issue with:

env = which('env')

env()

Both methods give the correct GNU prefix:

/opt/cray/libsci/13.2.0/GNU/5.1/x86_64

so it does not look like it is this.

Actually, I have diff'd the full env from the by-hand (working method) and the Spack version and they are exactly the same apart from PATH which has the extra paths to Spack stuff so it seems there is something more subtle going on here. I wonder if the "make" method itself is doing something with the environment? I will have a dig.

Thanks for the help

Andy

[Mario Melara Jr.]

Try with a spack -d install, it outputs the order of the modules being loaded and unloaded. 

[Andrew Turner]

Mario,

Thanks, that is exactly what I was looking for. Matching the loads and unloads by hand still does not replicate the error so I will have to look at the env settings from within the 'make' method call itself. I will let you know how I get on.

Cheers

Andy

[Andrew Turner]

Hi All,

OK, so I think I have got to the bottom of this though I need some input on how to fix it.

I was puzzled that compiling C code worked fine (e.g. Gromacs) but Fortran failed with the error described above and have found that it is due to the sequence of calls of various compiler wrappers.

For C:

The Spack C compiler wrapper 'cc' happens to have the same name as the Cray C compiler wrapper (is this intentional?) so the call sequence looks like:

1. Actual command: cc -o flie.o file.c

2. Spack compiler wrapper is first in path so (spack)cc -o file.o file.c

3. Spack compiler wrapper recognises "cc" as a C compiler

3. Spack compiler wrapper removes itself form path so (Cray)cc -o file.o file.c

and this all works fine as far as I can tell.

For Fortran:

There is no link from 'ftn' (the Cray Fortran wrapper command) to Spack 'cc' so the call sequence looks like:

1. Actual command: ftn -o flie.o file.f90

2. Cray compiler wrapper is first in path so (Cray)ftn -o file.o file.f90

3. Cray compiler wrapper calls 'gfortran' so gfortran -o file.o file.f90 + loads of options added by (Cray)ftn

4. Spack link gfortran -> cc first in path so (Spack)cc -o file.o file.f90 + loads of options added by (Cray)ftn

5. Spack compiler wrapper recognises "gfortran" as a Fortran compiler

6. Spack compiler wrapper removes itself form path so (Cray)gfortran -o file.o file.c + loads of options added by (Cray)ftn

Somewhere in the second process the environment gets corrupted so that Fortran stuff cannot compile.

So, testing my theory:

cd lib/spack/env

ln -s cc ftn

and I modified "cc" to add "ftn" to the list of recognised Fortran compilers. Everything now works as expected. 

Outstanding things:

1. Why does the env get mucked up in the first place - I cannot see why the original calling sequence for Fortran would not work? Do we care now we have a workaround?

2. Did C just work by chance as the Spack wrapper and Cray wrapper have the same name or was this by design?

3. I imagine the same is going to happen for C++ (the Cray wrapper script is called "CC") so this needs to be added in the same way.

Thanks for all you help in tracking this down. As the changes are minor, it is probably best that Todd just makes them rather than go through all the PR stuff.

Ta
Andy

[Robert French]

So on Titan (and I assume everywhere else with a modern CrayPE) there is just one "wrapper" program at /opt/cray/craype/<version>/bin/driver. The CC, cc, and ftn programs in userspace are just symlinks, so I reckon that the wrapper looks at $0 to figure out what compiler it should behave as. Here is some stuff to back up what Andy is saying:

```
frenchrd@titan-ext5:~> ftn -craype-verbose
pgf90 -tp=bulldozer -Bstatic -D__CRAYXE -D__CRAY_INTERLAGOS blah blah blah...
frenchrd@titan-ext5:~> ln -s `which ftn` cc # make a local cc link in my home dir
frenchrd@titan-ext5:~> ./cc -craype-verbose
pgcc -tp=bulldozer -Bstatic -D__CRAYXE -D__CRAY_INTERLAGOS blah blah blah...
frenchrd@titan-ext5:~> ls -AhlF `which ftn`
lrwxrwxrwx 1 bin bin 6 Oct  6  2015 /opt/cray/craype/2.4.2/bin/ftn -> driver*
frenchrd@titan-ext5:~>
```

So if I invoke "ftn" via a "cc" symlink in my homedir, argv[0] will be set to "cc" and it will work as a c compiler. However, there is no reasonable fallback when the executable name is set to something that the Cray driver doesn't expect:

```
frenchrd@titan-ext5:~> ln -s `which ftn` f77
frenchrd@titan-ext5:~> ./f77 -craype-verbose
Use either: 'cc', 'CC', or 'ftn'.
```

Is it okay to pollute lib/spack/env with this one Cray-specific link like Andy is suggesting?

Robert D. French

--------------------------

Web: http://www.robertdfrench.me

Cell: (615) 973-4002

Email: rob...@robertdfrench.me