Module files generation

[mart...@gmail.com]

Hi everyone,

I am having some struggle with Spack generating module files in the way we want, which is  using Lmod compiler-MPI hierarchy, as "CPU architecture / compiler / compiler version / MPI / MPI version" .

In Spack's modules.yaml, I have:

  lmod:
    hash_length: 0
    hierarchy:
      - mpi

    projections:
      all: '{architecture}/{name}/{version}'

which creates the Core modules as "Core/linux-centos7-nehalem/". This is OK though I'd prefer "linux-centos7-nehalem/Core/".

Similarly with the Compiler modules (=packages with compiler dependency), I get e.g. "intel/19.0.5.281/linux-centos7-nehalem/", where I'd prefer "linux-centos7-nehalem/Compiler/intel/19.0.5.281". I presume we could create a projection for the compilers as well, but, I am not seeing an example of that anywhere in the docs. I tried projection:

compiler: '{architecture}/{compiler.name}/{compiler.version}/{name}/{version}'

but it's ignored.

Finally, with the MPI (= packages dependent on MPI and a compiler), the Spack's default order is such where one can't have the same MPI version for different compilers by just using the MPI's version number, since the directory hierarchy is "MPI / MPI version / compiler / compiler version". Thus e.g. having intel-mpi 19.0.8 with both Intel 19.0.5 and 2021.1 requires e.g. the hash in the MPI version to differentiate those two MPI versions:

$ ls intel-mpi/
2019.8.254-kvtpiwf  2019.8.254-l3no3nl

(the first one for the compiler version 19.0.5 and the other for 2021.1).

What we'd want is e.g. "intel/19.0.5.281/intel-mpi/2019.8.254" and "intel/2021.1/intel-mpi/2019.8.254", which does not require to have the hash in the module name.

I attempted that with adding another projection into modules.yaml:

      ^mpi: 'MPI/{architecture]/{compiler.name}/{compiler.version}/{^mpi.name}/{^mpi.version}/{name}/{version}'

That does the hierarchy I want, but, this hierarchy gets appended to the Spack generated hierarchy, e.g. "intel-mpi/2019.8.254-kvtpiwf/intel/19.0.5.281/MPI/linux-centos7-nehalem/intel/19.0.5.281/intel-mpi/2019.8.254". What I'd like is to remove the "intel-mpi/2019.8.254-kvtpiwf/intel/19.0.5.281/" and keep just the "MPI/linux-centos7-nehalem/intel/19.0.5.281/intel-mpi/2019.8.254".

So, bottom line, I think what would help us the most is if we could remove all the paths that Spack puts in by default (e.g. the "intel-mpi/2019.8.254-kvtpiwf/intel/19.0.5.281") for the MPI, and specify our own full modules hierarchy paths with the projections.

I did try to remove the Lmod mpi hierarchy from the modules.yaml, but at no avail.

I appreciate any thoughts on that.

MC

PS: The "spack module refresh" command is awesome for quick testing of this. Thanks a lot for that.

[mart...@gmail.com]

Answering my own question.

The hierarchy and projections are to certain extent exclusive. If we use the "mpi" hierarchy, Spack builds the mpi.name/mpi.version/compiler.name/compiler.version hierarchy path, and to it appends whatever comes from the projection.

I hacked a few lines into the lib/spack/spack/modules/lmod.py to fix this for us. More details are at  https://github.com/CHPC-UofU/spack-config#chpc-lmod-integration.

I suppose I could have not used the "mpi" hierarchy and just use the projection, but, then I can't define a "compiler" projection, that I'd need for the compiler based hierarchy. The hack above is short and does what we need so I am OK with that for now.

I think it'd be more beneficial, in future Spack releases, to build the MPI hierarchy as compiler.name/compiler.version/mpi.name/mpi.version, since that's unique, as opposed to the default now. Also, if one uses the ^mpi projection, don't also add the spack built hierarchy_name into the module file path.

Thanks,

MC