Will Spack choose already-installed dependencies with non-default options?

[Chris Turner]

Hello,

I just started looking to potentially shift our HPC environment to Spack.  Setting this up in a test environment, I have noticed that it seems Spack will choose to install dependencies with default options rather than one that is already installed with non-default options.  Take OpenMPI an example.  I have installed OpenMPI as such:

spack install openmpi%gcc fabrics=verbs schedulers=slurm +pmi +legacylaunchers

However, if i then try to install, say, mpileaks (spack install mpileaks%gcc), it will try to install OpenMPI with default options (fabrics=none, schedulers=none, etc.).  It seems like it would make more sense for it to choose the version of OpenMPI I have installed.  I know I can force it to choose my version with the hash as such:

spack install mpileaks%gcc ^openmpi/td73nqj

However, this would seem to get burdensome for packages that have multiple dependencies with options (i.e. having to look up the hash for each one to make sure the one you've already installed gets used).

Am I understanding this correctly?  Or did I possibly miss something in a tutorial or in the documentation? Can this be controlled in a config?

Thanks,

Chris

[Ewan Roche]

Hi Chris,
our way of making this kind of thing less painful is by using environments and some definitions and lists to be used in spec matrices:

definitions:

- mygcc: [ g...@8.3.0 ]
- mybasegcc: [ g...@4.8.5 ]
- mygccmpi: [ mpich+slurm+verbs device=ch4 netmod=ofi pmi=pmi2 ]
- mypython3: [ pyt...@3.7.6 ]
- mypython2: [ pyt...@2.7.16 ]

- rootpackages:
- cm...@3.16.2
- g...@2.25.0
- basepackages:
- angsd@20191105
- bedt...@2.27.1
- blast...@2.9.0
..
..
- mpipackages:
- openfoam@1912
- elmerfem@8.4+mumps+mpi+openmp
- ff...@3.3.8
- gromacs@2020
..
..

specs:

- matrix:
- [$mygcc]
- [$%mybasegcc]

# BLAS and MPI for GCC
- matrix:
- [$gccblasmpi]
- [$%mygcc]

# Packages that don't use MPI
- matrix:
- [$basepackages]
- [$%mygcc]

# MPI Packages
- matrix:
- [$mpipackages]
- [$^mygccmpi]
- [$%mygcc]

# Python packages
- matrix:
- [$pypackages]
- [$^mypython3]
- [$%mygcc]
..
etc


If you’re interested our site specific configuration can be seen at https://c4science.ch/source/dcsr-spack-site

It may well be possible to set a preference for a particular MPI flavour but at least this way we are sure what is going to be installed.

Thanks

Ewan Roche

Division Calcul et Support à la Recherche
UNIL | Université de Lausanne

> --
> You received this message because you are subscribed to the Google Groups "Spack" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/spack/05547809-dccf-4922-972b-f6d5faf0aa61%40googlegroups.com.

[Chris Turner]

Hi Ewan,

Thanks for the info. I'll go take a look at your config.

I was able to partially answer my question, but have a follow up. I can see that using packages.yaml, we can specify variants for packages anytime that package is used--essentially changing the defaults. (https://spack-tutorial.readthedocs.io/en/latest/tutorial_configuration.html#variant-preferences) However, sticking with my original example, OpenMPI has a number of dependencies any of which can get updated at any given time.  Since Spack picks the latest versions, this would necessitate Spack creating an entirely new DAG/hash if, when concretizing, it sees newer versions of one or more dependencies.  I think this could create some instability in the software stack.  Is it possible to specify a particular hash in package.yaml thereby "freezing" the DAG anytime I pick OpenMPI%compiler?

Ewan, is your setup immune to this type of issue? If so, how?

Thanks,

Chris

> To unsubscribe from this group and stop receiving emails from it, send an email to sp...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/spack/05547809-dccf-4922-972b-f6d5faf0aa61%40googlegroups.com.

> To unsubscribe from this group and stop receiving emails from it, send an email to sp...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/spack/05547809-dccf-4922-972b-f6d5faf0aa61%40googlegroups.com.

> To unsubscribe from this group and stop receiving emails from it, send an email to sp...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/spack/05547809-dccf-4922-972b-f6d5faf0aa61%40googlegroups.com.

> To unsubscribe from this group and stop receiving emails from it, send an email to sp...@googlegroups.com.

[Ewan Roche]

Hi,
It’s pretty immune because all the versions can be fixed - I’ve just noticed that I don’t actually fix the version of mpich so will change that (and have done below)!

The definitions fix the exact version and option of the base components:

- mygcc: [ g...@8.3.0 ]
- mygccmpi: [ mp...@3.3.2+slurm+verbs device=ch4 netmod=ofi pmi=pmi2 ]

The matrix

- matrix:
- [$mpipackages]
- [$^mygccmpi]
- [$%mygcc]

the expands the cross product of the definitions so it’s a convenient way of saying

packageA%g...@8.3.0 ^mp...@3.3.2+slurm+verbs device=ch4 netmod=ofi pmi=pmi2
packageB%g...@8.3.0 ^mp...@3.3.2+slurm+verbs device=ch4 netmod=ofi pmi=pmi2
etc

In effect this freezes the DAG without having to resort of hashes as Spack in intelligent enough to know that things haven’t changed.

If I want to install a new package I add it to the list definition $mpipackages then run spack install inside the environment.

Thanks

Ewan Roche

Division Calcul et Support à la Recherche
UNIL | Université de Lausanne

> To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/spack/9ca86b66-0395-4009-a5c8-9ed0db0c787d%40googlegroups.com.

[Erik Schnetter]

On Tue, Mar 31, 2020 at 12:32 PM Chris Turner <chtur...@gmail.com> wrote:
>
> Hi Ewan,
>
> Thanks for the info. I'll go take a look at your config.
>
> I was able to partially answer my question, but have a follow up. I can see that using packages.yaml, we can specify variants for packages anytime that package is used--essentially changing the defaults. (https://spack-tutorial.readthedocs.io/en/latest/tutorial_configuration.html#variant-preferences) However, sticking with my original example, OpenMPI has a number of dependencies any of which can get updated at any given time. Since Spack picks the latest versions, this would necessitate Spack creating an entirely new DAG/hash if, when concretizing, it sees newer versions of one or more dependencies. I think this could create some instability in the software stack. Is it possible to specify a particular hash in package.yaml thereby "freezing" the DAG anytime I pick OpenMPI%compiler?
>
> Ewan, is your setup immune to this type of issue? If so, how?

Spack specifies software setups declaratively. Similar to containers,
this means that it's not easily possible to update one piece and
expect the remainder to remain self-consistent. The usual workflow is
to rebuild the whole stack when something needs to be updated. The
beauty of Spack is that this is relatively risk-free and pain-free.

You can specify (lock down) individual package versions as described
by Ewan Roche, and thus reduce the work that needs to be done during a
global rebuild. However, if you then rebuild only individual packages,
you risk inconsistencies the way to describe – I always rebuild the
full stack.

-erik

--
Erik Schnetter <schn...@gmail.com>
http://www.perimeterinstitute.ca/personal/eschnetter/

[Chris Turner]

I wanted to update this thread with more information.  

First, thanks to Erik and Ewan for the helpful comments.  Ewan, I have been following your recipe, more or less, to set up a test environment.  Thank you for sharing that!

I believe Todd Gamblin is working on new concretizer that will address the issue I described.  Much discussion can be found in this issue: https://github.com/spack/spack/issues/311. 

Chris