Propagated variants

[Florent Pruvost]

Hi,

I have a question relative to variants than could be propagated.

Let's say I have a package A depending on B and both have an MPI variant, OFF by default.

If I activate the MPI variant at the level of package A, e.g

$ spack install A+mpi

Spack will install A with MPI but B without because the MPI variant is OFF by default and because I have not added +mpi to B explicitly like this

$ spack install A+mpi ^B+mpi

It could be nice to have a way to propagate the MPI activation to B automatically (expressing this possibility in package B)

Can we imagine a system of propagated variants or is it already possible? 

Thanks!

Florent

[Todd Gamblin]

Hi Florent,

I guess I didn't respond to the list before.  Adding the google group so that people can read this.

Your point below about combinatoric dependencies is a good one.   For dependencies like mpi and cuda, it would be useful to "push" variants down the tree.  Currently the way to do this is:

class A(Package):

variant('mpi', default=True, description='build with MPI')

depends_on('mpi', when='+mpi')

depends_on('B+mpi', when='+mpi')

depends_on('B~mpi', when='~mpi')

And as you were saying, that quickly turns into:

depends_on('B+mpi', when='+mpi')
depends_on('B+cuda', when='+cuda')
depends_on('B+opencl', when='+opencl')
depends_on('B+mpi+cuda', when='+mpi+cuda')
depends_on('B+cuda+opencl', when='+cuda+opencl')
depends_on('B+mpi+opencl', when='+mpi+opencl')
depends_on('B+mpi+cuda+opencl', when='+mpi+cuda+opencl')

Yuck.  It seems like you want a specification more like this:

class A(Package):

variant('mpi', default=True, description='Build with MPI')
variant('cuda', default=True, description='Build with CUDA')

variant('opencl', default=False, description='Build with OpenCL')

depends_on('mpi', when='+mpi')

depends_on('B', match_variants=['mpi', 'cuda', 'opencl'])

That would tell the concretizer to make sure the mpi, cuda, and opencl variants are the same in B as they are in A, and then B can do what it wants with them.  

You asked about doing this at the level of B.  Right now there's nothing in a package that fetches spec properties from dependents.  However, there *is* support for this in the concretization process.  Right now if you build A and A depends on B, and B doesn't specify a compiler, then B will inherit A's compiler.  That's why this ends up building you a consistent intel stack:

spack install mpileaks %intel

All of mpileaks' dependencies inherit its compilers if nothing in the packages specifies a requirement for a particular compiler.  The tricky part of this is that if you start having too many things that look up and down the DAG, then Spack starts needing a more sophisticated constraint solver.  Either that or you settle for occasional bad greedy decisions, like Spack has now.

I am not sure I need to tackle that problem quite yet.  I'm assuming that you want to look up the tree for dependent packages with +mpi in order to enable its own +mpi.  The case above handles that if A depends directly on B, but it wouldn't handle a case where A -> B -> C and C supports +mpi but B does not.  Maybe this would be better:

class A(Package):

variant('mpi', default=True, description='Build with MPI')
variant('cuda', default=True, description='Build with CUDA')

variant('opencl', default=False, description='Build with OpenCL', conflicts='cuda')

depends_on('mpi', when='+mpi')

depends_on('B')

match_dependency_variants('mpi', 'cuda', 'opencl')

Where match_dependency_variants adds a rule during concretization that would cause *any* dependencies (direct or indirect) that support those particular variants to match their parent's settings (or fail with an unsatisfiable constraint error).  Would that work?

The cases of MPI, CUDA, and OpenCL are interesting because they're actually cross-cutting concerns to a lot of packages.  It might be a good idea to have some default concretization rules for Spack that cause particular variants to be matched among packages in a DAG.  That would mean that package authors can't redefine the variants to suit their own needs (e.g. If I want +cuda to mean something other than "build with cuda"), but I think there's value in making these particular variants consistent across the DAG.

Finally, another thing that might be useful would be a way for a package to say that particular variants conflict with others, e.g.:

variant('cuda', default=True, description='Build with CUDA', conflicts='opencl')

variant('opencl', default=False, description='Build with OpenCL', conflicts='cuda')

Or something to say that particular variants are mutually exclusive:

mutually_exclusive('cuds', 'opencl')

The latter would scale better.

What do you think?

-Todd

From: Florent Pruvost <florent...@inria.fr>
Date: Tuesday, June 16, 2015 at 2:32 AM
To: Todd Gamblin <gamb...@llnl.gov>
Subject: Re: [spack] Propagated variants

Ok,

so If I understand well, I have to express all combinations in package A.
what about several options?
In A:
depends_on('B+mpi', when='+mpi')
depends_on('B+cuda', when='+cuda')
depends_on('B+opencl', when='+opencl')
depends_on('B+mpi+cuda', when='+mpi+cuda')
depends_on('B+cuda+opencl', when='+cuda+opencl')
depends_on('B+mpi+opencl', when='+mpi+opencl')
depends_on('B+mpi+cuda+opencl', when='+mpi+cuda+opencl')

Number of combinations grows fast!
What about an expression at the level of B instead, like activation of MPI in B if activated at the upper level in A?

Thanks,
Florent

Le 16/06/2015 09:30, Gamblin, Todd a écrit :

In A:
depends_on('B+mpi', when='+mpi')
depends_on('B~mpi', when='~mpi')

Should work.



Sent with Good (www.good.com)

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at http://groups.google.com/group/spack.
For more options, visit https://groups.google.com/d/optout.

[Florent Pruvost]

Hi Todd,

Your suggestions are very interesting.

I think both features:

1. depends_on('B', match_variants=['mpi', 'cuda', 'opencl']) for a specific propagation of level 1 and not for all dependencies, and
   
2. match_dependency_variants('mpi', 'cuda', 'opencl') for a propagation to all dependencies would be useful. 

Sometimes we need to propagate the variant for all the stack and sometimes not.

I'm not sure that "to have some default concretization rules for Spack that cause particular variants to be matched among packages in a DAG." is a good idea.
It may represent a lot of effort for you to guess what are the best default behaviors and not sure they will be widely used... Lost of time?
Let's the packager expresses the behaviors he wants for its stack (what should be propagated to direct deps. and what to all the stack). If the features exist it is already a very good point :)

The conflict feature would be nice also yes. It may be useful to have the feature for any number of conflicting dependencies?
    mutually_exclusive('dep1', 'dep2', 'dep3')
means just one of the list can be enabled.

Thanks,
Florent

[Todd Gamblin]

Florent,

From: Florent Pruvost <florent...@inria.fr>
Date: Wednesday, June 17, 2015 at 2:44 AM
To: Todd Gamblin <tgam...@llnl.gov>
Cc: "sp...@googlegroups.com" <sp...@googlegroups.com>
Subject: Re: [spack] Propagated variants

Hi Todd,

Your suggestions are very interesting.

I think both features:

1. depends_on('B', match_variants=['mpi', 'cuda', 'opencl']) for a specific propagation of level 1 and not for all dependencies, and
   
2. match_dependency_variants('mpi', 'cuda', 'opencl') for a propagation to all dependencies would be useful. 

Sometimes we need to propagate the variant for all the stack and sometimes not.

That makes sense.  I can look into how easy this would be to add.  I think it's definitely needed, but it may not be something that makes it into 0.9.  Something like this to address the combinatoric issue should definitely be in 1.0.

I'm not sure that "to have some default concretization rules for Spack that cause particular variants to be matched among packages in a DAG." is a good idea.

It may represent a lot of effort for you to guess what are the best default behaviors and not sure they will be widely used... Lost of time?
Let's the packager expresses the behaviors he wants for its stack (what should be propagated to direct deps. and what to all the stack). If the features exist it is already a very good point :)

Fair enough.  FWIW, with the preferences feature we're implementing:

https://github.com/scalability-llnl/spack/pull/47

It should be possible to address some preferences like this during concretization.  I don't believe Matt has pushed the docs for those features yet, but they allow you to choose how virtual dependencies are chosen, as well as how versions are chosen during concretization.  Adding something for variants actually isn't that much more work.  So there may be something like this in there for v0.9.  I still need to integrate that PR (sorry Matt!), so I will look at the variant propagations stuff when I do.

The conflict feature would be nice also yes. It may be useful to have the feature for any number of conflicting dependencies?

    mutually_exclusive('dep1', 'dep2', 'dep3')
means just one of the list can be enabled.

Yep -- this may be more involved to implement, in that once you add conflicts, you really need some form of backtracking in the constraint resolution process.  I don't currently have that.  This will likely go in sometime after v0.9, maybe for 1.0.

-Todd

[Mark Miller]

Let me see if I can understand this. We have two packages, A and B. A depends on B. Both packages have useful mpi and non-mpi variants. Does package A's mpi-ability depend on package B also having its mpi-abilities enabled? It might, it might not. In other words, A+mpi might depend on B or it might depend on B+mpi or it might not matter in that either B or B+mpi will do. If A+mpi depends on B+mpi, then I think 'spack install A+mpi' should install B+mpi too. In other words, the variant gets propagated even though the default for B is to NOT build with mpi. If it depends on B or it doesn't matter, then there is no need for spack to propagate the variant and building B its default way is fine.

Is there a way of differentiating dependencies for A+mpi from those for A alone? Maybe A alone has no dependence on B. But A+mpi depends on B/B+mpi.

Maybe what we need is a way of saying, when creating this variant, these are the dependences and when creating that variant, these are the dependencies like this…

for variant(+mpi):

    depends_on(B+mpi)

    depends_on( C)

for variant():

    depends_on(D)

I have a similar situation. I've worked on the hdf5 package and added a +fortran variant. Its on by default. I installed hdf5~fortran on my OS X system because I didn't have fortran compilers handy. The netcdf package depends_on('hdf5'). When I went to install netcdf, 'hdf5~fortran' didn't satisfy netcdf's depends_on('hdf5'). So, spack went to install the default HDF5 which requires fortran compilers and it failed.

It smells to me like Spack is being frustratingly strict about its dependencies. If I say package A depends on package B and I don't specify any variants of B, then I think that ought to mean that *any* available variants of B that can be found to satisfy the depends_on should be fine. And, when there is a choice, spack should choose the variant that will lead to the fewest number of additional dependencies.

So, in my case, suppose there was an HDF5 with zlib and another without zlib. I didn't explicitly request zlib in my spack command. The netcdf package.py doesn't either. So, spack should be allowed to choose which. I think it should pick the HDF5 without zlib to resolve any depends_on('hdf5'). But, maybe thats some kind of a global preference one can set in the way spack should operate in any given installation.

Doesn't this same problem happen with virtual dependencies. Consider blas. If I have a package that depends_on('blas') and there are multiple implementations of blas avilable, how will spack resolve that? Does there need to be an explicit request for a specific implementation of blas? Can there be a preferred implementation if none is specified? Can a given package

Hope I am not adding to confusion.

[Todd Gamblin]

From: <sp...@googlegroups.com> on behalf of Mark Miller <markcm...@gmail.com>
Date: Monday, October 12, 2015 at 8:25 PM
To: Spack <sp...@googlegroups.com>
Subject: [spack] Re: Propagated variants

Let me see if I can understand this. We have two packages, A and B. A depends on B. Both packages have useful mpi and non-mpi variants. Does package A's mpi-ability depend on package B also having its mpi-abilities enabled? It might, it might not. In other words, A+mpi might depend on B or it might depend on B+mpi or it might not matter in that either B or B+mpi will do. If A+mpi depends on B+mpi, then I think 'spack install A+mpi' should install B+mpi too. In other words, the variant gets propagated even though the default for B is to NOT build with mpi. If it depends on B or it doesn't matter, then there is no need for spack to propagate the variant and building B its default way is fine.

Yep this works ok for one variant.  As Florent pointed out earlier, it isn't so great for multiple variants -- the specification gets combinatorial...

Is there a way of differentiating dependencies for A+mpi from those for A alone? Maybe A alone has no dependence on B. But A+mpi depends on B/B+mpi.

You can do:

depends_on('A+mpi', when='+mpi')

depends_on('A~mpi', when='~mpi')

This is not particularly satisfying because you really want to just say "always propagate the mpi value to the packages that support it".  We need some syntax to allow that.  Gentoo has some good ways to do this with its sort of unintuitively named "use flags" that I'm thinking I'll adopt (at least semantically if not their syntax).

Maybe what we need is a way of saying, when creating this variant, these are the dependences and when creating that variant, these are the dependencies like this…

for variant(+mpi):

    depends_on(B+mpi)

    depends_on( C)

for variant():

    depends_on(D)

I think in general there needs to be a way of grouping conditional logic for dependencies.  Right now all you've got is the 'when' clause in each depends_on() but that can get pretty redundant.  I need to think about how to work this stuff into a pure python dsl.  Overloading 'for' is kind of an interesting idea, although the above is not pure python and couldn't be in an embedded DSL like Spack.  We could do something like this with the "with" statement, though.  I could also see about implementing some functions that could be called as part of if/else that would allow nested conditions in a non-painful way.

I have a similar situation. I've worked on the hdf5 package and added a +fortran variant. Its on by default. I installed hdf5~fortran on my OS X system because I didn't have fortran compilers handy. The netcdf package depends_on('hdf5'). When I went to install netcdf, 'hdf5~fortran' didn't satisfy netcdf's depends_on('hdf5'). So, spack went to install the default HDF5 which requires fortran compilers and it failed.

This one would require us to do slightly smarter things in concretization to better reuse existing installs.  It doesn't actually require full on variant propagation, though -- we just have to be more liberal about finding existing packages that satisfy dependencies in specs.  

It smells to me like Spack is being frustratingly strict about its dependencies.

Nope -- it's just being stupid here :).  In these instances, it concretizes without looking at what is installed, THEN tries to find a match, instead of the other way around.  Adding a little search to do this wouldn't be too hard.

If I say package A depends on package B and I don't specify any variants of B, then I think that ought to mean that *any* available variants of B that can be found to satisfy the depends_on should be fine. And, when there is a choice, spack should choose the variant that will lead to the fewest number of additional dependencies.

Ok, so fewest deps is an interesting and potentially useful criterion.  However, I was thinking of taking this one step further.  Once we actually have vetted software stacks (I.e. sets of packages, or "stacks" that are known to work well), we could prefer packages that come from the stable build list over ones that don't.  I was thinking that would be a better way to pick a default.

So, in my case, suppose there was an HDF5 with zlib and another without zlib. I didn't explicitly request zlib in my spack command. The netcdf package.py doesn't either. So, spack should be allowed to choose which. I think it should pick the HDF5 without zlib to resolve any depends_on('hdf5'). But, maybe thats some kind of a global preference one can set in the way spack should operate in any given installation.

Doesn't this same problem happen with virtual dependencies. Consider blas. If I have a package that depends_on('blas') and there are multiple implementations of blas avilable, how will spack resolve that? Does there need to be an explicit request for a specific implementation of blas?

No -- see the paper: http://tgamblin.github.io/pubs/spack-sc15.pdf

That gets resolved by choosing an implementation for the virtual dep from the available ones.  Spack currently isn't too smart about this -- it doesn't have a preference about which one it picks -- there is a PR that provides that ability though.

Can there be a preferred implementation if none is specified? 

This is in the PR I mentioned (one of Matt's).  He has config files that allow you to specify concretization preferences.

-Todd

Hope I am not adding to confusion.

On Tuesday, June 16, 2015 at 12:22:05 AM UTC-7, Florent Pruvost wrote:

Hi,

I have a question relative to variants than could be propagated.

Let's say I have a package A depending on B and both have an MPI variant, OFF by default.

If I activate the MPI variant at the level of package A, e.g

$ spack install A+mpi

Spack will install A with MPI but B without because the MPI variant is OFF by default and because I have not added +mpi to B explicitly like this

$ spack install A+mpi ^B+mpi

It could be nice to have a way to propagate the MPI activation to B automatically (expressing this possibility in package B)

Can we imagine a system of propagated variants or is it already possible? 

Thanks!

Florent