Parameter system revamp

[Dag Sverre Seljebotn]

I hope I can have your +1/-1 on the following ASAP, I'll start work on
Monday and any change of direction after then will be more difficult.

As part of my small Hashdist project I am thinking the first piece I
need to attack is the parameter system. Everything else builds on top of
that.

1)

I was thinking I'd replace the 'default:' section in spec YAML files
with something more like this:

parameters:
optlevel:
type: str
default: "-O0"
version:
type: version # in Python code this will have a Version class

or something like that; being a bit more explicit is easier to relate to
when coding, sets us up for SAT solving and so on. The rewrite of
package specs should hopefully be fairly mechanical though if you wish
I'll throw in backwards compatability.

2)

Every dependency will also be an implicit parameter, i.e., if you depend
on "numpy" then "numpy" is a parameter (of type "numpy" ... think of it
as numpy(version=1.2.3) being one instantiation of "numpy").

Then, you should be able to refer to parameters of dependencies,
${numpy.version} or something like that, I'll figure out the details as
I go.

3)

I was thinking that root-level parameters in profile specs are passed on
to packages only if they are declared in parameters. So "HOST_MPICC"
won't be available unless it is explicitly declared in the parameters:
dict. If it is declared it is passed on exactly like today. This is more
verbose but also doubles as nice documentation... agreed/disagreed?


4)

The {{build_with}} trick is effective but a bit crude, since it happens
before the YAML parse stage, so it could be a bit difficult to work
with. Today it is used like this in petcs.yaml:

dependencies:
build: [blas, mpi, python, {{build_with}}]

Any alternative that doesn't rely on the string expansion to turn into
YAML markup would be preferred (keep the code simpler at least). Ideas?

Here's my proposal:

dependencies:
build: [blas, mpi, python, parmetis?, scotch?, ... ]

i.e., any package. Then you can check if they are null/None. In the
profile spec it would work like this:


packages:
parmetis:
...
petsc:
# by default, "parmetis: parmetis" is passed in

However if you want a stack that does contain parmetis, but petsc
shouldn't use it, you can explicitly pass none:


packages:
parmetis:
...
petsc:
parmetis: none


If I can make this work I'll then rip out the ability to embed YAML
snippets in parameters.

Dag Sverre

[Dag Sverre Seljebotn]

Sorry. i.e., any package can have the '?' suffix to indicate it is an
optional dependency.

DS

[Dag Sverre Seljebotn]

On 05/14/2015 05:46 PM, Dag Sverre Seljebotn wrote:

Ehrm, and this should have been the YAML 'null', not none, sorry.

Dag Sverre

[Ondřej Čertík]

That all sounds ok to me.

One problem with dependencies is that currently we only depend on the
package by name. Spack, on the other hand, allows a package to depend
on other packages not only by name, but also by their specific
version, compiler, parameters (like debug and other parameters that we
currently use in Hashstack, perhaps applying or not applying some
patch, etc., that's package specific).

Ondrej

P.S. Dag, can you please make me an owner of the hashdist mailinglist?
So that I can set it to reply to the list by default.

[Dag Sverre Seljebotn]

Right, my thoughts there was to eventually supplement the above with a
list of constraints. So in a package you might have

constraints:
- 1.2.3 <= numpy.version <= 1.4.x
- not (blas.kind == 'ATLAS' and numpy.version == 1.2.4)

etc., so every line has to be true for the combination to be valid.
Sounds ok?

I guess you could throw in syntax candy, like

build:
- numpy(1.2.3<=version<=1.4.x)

which would mean the same but would be unable to express complex
dependencies between packages.

> P.S. Dag, can you please make me an owner of the hashdist mailinglist?
> So that I can set it to reply to the list by default.

Will do.


Dag Sverre

[Jimmy Tang]

+1 to all of the above.

[Chris Kees]

I'm +1 too. What kind of effort will it require if we don't worry about backward compatibility with the existing default: section? Is this just a one time operation on any stack repositories?

I like the ? syntax for optional dependencies and the parameters section.

I agree on #3. I suppose it  will also require an update to hashstack, but I don't think it should be too bad.

--
You received this message because you are subscribed to the Google Groups "hashdist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hashdist+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dag Sverre Seljebotn]

BTW, is the run vs build deps distinction a) useful and b) used properly in hashstack?

DS

[Chris Kees]

I think the distinction is useful, but we need to clarify the distinction and possibly check the current behavior on how these effect the profile, chaining of dependencies, etc. I'd like to support the case where X and Y are both in a profile but X needs  to build and link to a special version of Y that shouldn't be included in the profile.  I don't have a good grasp of how the run/build distinction handles this and similar cases.