Force git versioning of stack descriptions?

[Dag Sverre Seljebotn]

Ondrej started a discussion at
https://github.com/hashdist/hashdist/pull/337 which I'd like to open up
on the mailing list. Brief summary:

Ondrej's working on adding synonyms for profiles, so that you can do
`hit activate myprofile` (or whatever) and we have a global alias from
myprofile to the built profile.

Then the question is: Do we also add a history mechanism; `hit
rollback`, etc?

To me it feels a bit like duplicating git, and that all the features of
git (branching, etc. etc.) would be beneficial for managing stacks as well.

Some brainstorming:

We could have a special git repo at ~/.hashdist/stacks. This is *the*
Hashdist stack repository. You can use other repos and use `hit build`
as today, but if you want to do things like

hit use mybranch
hit install numpy
hit update pandas

then ~/.hashdist/stacks is what you work with. Then:

a) Every branch is one profile (first try $(BRANCH).yaml, then
default.yaml, in the repo)

b) Parsing of stack spec is done with clean checkout to ensure all
referenced files are committed in fresh version

c) Perhaps: As post-commit hook we make sure to rebuild the profile

d) Everywhere on your system, you can use the "porcelain" commands like:

hit use mybranch
use whatever profile is in 'mybranch' branch of
~/.hashdist/stacks repo

hit install numpy
check out the branch of the Hashdist profile in use in the
current shell's environment, modify profile YAML file,
make commit, build the profile, change branch back if we
were on another branch

etc.

So this is adding "global profiles" in an opiniated way, without
changing the existing flow at all, and I propose it as an alternative to
the kind of list of symlinks discussed in
https://github.com/hashdist/hashdist/pull/337.

Dag Sverre

[Dag Sverre Seljebotn]

BTW, I would think that ~/.hashdist/stacks would be a copy of the
"profile-template" repo, i.e. you pull in hashstack by extending the
hashstack profile. Or at least pull in the hashstack package set through
git.

Hashstack power users could just push into it a copy of hashstack itself
though, and keep rebasing their profile branches on top of the official
hashstack..

Dag Sverre

[Ondřej Čertík]

On Thu, May 21, 2015 at 12:42 AM, Dag Sverre Seljebotn
<d.s.se...@astro.uio.no> wrote:
> Ondrej started a discussion at https://github.com/hashdist/hashdist/pull/337
> which I'd like to open up on the mailing list. Brief summary:
>
> Ondrej's working on adding synonyms for profiles, so that you can do `hit
> activate myprofile` (or whatever) and we have a global alias from myprofile
> to the built profile.

It's `hit load myprofile`.

>
> Then the question is: Do we also add a history mechanism; `hit rollback`,
> etc?

For sure ---- it's already there, the old profiles stay, as hashes. We
just need to provide nice tools for the user to use them.
One of them is to print old hashes that were once called "myprofile",
so that you can revert to it.

> To me it feels a bit like duplicating git, and that all the features of git
> (branching, etc. etc.) would be beneficial for managing stacks as well.

The git analogy is not perfect ---- the similarity is that the
myprofile is like a branch, and a profile hash is like a commit. But
the difference is that git commits depend on each other, in
succession, while profile hashes are completely independent. You can
order them by the time they were created, but that's not how git works
--- you can create the last commit first, and first commit last, and
rebase on top. The time of commits will show this, i.e. that the
commits are not ordered by time (necessarily). So in our case, the
profile history is really just a set of hashes, with no intrinsic
ordering/dependencies.

I am not sure I follow. Essentially you are saying to put the
Hashstack repository inside ~/.hashdist, and let it be managed by
`hit`? And the profile yaml, let's say the default.yaml, will also be
managed by git, inside ~/.hashdist, and `hit` will provide some nice
tools on top?

So if I have 10 different profiles, those will be in 10 different
branches? A typical thing that I do is that I update one package, and
then rebuild all profiles. So I would need to rebase the 10 profile
branches first on top of the latest Hashstack to rebuild?

Ondrej

[Chris Kees]

On Thu, May 21, 2015 at 1:46 AM, Dag Sverre Seljebotn <d.s.se...@astro.uio.no> wrote:

On 05/21/2015 08:42 AM, Dag Sverre Seljebotn wrote:

Then the question is: Do we also add a history mechanism; `hit
rollback`, etc?

+1

hit use mybranch

+1

hit install numpy

+10

hit update pandas

+10
 

I' don't have time to think through the difference  in the .hashdist git repo vs symlinks issue this morning, but I think this  fits with the original vision.  I  find doing things  like the following get old fast:

cd hashstack

hit skeleton-pypi some_simple_pure_python_package

emacs examples/proteus.linux2.yaml #add new package

cp examples/proteus.linux2.yaml default.yaml

hit build default

Rolling back to previous profiles is also a bit clunky.

Hashstack power users could just push into it a copy of hashstack itself though, and keep rebasing their profile branches on top of the official hashstack..

Dag Sverre

--
You received this message because you are subscribed to the Google Groups "hashdist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hashdist+u...@googlegroups.com.
To post to this group, send email to hash...@googlegroups.com.
Visit this group at http://groups.google.com/group/hashdist.
To view this discussion on the web visit https://groups.google.com/d/msgid/hashdist/555D7F44.1060104%40astro.uio.no.

For more options, visit https://groups.google.com/d/optout.

[Ondřej Čertík]

On Thu, May 21, 2015 at 9:32 AM, Chris Kees <cek...@gmail.com> wrote:
>
>
> On Thu, May 21, 2015 at 1:46 AM, Dag Sverre Seljebotn
> <d.s.se...@astro.uio.no> wrote:
>>
>> On 05/21/2015 08:42 AM, Dag Sverre Seljebotn wrote:
>>>
>>>
>>> Then the question is: Do we also add a history mechanism; `hit
>>> rollback`, etc?
>>>
> +1
>>>
>>> hit use mybranch
>
> +1
>>>
>>> hit install numpy
>
> +10
>>>
>>> hit update pandas
>
> +10
>
> I' don't have time to think through the difference in the .hashdist git
> repo vs symlinks issue this morning, but I think this fits with the
> original vision. I find doing things like the following get old fast:
>
> cd hashstack
> hit skeleton-pypi some_simple_pure_python_package

My understanding is that you would still need to do this, i.e. create
a new yaml file for each package you want to install.

> emacs examples/proteus.linux2.yaml #add new package

'hit install numpy' would add it for you in the profile, is that right?

> https://groups.google.com/d/msgid/hashdist/CAOVFbFjPXrW5ohhD9MgD6hMhrO493ZxP%2B_vf42cd7bzU2Nmfxg%40mail.gmail.com.

[Chris Kees]

On Thu, May 21, 2015 at 10:36 AM, Ondřej Čertík <ondrej...@gmail.com> wrote:

On Thu, May 21, 2015 at 9:32 AM, Chris Kees <cek...@gmail.com> wrote:
>
>
> On Thu, May 21, 2015 at 1:46 AM, Dag Sverre Seljebotn
> <d.s.se...@astro.uio.no> wrote:
>>
>> On 05/21/2015 08:42 AM, Dag Sverre Seljebotn wrote:
>>>
>>>
>>> Then the question is: Do we also add a history mechanism; `hit
>>> rollback`, etc?
>>>
> +1
>>>
>>> hit use mybranch
>
> +1
>>>
>>> hit install numpy
>
> +10
>>>
>>> hit update pandas
>
> +10
>
> I' don't have time to think through the difference  in the .hashdist git
> repo vs symlinks issue this morning, but I think this  fits with the
> original vision.  I  find doing things  like the following get old fast:
>
> cd hashstack
> hit skeleton-pypi some_simple_pure_python_package

My understanding is that you would still need to do this, i.e. create
a new yaml file for each package you want to install.

I realize that initially `install`  wouldn't  do  that for you, but I think it  would be easy to add  once  we have the basic  ability to install existing packages into existing profiles and update existing package specs. If `install`  can't find  the package then it could offer  "Package  not found. Attempt to build package description from other source? [pypi,brew, conda] default=pypi?"
 

To view this discussion on the web visit https://groups.google.com/d/msgid/hashdist/CADDwiVC62CtMXrzxXCb-h6F-bRGJ1_LPbmFQ4APhUSETJgnGmg%40mail.gmail.com.

[Dag Sverre Seljebotn]

On 05/21/2015 05:23 PM, Ondřej Čertík wrote:
> On Thu, May 21, 2015 at 12:42 AM, Dag Sverre Seljebotn
> <d.s.se...@astro.uio.no> wrote:
>> Ondrej started a discussion at https://github.com/hashdist/hashdist/pull/337
>> which I'd like to open up on the mailing list. Brief summary:
>>
>> Ondrej's working on adding synonyms for profiles, so that you can do `hit
>> activate myprofile` (or whatever) and we have a global alias from myprofile
>> to the built profile.
>
> It's `hit load myprofile`.
>
>>
>> Then the question is: Do we also add a history mechanism; `hit rollback`,
>> etc?
>
> For sure ---- it's already there, the old profiles stay, as hashes. We
> just need to provide nice tools for the user to use them.
> One of them is to print old hashes that were once called "myprofile",
> so that you can revert to it.
>
>> To me it feels a bit like duplicating git, and that all the features of git
>> (branching, etc. etc.) would be beneficial for managing stacks as well.
>
> The git analogy is not perfect ---- the similarity is that the
> myprofile is like a branch, and a profile hash is like a commit. But
> the difference is that git commits depend on each other, in
> succession, while profile hashes are completely independent. You can

The hashes are independent, but the changes you do to profiles typically
build on one another, and don't underestimate the power of the metadata
(integrity hash, timestamp, author, log message), even if that is stuff
we could duplicate too...

If we get "hit install numpy", then "hit install scipy", I would really
like that to

a) be implemented as successive changes to a profile YAML file
b) each command I run should be a seperate commit (being a machine-based
change it needs to be well tracked, easily revertable etc.)

I guess that could also be satisfied by using git in a very restricted
mode (like, only one branch), instead. But I am rather convinced that
using git in one way or another should play a part, life is too short to
make our own history tracking system, and we don't have to use or
support all of git's features.

> order them by the time they were created, but that's not how git works
> --- you can create the last commit first, and first commit last, and
> rebase on top. The time of commits will show this, i.e. that the
> commits are not ordered by time (necessarily). So in our case, the
> profile history is really just a set of hashes, with no intrinsic
> ordering/dependencies.

The editing history of your profile YAML files *is* ordered in times and
git commits will typically match that. Fancy rebasing is not something
that happens by default. If you do, you of course can't complain
afterwards that "hit rollback" reset the last commit even if didn't have
newest timestamp; that is what you expect...

If you do "hit install numpy; hit uninstall numpy; hit install numpy",
there are 3 commits even if you end up with the original profile hash.

> I am not sure I follow. Essentially you are saying to put the
> Hashstack repository inside ~/.hashdist, and let it be managed by
> `hit`? And the profile yaml, let's say the default.yaml, will also be
> managed by git, inside ~/.hashdist, and `hit` will provide some nice
> tools on top?

By default, I would put a small repo that links to hashstack, like this one:

https://github.com/hashdist/profile-template

> So if I have 10 different profiles, those will be in 10 different
> branches? A typical thing that I do is that I update one package, and
> then rebuild all profiles. So I would need to rebase the 10 profile
> branches first on top of the latest Hashstack to rebuild?

Yes, it doesn't quite work, does it? I'm brainstorming :)

So what about (typically) having a single branch, and each profile is a
YAML file? So if I do

hit load prof1
hit install numpy # commit (1) changes prof1.yaml
hit load prof2
hit install pandas # commit (2) changes prof2.yaml

then there are 2 commits to the repo, one that changes prof1.yaml and
one that changes prof2.yaml.

"hit rollback" needs some thought here though. For the commits above it
could use "git log prof1.yaml" and do a "git revert", but it's not clear
what it should do, or whether it should raise an error, in your case
where you change package definition and wants to rebuild all the profiles.

I think for typical non-power-user cases, "prof1.yaml" contains the sha1
of a specific Hashstack commit. And if you want new package definitions,
that could be done with

hit load prof1
hit fetch-package-definitions # commit (3) changes prof1.yaml

which again lends itself nicely to a rollback git commit.

Anyway, I'm not sure just how important the rollback command is. The
important thing is there's an easy to understand history system that
tracks everything in a nice way.

So, how do you feel about one repo with one YAML file per "globally
available" profile? Files like "debian.yaml" should either be in other
repos included via a git hash, or in the same repo in a sub-directory
(so "debian.yaml" would be a profile named "debian", but just move it to
"support/debian.yaml" and it doesn't show up in the profile list).

Dag Sverre

[Dag Sverre Seljebotn]

To be clear, what this command does is update the "key:" property
containing the sha1 of hashstack, to move it to the latest one.

Dag Sverre

[Dag Sverre Seljebotn]

On 05/21/2015 05:32 PM, Chris Kees wrote:
>
>
> On Thu, May 21, 2015 at 1:46 AM, Dag Sverre Seljebotn
> <d.s.se...@astro.uio.no <mailto:d.s.se...@astro.uio.no>> wrote:
>
> On 05/21/2015 08:42 AM, Dag Sverre Seljebotn wrote:
>
>
> Then the question is: Do we also add a history mechanism; `hit
> rollback`, etc?
>
> +1
>
> hit use mybranch
>
> +1
>
> hit install numpy
>
> +10
>
> hit update pandas
>
> +10
>
> I' don't have time to think through the difference in the .hashdist git
> repo vs symlinks issue this morning, but I think this fits with the
> original vision. I find doing things like the following get old fast:
>
> cd hashstack
> hit skeleton-pypi some_simple_pure_python_package
> emacs examples/proteus.linux2.yaml #add new package
> cp examples/proteus.linux2.yaml default.yaml
> hit build default

Out of curiosity, is there a reason you didn't do

ln -s examples/proteus.linux2.yaml default.yaml

to eliminate at least one of these steps?

As for PyPI, the solution should be to elide the YAML file entirely from
the equation, rather than auto-generate it on "hit install", IMO. I
really really want "hit install foo" to only modify the right profile
YAML file and add a single line "foo:" under "packages:", everything
else that "hit install" could do we should instead figure out when you
use the profile YAML...

Anyway I think "hit install" etc. is definitely for later, I just wanted
the way we define "global system profiles" to align with some vision for
this.

Dag Sverre

[Dag Sverre Seljebotn]

Or more typically, your default.yaml could just contain this?:

extends:
- file: examples/proteus.linux2.yaml

or something like that. Then it's also possible with overrides. If this
doesn't work we need to make it :)

DS

[Ondřej Čertík]

What if you need to modify the build script, e.g. for packages like
pytables (https://github.com/hashdist/hashstack/blob/d978b36df6dcdaab43c18996e9986cbdce26c618/pkgs/pytables/pytables.yaml)?
Most packages work out of the box with our default template, but some
don't. Also, what about dependencies?

>
> Anyway I think "hit install" etc. is definitely for later, I just wanted the
> way we define "global system profiles" to align with some vision for this.
>
> Dag Sverre
>

> --
> You received this message because you are subscribed to the Google Groups
> "hashdist" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to hashdist+u...@googlegroups.com.
> To post to this group, send email to hash...@googlegroups.com.
> Visit this group at http://groups.google.com/group/hashdist.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/hashdist/555E17CB.7050604%40astro.uio.no.

[Ondřej Čertík]

On Thu, May 21, 2015 at 11:29 AM, Dag Sverre Seljebotn

I think the idea to use git is good, if we can figure all the details.
Obviously I don't want to reimplement git, at the same time, `hit` is
modeled after git, so it's already duplicating the philosophy at
least. So whether we can leverage the actual git, or implement things
ourselves depends on the details.

>
> If we get "hit install numpy", then "hit install scipy", I would really like
> that to
>
> a) be implemented as successive changes to a profile YAML file
> b) each command I run should be a seperate commit (being a machine-based
> change it needs to be well tracked, easily revertable etc.)

I agree.

>
> I guess that could also be satisfied by using git in a very restricted mode
> (like, only one branch), instead. But I am rather convinced that using git
> in one way or another should play a part, life is too short to make our own
> history tracking system, and we don't have to use or support all of git's
> features.
>
>> order them by the time they were created, but that's not how git works
>> --- you can create the last commit first, and first commit last, and
>> rebase on top. The time of commits will show this, i.e. that the
>> commits are not ordered by time (necessarily). So in our case, the
>> profile history is really just a set of hashes, with no intrinsic
>> ordering/dependencies.
>
>
> The editing history of your profile YAML files *is* ordered in times and git
> commits will typically match that. Fancy rebasing is not something that
> happens by default. If you do, you of course can't complain afterwards that
> "hit rollback" reset the last commit even if didn't have newest timestamp;
> that is what you expect...
>
> If you do "hit install numpy; hit uninstall numpy; hit install numpy", there
> are 3 commits even if you end up with the original profile hash.

So it seems you would version the profile.yaml files in git, and build
some nice `hit` interface on top? That's fine.

It seems to me the Hashstack git repo, that contains the package
definition files, should be separate from any possible profiles.yaml
git repo. Because the package definitions do not depend on the steps
that you install or uninstall packages.

>
>> I am not sure I follow. Essentially you are saying to put the
>> Hashstack repository inside ~/.hashdist, and let it be managed by
>> `hit`? And the profile yaml, let's say the default.yaml, will also be
>> managed by git, inside ~/.hashdist, and `hit` will provide some nice
>> tools on top?
>
>
> By default, I would put a small repo that links to hashstack, like this one:
>
> https://github.com/hashdist/profile-template
>
>> So if I have 10 different profiles, those will be in 10 different
>> branches? A typical thing that I do is that I update one package, and
>> then rebuild all profiles. So I would need to rebase the 10 profile
>> branches first on top of the latest Hashstack to rebuild?
>
>
> Yes, it doesn't quite work, does it? I'm brainstorming :)
>
> So what about (typically) having a single branch, and each profile is a YAML
> file? So if I do
>
> hit load prof1
> hit install numpy # commit (1) changes prof1.yaml
> hit load prof2
> hit install pandas # commit (2) changes prof2.yaml
>
> then there are 2 commits to the repo, one that changes prof1.yaml and one
> that changes prof2.yaml.

Yes. This git repo is separate from the Hashstack git repo.

>
> "hit rollback" needs some thought here though. For the commits above it
> could use "git log prof1.yaml" and do a "git revert", but it's not clear
> what it should do, or whether it should raise an error, in your case where
> you change package definition and wants to rebuild all the profiles.
>
> I think for typical non-power-user cases, "prof1.yaml" contains the sha1 of
> a specific Hashstack commit. And if you want new package definitions, that
> could be done with
>
> hit load prof1
> hit fetch-package-definitions # commit (3) changes prof1.yaml
>
> which again lends itself nicely to a rollback git commit.

That's fine.

>
> Anyway, I'm not sure just how important the rollback command is. The
> important thing is there's an easy to understand history system that tracks
> everything in a nice way.
>
> So, how do you feel about one repo with one YAML file per "globally
> available" profile? Files like "debian.yaml" should either be in other repos
> included via a git hash, or in the same repo in a sub-directory (so
> "debian.yaml" would be a profile named "debian", but just move it to
> "support/debian.yaml" and it doesn't show up in the profile list).

That's fine. Essentially what you are saying is to provide `hit`
commands to manage your profiles. Currently you have to edit them by
hand. You could store them in a git repo already, by hand.
That seems to be in line with what I proposed here:
https://github.com/hashdist/hashdist/issues/329

But there is still one fundamental design issue though. Currently the
yaml files are not part of the Hashdist database, i.e. once you
install a profile, the only info that is being stored is the
build.json and artifact.json and then it links installed packages into
the profile. Each package is similar --- only build.json and
artifact.json and the installed files. This is nice, because it is
simple. The yaml descriptions are separate from this and we are free
to change how it works. For example if we wanted to change the yaml
files to Python files, like Spack works, we can. Hashdist runtime
database does not depend on this at all.

If, however, we now put at least the profile.yaml files as part of the
Hashdist database, we are essentially committing to use them.
Essentially we are tying the sources (yaml) with the binaries (the
installed packages and profiles in the Hashdist database).

I guess that's fine --- now the Hashdist database will know not only
about the binary packages and their dependencies, but also how they
were actually built from sources.

Ondrej

[Dag Sverre Seljebotn]

For every release, we either break a particular users' setup and
workflow, or we don't -- I don't feel that what's "official" or not come
into it all that much.

It's still layered in exactly the same way as today though; we could
support Python profiles alongside YAML profiles down the road.

Also it's mainly the profiles YAML format, not the packages YAML format.
E.g., we could play with having the profiles YAML "extend" PyPI, and
there would be an automated PyPI adapter, that didn't know anything
about the YAML build specs, only the profile and property-passing logic
and (eventually) constraint solving.

Dag Sverre

[Dag Sverre Seljebotn]

To be clear, the "hashdist database" has these layers:

- src : Source cache. Independent of anything else.

- bld : Build artifacts. Builds on source cache. In theory two different
packaging formats can yield the same lookup in the cache, but I think
that possibility is highly theoretical, in practice every package
description format gives a different hash and artifact anyway.

And now it's proposed to add a third database:

- stacks : A git repo where the glob "/*.yaml" defines the "globally
available profiles", and which is managed by high-level commands.

Really, all of these are seperate databases; the addition of "stacks"
doesn't make "bld" or "src" commit to anything new.

Dag Sverre

[Ondřej Čertík]

On Thu, May 21, 2015 at 12:18 PM, Dag Sverre Seljebotn

That works and should be documented as such, i.e. we need to explain
this design well in our docs, as well as the artifact.json and
build.json.

Ondrej

[Chris Kees]

Out of curiosity, is there a reason you didn't do

ln -s examples/proteus.linux2.yaml default.yaml

to eliminate at least one of these steps?

I don't think there's any reason it couldn't be a symlink. @ahmadia wrote that part of the proteus Makefile, so he may have a different opinion. I'm getting ready to update the makefile once the issue of how to depend on the hashstack gcc that @ondrej has been working on. I will update this to a sym link then.
 

As for PyPI, the solution should be to elide the YAML file entirely from the equation, rather than auto-generate it on "hit install", IMO. I really really want "hit install foo" to only modify the right profile YAML file and add a single line "foo:" under "packages:", everything else that "hit install" could do we should instead figure out when you use the profile YAML...

For later...everything inside  .hashdist  would be as if  the YAML file existed, but we would just never generate the YAML.  I guess the url  and key would get written directly to the profile.
 

Anyway I think "hit install" etc. is definitely for later, I just wanted the way we define "global system profiles" to align with some vision for this.

Dag Sverre

--
You received this message because you are subscribed to the Google Groups "hashdist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hashdist+u...@googlegroups.com.
To post to this group, send email to hash...@googlegroups.com.
Visit this group at http://groups.google.com/group/hashdist.

To view this discussion on the web visit https://groups.google.com/d/msgid/hashdist/555E17CB.7050604%40astro.uio.no.

[Chris Kees]

I think that would work too. The issue is that we need default.yaml to point to a different  stack for each platform, so we're doing this from a makefile:

cp stack/examples/proteus.${PROTEUS_ARCH}.yaml stack/default.yaml

Changing that to a  sym link is trivial, but writing the  default.yaml would also be  easy if  there is some weird issue with links that I'm not thinking about.

As for PyPI, the solution should be to elide the YAML file entirely from
the equation, rather than auto-generate it on "hit install", IMO. I
really really want "hit install foo" to only modify the right profile
YAML file and add a single line "foo:" under "packages:", everything
else that "hit install" could do we should instead figure out when you
use the profile YAML...

Anyway I think "hit install" etc. is definitely for later, I just wanted
the way we define "global system profiles" to align with some vision for
this.

Dag Sverre

--
You received this message because you are subscribed to the Google Groups "hashdist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hashdist+u...@googlegroups.com.
To post to this group, send email to hash...@googlegroups.com.
Visit this group at http://groups.google.com/group/hashdist.

To view this discussion on the web visit https://groups.google.com/d/msgid/hashdist/555E1833.8010409%40astro.uio.no.