How to update/upgrade SPACK?

[Jim Prewett]

Hi All,

The answer to this question may be painfully obvious...  How do I keep SPACK at the latest version?

Is it as simple as the right incantation involving "git"?

Thanks!

Jim

[George Hartzell]

I think that it depends on what your goal is.

When I'm wearing my "deploy the standard set of applications" hat, I
don't ever update it. I clone a fresh tree and run a [standard
installation script][spack-install]. I keep the old trees around
until they're no longer needed (might be teams migrating or just me
ensuring that my personal kit works).

When I'm working on something and just want to bring an existing clone
of the repository up to date, I:

```
cd <spack directory>
git checkout develop
git fetch
# look at what was fetched, and if I'm happy
git merge origin/develop
# then do the following for each of my works in progress
git checkout some-wip-branch
git rebase develop
# deal with any fallout

# and then perhaps reinstall packages of interest.
```

That gets the source tree updated. But what about your installed
packages? You could delete everything and re-install, now you're back
at my "just clone a new tree". You could just start installing
additional packages without changing what you've already installed,
but see below.

What about installing packages that have had newer versions released
since you installed them? I don't think that Spack has any equivalent
of Homebrew's `brew outdated`, `brew upgrade`, or `brew update`. I
handle this via my 'clone a new tree' mechanism.

I've found that, except for hacking around, updating trees in place
leads to inconsistencies and eventual pain (though it works well
enough that it's worth doing in controlled short-term situations).

- Sometimes (rarely, lately) the way hashes are calculated changes and
you'll end up with multiple installed copies of a package+variants,
which leads to confusion.

- Even if you're installing particular versions of packages
(e.g. emacs@25.3+tls), you're unlikely to be pinning every package
in your complete dependency graph so previously built packages might
depend on one version of something while others depend on a newer
version. Depending on how you're generating your modulefiles,
taking the "default" (aka newest) may no longer get you what you're
expecting.

Having said all of that, what exactly are you trying to do?

g.

[spack-install]: https://gist.github.com/hartzell/d7d067e59695575c47e1c86b898b37fc

[George Hartzell]

I've added the Spack group back into this conversation, I *think* that
Jim didn't mean to close the group discussion out of the discussion
(nothing seems private). Hope I didn't overstep....

Jim Prewett writes:
>
> Thanks George for your detailed response!
>
> Right now, I'm thinking once software is installed it is essentially
> "there forever" so lib...@0.0.1%g...@4.8.5 will remain unchanged unless I
> find a bug in my libfoo/package.py file: Then I would reinstall everything
> that depended on that. After I retire they can worry about what to do
> with it all! ;)

Well, the concrete graph of things you get from
`lib...@0.0.1%g...@4.8.5` might not be the same before and after an
update. Two things might change:

1. it might depend on a package that has been updated and isn't pinned
in your package.yml, you'd pick up the newest version that is
compatible with the full set of constraints.

2. something might have changed in the package.py's of the involved
packages or the build system, you won't see this *unless* you
uninstall/reinstall but you might want it.

Even if you don't explicitly un/reinstall it, you might build
something new that wants `lib...@0.0.1%g...@4.5.8` and it might build
you *another* copy (e.g. if something changes the hash computations).
Hillarity will eventually ensue.

> What I fundamentally want, I think, is to continuously have the latest
> version of the SPACK package repository (and whatever updates to the SPACK
> software itself that may be needed to install these packages). I'll try
> to digest your answer properly: I hope to have a better formed question to
> ask! :)

This sounds a bit closer to the Homebrew goal (give me the current set
of things, update me to the current set of things, emphasis on the
current set of things) than the spack focus (give me a predictable set
of things that might include older versions).

I wear two hats, which want different things:

The stated goal of hat one is to have a copy of "all of my apps" built
using the latest copy of the tree. I pin the app versions. I pin a
few dependency versions. I set up a modules.yml file. Then I build
all of my things. If it works, I start using it. I may keep the old
copy around for "reproducibility".

The other hat is me hacking around in a tree. I'll happily fetch new
commits from wherever I feel like, build stuff, and YAY. If (when) it
becomes too messed up of an environment (see above), I uninstall
everything and reinstall everything.

The first hat is generally my building sets of apps that are used by
other folks or in real work.

The second hack is me hammering on an idea until I'm happy with it and
can PR it (or abandon it).

g.