How to update Spack itself?

[Henrik Bengtsson]

Installing Spack is a simple Git clone:

    git clone https://github.com/llnl/spack.git

    . spack/share/spack/setup-env.sh

What's the recommended way to upgrade Spack?  Should I just do a Git pull (on master):

    cd spack

    git pull

?

Do I have to worry about existing software installations not being compatible across updates?  For instance, say I've installed pyt...@3.5.1 using spack 0.9.1, then update to spack 0.9.2 and ask to install pyt...@3.5.1 again, will I get the same hashes?

Thanks,

Henrik

[Gamblin, Todd]

Hi Henrik,

 

At the moment, `git pull` is the way to update. If you’re deploying Spack in production, at the moment I would recommend freezing your version and deploying an updated Spack version *alongside* the older version.  I’d like to have a more stable upgrade path eventually, and make pulling reliable at least for `master`.  But at the moment the pace of development is very rapid.

 

FYI, per some recent telcons and mails on this list, the current roadmap looks like this:

-          Spack v0.10 will be out this week, fixing a large set of bugs we identified shortly before SC16.

o    I plan to at least backport breaking package issues from `develop` to the 0.10 branch, so there will be point releases.

-          Spack v1.0 will be out at the end of February.  The two main goals with 1.0 will be:

o    Rigorous package testing across OS’s and compilers, with test dashboards at http://spack.io/cdash

o    An improved concretization algorithm to better manage constraints among packages, and to make Spack more aggressive about reusing installed software.  There are pros and cons to the second point here. Reusing packages makes installation nondeterministic, and we’ll keep an option to install deterministically.

 

I think these will go a long way towards increasing stability.  Longer term, once we have a more stable package API, I would like to look at separating the core package repo from the Spack core (like homebrew). But there are a fair number of features I would like to get into the core before I’d be comfortable doing that.

 

-Todd

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at http://secure-web.cisco.com/1VGUV4ofZgNOSfOdBBweTxMTYDio0LHq5_fT4I0eUEjLJKuWKquX3jA5NrMTnPL2jQsftQsYM6pksObtLZSofSONfJRlQyPNwMe9jTlvwkKZq1_1A5dZn9KP6Btufbb0b-s2jGNQ4B_Jx2KsPQZT6VJLRnQ7ljR7Ht_uwzYVwvnypHHdCf__RZlWCXn1wVGIVVR2x3RxwI0gAO756FPhZG4iOSXKryrOGi-P3u_YPZgajUDCAtm7kPXJ19JwJ3wbhIGMUI7eilWZvB9BaEJyE_iYer2WSXysmpWSbq8p5VtOgYckzEo3WT4zKqW5NJuLXGYGsRI4XU9VgSx9sr8PLeCmsL1bdccPlfrwpZknSjnzvVQVO7fvO4rE51255AvUpWxZW_YCwBblZfqVoH-f56gBAFGI83WPmmFqP5fZ6rUE/l37%3Ahttps%3A%2F%2Fgroups.google.com%2Fgroup%2Fspacke.
For more options, visit http://secure-web.cisco.com/1Qdk3B4WilCvDIuYLohEzHKYUOlPa-Y6vFZRiehWc3SP7jfZMAP_zG4iBQswd98n9jVNyuvrKORjpQ53QWwQEIAbQKtgULmpL-Yl8LgDlf9RoNSYf7rCcOOjB0Ju7TqvJjkVV9riD5E5aMD0pXBg8Bwhs65uIshwfwrMg7WaHzFzVvmpfiQwfDFEcq1rU7rCByIn1u4dtW4J_G__ohsY_Qyv0388UXTLL2Ueq7UyMX1CNT5tMoobtT4Cx3vQKCccXzMtdl1sstzNqwBrqF216Xz_G74HM5l2vXBXFhrM-_9H19bzyy4HQuvKjbZfo8gBU9Z25iWW6rOY5scId847f1mpIBPnGknW7FsiOd_F1Esf7luN4KrgXKBCVdb1D-Yzju4cG2kl2K9y3bwsOqHuhjTekc9B3eoPZo_aBw3hFwIA/l34%3Ahttps%3A%2F%2Fgroups.google.com%2Fd%2Foptoute.

[Henrik Bengtsson]

Thank you for a clear explanation and what the plans are - this is all
great and exciting news.

/Henrik

[Alan Sill]

Just a quick check on this topic. We started using Spack in earnest only recently, and are wondering about the best practices for keeping Spack itself updated while working towards a production rollout as the primary method of building software for our clusters.

Is there advice I have missed in the documentation on this topic?

Thanks! This is an awesome project and a real service to the community.

Alan

To unsubscribe from this group and stop receiving emails from it, send an email to spack+unsubscribe@googlegroups.com.

[Gamblin, Todd]

Hi Alan,

Thanks for using Spack!  There’s not a lot of documentation on this, but I’d recommend getting something working on top of a release branch, then updating to the latest releases as they come out.  I’d be interested to hear what issues you encounter when doing that.

If you’re only *working towards* a release, I think you could do your builds on `develop`.  You can version a spack.yaml (https://spack-tutorial.readthedocs.io/en/latest/tutorial_stacks.html) description of your deployment in a repo, and use the new pipeline integration (https://spack.readthedocs.io/en/latest/pipelines.html) to test it, then freeze it *somewhere* (either some point in develop or some release) once you decide to go live.  This is what we’re working towards at LLNL right now. 

I’d be interested to know what your workflow looks like and what roadblocks you hit along the way.

-Todd

To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spack/db61bbec-dbe7-4a84-8ce9-8d97c01d8946%40googlegroups.com.

[George Hartzell]

'Gamblin, Todd' via Spack writes:
> [...]

> I’d be interested to know what your workflow looks like and what
> roadblocks you hit along the way.

One of the things that I like^H^H^H^Hlove about Spack is that it makes
it [simple] to follow the DevOps [cattle-not-pets] model of
maintenance (which boils down to rebuild, don't update).

Entire rebuilds can take some time, particularly if you're building a
compiler (and heaven help you if you need rust) but they run nicely
overnight. If it's really a problem there are now ways to "chain"
installations; e.g. you could periodically build a tree with your
compilers and then more quickly build a separate tree of your apps.

Back in the days before the chaining support and the environment
support, I used scripts like this [install] script in CI/CD jobs that
ran every night to check that my stuff still built with the current
Spack dev release. I'd also use the same script to build from a
tagged point so that I could make releases. I ended up with a tree of
versioned releases that my community used (policy abounded) and a
couple of recent dev builds.

I never updated publicly visible installations, always rolled out a
new one.

What happened in the privacy of my own home dir was another story and
I did periodically find that I had more than one copy of a particular
thing installed because hashes had changed or default versions of
dependencies had changed or .... I think that I also had some sneaky
trouble because the lmod logic for choosing a default version of a
module (dependency) to load wasn't the same as Spack's and so behavior
would change. Details are fuzzy and it's been a couple of months
since I've been in the game.

If you do try to update installations in place and it's critical that
it work, I'd recommend making a backup or snapshot, just in case.

g.


[cattle-not-pets]; https://medium.com/@Joachim8675309/devops-concepts-pets-vs-cattle-2380b5aab313
[install]: https://gist.github.com/hartzell/c08cf0a04fac87ea5a14321f3e8f6fbe
[simple]: https://www.infoq.com/presentations/Simple-Made-Easy/

[Bryce Allen]

I agree with the recommendation of using a release branch, rather then develop, for most use cases. For packages I need absolutely fresh versions, I am using another repository maintained in another github repository. PRs for package updates and new packages are generally merged quickly, but this is still not quickly enough for fast turn-around of building a spack workflow for running complex HPC software. I really see no advantage to using develop for anything other than developing spack and contributing PRs to spack.

The installation tutorial now recommends using a release branch:

https://spack-tutorial.readthedocs.io/en/latest/tutorial_basics.html#installing-spack

The main docs installation section still just uses the default branch:

https://spack.readthedocs.io/en/latest/getting_started.html#installation

Has there been discussion of updating the main docs as well, or at least adding a note about using release branches?

-Bryce