(Very) new user question: system wide install for all users

[a-jp]

Hi,

I want to get up and running with Spack on our cluster.  I want to install some common packages for access by a number of users.  I have cloned Spack into a system wide install path.  

I have admin rights but I understand it's best to operate Spack via a normal user account.  

If I run everything in my own user account, how can I ensure I install everything to a location where all users can see and use the packages?  That is to say I do not want to install packages into my own home space, but a user wide location, which I can also access.

Second question, once I've done this, are there any configs files that I need to provide to the other users that may wish to access these newly installed packages?  As in, if I log in as another user, even I can install everything system wide as per the above question, do users need additional config files that have been created/modified locally for me simply because I performed the installation and configuration under my user account:  are there files written to $HOME/.spack that I need to distribute?

Bit lost...

Cheers,

Andy

[George Hartzell]

Hi Andy,

a-jp writes:
> Hi,
>
> I want to get up and running with Spack on our cluster. I want to install
> some common packages for access by a number of users. I have cloned Spack
> into a system wide install path.

That's reasonable.

> I have admin rights but I understand it's best to operate Spack via a
> normal user account.

Yep, though there are one or two things in the tree (e.g. Singularity)
that need root priv's to complete the install. For Singularity, at
least, the pkg install message will tell you about a small script that
gets installed that you can examine and then run as root (it's
checking permissions and making a few bits setuid).

> If I run everything in my own user account, how can I ensure I install
> everything to a location where all users can see and use the packages?
> That is to say I do not want to install packages into my own home space,
> but a user wide location, which I can also access.

Let's suppose that you've cloned Spack into a directory named
`$spack_dir`. Unless you configure Spack to do it differently, all of
the things that you build and install will end up beneath
`$spack_dir/opt`. Various other things will end up in other subdirs
(e.g. modulefiles in `$spack_dir/share`). If you ensure that the
account doing the cloning and building/installing has a
sufficiently/appropriately permissive umask, then users will be able
to use the applications.

> Second question, once I've done this, are there any configs files that I
> need to provide to the other users that may wish to access these newly
> installed packages? As in, if I log in as another user, even I can install
> everything system wide as per the above question, do users need additional
> config files that have been created/modified locally for me simply because
> I performed the installation and configuration under my user account: are
> there files written to $HOME/.spack that I need to distribute?
>
> Bit lost...

I've been using Spack for a while so some of my practices pre-date
recent additions like "environments" and using "spack load" to manage
environments. You can read about the new-fangled things in the user
docs. If you haven't already, it's really worth running through the
[tutorial].

Here are some thoughts:

- It's relatively safe to clone a tree, install a bunch of things,
then come back later and install a few new things into the tree.
The only surprising bits is that if users were loading default
versions of applications via modules, the newly installed
applications might change the module system's idea of the default
versions. If they're explicitly loading particular versions or if
you've defined a default, then it's safe.

- It's not safe to clone a tree, install a bunch of things, use `git
pull` to update the tree, and then install more things. Often
everything will work. Sometimes you'll reinstall things that you
already have in the tree because something has changed the package
hashes for those versions. Occasionally everything will go
kerblooie and you'll be sad....

My approach to this is to:

- Use a script to install All The Things (see below).

While it's probably possible to build/deploy a single tree with
all the things that everyone might ever need and then load the
interesting bits via Lmod, building a great big tree might take
a long time (I had a 100+ core box to play with, so wheee....).
You might be better off building smaller team/org specific
trees.

- Keep personal/development trees where I figure out what I want
to install (versions, variants, ...) and test out the install
script. If it breaks, no harm/no foul.

- Do regular test runs of the install script (I used a Jenkins
installation, but you could do them by hand if you wanted). I
also updated the underlying Spack tree daily so that I could
catch upstream changes that [GASP] broke the build.

- Use the script to install **production releases** of the tree,
which are considered read-only, I *never* install additional
apps into a released tree.

Here is a slightly simplified version of my [install.sh] script (the
real version also supported a local mirror for all of the source
tarballs and had various incriminating comments...). It's a bit
idiosyncratic, but Works For Me. Your Mileage May Vary, etc....

- It indiscriminately splats a `packages.yaml` file into the tree that
pins versions and variants for nearly all of the packages so that I
get what I expect. This was old school, I think that *environments*
are the modern way to do this....

- Compilers....

- Various apps that we needed didn't (don't?) compile with newer
gcc's (C++ default changes) and Lmod's hierarchical structure
makes it difficult to simultaneously load apps that were built
with different compilers, so we just built with g...@5.4.0. You
can probably use a newer compiler.

- We were working on CentOS 7, different sub-versions supplied
different versions of gcc as the system compiler. This accounted
for those. See the compiler comments re modulefiles below.

I used Lmod modulefiles to give my users access to the various
applications.

- The [install.sh] script indiscriminately splats a `modules.yaml`
into the tree to configure things the way I wanted them. The
Modules section of the [tutorial] is very useful to understand
what's going on. Things to pay particular attention to include:

- I only wanted to generate modulefiles for applications built with
g...@5.4.0. So I define the system compilers as "Core" and
blacklist the things built with them, except for the copy of
g...@5.4.0 that they built.

- I don't include LD_LIBRARY_PATH and friends in the modulefiles,
except for Python things. Spack's RPATH support makes it
unnecessary (except for some Python things, sigh...) and I find
it's fragile (e.g. it causes `yum` to malfunction). On the other
hand, it's apparently useful if you have users linking their own
applications against libraries within the Spack tree.

I installed a small BASH snippet [app-init.sh] that users could source
to set up their account to use Lmod to access one of my production
releases. There may be a more modern Spack approach to this
(e.g. sourcing `share/spack/setup-env.sh`), but my users were already
Lmod savvy and a) I didn't want to make them learn about Spack and b)
didn't want Spack running every time they started a shell/.... I
might be biased/grumpy.... The snippet has useful comments at the
top, but keep in mind that...

- All of my production trees were installed in a `spack` directory
beneath in a single directory. `_APPS_TOP_DIR` should be set to
that directory.

- Users could set `APPS_DIR` to point to a particular tree, otherwise
they would get the default/"current" tree, the full path to which
was defined in the file `${_APPS_TOP_DIR}/spack/current)`.

- Users could set `APPS_MODULES` to a list of modulefiles to load,
otherwise they'd get the default list set in the [install.sh].

- There are a few other knobs that sophisticated users could set.

Phew.

Happy to help and/or explain things further. If you're at a location
with a decent bank balance, I'm also open to consulting work....

Go get 'em,

g.

[app-init.sh]: https://gist.github.com/hartzell/5a759b79b73c9e435cf1b8e3fa0a7370
[install.sh]: https://gist.github.com/hartzell/75407b75bac55113cf670264137fb37e
[tutorial]: https://spack-tutorial.readthedocs.io/en/latest/

[Eric Berquist]

Hello George and others,

I am surprised that there is no best-practices documentation for things like this; for example, how do LLNL and other labs do it? Should I provide a global environment?

Eric