Resolving conflicts of identically named packages in spack builtin repository and additional repositories

[Pariksheet Nanda]

Hi folks,

Prior to graduating earlier this year, I setup and managed an academic research lab server in 2019 shared among a few graduate students, which has the spack git repository cloned to /opt/spack and at midnight a cron job runs git pull to update to the latest develop branch that is also checked out.    Occasionally, some of my lab mates would need me to modify a spack package with some temporary change or apply a change immediately to fix something and I didn't have a time window to also upstream with a pull request.  Or sometimes I needed to experiment with new packages.  To do this I created a new repository in /var/spack/repos/corelab and made a copy of the package directory of the builtin repository to modify the package in my repository.

However, I eventually learned that as time goes on and the package diverges from the original copy from the builtin repository, the package becomes invalid as may be expected due to changes to spack core, etc.  The latest example of this is "from spack import *" was changed to "from spack.package import *".

What I was not aware of was this divergence then breaks all installed packages that have a dependency on the package from this repository namespace.  My expectation using Gentoo Linux Overlays is that the primary recipe for a package is the one with the latest version.  So a Gentoo Overlays if I created a copy of an upstream package and bump it with a newer patch level than upstream, my package is preferred.  When upstream bumps their revision to be ahead of mine, upstream is preferred.  Similarly, Debian has a priority system to honor choosing among various package sources.  But that is not how spack works.  Spack humbly always allows other packages to have higher priority than additional repositories irrespective of how they are ordered in /etc/spack/repos.yaml etc and as far as I am aware from the last time I read the documentations when I was struggling with this issue there is no way to change this behavior.

Of course, Spack's concretizer is more sophisticated than most GNU/Linux distributions and so I don't expect it to have identical behavior.  I mention those experiences because that was my mental model of mixing repositories with identical package names.  I don't know what the intended workflow is for managing identically named packages in repositories other than builtin.

1. One option would be to modify packages in builtin for an installation and discard those changes to continue pulling from develop.  This may appear to be unreproducible, but spack creates a copy of the package.py etc in the install prefix under a hidden .spack/repos directory. 

2. Another option would be to have a priority system where the package has a lower priority than anything in builtin and so the package from the additional repository would need it's name explicitly specified with the package namespace.  But this may introduce problems with multiple packages sharing a dependency with each needing their dependency specified to the namespace package.

So if I had to do anything similar now, I would lean toward option 1.

In any case, to sustainably resolve the version divergence issues I introduced with my temporary packages, I've replaced entire package directories in my repository with symlinks to the copy in builtin, so that pulls from the develop branch no longer break existing installations:

lmw09001@corelab2:/var/spack/repos/corelab/packages$ ls -l
total 84
drwxr-sr-x 3 pan14001 users  5 May 31 20:24 basespace-cli
lrwxrwxrwx 1 pan14001 users 49 May 31 20:22 bzip2 -> /opt/spack/var/spack/repos/builtin/packages/bzip2
lrwxrwxrwx 1 pan14001 users 57 May 31 20:22 fastx-toolkit -> /opt/spack/var/spack/repos/builtin/packages/fastx-toolkit
lrwxrwxrwx 1 pan14001 users 47 May 31 20:22 jdk -> /opt/spack/var/spack/repos/builtin/packages/jdk
lrwxrwxrwx 1 pan14001 users 48 May 31 20:22 meme -> /opt/spack/var/spack/repos/builtin/packages/meme
drwxr-sr-x 3 pan14001 users  5 May 31 20:11 ncbi-toolkit
lrwxrwxrwx 1 pan14001 users 48 May 31 20:21 perl -> /opt/spack/var/spack/repos/builtin/packages/perl
drwxr-sr-x 3 pan14001 users  5 May 31 20:23 perl-pfamscan
lrwxrwxrwx 1 pan14001 users 50 May 31 20:21 picard -> /opt/spack/var/spack/repos/builtin/packages/picard
drwxr-sr-x 3 pan14001 users  5 May 31 20:11 py-bart2
lrwxrwxrwx 1 pan14001 users 56 May 31 20:21 py-deeptools -> /opt/spack/var/spack/repos/builtin/packages/py-deeptools
lrwxrwxrwx 1 pan14001 users 65 May 31 20:21 py-deeptoolsintervals -> /opt/spack/var/spack/repos/builtin/packages/py-deeptoolsintervals
lrwxrwxrwx 1 pan14001 users 53 May 31 20:21 py-plotly -> /opt/spack/var/spack/repos/builtin/packages/py-plotly
lrwxrwxrwx 1 pan14001 users 55 May 31 20:18 py-pybigwig -> /opt/spack/var/spack/repos/builtin/packages/py-pybigwig
lrwxrwxrwx 1 pan14001 users 54 May 31 20:21 py-pyfaidx -> /opt/spack/var/spack/repos/builtin/packages/py-pyfaidx
lrwxrwxrwx 1 pan14001 users 52 May 31 20:20 py-pysam -> /opt/spack/var/spack/repos/builtin/packages/py-pysam

Below is an example of how a package broke with the "from spack import *" to "from spack.package import *" and how I diagnosed by adding the -d / --debug flag to help people doing web searches with a similar issue if they have the same brain dead setup as me.

Pariksheet

lmw09001@corelab2:/var/spack/repos/corelab/packages$ spack load me...@5.1.1
==> Error: Package 'meme' not found.

lmw09001@corelab2:/var/spack/repos/corelab/packages$ spack -d load me...@5.1.1
==> [2022-05-31-20:06:27.148502] Imported load from built-in commands
==> [2022-05-31-20:06:27.149325] Imported load from built-in commands
==> [2022-05-31-20:06:27.150930] Reading config file /opt/spack/etc/spack/defaults/config.yaml
==> [2022-05-31-20:06:27.176288] Reading config file /opt/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-05-31-20:06:27.184594] DATABASE LOCK TIMEOUT: 3s
==> [2022-05-31-20:06:27.184649] PACKAGE LOCK TIMEOUT: No timeout
==> [2022-05-31-20:06:27.346348] Reading config file /opt/spack/etc/spack/defaults/repos.yaml
==> [2022-05-31-20:06:27.348870] Reading config file /etc/spack/repos.yaml
==> [2022-05-31-20:06:27.437298] UnknownPackageError: Package 'meme' not found.
==> [2022-05-31-20:06:27.437365] Error: Package 'meme' not found.
Traceback (most recent call last):
  File "/opt/spack/lib/spack/spack/repo.py", line 1298, in get_pkg_class
    module = importlib.import_module(fullname)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/spack/repos/corelab/packages/meme/package.py", line 1, in <module>
    # Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
ModuleNotFoundError: No module named 'spack.pkgkit'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/spack/lib/spack/spack/main.py", line 893, in main
    return _main(argv)
  File "/opt/spack/lib/spack/spack/main.py", line 848, in _main
    return finish_parse_and_run(parser, cmd_name, env_format_error)
  File "/opt/spack/lib/spack/spack/main.py", line 876, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/opt/spack/lib/spack/spack/main.py", line 533, in _invoke_command
    return_val = command(parser, args)
  File "/opt/spack/lib/spack/spack/cmd/load.py", line 78, in load
    for spec in spack.cmd.parse_specs(args.constraint)]
  File "/opt/spack/lib/spack/spack/cmd/load.py", line 78, in <listcomp>
    for spec in spack.cmd.parse_specs(args.constraint)]
  File "/opt/spack/lib/spack/spack/cmd/__init__.py", line 197, in disambiguate_spec
    return disambiguate_spec_from_hashes(spec, hashes, local, installed, first)
  File "/opt/spack/lib/spack/spack/cmd/__init__.py", line 217, in disambiguate_spec_from_hashes
    installed=installed)
  File "/opt/spack/lib/spack/spack/database.py", line 1565, in query
    local_results = set(self.query_local(*args, **kwargs))
  File "/opt/spack/lib/spack/spack/database.py", line 1550, in query_local
    return sorted(self._query(*args, **kwargs))
  File "/opt/spack/lib/spack/spack/database.py", line 1538, in _query
    rec.spec.satisfies(query_spec, strict=True)):
  File "/opt/spack/lib/spack/spack/spec.py", line 3628, in satisfies
    if not self.virtual and other.virtual:
  File "/opt/spack/lib/spack/spack/spec.py", line 1536, in virtual
    return spack.repo.path.is_virtual(self.name, use_index=False)
  File "/opt/spack/lib/spack/spack/repo.py", line 992, in is_virtual
    self.get_pkg_class(pkg_name).virtual)
  File "/opt/spack/lib/spack/spack/repo.py", line 955, in get_pkg_class
    return self.repo_for_pkg(pkg_name).get_pkg_class(pkg_name)
  File "/opt/spack/lib/spack/spack/repo.py", line 1300, in get_pkg_class
    raise UnknownPackageError(pkg_name)
spack.repo.UnknownPackageError: Package 'meme' not found.

[Matthieu Dorier]

Not sure if it can help, but in my team we also occasionally need to edit packages, to add versions/variants and have access to them before they are upstreamed (or versions that will never be upstreamed, like release candidates), to have some versions pointing to forks of a github repo, and so on. Rather than copying the original package file, we inherit from it. For example this is what we have for the Libfabric package:

from spack.pkg.builtin.libfabric import Libfabric as BuiltinLibfabric

class Libfabric(BuiltinLibfabric):

   version('x',...) # this version is not upstreamed yet

   ... # some other modifications

By doing so you can add version, variants, override the way the package is built, etc. and we also get whatever is added upstream when updating spack (i.e. if a new variant appears upstream, we don't have to modify our own package).

Hope this helps.

Matthieu

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spack/e811c2d0-be62-4834-9018-bcea4ea1ac4cn%40googlegroups.com.

[Pariksheet Nanda]

Hi Matthieu,

On Wednesday, June 1, 2022 at 6:58:02 AM UTC-4 Matthieu Dorier wrote:

Not sure if it can help, but in my team we also occasionally need to edit packages, to add versions/variants and have access to them before they are upstreamed (or versions that will never be upstreamed, like release candidates), to have some versions pointing to forks of a github repo, and so on. Rather than copying the original package file, we inherit from it. For example this is what we have for the Libfabric package:

from spack.pkg.builtin.libfabric import Libfabric as BuiltinLibfabric

class Libfabric(BuiltinLibfabric):

   version('x',...) # this version is not upstreamed yet

   ... # some other modifications

By doing so you can add version, variants, override the way the package is built, etc. and we also get whatever is added upstream when updating spack (i.e. if a new variant appears upstream, we don't have to modify our own package).

Hope this helps.

That's actually ingenious.  It never occurred to me to work at the python API level.  Thank you for sharing your approach!

Matthieu

Pariksheet