Aborting failed builds

[Thomas M. Payerle]

I have just upgraded from spack 0.14 to 0.16, and I am seeing a new feature in that spack seems to keep trying to rebuild failed builds.  While I can see this being useful in some cases, it can also be quite annoying.

E.g.,  I was building some packages in an environment, and due to a misconfiguration of my packages.yaml, a package tried to build with cray-mpich instead of openmpi (I am not on a cray system and want to use openmpi).  But now whenever I build something, it keeps trying to install that package again (using the problematic spec). 

I.e., I already have hdf5 installed in the environment, but if I issue the command "spack install hdf5", it tries to install the failed version of arpack-ng (using cray-mpich):

devspack:dev-2:~$ spack debug report
* **Spack:** 0.16.1-2432-16111354aa
* **Python:** 3.6.8
* **Platform:** linux-rhel8-x86_64
* **Concretizer:** original
devspack:dev-2:~$ spack clean -a

==> Removing all temporary build stages
==> Removing cached downloads
==> Removing install failure marks
==> Removing cached information on repositories
==> Removing python cache files
==> Removing software in "/home/spackswinst/.spack/bootstrap/store"

devspack:dev-2:~$ spack --env gcc840-ompi315 install hdf5
==> Warning: included configuration files should be updated manually [files=/software/spack/dev-environments/gcc840-ompi315/packages-gcc840-ompi315.yaml, /software/spack/dev-environments/gcc840/packages-gcc840.yaml, /software/spack/dev-environments/common/packages-common.yaml]
==> Warning: Missing a source id for cray-...@8.1.0
[+] /software/spack-software/2021.04.28/linux-rhel8-x86_64/gcc-8.4.0/pkg-config-0.29.2-2ykulq6prynprhyyj6ony4iukwgmjbgn
[+] /software/spack-software/2021.04.28/linux-rhel8-x86_64/gcc-8.4.0/berkeley-db-18.1.40-5xksxkjdqerdnyzbcjz3gaqemjwmhz3j
[+] /software/spack-software/2021.04.28/linux-rhel8-x86_64/gcc-8.4.0/zlib-1.2.11-wwi45nrodqbaui747cuyzg2qyz3ktcdw
==> Installing cray-mpich-8.1.0-f4kdrndvzxbbfjxcznargmumtnj3nssy
==> No binary for cray-mpich-8.1.0-f4kdrndvzxbbfjxcznargmumtnj3nssy found: installing from source
==> Warning: Missing a source id for cray-...@8.1.0
==> No patches needed for cray-mpich
==> cray-mpich: Executing phase: 'install'
==> Error: InstallError: cray-mpich is not installable, you need to specify it as an external package in packages.yaml

/software/spack/git.2021.04.28/var/spack/repos/builtin/packages/cray-mpich/package.py:59, in install:
         57    def install(self, spec, prefix):
         58        raise InstallError(
  >>     59            self.spec.format('{name} is not installable, you need to specify '
         60                             'it as an external package in packages.yaml'))

See build log for details:
  /tmp/spackswinst/spack-stage/spack-stage-cray-mpich-8.1.0-f4kdrndvzxbbfjxcznargmumtnj3nssy/spack-build-out.txt

==> Warning: Skipping build of arpack-ng-3.7.0-egfz6biz6enayuhynblxjokpp3nn26ax since cray-mpich-8.1.0-f4kdrndvzxbbfjxcznargmumtnj3nssy failed
[+] /software/spack-software/2021.04.28/linux-rhel8-x86_64/gcc-8.4.0/ncurses-6.2-f5r24q7skhpnvmzjkksifsqu4fn6f5et
[+] /software/spack-software/2021.04.28/linux-rhel8-x86_64/gcc-8.4.0/readline-8.1-ppc6pwisx4ee7v5ypee5sd7rfvvh46ik
[+] /software/spack-software/2021.04.28/linux-rhel8-x86_64/gcc-8.4.0/gdbm-1.19-dw52ox6pjoytwdz2hutazymd7q5ikl2e
[+] /software/spack-software/2021.04.28/linux-rhel8-x86_64/gcc-8.4.0/perl-5.30.1-czgyu5rwxma55fjbce3glxxe4r7ijzwd
[+] /software/spack-software/2021.04.28/linux-rhel8-x86_64/gcc-8.4.0/openssl-1.1.1k-gfxnuznk6jpiyb6c4gsiveeny6726svp
[+] /software/spack-software/2021.04.28/linux-rhel8-x86_64/gcc-8.4.0/openblas-0.3.9-nfmto7pdvioa3cqfy4pgzatpmk35dgrc
[+] /software/spack-software/2021.04.28/linux-rhel8-x86_64/gcc-8.4.0/cmake-3.16.5-ls4bxwc6alo76et5lx4tumcixzia5jie
==> Error: arpack-ng-3.7.0-egfz6biz6enayuhynblxjokpp3nn26ax: Package was not installed
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

If I try to install a new package foo, the foo install will work, but it always attempts to build the failed arpack-ng.

Is there a way to tell spack to forget about the arpack-ng ^cray-mpich once and for all?

--

Tom Payerle
DIT-ACIGS/Mid-Atlantic Crossroads        pay...@umd.edu

5825 University Research Park               (301) 405-6135

University of Maryland
College Park, MD 20740-3831

[Rob Groner]

If you're using a spack environment (and it sounds like you are), then look in your spack.yaml file.  I would guess it put that failed package spec in there, so every time you tell it spack install, it tries to build everything listed in the spack.yaml.  That's what happens to me anyway.  So just go in and remove that spec and save the spack.yaml file and it should stop trying.

[Thomas M. Payerle]

Thank you for your comment.

I am using environments, and I suspect that that might have something to do with it, as deleting the environment and recreating resolves the issue (but such seems a bit drastic).

I am familiar with the spack behavior that

spack --env foo install

will attempt to install all of the root specs listed in the spack.yaml file for the foo environment.

But I do not believe this is the cause of the problem I am seeing, because in my past experience (admittedly, mainly with sp...@0.16)

i) while it is true 'spack install' in an environment will attempt to install all root specs for the environment if no spec is provided on the command line, it does *not* try to install all root specs if a spec is given on the command line.

ii) when a spec is given to the spack install on the command line, it will add the spec to the root specs for the environment (I _think_ this is only done after a successful install, but I am not certain).  But in my experience the spec added is the spec given on the command line, not the fully concretized spec.

I think I checked the spack.yaml for such before my initial posting, but I deleted and recreated the environment which had the issue with attempting to install arpack-ng with cray-mpich before reading your response, so I cannot confirm.  (Again, deleting and recreating the environment "fixed" the issue, but that seems drastic to me.)  In that case, I never explicitly requested cray-mpich; that was done by concretizer (because I erroneously set all other MPIs to not be buildable in the packages hash for the environment).  But even after I fixed the packages hash, and actually successfully installed arpack-ng with openmpi, every time I try to do a spack install in the environment, after installing the requested package, spack tries again to install the failed package, using the erroneous settings which are no longer in the spack.yaml file.  These leads to a lot of extraneous error messages in spack output, which is annoying and can hide real errors if the desired build was unsuccessful.

And this is not an one-time fluke event.  Just now I had a package install fail because the package was attempting to use the system installed rather than the spack installed python (I believe this is an error in the spack recipe, but that is a separate matter).  In an attempt to fix the issue, I modified the package.py for the problematic package adding an explicit python dependency, which changed the hash (and seems to have resolved the installation issue).  But even though I "fixed" the issue and successfully installed the package in question, whenever I do a spack install in that environment, after installing the packages I specify on the command line, it then proceeds to try to install the problematic package, using the system python despite the fact that the package.py for the package now explicitly depends on the spack-installed python.  I.e., because I editted the package.py for the package, I don't think I can even write a spec for the package that spack is now trying to install, but it is still trying to install it.  This happens even if I give a spack install command for a package that is already installed (it recognized the given package as already installed, then proceeds to install the failed package).

I expect it is something in the spack.lock  or in something under the .spack-env in the environment, but those aren't well-documented and I suspect are not meant for end user consumption

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spack/2d52bf94-30c3-433c-b4a6-8906d1262bban%40googlegroups.com.

[Groner, Rob]

Sounds like you've already tried more things than I would have suggested.  I do recall quite a bit of annoyance at one point because something had gotten into the spack config files in my home directory under .spack (that I don't recall putting there), and so it kept popping up and I couldn't figure out why.  The "spack config --blame" is of SOME use, but wasn't in this case.  So deleting the ~/.spack dir helped me.  

The only other thing I can suggest is to use the -d flag with spack for debug, and -vvv for install for verbose, and then try to parse the mountain of output to see if you can figure out where Spack is making that decision from.  I then usually go in and find the python responsible in spack/lib/spack and add my own output to see what's really going on.

---

From: Thomas M. Payerle <pay...@umd.edu>
Sent: Tuesday, May 4, 2021 4:01 PM
To: Groner, Rob <rug...@psu.edu>
Cc: Spack <sp...@googlegroups.com>
Subject: Re: [spack] Re: Aborting failed builds

 

[Gamblin, Todd]

Does running:

spack concretize —force

Help here? That *should* remove any existing concrete specs from your spack.lock and re-concretize everything, which I believe will get rid of this issue.

-Todd

To view this discussion on the web visit https://groups.google.com/d/msgid/spack/CAHJ2ZQ-AZa%2Bvv5M2ZtRLP1KqcwktrhpL5UYkmMnZ6aszY0SUPA%40mail.gmail.com.

[Gamblin, Todd]

Great!  Thanks for following up on this.

We still shouldn’t be retrying endlessly — I think there is another underlying bug in here, but at least you do not have to start from scratch.

On May 10, 2021, at 2:09 PM, Thomas M. Payerle <pay...@umd.edu> wrote:

Sorry about the delay in responding.

spack concretize -f

does seem to resolve the issue.

To view this discussion on the web visit https://groups.google.com/d/msgid/spack/95AA956C-9481-44AB-8B6F-DF58E175B693%40llnl.gov.