spack find not showing manually installed packages

[Mark Miller]

I had a couple of packages that failed to install with 'spack install'.

I wound up doing the spack cd pkg, spack env pkg tcsh thing, adjusting CMakeCache.txt, re-cmaking and then 'make install' there and it succeeded.

I can see the packages in the 'opt' dir.

But, spack find doesn't show they are there.

mcmiller@vestalac1 metis-5.1.0]$ spack env metis tcsh

[mcmiller@vestalac1 metis-5.1.0]$ spack install metis

==> metis is already installed in /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2015/spack/opt/spack/powerpc64/gcc-4.4.7-mpi/metis-5.1.0-6d3gprsxspo5amq4m3pwcy2ptvq7vw4s.

[mcmiller@vestalac1 metis-5.1.0]$ spack find

==> 6 installed packages.

-- powerpc64 / g...@4.4.7-mpi ------------------------------------

cblas@2015-06-06  hd...@1.8.13  hy...@2.10.0b  lap...@3.5.0  netli...@3.5.0  zl...@1.2.8

[mcmiller@vestalac1 metis-5.1.0]$ make install

[ 75%] Built target metis

[ 79%] Built target cmpfillin

[ 83%] Built target gpmetis

[ 86%] Built target graphchk

[ 90%] Built target m2gmetis

[ 95%] Built target mpmetis

[100%] Built target ndmetis

Install the project...

-- Install configuration: "RelWithDebInfo"

-- Up-to-date: /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2015/spack/opt/spack/powerpc64/gcc-4.4.7-mpi/metis-5.1.0-6d3gprsxspo5amq4m3pwcy2ptvq7vw4s/include/metis.h

-- Up-to-date: /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2015/spack/opt/spack/powerpc64/gcc-4.4.7-mpi/metis-5.1.0-6d3gprsxspo5amq4m3pwcy2ptvq7vw4s/lib/libmetis.a

-- Up-to-date: /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2015/spack/opt/spack/powerpc64/gcc-4.4.7-mpi/metis-5.1.0-6d3gprsxspo5amq4m3pwcy2ptvq7vw4s/bin/gpmetis

-- Up-to-date: /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2015/spack/opt/spack/powerpc64/gcc-4.4.7-mpi/metis-5.1.0-6d3gprsxspo5amq4m3pwcy2ptvq7vw4s/bin/ndmetis

-- Up-to-date: /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2015/spack/opt/spack/powerpc64/gcc-4.4.7-mpi/metis-5.1.0-6d3gprsxspo5amq4m3pwcy2ptvq7vw4s/bin/mpmetis

-- Up-to-date: /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2015/spack/opt/spack/powerpc64/gcc-4.4.7-mpi/metis-5.1.0-6d3gprsxspo5amq4m3pwcy2ptvq7vw4s/bin/m2gmetis

-- Up-to-date: /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2015/spack/opt/spack/powerpc64/gcc-4.4.7-mpi/metis-5.1.0-6d3gprsxspo5amq4m3pwcy2ptvq7vw4s/bin/graphchk

-- Up-to-date: /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2015/spack/opt/spack/powerpc64/gcc-4.4.7-mpi/metis-5.1.0-6d3gprsxspo5amq4m3pwcy2ptvq7vw4s/bin/cmpfillin

[mcmiller@vestalac1 metis-5.1.0]$ spack find

==> 6 installed packages.

-- powerpc64 / g...@4.4.7-mpi ------------------------------------

cblas@2015-06-06  hd...@1.8.13  hy...@2.10.0b  lap...@3.5.0  netli...@3.5.0  zl...@1.2.8

[mcmiller@vestalac1 metis-5.1.0]$ ls /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2015/spack/opt/spack/powerpc64/gcc-4.4.7-mpi/

cblas-2015-06-06-44d2jufongtd7surxuxovglplr3r5cgj  metis-5.1.0-6d3gprsxspo5amq4m3pwcy2ptvq7vw4s

hdf5-1.8.13-myvlq4zc5axi72xrqguk6nxocibq5ida       netlib-blas-3.5.0-gxwqxkcdgbiuj5sqpoccdd2vbyytmfs3

hypre-2.10.0b-wdfd3csffrkhdcmxpd52qahka6zqgdlh     parmetis-4.0.3-zqp4jk3lc26fo7okduqteyysan23olzl

lapack-3.5.0-jqvtkxgdos4ozunxyl3lvpjwzbi2gren      zlib-1.2.8-2tlgo2lduu3cbmvqq3w6ljnpywh75vt5

[mcmiller@vestalac1 metis-5.1.0]$ 

[Todd Gamblin]

Mark,

`spack install` creates a .spack directory, which contains a spec.yaml file.  This contains all the metadata about each build, and there is also some other provenance information about the package in there.

Spack needs this file in order for `spack find` to locate a package.  If you install manually, .spack doesn't get created with all that information, and Spack can't know what libraries the package was built with, which makes it hard to use as a dependency of *other* packages.  

Generally if you modify CMakeCache.txt, you can get that into the package file through -D args to CMake right?  The right way to do this would be to get the package file into shape so you could actually reproduce the build later...

Do you think that Spack *should* understand arbitrary manually installed packages?  I don't.

-Todd

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at http://groups.google.com/group/spack.
For more options, visit https://groups.google.com/d/optout.

[Todd Gamblin]

> Do you think that Spack *should* understand arbitrary manually installed packages?  I don't.

I'll add a caveat to this.  Matt is already implementing a way to register packages that are *external* to Spack, and have it build internal packages with them as dependencies.  However, this wasn't intended for packages that are manually installed from within a spack build.  The package you're installing goes into $spack/opt/spack, which I think would be confusing for users, because it's really NOT a spack-managed package -- it's a tweaked one.

The way we're planning to support manually built packages is through externals, where you would give Spack a prefix *outside* its own install location, and tell Spack enough about the library that it can link it with others.  We'll have more on this once Matt finished the feature.

-Todd

[Mark Miller]

On Thursday, July 30, 2015 at 12:54:40 AM UTC-7, Todd Gamblin wrote:

Mark,

`spack install` creates a .spack directory, which contains a spec.yaml file.  This contains all the metadata about each build, and there is also some other provenance information about the package in there.

Just curious but is this a relatively easy file to jin up if one knows what they're doing?

 

Spack needs this file in order for `spack find` to locate a package.  If you install manually, .spack doesn't get created with all that information, and Spack can't know what libraries the package was built with, which makes it hard to use as a dependency of *other* packages.  

Generally if you modify CMakeCache.txt, you can get that into the package file through -D args to CMake right?

Yes and no. IME, you often windup fighting with the build tools to get everything to work from the tool's command-line. CMakeLists.txt files and configure.in files do processing that can wind up undoing or getting flubbed by attempts to feed stuff through via CL args.

 

The right way to do this would be to get the package file into shape so you could actually reproduce the build later...

Agreed

 

Do you think that Spack *should* understand arbitrary manually installed packages?  I don't.

Not necessarily. But, it might be useful to be able to 'import' a manually installed package into a Spack managed install point. Maybe the resulting package is more limited in how it behaves in Spack-land there as a result and that would be fine. But, having it in the *same place* as everything else would be convenient.

Something related (I don't know if this has been discussed) but what about proprietary packages for which maybe only binaries are available. We have a couple of these in VisIt. We don't get sources. We get pre-compiled binaries for os x, windows and Linux and we've got to just use the right ones in the right places. Currently, I don't think we deal with too many package dependencies in this case but, if so, the company would build their binaries using the same dependent package versions and compiler(s) as build_visit would use. Is Spack able to deal with that? It seems like a similar problem. The package is compiled outside of Spack but kinda needs to join in the Spack game. Just a curiosity at this point.

[Todd Gamblin]

Mark,

Resurrecting and older email from you that I neglected to reply to.  Sorry!

From: Mark Miller <markcm...@gmail.com>
Date: Thursday, July 30, 2015 at 10:46 AM
To: Spack <sp...@googlegroups.com>
Cc: Todd Gamblin <tgam...@llnl.gov>
Subject: Re: [spack] spack find not showing manually installed packages

On Thursday, July 30, 2015 at 12:54:40 AM UTC-7, Todd Gamblin wrote:

Mark,

`spack install` creates a .spack directory, which contains a spec.yaml file.  This contains all the metadata about each build, and there is also some other provenance information about the package in there.

Just curious but is this a relatively easy file to jin up if one knows what they're doing?

Yes.

Spack needs this file in order for `spack find` to locate a package.  If you install manually, .spack doesn't get created with all that information, and Spack can't know what libraries the package was built with, which makes it hard to use as a dependency of *other* packages.  

Generally if you modify CMakeCache.txt, you can get that into the package file through -D args to CMake right?

Yes and no. IME, you often windup fighting with the build tools to get everything to work from the tool's command-line. CMakeLists.txt files and configure.in files do processing that can wind up undoing or getting flubbed by attempts to feed stuff through via CL args.

filter_file should help here... I think I've shown you a few builds that use that, but this email is from a while ago.

Do you think that Spack *should* understand arbitrary manually installed packages?  I don't.

Not necessarily. But, it might be useful to be able to 'import' a manually installed package into a Spack managed install point. Maybe the resulting package is more limited in how it behaves in Spack-land there as a result and that would be fine. But, having it in the *same place* as everything else would be convenient.

Matt's actually almost done with support for linking with external packages, which should help here.  Right now it's basically "use this MPI in its current location" so you have to tell Spack about where the external package lives.  What I would like to have is custom detectors for these kinds o things, maybe per-architecture or per type of MPI, so that they work kind of like compilers.  That's a ways off, but at least you can use them now.

If you want specifics (they get fairly involved w.r.t. concretization) then we can sit down with Matt sometime.

The good news is that Matt's external package support will help a lot with using MPI in a consistent way across packages.

Something related (I don't know if this has been discussed) but what about proprietary packages for which maybe only binaries are available. We have a couple of these in VisIt. We don't get sources. We get pre-compiled binaries for os x, windows and Linux and we've got to just use the right ones in the right places. Currently, I don't think we deal with too many package dependencies in this case but, if so, the company would build their binaries using the same dependent package versions and compiler(s) as build_visit would use. Is Spack able to deal with that? It seems like a similar problem. The package is compiled outside of Spack but kinda needs to join in the Spack game. Just a curiosity at this point.

Matt, because he is pretty awesome, was telling me that he has some rudimentary compiler ABI matching rules in there.  So you can register an external package and say what compiler it was built with, and Spack can be loose or strict (according to preferences) with how it matches compilers.  So, e.g., he's got it so you can say you want to allow gccs with the same major and minor versions to match, regardless of what the patch version is.  This is all in the "coming soon" category but I thought I would mention it, as it would let you, say,  register a factory MPI installation and build against it with a number of different compilers safely.

-Todd