Differences between Spack pip and system's local pip

[Itaru Kitayama]

Hi,

Can we safely mix Python packages installed either Spack or system's pip?

[Gamblin, Todd]

You can’t install Spack packages in a pip environment (that I know of  — I suppose you could try to point a Spack view there but I think it’ll be awkward).  You can use pip within a spack environment — Python is linked into the view in a similar way to how virtualenv works, so pip will assume that Python lives in the view.  

Note that when the view is regenerated (on concretization and after each installation), we currently destroy the view directory and re-create it. So you’ll need to reinstall your pip packages.

On May 7, 2021, at 12:18 AM, Itaru Kitayama <itaru.k...@gmail.com> wrote:

Hi,

Can we safely mix Python packages installed either Spack or system's pip?

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spack/073d6f4c-c944-4502-8579-b6be48fc163an%40googlegroups.com.

[Elizabeth Fischer]

Note that when the view is regenerated (on concretization and after each installation), we currently destroy the view directory and re-create it. So you’ll need to reinstall your pip packages.

BTW, this behind-the-scenes regeneration of views, multiple times during the process of building a Spack environment, is a royal PITA.

-- Elizabeth

[Gamblin, Todd]

How would you change it?

You can disable the view entirely with 

view: False

---
Sent from Workspace ONE Boxer

[Elizabeth Fischer]

On Sat, May 8, 2021 at 2:09 PM Gamblin, Todd <gamb...@llnl.gov> wrote:

How would you change it?

Think of how we write and use an individual Spack package:

1. Write mypackage/package.py

2. Run "spack install mypackage"

3. Deal with the errors, iterate step 2 until it works.

4. Run "spack load mypackage"... then use and enjoy!!

5. Later on, an update is needed on the package.  So edit package.py and repeat steps 2-4.

Now what is a Spack environment?  It's simply a set of Spack packages that are used and (frequently) concretized together.  So here is how I would like to write and use a Spack environment.  It's really the

1. Write mypackage/spack.yaml.  [NOTE: I DO NOT want Spack to rewrite my spack.yaml source code.  Does it overwrite package.py?   No!  So why should it overwrite spack.yaml?]

2. Run "spack env concretize" [If the new concretizer is fast enough, maybe this no longer needs to be a separate step]

3. Run "spack env install"

4. Iterate on 1..3 (and also on fixing individual packages) until step 3 works.

5. "Render" the environment.  This should be a matter of either:

   (a) Create a file we can "source" (or a single mega-module) to load env vars correctly.   or...

   (b) Link all the environment's packages into a view, and create a file we can "source" (or a single module) to load env vars correctly.  The need for setting env vars is GREATLY reduced with view vs. modules; but some env vars still need to be set, for a few packages.  Plus, PATH, PYTHONPATH, etc. needs to be set to the view.

NOTE: The choice of modules vs. view should not be an intrinsic part of the environment, and therefore should not be in spack.yaml.  It should simply be a choice one makes at the end, whether to render as modules or view (or both).  And if rendering as a view, we should give the location of where we want the view rendered.

6. Load the environment by sourcing the file created in step 4.

Spack Environments as they are currently formulated (AFAIK) are all wrong for this way of working:

1. Spack wants a "current" environment, which makes no sense here.  There is no "current" package while we are debugging a recipe.

2. Spack likes to overwrite my spack.yaml file.  Yuck.  Doesn't play well with git, among other things.

3. If a view is involved, spack wastes enormous amounts of time rebuilding the view while I'm still trying to debug the whole thing.  The view only needs to be created at the end, when I'm ready to actually try it out.

4. Spack makes the view/module thing an intrinsic feature of the environment.  which it shouldn't be.

5. Spack modules are a "hack" for environments.  If you have a Spack environment, you'll never be loading modules one by one.  Just make a single module to do it all.  Or just a script you source; because we're not going to load /unload individual modules when we have an environment set up.

6. Spack fails to give us any help setting env vars if we choose a view.  But some env vars still need to be set, and it's not OK to have to do this manually.

I've hacked around all these issues.  But it involves a lot of ugly hacks, ugly extra scripts, etc.  Not fun at all, and not something that can be easily shared or recommended with others.

I have ZERO interest in the idea that an environment is something you interact with and build incrementally on the command line.  All the stuff that gives you a "current" environment, auto-updates the spack.yaml file when you say "spack install", etc. etc. is all NOT what I want.  It's the wrong use case model.

The current Spack environments are so NOT how I want to use them, that I imagine creating a new set of Spack environment commands that do this stuff the way I need.  Let's call it "batch environments" or something, instead of "interactive environments" which is what we have now.

-- Elizabeth

.. Elizabeth

--

Elizabeth Ashley Fischer, she/her

University of Alaska, Geophysical Institute

cell: 617-308-0436

UA is an AA/EO employer and educational institution and prohibits illegal discrimination against any individual: www.alaska.edu/nondiscrimination.

[Groner, Rob]

Just to butt-in on this conversation....I completely agree with hating that the spack.yaml file gets edited for me.  I've numerous reasons why, but I'll just leave it at that.

---

From: sp...@googlegroups.com <sp...@googlegroups.com> on behalf of Elizabeth Fischer <eafis...@alaska.edu>
Sent: Saturday, May 8, 2021 6:36 PM
To: Gamblin, Todd <gamb...@llnl.gov>
Cc: Itaru Kitayama <itaru.k...@gmail.com>; Spack <sp...@googlegroups.com>
Subject: Re: [spack] Differences between Spack pip and system's local pip

 

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/spack/CAHb-Use43EVbsfniEtjiOT3u4YUhGQCVf9bGJoP8Rvd-MJL%3DSg%40mail.gmail.com.

[Thomas M. Payerle]

I too dislike the automatic editing of spack.yaml in environments, especially since I like to have comments re the package definitions.

FYI, I have discovered, however, that the editing of spack.yaml does not extend to files "included" in the spack.yaml, so I normally get around this automatic editing by creating relatively simply spack.yaml files consisting basically of

spack:

   spec:

      - root-spec1

     ...

     - root-specN

   view: ...

   include:

     - include-file1

     ...

and put all the 'packages' definitions in the include-file1, which spack does not overwrite.

To view this discussion on the web visit https://groups.google.com/d/msgid/spack/BL0PR02MB44993CF8035410E349177F6E80549%40BL0PR02MB4499.namprd02.prod.outlook.com.

--

Tom Payerle
DIT-ACIGS/Mid-Atlantic Crossroads        pay...@umd.edu

5825 University Research Park               (301) 405-6135

University of Maryland
College Park, MD 20740-3831

[Groner, Rob]

Oh, that's good to know.  Yes, the mangling of comment lines was one of my gripes.

Rob

---

From: Thomas M. Payerle <pay...@umd.edu>
Sent: Monday, May 10, 2021 9:37 AM
To: Groner, Rob <rug...@psu.edu>
Cc: Elizabeth Fischer <eafis...@alaska.edu>; Gamblin, Todd <gamb...@llnl.gov>; Itaru Kitayama <itaru.k...@gmail.com>; Spack <sp...@googlegroups.com>

[Elizabeth Fischer]

As I said... we're finding ways to hack around Spack Environment's anti-features.

-- Elizabeth