Avoid building newer versions as mush as can

90 views
Skip to first unread message

Itaru Kitayama

unread,
Apr 16, 2021, 8:06:27 PM4/16/21
to Spack
Hi,
It seems Spack initiates a build whenever there's a new version for
one of the dependents, how do I avoid that unless specifically required?
My motivation is that so I can keep inode usage minimal.

Rob Groner

unread,
Apr 21, 2021, 10:28:02 AM4/21/21
to Spack
Not sure if this answers your question or not, but we use a spack environment to "freeze" the versions of everything, so updates don't happen unless we want it to.  You might want to look into those to see if that meets your needs.

Itaru Kitayama

unread,
Apr 21, 2021, 8:29:14 PM4/21/21
to Rob Groner, Spack
Hi Rob, all,
Thanks for your comment. How do I freeze them in practice? I have actively maintaining Spac specs locally on the GPFS.

--
You received this message because you are subscribed to a topic in the Google Groups "Spack" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/spack/SlbfzL0vY1Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to spack+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spack/8209d394-b064-413d-b7d7-b80e3307975cn%40googlegroups.com.

Patrick E Gartung

unread,
Apr 21, 2021, 8:32:34 PM4/21/21
to Itaru Kitayama, Rob Groner, Spack

You can use the hash of the already installed packages for the dependencies , i.e.

 

Spack install packageA^/hashB^/hashC^/hashD

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spack/CANW9uytjLsBUQ2nigTWUUKtJtEatLz9B-snbGfN%3Dw3Y7oJ3syA%40mail.gmail.com.

Itaru Kitayama

unread,
Apr 21, 2021, 8:35:43 PM4/21/21
to Patrick E Gartung, Rob Groner, Spack
Patrick,
I feel it is a bit tedious, can we do that in config?

Patrick E Gartung

unread,
Apr 21, 2021, 8:38:01 PM4/21/21
to Itaru Kitayama, Rob Groner, Spack

Yes, you can specify the version and variants in the config file for all installed packages. This should force the concretizer to use the installed packages.

Kevin Huck

unread,
Apr 21, 2021, 9:10:21 PM4/21/21
to Patrick E Gartung, Itaru Kitayama, Rob Groner, Spack
I ran into this today...whenever I pull from Spack github and reinstall a development build, I get a new cmake installation.  I now have the 5 most recent cmake versions... but my package only depends_on(‘cm...@3.10.0:).  Can’t the concretizer decide my needs are met with 3.18, without installing 3.20?

Thanks -
Kevin

On Apr 21, 2021, at 5:38 PM, Patrick E Gartung <gar...@fnal.gov> wrote:



Patrick E Gartung

unread,
Apr 21, 2021, 10:36:54 PM4/21/21
to Kevin Huck, Itaru Kitayama, Rob Groner, Spack

That’s on the roadmap for v0.17 according to Todd.

Itaru Kitayama

unread,
Apr 21, 2021, 10:40:25 PM4/21/21
to Patrick E Gartung, Kevin Huck, Rob Groner, Spack
Until that happens, we have to Spack about all the packages version we
want to be integrated during build?

Patrick E Gartung

unread,
Apr 21, 2021, 10:43:28 PM4/21/21
to Itaru Kitayama, Kevin Huck, Rob Groner, Spack

Yes. That is one of the ways considered at Fermilab for fixing the versions in a software stack. The config file would be under version control and you check out the tag for the software stack with the versions you want.


> To unsubscribe from this group and all its topics, send an email to spack+un...@googlegroups.com.


>
> --
> You received this message because you are subscribed to the Google Groups "Spack" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.


>
> --
> You received this message because you are subscribed to the Google Groups "Spack" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.

Itaru Kitayama

unread,
Apr 21, 2021, 10:46:03 PM4/21/21
to Patrick E Gartung, Kevin Huck, Rob Groner, Spack
I'm on a shared filesystem and have hard upper limit of inode usage;
that's one of the reasons I'd like to avoid introducing new versions
whenever possible.

Gamblin, Todd

unread,
Apr 22, 2021, 10:36:51 AM4/22/21
to Itaru Kitayama, Patrick E Gartung, Huck, Kevin, Rob Groner, Spack
Hey folks: this feature (optimizing for reuse of installed dependencies) is the most wanted feature from our recent spack user survey. See here:


It’s been the most wanted feature for a while, and with the new concretizer becoming default in 0.17, we’ll be able to address it then. 

I think you can expect that in 0.17, by default, installs will try to reuse what’s already installed on your machine. Likewise, by default we’ll prefer stuff in buildcaches to concretizing the latest thing from package files.  So there will be a lot less rebuilding.

That said, we’ll also keep an option to keep the current “deterministic” (according to the state of package files in the repository) installation method. There are still users who want this because it’s not dependent on what is installed in your particular spack installation — and it gives you the same installation for the same spack commit, regardless of what you’ve installed.

We don’t want to lose determinism, but we do want to allow people to effectively use what’s installed on their machine, so we’ll offer them both as options.

Here’s an example to think about. One kind of cool/kind of not cool thing about optimizing for reuse is that it means installing packages in a different order will give you different results. E.g., if you first install mpich, and you don’t have any MPI provider prefs in packages.yaml, spack will reuse mpich as the MPI for subsequent installs. 

So it enables you to build a stack constructively and avoid typing long specs, which is good. It can also be confusing if you tell someone that you got one thing when you said “spack install hdf5” but someone else got a different build for the same command. 

There are also questions around precedence for concretization preferences. Should we respect packages.yaml or environment configuration before choosing to reuse existing installs? Or should we always prefer to reuse? We’ll probably need some options to allow users to order preferences like this.

Anyway, looking forward to 0.17, as I think a lot of people would like this feature.  But keep in mind that it does make your build less deterministic. I think it’ll depend on the use case whether it’s actually what you want, so having an option is important.

Todd


---
Sent from Workspace ONE Boxer


> > To unsubscribe from this group and all its topics, send an email to spack+un...@googlegroups.com.

> >
> > --
> > You received this message because you are subscribed to the Google Groups "Spack" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.

> >
> > --
> > You received this message because you are subscribed to the Google Groups "Spack" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.

George Hartzell

unread,
Apr 22, 2021, 5:40:22 PM4/22/21
to Gamblin, Todd, Itaru Kitayama, Patrick E Gartung, Huck, Kevin, Rob Groner, Spack
'Gamblin, Todd' via Spack writes:
> [...]
> That said, we’ll also keep an option to keep the current
> “deterministic” (according to the state of package files in the
> repository) installation method. There are still users who want
> this because it’s not dependent on what is installed in your
> particular spack installation — and it gives you the same
> installation for the same spack commit, regardless of what you’ve
> installed.
> [...]

Just to throw my vote onto one side of the see-saw, Spack's
determinism is absolutely critical for our use cases.

I'm glad to hear that it won't be going out the window.

Thanks!

g.
Reply all
Reply to author
Forward
0 new messages