Any committers interested in Singularity update?

64 views
Skip to first unread message

George Hartzell

unread,
May 2, 2019, 5:21:40 PM5/2/19
to sp...@googlegroups.com

Hi All,

@vsoch submitted a PR to update Singularity to the new, Go-based,
Singularity v3.0.

https://github.com/spack/spack/pull/11094

It's languishing and I'd love to see it get merged.

There's a very long conversation thread, but it's mostly us trying to
figure out a bug that I had written and was effectively blind to. The
PR is ready to go as is. If/when something like #11331 or #10412 is
merged, this package can take advantage of it, but there's no reason
to wait.

Thanks,

g.

ps, a bit more detail...

The only thing unusual about the package is that it extends the
`do_stage` step and moves the source directory into
`src/github.com/sylabs/singularity` after unpacking it (and sets
GOPATH) so that the Go machinery works as expected.

There's a long bit in the conversation about problems she was seeing
that I wasn't; in a nutshell I was nervously creating
`src/github.com/sylabs/singularity` before calling `shutils.move` and
that was messing things up when we called `source_path` in the
arguments to `shutil.move`. Whether it succeeded or failed depended
on the order things were returned in a directory listing.

Eventually @vsoch figured it out and I achieved understanding. I
appreciate her use of the Nerf clue stick.

v

unread,
May 2, 2019, 5:36:54 PM5/2/19
to Spack
+1! :)

When this PR is merged I'd be interested to chat about how we can include Singularity builds for any software. The general idea would be that spack uses it's dependency graph to build the same software... but in a container. I started thinking about this today, figured out the basics to add a command, but had some basic questions about paths / function for spack (noob here!) https://github.com/spack/spack/issues/11349 Apparently, with a Nerf clue stick (@hartzell what is that?)

Greg Burgreen

unread,
May 2, 2019, 5:44:49 PM5/2/19
to v, Spack
Spack + Singularity is the way to go (or at least, that is the direction I am heading).


--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at https://groups.google.com/group/spack.
For more options, visit https://groups.google.com/d/optout.

v

unread,
May 2, 2019, 5:46:50 PM5/2/19
to Greg Burgreen, Spack
🦖🥑❤️🎉!!
--
Vanessa Villamia Sochat
Stanford University '16

George Hartzell

unread,
May 2, 2019, 7:32:25 PM5/2/19
to v, Spack

[resent, including the group]

@v, you might be interested in @alalazo's "Build farm in a box", https://github.com/spack/spack/pull/7204

[Here's a def'n of a clue
stick](https://idioms.thefreedictionary.com/clue+stick) (aka "clue
bat").

The Nerf version is used amongst friends and cow-orkers; it's
analogous to various other safe-for-the-workplace weaponry.

g.
> > `src/github.com/sylabs/singularity` <http://github.com/sylabs/singularity>
> > after unpacking it (and sets
> > GOPATH) so that the Go machinery works as expected.
> >
> > There's a long bit in the conversation about problems she was seeing
> > that I wasn't; in a nutshell I was nervously creating
> > `src/github.com/sylabs/singularity` <http://github.com/sylabs/singularity>
> > before calling `shutils.move` and
> > that was messing things up when we called `source_path` in the
> > arguments to `shutil.move`. Whether it succeeded or failed depended
> > on the order things were returned in a directory listing.
> >
> > Eventually @vsoch figured it out and I achieved understanding. I
> > appreciate her use of the Nerf clue stick.
> >
>

Joseph Schoonover

unread,
May 3, 2019, 8:56:52 AM5/3/19
to hart...@alerce.com, v, Spack
This would be amazing

v

unread,
May 3, 2019, 10:50:01 AM5/3/19
to hart...@alerce.com, Spack
This is great! I'll futz around with trying the same with Singularity. To answer this question:

Docker seems to me more flexible and more widely used outside of the HPC world (e.g. services like Travis support docker images, I doubt they will add support for singularity images) + from Singularity homepage:

This is accurate that Docker isn't fit for HPC - a container technology with a root daemon is a non starter. But it's pretty great for enterprise (cloud, non HPC) use. Also, if anyone needs Travis recipes for building Singularity, I've created those here (2.* family in a different branch). The Singularity docs are also now here - the company Sylabs was founded a little over a year ago, and many of the original developers (myself not included) signed on. 

v

unread,
May 3, 2019, 2:26:46 PM5/3/19
to hart...@alerce.com, Spack
Hey everyone,

Your wish is my command! Here is a simple command group, "singularity" that will generate a recipe for you, complete with spack and one or more packages of interest (or I think you call them specs)

$ spack singularity recipe --from ubuntu:16.04 curl

That will sploot out a recipe file with an ubuntu:16.04 base, with curl installed. You can list as many packages as you want, I just was too impatient to wait for more than one :) 

If you are able to be sudo, you can generate a recipe and build it. By default it will build in the same directory as the recipe is written to in /tmp, and we can have discussion around customizing this (for example, why not allow spack to have a location for installed containers and the recipes that generated them? Here is what that command looks like:

$ sudo bin/spack singularity build --from ubuntu:16.04 curl
To show you the recipe (and then a manual build using the command printed) here is an asciinema: https://asciinema.org/a/244183?speed=3

And there are lots of questions / discussion to be had! I want to note that I chose the simplest solution (no additional dependencies needed to install) because I think this is best, but of course there might be other ways. I also haven't written tests yet (smirk). Regardless, we have a start! Jump over to the PR to join in :)


Best,

Vanessa
Reply all
Reply to author
Forward
0 new messages