Updated the Singularity FAQs

[Gregory M. Kurtzer]

I updated the FAQ entries and layout. Please take a look at it and let me know if I messed anything up or if there are any other good questions to add.

http://singularity.lbl.gov/#faq

Thanks!

--

Gregory M. Kurtzer
High Performance Computing Services (HPCS)
University of California
Lawrence Berkeley National Laboratory
One Cyclotron Road, Berkeley, CA 94720

[Dave Love]

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

> I updated the FAQ entries and layout. Please take a look at it and let me
> know if I messed anything up or if there are any other good questions to
> add.

[Why, oh why do we have to use JavaScript to view text?]

* One obvious question is "why the name?". To me it might mean
mathematical ill-behaviour or something that may always be hidden by
cosmic censorship.

* I don't know if it makes sense to compare with Snappy and Flatpak
packaging.

* If it's actually possible simply to import a Docker image, maybe say
so in the Docker or Shifter items? Another important point about
Docker in HPC is that the containers are not launched by the resource
manager, so it can't directly manage them.

* I'd be inclined to mention somewhere the difference between
"containers" as application packaging and operating system containers
of old (zones, jails, WPARs, maybe OpenVz/LXC...).

[Gregory M. Kurtzer]

On Wed, Jun 15, 2016 at 9:03 AM, Dave Love <d.l...@liverpool.ac.uk> wrote:

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

> I updated the FAQ entries and layout. Please take a look at it and let me
> know if I messed anything up or if there are any other good questions to
> add.

[Why, oh why do we have to use JavaScript to view text?]

Haha, because it is the future^H^H^H^H^H^Hpresent of html technology? ;-)

 

* One obvious question is "why the name?".  To me it might mean
  mathematical ill-behaviour or something that may always be hidden by
  cosmic censorship.

Ahh, ok. I'll add that!

 

* I don't know if it makes sense to compare with Snappy and Flatpak
  packaging.

I'm not familiar with Snappy, but I've read through the Flatpak pages so I can take a stab at it. But if you gawk at what I write, please feel free to fix it. lol

 

* If it's actually possible simply to import a Docker image, maybe say
  so in the Docker or Shifter items?  Another important point about
  Docker in HPC is that the containers are not launched by the resource
  manager, so it can't directly manage them.

Good points!

 

* I'd be inclined to mention somewhere the difference between
  "containers" as application packaging and operating system containers
  of old (zones, jails, WPARs, maybe OpenVz/LXC...).

Other good points!

I will work on these thank you!

Greg

[Gregory M. Kurtzer]

Updated! Please let me know what you think!

[Rémy Dernat]

Hi,

Concerning flatpak (http://flatpak.org/), you wrote "flatbak", and the pop-up does not work on the website actually.

Now, I think, there are other techs that could be compare in the faq:

- app-image: http://appimage.org/

- rkt: https://coreos.com/rkt (you can also report to rkt-vs-others: https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.html )

- orbital-app: https://www.orbital-apps.com/blog/category/orbital-apps-com

- runC: https://runc.io/

- snappy: https://developer.ubuntu.com/en/snappy/

Another question that could be added ? Can I run nested container within singularity ? (singularity in singularity or another container techs within singularity). It is used by lxd/lxc to run docker images as non-root user. I think you can imagine some other use case.

Best,

Rémy

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Rémy Dernat]

I forget to say that there are other techs compared in the webpage rkt-vs-other (systemd-nspawn, machinectl...). But I think, you can also just refer to this page in the Faq.

For runC, as you already compared runC with singularity in the mailing list, this would be easy.

Cheers.

[Adam Huffman]

Might be worth adding the question about GPUs, that was asked on IRC
the other day.

Cheers,
Adam

[Dave Love]

Rémy Dernat <rem...@gmail.com> writes:

> Hi,
>
> Concerning flatpak (http://flatpak.org/), you wrote "flatbak", and the
> pop-up does not work on the website actually.

It's the future^Wpresent...

> Now, I think, there are other techs that could be compare in the faq:
> - app-image: http://appimage.org/
> - rkt: https://coreos.com/rkt (you can also report to rkt-vs-others:
> https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.html )
> - orbital-app: https://www.orbital-apps.com/blog/category/orbital-apps-com
> - runC: https://runc.io/
> - snappy: https://developer.ubuntu.com/en/snappy/

Yes, I'd forgotten about some of the other things like that which might
have been appropriate to mention. I don't know much about them, but it
seems from discussion I've seen that flatpak is only for graphical
applications, unlike snappy.

There are clearly too many of these things...

The crucial general point of comparison from my point of view is how the
processes are started, i.e. by privileged daemon access or as a normal
process, potentially properly controlled by, say, an HPC resource
manager.

I'd be inclined to write generally in terms of separate concerns:
"containing" the necessary components required to run something
c.f. maybe "containing" the behaviour of the processes when it runs.

> Another question that could be added ? Can I run nested container within
> singularity ? (singularity in singularity or another container techs within
> singularity). It is used by lxd/lxc to run docker images as non-root user.
> I think you can imagine some other use case.

Right, related to the separation of concerns and how things compose. As
a simple example you could consider running singularity under, say,
bubblewrap <https://github.com/projectatomic/bubblewrap> (ignoring
singularity --contain).

[Gregory M. Kurtzer]

Oh goodness, these are all great ideas but most are outside my reach (with the exception of GPUs). So, I ask for help... Can people either send the question and answer to me, and I will throw it into the FAQ, or send me a GitHub pull request that has the updates?

With that said, I will work on the GPU question and will fix the typos related to Flatpak.

Thanks!!

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Dave Love]

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

> Oh goodness, these are all great ideas but most are outside my reach (with
> the exception of GPUs). So, I ask for help... Can people either send the
> question and answer to me, and I will throw it into the FAQ, or send me a
> GitHub pull request that has the updates?

I think it would help to have context for comparisons, which might allow
tabulating comparisons, such as packaging vs. sandboxing, privileged
v. unprivileged, daemonless v. daemonful, server v. desktop, mutable
v. immutable, nestable v. not...

I could have a go. Is anyone else working on it, or likely to?

I think it also needs something on MPI, like "Can I/How do I run
something in a container built against a different MPI than a system one
outside the container". The implication seems to be that you can,
surprisingly. I had a brief look, knowing a bit about openmpi, and it's
not clear to me how you can at all straightforwardly, or what the
integration in openmpi is all about, especially as that seems to be for
an obsolete singularity. [I didn't expect to find openmpi documentation
on a new feature like that, or even a new MCA framework, of course.]

[Gregory M. Kurtzer]

On Monday, June 20, 2016, Dave Love <d.l...@liverpool.ac.uk> wrote:

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

> Oh goodness, these are all great ideas but most are outside my reach (with
> the exception of GPUs). So, I ask for help... Can people either send the
> question and answer to me, and I will throw it into the FAQ, or send me a
> GitHub pull request that has the updates?

I think it would help to have context for comparisons, which might allow
tabulating comparisons, such as packaging vs. sandboxing, privileged
v. unprivileged, daemonless v. daemonful, server v. desktop, mutable
v. immutable, nestable v. not...

I could have a go.  Is anyone else working on it, or likely to?

I think this is a fantastic idea! I'd happily not work on it if you were interested. 

Can you make it another content page? We can make it as big as we like then and directly referable from the FAQs. 

 

I think it also needs something on MPI, like "Can I/How do I run
something in a container built against a different MPI than a system one
outside the container".  The implication seems to be that you can,
surprisingly.  I had a brief look, knowing a bit about openmpi, and it's
not clear to me how you can at all straightforwardly, or what the
integration in openmpi is all about, especially as that seems to be for
an obsolete singularity.  [I didn't expect to find openmpi documentation
on a new feature like that, or even a new MCA framework, of course.]

I will work on this one. I can add an FAQ as well as extend on the HPC page that already exists and link between them. 

BTW, there is the ability to use different MPI versions inside and outside the container provided that the MPI outside (on the host) is newer or equal to what is in the container. Additionally (as I understand it) it will be theoretically possible to run different MPI implementations over PMIx. I will add this to the info on the site. 

Thanks again!!

[Rémy Dernat]

Hi,

If I can find some time (...), I will try to help in that comparison with other techs.

Regards.

[Jai Dayal]

So, in regards to nested containers, can I indeed have a container w/in a container and run them via singularity? Meaning, Singularity is installed on the host and also in the first container, and then run the app in the second level container from the top level container?

Regards,

Jai

[vanessa s]

Yes, I've done this before. What is your particular use case?

Best,

Vanessa

--

[Chris Hines]

Hi Vanessa,

I can't speak to Jai;'s use case, but I'd like to try to put a linux desktop environement (vnc server, window manager etc) in a container, allow people to start a vnc server and use the menus to launch subsequent containers (neuroimaging workflows come to mind, but really any container)

I'm unclear as to how this works, unless there is more magic in the more recent singularity releases than I realised, wouldn't bind mounting the image drop the SUID bits on the singularity executable within the container?

Cheers,

--

Chris.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[vanessa s]

Hey Chris,

Right now, you could make a singularity container with singularity installed, and issue commands to it (with appropriate permissions) to do some action with Singularity internally. The same rules apply - the user inside the container is the user outside the container. What you can do now is execute running commands that aren't akin to services, like a scientific analysis, job, etc. What you can't yet do (at least without producing some ghost processes that you lose control of) is start a service inside a container (that itself starts and stops) and expect that the service that you've launched will be under your control. What you could do is have a single container consistently running (eg, the user is shelled into it) and then from there run additional containers. As for the "launching" part this would make most sense in some kind of job manager context, in which case you would just be issuing a command to some node to load singularity and run a job. Again, we are already sort of outside the container.

The functionality that is needed to seriously consider these kinds of ideas is under development (the feature-daemon branch), Michael Bauer is lead and we are working together to test and make better - your feedback and comments would be greatly appreciated!

Best,

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Jai Dayal]

To answer the previous question, the scenario you describe here is what I was asking about:

>What you could do is have a single container consistently running (eg, the user is shelled into it) and then from there run additional containers.

Thanks,

Jai

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[vanessa s]

yep, the only limitation / thing to be careful about is space (each container is going to take up room) and then being smart about sharing files / libraries, etc.

https://asciinema.org/a/130622?speed=3

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.