Unprivileged Singularity containers? Oh yes!

[Gregory M. Kurtzer]

Singularity can now avoid running as SUID iff you have a new enough kernel and don't need to support Singularity images (oh yeah, Singularity now also supports directories and soon archive images)... So how does it work? From your Git clone of Singularity:

$ make maintainer-clean # Only necessary if you have a previous build

$ git pull

$ git checkout devel

$ ./autogen.sh

$ ./configure --prefix=$HOME/singularity-test

$ make

$ make install

Some background info:

$ whoami

gmk

$ cat /etc/debian_version 

jessie/sid

$ cat /etc/redhat-release

cat: /etc/redhat-release: No such file or directory

As I mentioned, it doesn't work with images:

$ ~/singularity-test/bin/singularity shell ~/container.img 

ERROR  : Singularity must be executed in privileged mode to use images

And here it is in action:

$ ~/singularity-test/bin/singularity exec /var/tmp/Centos-7/ cat /etc/redhat-release

CentOS Linux release 7.2.1511 (Core) 

$ ~/singularity-test/bin/singularity exec /var/tmp/Centos-7/ whoami

gmk

$ ~/singularity-test/bin/singularity shell /var/tmp/Centos-7/

Singularity.Centos-7> ps aux

USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

gmk           1  0.0  0.8 115388  3120 pts/1    S    20:31   0:00 /bin/sh

gmk           2  0.0  0.8 139508  3152 pts/1    R+   20:31   0:00 ps aux

Singularity.Centos-7> exit

And of course, everything still works exactly as it did before when run or installed by root and on older distributions that do not support the user namespace properly. But, in either of those cases, it is very easy to disable or omit the SUID bits if the system administrator wishes.

Expect to see all of these new features in the upcoming Singularity v2.2 release!

--

Gregory M. Kurtzer
High Performance Computing Services (HPCS)
University of California
Lawrence Berkeley National Laboratory
One Cyclotron Road, Berkeley, CA 94720

[Bernard Li]

Cool beans!

Cheers,

Bernard

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Adam DeConinck]

Cool! Happy to see this. :) What is the kernel requirement? I.e. version, namespace support, etc?

Cheers,
Adam

--

[Gregory M. Kurtzer]

Hi Adam,

The kernel version requirement is difficult to say because some kernel's have features back ported. Singularity will automatically figure out if it can do it or not at compile and runtime and gives the system administrator the ability to toggle options/features if it is a privileged install.

BTW, what I have seen from trial and error... RHEL7 user namespaces is non-functional, and current Ubuntu works fine. I haven't tested with too much more then that, but once the release approaches, I will collect feedback and assemble a support matrix.

Thanks!

--

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Igor Yakushin]

Hi Greg,

I am not sure I understand. From reading the documentation I got an impression that singularity works only with images. But from the above, do I understand correctly that one can simply make a container installation in a directory without creating an image? Any documentation describing how the commands change to support this? create, bootstrap? Starting from what kernel version is it supported? How portable is such a container? Can I just move the directory to a different machine with the sufficiently new kernel? Should it be under the same path on a different machine?

Thank you,

Igor

--

[Igor Yakushin]

Another question: is there any performance benefit using directories vs images?

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Gregory M. Kurtzer]

Hi Igor,

On Sat, Aug 13, 2016 at 7:39 PM, Igor Yakushin <igor...@gmail.com> wrote:

Hi Greg,

I am not sure I understand. From reading the documentation I got an impression that singularity works only with images. But from the above, do I understand correctly that one can simply make a container installation in a directory without creating an image?

This is in the development version of Singularity and will be a feature available in the 2.2 release. Yes, it will work with a standard directory that contains an installation of Linux.

I am moving Singularity to a stacked library architecture which is very modular and easy to add features without changing or impacting others. So adding support other modules for the "rootfs" is actually pretty easy.

 

Any documentation describing how the commands change to support this?

No docs on this yet as it is still in development. In a nutshell (and as you can see from my snippet example) you simply replace the image path that you currently use in Singularity for the directory path.

 

create, bootstrap? Starting from what kernel version is it supported?

The directory feature will work with all Kernel versions.

 

How portable is such a container? Can I just move the directory to a different machine with the sufficiently new kernel?

Yes, exactly. Singularity will also have the ability to deal with archives (tar.gz files, etc..). This feature isn't present yet, but as I mentioned above, will be fairly trivial to add.

 

Should it be under the same path on a different machine?

Nope, path of the directory does not matter from host to host.

Thank you,

Igor

On Sat, Aug 13, 2016 at 8:06 PM, Igor Yakushin <igor...@gmail.com> wrote:

Another question: is there any performance benefit using directories vs images?

Well, yes and no. Yes in that there is slightly more overhead in the loop bind of the image which will save about 5-7 thousandths of a second when using a base directory, but you may quickly loose that benefit with dealing with parallel container execution when the container is on a network file system (especially a parallel file system) as metadata lookups are expensive. For large scale parallel runs, it makes much more sense to use images.

Hopefully that helps, but let me know if anything else needs clarification.

Greg

 

--

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Chris Gorgolewski]

Speaking of container formats (images, directories, archives). I wonder if it would be worthwhile looking into supporting OCI images: https://github.com/opencontainers/image-spec It has a growing support in the container community (see https://coreos.com/blog/oci-image-specification.html).

Best,

Chris

Hi Igor,

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Gregory M. Kurtzer
High Performance Computing Services (HPCS)
University of California
Lawrence Berkeley National Laboratory
One Cyclotron Road, Berkeley, CA 94720

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Gregory M. Kurtzer]

I too have been keeping an eye on the OCI, but I'm not sure how well the format translates into Singularity concepts.

The runtime specification is based on a JSON file that defines various bits about how the container should run (e.g. cores, memory, network ports, user, as well as environment, entrypoint/command, etc..). In Singularity, the first bits are not relevant and the latter Singularity currently handles differently. With that said, there is no reason why Singularity can't be extended to include the JSON file (if it exists).

As far as the packaging of the image (serialization), that also is very different in that (as I understand it), OCI builds images based on layers of tarball diffs. While in theory, this too should be able to be leveraged, but it doesn't strictly apply to the current Singularity model. I have plans to support a URI which can possibly be used for this as well a workflow or container ID in the Singularity-Hub.

I love the ideas, and if you see this clearer then I do, or if I'm wrong about my understanding of the OCI, please let me know. :)

Thank you!

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Kenneth Hoste]

Hi Gregory,

On 13/08/16 23:26, Gregory M. Kurtzer wrote:

Hi Adam,

The kernel version requirement is difficult to say because some kernel's have features back ported. Singularity will automatically figure out if it can do it or not at compile and runtime and gives the system administrator the ability to toggle options/features if it is a privileged install.

BTW, what I have seen from trial and error... RHEL7 user namespaces is non-functional, and current Ubuntu works fine. I haven't tested with too much more then that, but once the release approaches, I will collect feedback and assemble a support matrix.

Can you elaborate on the "RHEL7 user namespaces is non-functional"?

Is RHEL7 supposed to support user namespaces, but are you seeing it doesn't work as it should be?
Is this a bug that should be reported upstream?

I'm excited about Singularity not requiring special privileges (it kind of was a blocker for me), and it would be great if it worked on RHEL7 too in the not-so-distant future...


regards,

Kenneth

[Gregory M. Kurtzer]

On Sun, Aug 14, 2016 at 1:51 AM, Kenneth Hoste <kennet...@gmail.com> wrote:

Hi Gregory,

On 13/08/16 23:26, Gregory M. Kurtzer wrote:

Hi Adam,

The kernel version requirement is difficult to say because some kernel's have features back ported. Singularity will automatically figure out if it can do it or not at compile and runtime and gives the system administrator the ability to toggle options/features if it is a privileged install.

BTW, what I have seen from trial and error... RHEL7 user namespaces is non-functional, and current Ubuntu works fine. I haven't tested with too much more then that, but once the release approaches, I will collect feedback and assemble a support matrix.

Can you elaborate on the "RHEL7 user namespaces is non-functional"?

Sure thing. Invocation of unshare(CLONE_NEWUSER) results in an error 22 (Invalid argument). The same system call works on other distros which support user namespaces.

 

Is RHEL7 supposed to support user namespaces, but are you seeing it doesn't work as it should be?

It is quite misleading:

$ grep CONFIG_USER_NS /boot/config-3.10.0-327.28.2.el7.x86_64 

CONFIG_USER_NS=y

$ grep CLONE_NEWUSER /usr/include/linux/sched.h 

#define CLONE_NEWUSER           0x10000000      /* New user namespace */

Is this a bug that should be reported upstream?

Nope, not according to this: (http://rhelblog.redhat.com/2015/07/07/whats-next-for-containers-user-namespaces/)... It was done on purpose:

"That said, Red Hat disabled them, because we think that user namespaces need to incubate in the upstream community longer to fully understand the security implications ..."

 

I'm excited about Singularity not requiring special privileges (it kind of was a blocker for me), and it would be great if it worked on RHEL7 too in the not-so-distant future...

If you don't mind installing a Fedora kernel, it should work just fine as the user space requirements seem present.

 

regards,

Kenneth

Hope that helps!

Greg

 

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Gregory M. Kurtzer
High Performance Computing Services (HPCS)
University of California
Lawrence Berkeley National Laboratory
One Cyclotron Road, Berkeley, CA 94720

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[vanessa s]

If Singularity has something to benefit from adopting the spec to whatever extent that it can (integration into applications that support it) or in that it helps to capture things like history, +1 that it wouldn't hurt to include. There isn't much overlap between boutiques (which captures input and outputs) and this (which captures runtime system information), but I wonder if the two groups might talk to one another so generally we wouldn't be sticking out fingers into multiple egg baskets :)

Greg - the URI would be very useful for Singularity Hub - right now a "unique id" (which is just a number) is assigned to an image when it's uploaded, but that of course would only serve to live with the image while it lives in the Hub. We could still maintain this id (it's really just an application table key), but have a more proper one to live with the container.

Chris - I was rethinking our earlier conversation about workflows, and I think starting with running a single container with inputs / outputs (a single node workflow) would work to start, if you could direct me to a suggested one for that! :O)

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Chris Gorgolewski]

Another advantage of adopting the OCI Image spec is that soon repositories such as docker hub and quay will start distributing compatible images. It would be great if a singularity user could download such image from one of those repositories and run it (without needing root access to convert it to Singularity .img).

@Vanessa: have a look at https://github.com/BIDS-Apps/freesurfer or https://github.com/BIDS-Apps/example.

Best,

Chris

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[vanessa s]

Shweet, thanks! :)

Best,

Vanessa

[rd]

Hello Greg,

Is there any documentation on how to create a container directory? I need to make singularity work from centos 6/7 without root installation.

Thanks,

rd

Hi Igor,

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[vanessa s]

Hi Rd,

With the development branch, you don't need sudo to create a container or import, eg:

singularity create container.img
singularity import container.img docker://centos:7

Singularity supports building containers into directories, if you want to test things out, for example, but it must be done with bootstrap (which requires sudo)

mkdir container
sudo singularity bootstrap container Singularity

where Singularity is the bootstrap definition file.

If you could provide more information about your use case, I'd be glad to help. Generally I wouldn't recommend building into a directory, but instead creating a container proper, and then mounting it to see inside (like a folder). We have docs that aren't published yet, see:

https://github.com/singularityware/singularityware.github.io/blob/docs/2.3/pages/docs/user-docs/docs-create-an-image.md#mounting-an-image

Best,

Vanessa 

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[rd]

Hi Vanessa,

Thanks for your reply. I can create the image/directory on a centos box where I have root access, however I would like to use the image/dir on a system (centos 6.8) where I do not have root privileges and where I have installed singularity (2.2.1) as a regular user (as installing singularity from root is not supported).

Do I have to install the devel version of singularity on the host where I do not have root access or on the host where I create the img/directory (and where I have root access)?

With the current singularity version 2.2.1 using the "singularity mount" command without sudo does not work:

$ singularity mount ubuntu.img /tmp/container/
ERROR: Calling user must be root!

nor does:

$ singularity shell /tmp/ubuntu/
ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT  : Retval = 255

where:

$ ls /tmp/ubuntu/
ubuntu.img

or (as expected):

$ singularity shell /tmp/ubuntu/
ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT  : Retval = 255


Any chances that I can get singularity (installed by my regular user) to load an image (created elsewhere) on the centos 6.8 box where I do not have root access?

Thanks,

rd

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[vanessa s]

Hi Rd,

All of the non root functionality is in the upcoming 2.3 (development branch) and not supported in 2.2, so you are out of luck for running these commands on your cluster as is. I would recommend taking the old school Singularity approach - develop your image locally where you have sudo/root and can do what you like, transfer to your cluster, and then use.

Best,

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[rd]

Hello again,

The problem is that on the system where I do not have root/sudo access singularity can only be installed at the user level. So yes I can create and manipulate the image on a different system where I have root/sudo privileges but I ultimately need to use such image on the system where I do not have root/sudo privileges. Currently with my user installed version of singularity on the system where I do not have root/sudo access I am unable to use the image created elsewhere (where I do have root privileges).

Will any current development version of singularity installed as a simple user on a system where that user does not have root/sudo abilities be able to use an image created elsewhere?

Or as long as I am on a centos 6.8 box the only way to make singularity able to use an image (created elsewhere) is to have singularity installed by root?

Thanks,

rd

[vanessa s]

I don't understand - why are you not able to use your image on your cluster? All images can be run with your same user permissions / privileges and you should not need sudo for this.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[rd]

With the singularity 2.2.1 (installed by my simple user) on the system (centos 6.8) where I do not  have  root/sudo access I cannot run the image, if I type:

singularity shell ubuntu.img

I get:

ERROR  : Singularity must be executed in privileged mode to use images

ABORT  : Retval = 255

Where can I get the devel package?

Thanks,

rd

[vanessa s]

I think on the cluster Singularity would need to be installed by your administrator, and not locally by you. Is it correct that you installed this yourself?

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Gregory M. Kurtzer]

It sounds like you didn't install Singularity properly. Did you run `make install` as root or with sudo?

devel package == installation from the development GitHub branch

Hope that helps!

On Mon, Apr 17, 2017 at 6:13 PM, rd <rda...@g.ucla.edu> wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[rd]

Yes, I did install it as a simple user and this is the whole point: being able to use singularity entirely in the userspace. Will 2.3 or the devel package support non root installation (just to run an image not to create one)?

Thanks,

rd

[vanessa s]

Hi Rd,

The point of Singularity is not to make you root on your shared cluster. The point is that, given Singularity is installed by a cluster administrator, you are empowered to run your own portable environments, Singularity images, that were generated by you (on your local resource where you have sudo, for version 2.2) with any kind of content inside.

Thus, you should direct your cluster administrator to these docs: http://singularity.lbl.gov/admin-guide and then run your images. A non root installation is not supported (and afaik, doesn't really make sense to support) in the near future. Please let me know if you are still confused.

Best,

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[rd]

Hi Vanessa,

Yes, of course the point is not to become root anywhere but just to run code more easily available for a different OS than the one you are currently stuck with. The non-root installation I think would precisely avoid potential privilege escalation. Any manipulation of the image can be done on a system with root access while the image could be run on a system without having to use any setuid code which makes many administrators nervous. However, I guess that being on Centos 6.8 there is no way to leverage the namespace features and work totally in user space. I was hoping this was not the case.

Thanks,

rd

[Gregory M. Kurtzer]

Mounting images are not supported by the user namespace in the kernel, thus even if you install as a user, have user namespace support in the host kernel, and do all of your image manipulation on a system where you have root access to, the image could *NOT* run on a system without having to use setuid code.

If the administrators are nervous about setuid code, then you might consider forwarding them our registered systems list as sometimes there is comfort in numbers. lol

Good luck with them!

Greg

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.