Workflow for converting Docker container to work with non-priveleged Singularity

[Peter van Heusden]

Hi there

I've got a workflow that relies on a series of Docker containers. I'm working on replacing these Docker containers with Singularity for use on a cluster where priveleged operations are not possible (the Centre for High Performance Computer's lengau cluster in South Africa).

I can convert to Singularity images using docker2singularity (with a few minor changes, see the PR I submitted). And then exported the image using singularity export. After moving the tar file to the CHPC cluster and unpacking it, I cannot run singularity with the error:

[pheusden@login1 scratch]$ singularity shell $(pwd)/fastqc
ERROR  : User namespace not supported, and program not running privileged.
ABORT  : Retval = 255

The environment here is CentOS 7.2.1511 with kernel 3.10.0-327.36.3.el7.x86_64.

Is it possible to run singularity as an unprivileged user on such a system?

Thanks!

Peter

[Stefan Kombrink]

Hi Peter,

 I struggled with this as well.
I made it work without further modifications on my Linux Mint 17.1 laptop which apparently has namespaces enabled.

Centos 7.2 is however a much older environment.
Its user namespace support is still experimental and when I enable it (you can force it to enabled by passing a kernel param) it fails due to a non-working mount namespace.
Really no idea when this will officially work on RHEL/CentOS but would be definitely appreciated.

greets Stefan

[Stefan Kombrink]

To answer your question better:

Yes, you can let unprivileged users run singularity containers.
For that to work, however, you need to have sexec-suid have equipped with the suid bit and have allow setuid = yes enabled in singularity.conf.
I believe without that there is no chance since namespaces are broken.

[vanessa s]

Hi Peter,

The docker2singularity conversion isn't (anymore) the best way to do the conversion. I don't see your PR - could you direct me to the Dockerfile (on Docker Hub?) of the image you are trying to convert, and we will go from there? Ideally you would want:

- add the image to Docker Hub, making it available on the docker registry

- create a bootstrap file, or run/shell the image on demand. A bootstrap build file looks something like this:

#Singularity

BootStrap: docker

From: ubuntu:latest

%runscript

exec echo "Hello World" "$@"

%post

apt-get install git

Then you would create an image and do the bootstrap, on a local machine where you have sudo

sudo singularity create workflow.img

sudo singularity bootstrap workflow.img Singularity

The other option is just to run/shell directly, something like

singularity run docker://ubuntu:latest

singularity shell docker://ubuntu:latest

There are other things you can do like binding directories, executing commands, that may be more fit for your particular use case. If you can give me specifics about your images (and a github repo where we can work together would be great!), I would be happy to help get this moving along. It's a bit late here (after midnight) but I'll be going to sleep soon and waking up... sometime later today, haha.

Best,

Vanessa

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Peter van Heusden]

Hi Vanessa

The images in question are not on Docker Hub - I can upload most of them but one of them I cannot due to licensing issues. This one is on a hub though, so it could be a test case:

quay.io/biocontainers/fastx_toolkit

from

https://quay.io/repository/biocontainers/fastx_toolkit

This is one of the containers from the bioconda project. I had problems with that so tried pushing one of our containers to Docker Hub, yielding this bootstrap file:

#Singularity

BootStrap: docker
From: pvanheus/qiime:latest

%runscript

exec echo "Hello World" "$@"

Then tried to bootstrap:

pvh@gabber:/tmp$ sudo singularity create -F workflow.img
Creating a new image with a maximum size of 768MiB...
Executing image create helper
Formatting image with ext3 file system
Done.
pvh@gabber:/tmp$ sudo singularity bootstrap workflow.img Singularity
Bootstrap initialization
Checking bootstrap definition
Executing Prebootstrap module
Executing Bootstrap 'docker' module
From: pvanheus/qiime:latest
pvanheus/qiime:latest
Error getting tags using url https://registry-1.docker.io/v2/pvanheus/qiime/tags/list
Executing Postbootstrap module
ERROR: Container does not contain the valid minimum requirement of /bin/sh

I'm not sure which container doesn't contain /bin/sh but the pvanheus/qiiime:latest one certainly does.

Peter

(This github repo for the container in question is here: https://github.com/h3abionet/h3abionet16S/tree/master/dockerfiles/qiime)

On Mon, 5 Dec 2016 at 10:20 vanessa s <vso...@gmail.com> wrote:

Hi Peter,

The docker2singularity conversion isn't (anymore) the best way to do the conversion. I don't see your PR - could you direct me to the Dockerfile (on Docker Hub?) of the image you are trying to convert, and we will go from there? Ideally you would want:

- add the image to Docker Hub, making it available on the docker registry

- create a bootstrap file, or run/shell the image on demand. A bootstrap build file looks something like this:

#Singularity

BootStrap: docker

From: pvanheus/qiime:latest

%runscript

PATH=/opt/conda/bin:$PATH
export PATH
source activate qiime1

exec "$@" $*

Then you would create an image and do the bootstrap, on a local machine where you have sudo

sudo singularity create workflow.img

sudo singularity bootstrap workflow.img Singularity

The other option is just to run/shell directly, something like

singularity run docker://ubuntu:latest

singularity shell docker://ubuntu:latest

There are other things you can do like binding directories, executing commands, that may be more fit for your particular use case. If you can give me specifics about your images (and a github repo where we can work together would be great!), I would be happy to help get this moving along. It's a bit late here (after midnight) but I'll be going to sleep soon and waking up... sometime later today, haha.

Best,

Vanessa

On Sun, Dec 4, 2016 at 11:16 PM, Peter van Heusden <p...@sanbi.ac.za> wrote:

Hi there

I've got a workflow that relies on a series of Docker containers. I'm working on replacing these Docker containers with Singularity for use on a cluster where priveleged operations are not possible (the Centre for High Performance Computer's lengau cluster in South Africa).

I can convert to Singularity images using docker2singularity (with a few minor changes, see the PR I submitted). And then exported the image using singularity export. After moving the tar file to the CHPC cluster and unpacking it, I cannot run singularity with the error:

[pheusden@login1 scratch]$ singularity shell $(pwd)/fastqc
ERROR  : User namespace not supported, and program not running privileged.
ABORT  : Retval = 255

The environment here is CentOS 7.2.1511 with kernel 3.10.0-327.36.3.el7.x86_64.

Is it possible to run singularity as an unprivileged user on such a system?

Thanks!

Peter

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

--

You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Michael Bauer]

Hi Peter,

This is unfortunately an issue with Singularity version 2.2 that you are using. Our API requests to Docker Hub did not properly implement support for different "library" names, and thus you are unable to get a Docker image with a non-standard tag. The current branch on the GitHub no longer has this issue, and we're working hard to push out a new version of Singularity that fixes this (and many other) standing issues with version 2.2 of Singularity. In the mean time, there is a workaround that you can employ which might help you get up and running:

1. Create a local private registry by following instructions here: https://docs.docker.com/registry/deploying/
2. Push the Docker images you wish to bootstrap to your local registry (tag should be "$IP:5000/qiime:latest")
3. Bootstrap your Singularity image as follows

BootStrap: docker 

From: qiime:latest

IncludeCmd: no

Registry: http://$IP:5000

Token: no

Please try this out and let me know if this solves your issues. 

Cheers,

Michael

 

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Peter van Heusden]

Thanks Stefan

Do you know if anyone has done any kind of security audit on sexec-suid? I think in the current context it will be hard to convince the cluster admins to install a novel setuid program.

Peter

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[vanessa s]

Hi Peter,

Michael is correct! And if you are able to download and use the latest from master, the minimum header you would need for the file is the following:

BootStrap: docker

From: biocontainers/fastx_toolkit:0.0.14--2

Registry: quay.io

You should be able to bootstrap that, I just tested (with a slightly older version) and it seemed to work ok.

sudo singularity bootstrap workflow.img Singularity

Bootstrap initialization

Checking bootstrap definition

Executing Prebootstrap module

Executing Bootstrap 'docker' module

From: biocontainers/fastx_toolkit:0.0.14--2

Registry: quay.io

Cache folder set to /root/.singularity/docker

Downloading layer sha256:2c07bb1526a59386037550d4a4cb589c97a518b5bc07b9a9011b60aaf46278da

Extracting /root/.singularity/docker/sha256:2c07bb1526a59386037550d4a4cb589c97a518b5bc07b9a9011b60aaf46278da.tar.gz

Extracting /root/.singularity/docker/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz

Downloading layer sha256:10c3bb32200bdb5006b484c59b5f0c71b4dbab611d33fca816cd44f9f5ce9e3c

Extracting /root/.singularity/docker/sha256:10c3bb32200bdb5006b484c59b5f0c71b4dbab611d33fca816cd44f9f5ce9e3c.tar.gz

Downloading layer sha256:dfda3e01f2b637b7b89adb401f2f763d592fcedd2937240e2eb3286fabce55f0

Extracting /root/.singularity/docker/sha256:dfda3e01f2b637b7b89adb401f2f763d592fcedd2937240e2eb3286fabce55f0.tar.gz

Downloading layer sha256:d2ba336f2e4458a9223203bf17cc88d77e3006d9cbf4f0b24a1618d0a5b82053

Extracting /root/.singularity/docker/sha256:d2ba336f2e4458a9223203bf17cc88d77e3006d9cbf4f0b24a1618d0a5b82053.tar.gz

Downloading layer sha256:7ff999a2256f84141f17d07d26539acea8a4d9c149fefbbcc9a8b4d15ea32de7

Extracting /root/.singularity/docker/sha256:7ff999a2256f84141f17d07d26539acea8a4d9c149fefbbcc9a8b4d15ea32de7.tar.gz

Downloading layer sha256:00cf8b9f3d2a08745635830064530c931d16f549d031013a9b7c6535e7107b88

Extracting /root/.singularity/docker/sha256:00cf8b9f3d2a08745635830064530c931d16f549d031013a9b7c6535e7107b88.tar.gz

Downloading layer sha256:3aaade50789a6510c60e536f5e75fe8b8fc84801620e575cb0435e2654ffd7f6

Extracting /root/.singularity/docker/sha256:3aaade50789a6510c60e536f5e75fe8b8fc84801620e575cb0435e2654ffd7f6.tar.gz

Downloading layer sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500

Extracting /root/.singularity/docker/sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500.tar.gz

Executing Postbootstrap module

Done.

Best,

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Sadanand Warrier]

Hello

So I went through the answers but there is something that is a little puzzling. I'm trying to convert a docker image to a Singularity container image. So I followed the process that the "singularity --help import" suggests by creating a tar file from the image using 

"docker save -o <tarfilename> <image file>

 and then  trying to convert it by using

sudo singularity import <singualrityshell.img> <tarfilename>.

Tried various things including increasing the singularityshell.img size from default 768M to 2048M (the docker container is 732M in size)  but always got this error

Bootstrap initialization

No bootstrap definition passed, updating container

Executing Prebootstrap module

Executing Postbootstrap module

ERROR: Container does not contain the valid minimum requirement of /bin/sh

There was actually a /bin/sh in the docker container so I think this is some generic error.

I finally managed to do the conversion by running the image interactively and then doing this

docker export <containername> | sudo singularity import <singularityshell.img>

Looks like it doesn't work with a tar file.

I did not try creating a local registry and pushing the image to it etc.

My question is why doesn't the tar method work??

Thanks

S