A new easy way to convert docker images to singularity images

[Chris Gorgolewski]

Dear Singularity Users,

Same as many of you I have been very excited about bringing my own environment to HPC clusters available at my institution using Singularity Containers. So far I've been using Docker to create my containers (it works well on Mac and Windows, has caching of layers, public repository of images etc.). Like many other scientists I'm using Mac and Windows for my personal computers and I don't have easy access to root on a linux box (which is required to create or convert to a Singularity image).

To be able to keep using Docker to develop images and make the conversion from docker to singularity as easy as possible I have created a special Docker image that taps into the host docker daemon and performs the conversion. All you need to do is:

docker run \

  -v /var/run/docker.sock:/var/run/docker.sock 

  -v D:\host\path\where\to\output\singularity\image:/output 

  --privileged -t --rm 

  filo/docker2singularity 

  ubuntu:14.04

Replace D:\host\path\where\to\output\singularity\image with a path in the host filesystem where your Singularity image will be created. Replace ubuntu:14.04 with the docker image name you wish to convert (it will be pulled from Docker Hub if it does not exist on your host system).

You can read more and submit issues or patches at https://github.com/chrisfilo/docker2singularity

I hope this helps!

Best,

Chris

[Gregory M. Kurtzer]

Whoa... That is cool!!!!

So if I'm understanding correctly, this is a docker container itself which will create another docker container and import to Singularity within that docker container? Does the --privileged option allow you to run Singularity within a Docker environment?

Thanks!

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Gregory M. Kurtzer
High Performance Computing Services (HPCS)
University of California
Lawrence Berkeley National Laboratory
One Cyclotron Road, Berkeley, CA 94720

[Chris Gorgolewski]

On Sat, Aug 6, 2016 at 6:46 AM, Gregory M. Kurtzer <gmku...@lbl.gov> wrote:

Whoa... That is cool!!!!

So if I'm understanding correctly, this is a docker container itself which will create another docker container and import to Singularity within that docker container?

In general yes. To be more precise inside the container I am not running the docker daemon, but communicating with the host docker daemon via mounted socket. This allows me to access docker images built or pulled on the host system.

 

Does the --privileged option allow you to run Singularity within a Docker environment?

Yes - it was necessary to access the loop device.

 

Thanks!

np - I needed it myself!

 

On Fri, Aug 5, 2016 at 9:31 PM, Chris Gorgolewski <krzysztof.gorgolewski@gmail.com> wrote:

Dear Singularity Users,

Same as many of you I have been very excited about bringing my own environment to HPC clusters available at my institution using Singularity Containers. So far I've been using Docker to create my containers (it works well on Mac and Windows, has caching of layers, public repository of images etc.). Like many other scientists I'm using Mac and Windows for my personal computers and I don't have easy access to root on a linux box (which is required to create or convert to a Singularity image).

To be able to keep using Docker to develop images and make the conversion from docker to singularity as easy as possible I have created a special Docker image that taps into the host docker daemon and performs the conversion. All you need to do is:

docker run \

  -v /var/run/docker.sock:/var/run/docker.sock 

  -v D:\host\path\where\to\output\singularity\image:/output 

  --privileged -t --rm 

  filo/docker2singularity 

  ubuntu:14.04

Replace D:\host\path\where\to\output\singularity\image with a path in the host filesystem where your Singularity image will be created. Replace ubuntu:14.04 with the docker image name you wish to convert (it will be pulled from Docker Hub if it does not exist on your host system).

You can read more and submit issues or patches at https://github.com/chrisfilo/docker2singularity

I hope this helps!

Best,

Chris

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Gregory M. Kurtzer
High Performance Computing Services (HPCS)
University of California
Lawrence Berkeley National Laboratory
One Cyclotron Road, Berkeley, CA 94720

[Tristan Glatard]

That sounds very useful Chris, thanks!

    Tristan

On 08/06/2016 12:31 AM, Chris Gorgolewski wrote:

Dear Singularity Users,

Same as many of you I have been very excited about bringing my own environment to HPC clusters available at my institution using Singularity Containers. So far I've been using Docker to create my containers (it works well on Mac and Windows, has caching of layers, public repository of images etc.). Like many other scientists I'm using Mac and Windows for my personal computers and I don't have easy access to root on a linux box (which is required to create or convert to a Singularity image).

To be able to keep using Docker to develop images and make the conversion from docker to singularity as easy as possible I have created a special Docker image that taps into the host docker daemon and performs the conversion. All you need to do is:

docker run \

  -v /var/run/docker.sock:/var/run/docker.sock 

  -v D:\host\path\where\to\output\singularity\image:/output 

  --privileged -t --rm 

  filo/docker2singularity 

  ubuntu:14.04

Replace D:\host\path\where\to\output\singularity\image with a path in the host filesystem where your Singularity image will be created. Replace ubuntu:14.04 with the docker image name you wish to convert (it will be pulled from Docker Hub if it does not exist on your host system).

You can read more and submit issues or patches at https://github.com/chrisfilo/docker2singularity

I hope this helps!

Best,

Chris

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Tristan Glatard

CNRS Researcher, CREATIS, Lyon, France
Visiting Scholar at McGill University, Montreal Neurological Institute (MNI), Montreal, Canada

Web Google Scholar Research Gate Publons ORCID LinkedIn Twitter Github

[Oleksandr Moskalenko]

An example run on RHEL7 results in an error:

sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v /home/om/containers/images:/output --privileged -t --rm filo/docker2singularity docker.io/nunofonseca/irap_ubuntu

docker: Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.22).

See 'docker run --help'.

On Mon, Aug 8, 2016 at 10:12 AM, Tristan Glatard <tristan...@creatis.insa-lyon.fr> wrote:

That sounds very useful Chris, thanks!

    Tristan

On 08/06/2016 12:31 AM, Chris Gorgolewski wrote:

Dear Singularity Users,

Same as many of you I have been very excited about bringing my own environment to HPC clusters available at my institution using Singularity Containers. So far I've been using Docker to create my containers (it works well on Mac and Windows, has caching of layers, public repository of images etc.). Like many other scientists I'm using Mac and Windows for my personal computers and I don't have easy access to root on a linux box (which is required to create or convert to a Singularity image).

To be able to keep using Docker to develop images and make the conversion from docker to singularity as easy as possible I have created a special Docker image that taps into the host docker daemon and performs the conversion. All you need to do is:

docker run \

  -v /var/run/docker.sock:/var/run/docker.sock 

  -v D:\host\path\where\to\output\singularity\image:/output 

  --privileged -t --rm 

  filo/docker2singularity 

  ubuntu:14.04

Replace D:\host\path\where\to\output\singularity\image with a path in the host filesystem where your Singularity image will be created. Replace ubuntu:14.04 with the docker image name you wish to convert (it will be pulled from Docker Hub if it does not exist on your host system).

You can read more and submit issues or patches at https://github.com/chrisfilo/docker2singularity

I hope this helps!

Best,

Chris

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Tristan Glatard

CNRS Researcher, CREATIS, Lyon, France
Visiting Scholar at McGill University, Montreal Neurological Institute (MNI), Montreal, Canada

Web Google Scholar Research Gate Publons ORCID LinkedIn Twitter Github

--

You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Bennet Fauber]

I had a similar problem.  A colleague here at U Michigan suggested cloning the docker2singularity to local disk,

$ git clone g...@github.com:chrisfilo/docker2singularity.git

then modifying the Dockerfile so that a specific version of Docker is specified, e.g., on my CentOS 7,

FROM docker:1.11.2

Once that is done,

$ sudo docker build -t docker2singularity docker2singularity

after which docker2singularity seemed to work OK.

Copying and pasting from history.  I hope I got all that right.

-- bennet

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Tristan Glatard

CNRS Researcher, CREATIS, Lyon, France
Visiting Scholar at McGill University, Montreal Neurological Institute (MNI), Montreal, Canada

Web Google Scholar Research Gate Publons ORCID LinkedIn Twitter Github

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Chris Gorgolewski]

Thanks for the feedback. I created versions of docker2singularity compatible with older docker releases. See: https://github.com/chrisfilo/docker2singularity/blob/master/README.md#client-is-newer-than-server-error

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Rémy Dernat]

Hi Chris,

Thanks for this update on the script.

Cheers

Rémy

[Bennet Fauber]

Just in case it matters, it looks like the stock docker version for
RH/CentOS 7.2 is docker is:

$ yum list docker
Loaded plugins: langpacks
Available Packages
docker.x86_64 1.10.3-44.el7.centos extras

> email to singularity...@lbl.gov.

[Chris Gorgolewski]

I've added a 1.10 version (filo/docker2singularity:1.10). Could someone test it?

>> email to singularity+unsubscribe@lbl.gov.

>
>
> --
> You received this message because you are subscribed to the Google Groups
> "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to singularity+unsubscribe@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.