Recommended way to run several applications within Singularity

[Ivan Gonzalez]

Hi,

I would like to run a workflow that involves several applications running within a Singularity container. My question is if there is a way to "compose" a Singularity container from several Docker containers or, more generally which is the recommended way to deal with this case.

To be more specific, let say I have applications A and B which I can docker-pull from some registry. These applications are independent (so building does not require the other). I would like to build a workflow where A's output is used as B's input. Or, maybe slightly more complex, where A calls B as an external program.

One option would be to build a Docker image containing A and B, but this imposes some restrictions, such as the docker base image being the same for the two apps. Is this the recommended way? Or should I build one Singularity container per Docker container and then find a way to combine them in the host?

Thanks in advance!

Ivan

[vanessa s]

Hi Ivan,

Singularity doesn't yet have orchestration, and there is no "best practices" defined for this so I can walk you through the options. I am doing an analysis that implements a bunch of tools across genomics, and I found it easiest to install them into one container and then call the container as an executable for different functions (and if you need help with issuing commands with exec we can talk about this too). Your other option of course is to go with modularity, and create different contains for different software suites, which might make sense if you are using already modular docker containers. For orchestration I would suggest plugging the container as an executable into your workflow software of choice. I know that nextflow has already been used various times and there are examples to start with!

We have users and developers for nextflow also on our slack, so I would suggest jumping on and chatting with everyone to work through specific questions. I am currently traveling but can help via this list or slack if you want to discuss details.

Best,

Vanessa

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Paolo Di Tommaso]

Hi Ivan, 

As mentioned by Vanessa you may be interested to Nextflow. 

https://www.nextflow.io/blog/2016/more-fun-containers-hpc.html

In a nutshell it's a pipeline orchestration tool for scientific workflows. It handles transparently Docker and Singularity containers and allows you to scale your workload on different resource schedulers (SGE, SLURM, etc) and the cloud (AWS). 

Disclaimer, I'm the author. 

Cheers,
Paolo

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[tin h]

Thanks for the very interesting thread!  nextflow is very interesting. 

One thing that I still wonder is, how big should each of the container be?

For instance, CoreOS don't come with any package manager, and tiny containers contain perform very specific task.  Many containers may be strung together to perform a "not so complex task", just like one would string several pipes together on the unix command line.

For HPC workflow, many tiny containers may not make sense?  If the goal is to provide a single image file that contain the workflow, then this would be a fat container, more akin to a VM than to a package?   I think the default singularity image is 768MB, so tiny container is probably not what was intended.

The granularity question probably don't have one fast and hard answer, but it would be interesting to hear what container size others are building out there, and where is the line when one say "this should go to a separate container"?

For example, when I created a container for circos, I put gcc, gd, perl and a slew of CPAN modules in it.  The resulting image was ~1.7GB.  Was this a good stopping point?  If the user wanted other, say, bioinformatics tools, they should go on a different container rather than build on top of this one?

Cheers,

Tin

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Paolo Di Tommaso]

I think trying to reduce the container size is always a good idea.

Said that, IMO small containers are needed much more when working with Docker, because images need to be downloaded from the hub and stored locally in each node. So you would waste not just storage but also bandwidth. 

Since the Singularity image is shared between all nodes, I would not worry about the size image and go for a fat container. 

Cheers,
Paolo

[Michael]

Hi All,

I am faced with the exact same problem as Ivan originally posted: I have one host singularity container (because HPC) and within this container I would like to run various "guest" isolated containers (Docker or Singularity).

As far as I know Next Flow doesn't solve the original problem as it is still based on a single-layer of containers (what if two containers need different versions of common libraries?). Would anyone know a solution to this problem? It seems running singularity within singularity is not possible (would be glad to be told the opposite) and i saw no information on running docker containers within a singularity container.

Michael

To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.

[Thomas Hartmann]

Hi Michael,

if the need for isolation is not too large, one way might be to just use
(old fashioned) different environment setups in the container (if you
really need to have more than one workflow per container)

I.e.,
if all the use cases share somewhat the same libs/bins, you could add
env setup functions to the environment section [1] to get the paths in
the container right for each use case.

Then -for 'automatic' execution or so- one could update the action
scripts at the end of the %post section [2].
So, that one or the other setup-function is executed for singularity
shell/execute/run/..., e.g., by adding on SINGULARITYENV_ envvars, so
that auser can choose his/her desired environment in the container by
exporting the corresponding envvar before starting the container.

Vanessa has a very good guide on how to tweak the action scripts
https://vsoch.github.io/2019/motd/
which might be a good starting point

Cheers,
Thomas

[1]
> Singularity
%environment
function setupHelp {
echo "Available environments"
echo " * setupGrid : WLCG Grid UI environment, needs CVMFS"
echo " ...
...."
echo ""
echo "To automatically setup an environment, export before starting
the container the singularity environment variable to 'true', e.g.,"
echo " export SINGULARITYENV_SETUPGRID=\"true\""
}
function setupGrid ...


[2]
%post
...
sed -i '2iecho \"Welcome to the DESY Basic SL6 Container\"'
/.singularity.d/actions/shell
sed -i '3iecho \" based on: $(/bin/cat /etc/redhat-release)\"'
/.singularity.d/actions/shell
...

sed XYZ /.singularity.d/actions/exec ...
sed ... ( if SINGULARITYENV_FOO ) ... setupFoo()

>> <javascript:>.

>
> --
> You received this message because you are subscribed to
> the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails

> from it, send an email to singu...@lbl.gov <javascript:>.

>
>
> --
> You received this message because you are subscribed to the
> Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails

> from it, send an email to singu...@lbl.gov <javascript:>.

>
>
> --
> You received this message because you are subscribed to the
> Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from

> it, send an email to singu...@lbl.gov <javascript:>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to singularity...@lbl.gov
> <mailto:singularity...@lbl.gov>.

[Michael Dayan]

Hi Thomas,

The solution you propose is interesting and I believe i can make use of it in other situations, thank you. In the case I had in mind I am not sure it can be used: ideally I would have only one tool per "guest" container, and then connect them (and run them) with a worflow engine living in the "host" container. Therefore i would only run the container once (single command to run the workflow engine) so i assume i would not be able to play with the environment variables as you suggested (which would have been mostly changing the PATH variable to isolate the libraries of each tool i guess).

Michael

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.