Kubernets with singularity

[Silveira, Plinio (R&D Center - Brazil)]

Hello, guys.

 

Had someone tried or investigated Kubernetes integration with Singularity?

 

Is singularity somehow prepared to that? For example, does it have some kind of compatibility with CRI (Container Runtime Interface)? Or it was not thought by design?

 

Appreciate any input.

 

Plinio Silveira

 

[Gregory M. Kurtzer]

Hi Silveira,

Ohhh, have you been reading my todo list?! ;)

Yes, I am looking to integrate Singularity with Kubernetes and even able to make some hires on that front! Granted, I've been looking for people, and haven't been hugely successful of someone who would come with the necessary knowledge (so if you, or anyone on this list knows someone interested and qualified, please send them my way!).

Thanks!

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[ghildebrand]

+1 for the topic from my side.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Chris Hines]

Hi Greg et al,

I'm curious as to how you would use kubernetes to orchestrate singularity containers? What is the use case for this? 

So far we've been using singularity only to ease the pain of installing software with build instructions/dependency lists for only one os vendor (e.g. installing OpenFOAM via apt get instead of trying to get it to compile on CentOS). We then leave it up to the user to submit their container to a (slurm) queue (very old school HPC of me ;-). Clearly Kubernetes is a completely different beast and I'm interested in how other people are using (or thinking of using) singularity?

Cheers,

--

Chris.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[vanessa s]

Once we have data containers, and Singularity has support for services, it plugs in seamlessly to run workflows, and more complicated applications that have different services with reliance on one another (eg, more than a single thing run as a job on a cluster). At face value, it seems like more of a "commercial" use case, but given that a scientific tool is (or should be) like an any other modern, web-based tools (eg, having some kind of application, separate database, APIs, workers, etc) this is a good direction for scientific products to go. We don't want just single containers, we want legos that can be put together in different ways to make different pipelines.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Jianxiong Dong]

Hi,
The current singularity container does not virtualize
networking/CPU/memory. It may not be easy for Kubernete to control
the resource consumption.

Also need to build network layer for singularity.
See Flannel (from Coreos) (also in RHEL 7 kubernetes) example (there
is a flow diagram there)
https://github.com/coreos/flannel

Another concern: currently we build singularity container
independently (each one requires base images). The base image is not
shared across containers. Is it a concern when we run many services in
one machine if each service is packed in one container?

Thanks

Jianxiong

>>>>> *Plinio Silveira*

>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "singularity" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to singularity...@lbl.gov.
>>>>>
>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups
>>> "singularity" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an
>>> email to singularity...@lbl.gov.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "singularity" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to singularity...@lbl.gov.

>>
>
>
>
> --
> Vanessa Villamia Sochat
> Stanford University '16
> (603) 321-0676
>

[Paolo Di Tommaso]

Hi Chris, 

You may be interested in Nextflow which is a lightweight workflow framework with built-in support for containers (both Docker and Singularity). In a nutshell you can use it to orchestrate the execution of containerised workloads in legacy HPC batch schedulers (SLURM, SGE, PBS, etc) and the AWS cloud as well.

Between the other things it has been recently published on Nature Biotech. We use it regularly with Singularity to deploy large scale genomic pipelines both at CRG and the Barcelona Supercomputer Center. 

You can read about a quick benchmark we performed at this page.

Cheers,

Paolo

[Carlos Eduardo Arango Gutierrez]

+1

[James Gundy]

One of the advantages of Singularity is that there is no user context changes or root escalation (which is great for security reasons).  However, in Kubernetes, there is no concept that I see where you can enforce this.  A user can run a container as any user that the image allows.  Would this break Singularity?  If you can run a Singularity image as any user, does Singularity have any advantages over running images in Docker?

--

[CARLOS EDUARDO ARANGO GUTIERREZ]

Hi James Gundy

> Would this break Singularity? 

No. Any user that run a singularity istance will be the same "user" (with the same permissions over the system) inside the container. 

As long as the "user" don't have root access security is guaranteed.

> does Singularity have any advantages over running images in Docker?

You mean besides the computational performance?.

e.g Since Singularity main goal is for HPC environments, yes, a simple advantage example would be a MPI application, in docker (with Swarm or kubernetes) you will have to pass trough a lot of pain to set a virtual-cluster to give the images a virtual IP and the discovery service, in order to create the machinefile for the mpirun. With singularity they will just run over the native network and the machinefile wont be an issue.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

------------------------------------------------------------------------------------------------------------------------

CARLOS EDUARDO ARANGO GUTIERREZ.  

Cel : +57 3182545671

Computer Science PhD Student

UNIVERSIDAD DEL VALLE

Laboratorio de redes y sistemas distribuidos - LASCILAB

Tecnología, Información y Complejidad Cuántica - QuanTIC (GrupLAC) 

Centro de Investigación e Innovación en Bioinformática y Fotónica (CIBioFi)

                                                              

[Gregory M. Kurtzer]

These are all fantastic questions, great points, and I'm excited to see so much interest in this! So much in fact, that I want to hire a person or two to investigate this specifically!

Please (privately) send me your resume if you are interested and have the ability and desire to not only investigate this but also develop and contribute updates and necessary enhancements back to the K8s team ASAP (contract work or staff hires)! Let's do this!

Thanks!

Greg

On Wed, May 10, 2017 at 6:50 AM, CARLOS EDUARDO ARANGO GUTIERREZ <carlos.aran...@correounivalle.edu.co> wrote:

Hi James Gundy

> Would this break Singularity? 

No. Any user that run a singularity istance will be the same "user" (with the same permissions over the system) inside the container. 

As long as the "user" don't have root access security is guaranteed.

> does Singularity have any advantages over running images in Docker?

You mean besides the computational performance?.

e.g Since Singularity main goal is for HPC environments, yes, a simple advantage example would be a MPI application, in docker (with Swarm or kubernetes) you will have to pass trough a lot of pain to set a virtual-cluster to give the images a virtual IP and the discovery service, in order to create the machinefile for the mpirun. With singularity they will just run over the native network and the machinefile wont be an issue.

On 10 May 2017 at 08:20, James Gundy <jgu...@gmail.com> wrote:

One of the advantages of Singularity is that there is no user context changes or root escalation (which is great for security reasons).  However, in Kubernetes, there is no concept that I see where you can enforce this.  A user can run a container as any user that the image allows.  Would this break Singularity?  If you can run a Singularity image as any user, does Singularity have any advantages over running images in Docker?

On Tue, May 9, 2017 at 12:52 PM Carlos Eduardo Arango Gutierrez <arangog...@gmail.com> wrote:

+1

On Friday, 5 May 2017 10:58:39 UTC-5, Silveira, Plinio (R&D Center - Brazil) wrote:

Hello, guys.

 

Had someone tried or investigated Kubernetes integration with Singularity?

 

Is singularity somehow prepared to that? For example, does it have some kind of compatibility with CRI (Container Runtime Interface)? Or it was not thought by design?

 

Appreciate any input.

 

Plinio Silveira

 

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

------------------------------------------------------------------------------------------------------------------------

CARLOS EDUARDO ARANGO GUTIERREZ.  

Cel : +57 3182545671

Computer Science PhD Student

UNIVERSIDAD DEL VALLE

Laboratorio de redes y sistemas distribuidos - LASCILAB

Tecnología, Información y Complejidad Cuántica - QuanTIC (GrupLAC) 

Centro de Investigación e Innovación en Bioinformática y Fotónica (CIBioFi)

                                                              

--

[Rob Gardner]

Wondering if anyone got anywhere with this?

[Alan Sill]

There’s this - useful in his and several other contexts. But not yet a complete tool. (We’re working on this in the context of science gateways at the moment.)

https://github.com/dgruber/drmaa2os

Alan

--

[Bruce D'Amora]

Hi,

So this thread seems to have stopped last year. Was it continued in some other forum? Was there a conclusion reached?

[Gregory M. Kurtzer]

Hi Bruce,

Apologies that the thread got dropped, but our interest certainly has not changed. We have begun two projects which will help on this initiative:

1. An OCI compatible interface (both CLI and library) to Singularity. This is a good path forward for community compliance, but it won't support features like cryptographically signed containers via SIF or encryption as they are not OCI compliant.

2. Because OCI doesn't support all of our features, we are also developing a Kubernetes CRI gRPC shim which will allow us to interface Singularity into Kubernetes at the same level as Docker and Podman. This will allow us to support all of our features under K8s.

Also, please note, that we have also prototyped and even demo'ed Singularity running under HashiCorp Nomad for services and AI workflows.

The OCI, Kubernetes and the Nomad work in progress will be opened up in the coming weeks so stay tuned!

Thanks!

Greg

--

You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Gregory M. Kurtzer

CEO, Sylabs Inc.