Native SLURM integration

[Brian Bockelman]

Hi,

I've been tinkering with the idea of doing native SLURM integration for Singularity via SLURM's SPANK plugin interface.

The idea would be to ship it with a future version of Singularity and have it come in via a sub-RPM (or .deb or ...) of the main package.  Then, users could simply add the following to their submit file:

#SBATCH --image=docker://ubuntu/latest

And automatically get started in their relevant container.  My hope is that, at least locally, this would help us to change to RHEL7 before waiting on the last user to migrate their code.

Before I get too far down this line:

- is this of interest to anyone else?

- is anyone already doing this?

Thanks!

Brian

Sent from my iPhone

[vanessa s]

Holy cow YES. I am so down for this!!

Best,

Vanessa

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[r...@open-mpi.org]

Questions:

1. Why does the user have to put anything in their submit file? All we do in OMPI is check the file type, and if it is “singularity”, then treat it accordingly - which really means doing nothing more than prefacing the container name with the “singularity” command.

2. Why does Slurm care if it is a container or not? Either way, it is just an executable so far as the scheduler is concerned, and in a batch script, putting “singularity” in front of your image seems a pretty trivial thing to do.

[vanessa s]

I think he is talking about more of a use case to provide interactive (container) environments based on a set of images, for example to support different OS? E.g., the container environment is the idev node. Running the container (from the outside) as an executable is a bit different I think. I could be totally off, but I think this is an interesting idea for a a slightly different reason. I'm developing singularity hub, (the singularity equivalent of Docker hub) and it's going to work in a similar way - a command line tool takes some run command, looks to see if the containers are cached, and if not, pulls and runs. This kind of tool would then need different plugins for easily integrating into different running environments (SLURM being one of them, because you know, the other big players in the game have been such great supporters of HPC *smile*). The early use case is just downloading one container and running it,and we wouldn't need anything very sophisticated for that.  @rhc - you are right, a container in this case is just another executable. The later (cooler!) use case is asking to run a workflow, which would have dependencies and multiple containers. In the simplest implementation, it correctly sets up the workflow, assesses the queues that the user has access to, allocates correct memory and runtimes (possibly needing different queues), and then runs,and produces outputs. In a more sophisticated implementation, it captures different meta data about the analysis and run, and stores that somewhere (for reproducibility). For all of those out there interested in this idea of "push button HPC" from a browser, that could all happen from some web interface. Thus, a native SLURM plugin that can manage those dependencies and advanced use cases would be... super awesome :)

I think there are other potential cool ideas for having a SLURM integration, some of which may not be stumbled upon until we dig in a bit! The early set up, while maybe simple, will teach us things and possibly lead us there. Thus, I think this is definitely worth exploring.

On Sat, Oct 29, 2016 at 11:24 AM, r...@open-mpi.org <r...@open-mpi.org> wrote:

Questions:

1. Why does the user have to put anything in their submit file? All we do in OMPI is check the file type, and if it is “singularity”, then treat it accordingly - which really means doing nothing more than prefacing the container name with the “singularity” command.

2. Why does Slurm care if it is a container or not? Either way, it is just an executable so far as the scheduler is concerned, and in a batch script, putting “singularity” in front of your image seems a pretty trivial thing to do.

On Oct 29, 2016, at 11:19 AM, vanessa s <vso...@gmail.com> wrote:

Holy cow YES. I am so down for this!!

Best,

Vanessa

On Oct 29, 2016, at 11:16 AM, Brian Bockelman <bock...@gmail.com> wrote:

Hi,

I've been tinkering with the idea of doing native SLURM integration for Singularity via SLURM's SPANK plugin interface.

The idea would be to ship it with a future version of Singularity and have it come in via a sub-RPM (or .deb or ...) of the main package.  Then, users could simply add the following to their submit file:

#SBATCH --image=docker://ubuntu/latest

And automatically get started in their relevant container.  My hope is that, at least locally, this would help us to change to RHEL7 before waiting on the last user to migrate their code.

Before I get too far down this line:

- is this of interest to anyone else?

- is anyone already doing this?

Thanks!

Brian

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[r...@open-mpi.org]

On Oct 29, 2016, at 11:44 AM, vanessa s <vso...@gmail.com> wrote:

I think he is talking about more of a use case to provide interactive (container) environments based on a set of images, for example to support different OS? E.g., the container environment is the idev node. Running the container (from the outside) as an executable is a bit different I think. I could be totally off, but I think this is an interesting idea for a a slightly different reason. I'm developing singularity hub, (the singularity equivalent of Docker hub) and it's going to work in a similar way - a command line tool takes some run command, looks to see if the containers are cached, and if not, pulls and runs. This kind of tool would then need different plugins for easily integrating into different running environments (SLURM being one of them, because you know, the other big players in the game have been such great supporters of HPC *smile*). The early use case is just downloading one container and running it,and we wouldn't need anything very sophisticated for that.  @rhc - you are right, a container in this case is just another executable. The later (cooler!) use case is asking to run a workflow, which would have dependencies and multiple containers. In the simplest implementation, it correctly sets up the workflow, assesses the queues that the user has access to, allocates correct memory and runtimes (possibly needing different queues), and then runs,and produces outputs. In a more sophisticated implementation, it captures different meta data about the analysis and run, and stores that somewhere (for reproducibility). For all of those out there interested in this idea of "push button HPC" from a browser, that could all happen from some web interface. Thus, a native SLURM plugin that can manage those dependencies and advanced use cases would be... super awesome :)

FWIW: the PMIx community is working on a parser that will take as input a script or an executable, scan it for executables, determine the type of that executable (e.g., container, binary, Java, etc), and provide a list of all the library dependencies for it. This is being done to support schedulers so they can pre-position those dependencies, thus enabling faster startup. PMIx has already being integrated into many of the HPC schedulers to support fast startup, and so this is a natural extension.

Also FWIW: there are several good HPC schedulers out there, SLURM being only one of them. Those of us working on libraries to support the community generally try to remain RM-agnostic as they all have their pros/cons and dedicated market share. Ultimately, I think Greg would like to see Singularity equally well supported by them all, so if we can find mechanisms that don’t require RM-specific code, that would be best.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Brian Bockelman]

In this use case, I don't think of Singularity as something the user
invokes as part of a script in a job - but that the OS environment
becomes a job property, and it is the batch system's responsibility to
set things up. Singularity is just giving an assist to the batch
system.

For the inspiration, see NERSC's Shifter integration:

http://www.nersc.gov/users/software/using-shifter-and-docker/running-jobs-with-shifter/

This would take the idea one step further and remove all references to
the underlying implementation (ideally, someone could write an
equivalent SLURM plugin for Docker or RunC or .... and keep the same
interface).

To me, this is a more useful abstraction - oh, and you could then
deploy and use Singularity on the worker node without setuid, a BIG
plus.

As far as RM-specific code: indeed, where possible, it's useful to
push the integration to the RM itself. That's how it works with
HTCondor. However, the SLURM plugin system is a bit different.

I don't mind having RM-specific code, provided that it doesn't come at
the cost of different RM. I'd hope we'd accept a pull request from
Torque or SGE in the same manner. As a separate analogy - we have
lots of Docker-specific code, but it doesn't mean that we'd reject
that enables Great Format X (assuming it's maintainable, clean, has a
maintainer, etc, etc).

Brian