SingularityCE 4.0 and beyond - Bridging the gap between HPC and OCI
David Trudgian
https://sylabs.io/2022/02/singularityce-4-0-and-beyond
After SingularityCE was forked, Sylabs committed to placing a heavy emphasis on open and transparent discussion of the future direction of the project. Our SingularityCE roadmap was created as an open living document that we encourage users and developers to comment on and contribute to directly.
On this roadmap and in our monthly community calls, we previously explored what the next major version of SingularityCE, 4.0, would look like. Based on feedback from users, we held back from committing to a 4.0 release too early, which would bring changes that were primarily focused on overhauling the code, rather than addressing user needs.
Today we’ve updated the roadmap to capture an exciting new focus for SingularityCE, where we are committed to bridging the gap between HPC / batch computing and OCI.
-
In SingularityCE 3.10 we will begin work to integrate an established OCI runtime to replace Singularity’s own limited OCI runtime, used by the singularity oci commands.
-
In SingularityCE 3.11 we will allow this OCI runtime to run native OCI container images in a truly compatible manner via the existing simple shell/exec/run CLI that Singularity users are familiar with. We want to remove the headaches people sometimes experience when they need to bring a container built for Docker to their HPC cluster or other system. We also want to make it easy to continue to benefit from Singularity’s friendly defaults and unique design and approach of integration over isolation.
-
In SingularityCE 4.0 we will introduce SIF encapsulated OCI images, offering a way to leverage the performance and other benefits of SIF, while running a true OCI container image, rather than a translation to Singularity’s current format.
We want to move Singularity forward by recognizing both that OCI based container workflows are being increasingly adopted in HPC environments and that Singularity continues to have significant advantages as a runtime dedicated to batch and interactive work on shared systems, as opposed to running isolated persistent services.
Please let us know what you think! Comment on the roadmap, reach out in Slack or Google Groups, or get involved in the discussion at the next community call.
Read on for a bit more detail about the motivation for these plans, and some answers to the questions we anticipate that you might have…
What’s the gap between Singularity and OCI?
The OCI (Open Containers Initiative) specifications for container images, runtimes, and distribution are indisputably the industry-wide adopted standard for building, running, and sharing containers. Inside the OCI ecosystem, everyone benefits from being able to pull a container from Docker Hub, Quay.io, their on-premise Harbor installation, or any other registry, and knowing it will work with their runtime of choice.
Singularity strives to support fetching and running OCI containers, but it is currently not 100% compatible with all of the OCI standards (and possible container configurations) in its standard workflow. There are good reasons for this. Singularity enabled containers to be used safely on HPC and other shared systems at a time when the design of other runtimes, and lack of deployment of new operating system features, was a serious issue. Singularity was also designed for batch and interactive work, where users typically want to run an application that happens to be provided in a container rather than provision a service that needs to be isolated.
As Docker became widely used for software development and Docker Hub hosted containers became the norm for distributing complex software, functionality was added to Singularity to fetch and execute these containers. This was by means of translation into Singularity’s own container format, and while most Docker/OCI containers translate well, there are some unavoidable limitations.
With the adoption of more modern Linux installations, offering important kernel features, we could gradually adapt Singularity’s approaches to running native Singularity containers until they match or translate 1:1 with the OCI standards. However, HPC environments often move slowly due to the needs of multi-year projects, so significantly changing behavior that users have come to rely on is not a serious option. Our new roadmap aims to fill the gaps while preserving existing behavior for those who depend on it. The best way to do this is to bring an existing low-level OCI runtime into Singularity, just as Docker, podman, and other container platforms sit above runc or crun.
But I heard that Singularity is 100% OCI Compatible?
In some locations, Singularity has been promoted as 100% OCI compatible. It’s true that it does have a mode singularity oci that will run OCI container bundles (which fall under the OCI image-spec) according to the OCI runtime-spec. It can also interact with registries that support the OCI distribution-spec, to pull OCI container images, and can even push SIF images via ORAS.
However, what a user generally understands as OCI compatible is quite different from satisfying individual specifications. Interaction with singularity oci is at a low-level. Not many users expect to have to “mount an OCI bundle” or tackle a config.json file to run a container. To most users, OCI compatibility would sound more like "I can singularity run any Docker Hub container without issue”.
What is SIF encapsulated OCI?
The Singularity Image Format (SIF), introduced in Singularity 3.0, is used to hold a Singularity container’s SquashFS root filesystem but was also designed in a way that makes it flexible enough to hold other kinds of data. We’ve used SIF’s flexibility in the past to support embedded signatures and encrypted containers.
Looking forward to Singularity 4.0 we’re excited to introduce SIF images that directly encapsulate OCI containers. Pulling an OCI image into a SIF will embed the native OCI data and configuration into the SIF file. SingularityCE will be able to run the container straight out of the SIF, without any loss of fidelity from translation to Singularity’s own SquashFS based container format.
The goals of SIF encapsulated OCI images are to extend the performance and simplicity benefits of SIF to OCI containers. Rather than needing to manage local container image stores per host, extract layers to disk etc., containers will run from a single file that can be trivially copied, moved, and shared.
But why stick with Singularity, if it’s going OCI?
There are a number of OCI runtimes available, and they have all made strides toward unprivileged use, bringing them closer to deployment on HPC clusters and other shared systems. With our roadmap to SingularityCE 4.0 we want to deliver the important benefits of these other runtimes without compromising the reasons why users enjoy Singularity.
-
Singularity remains the only mainstream container runtime that is deliberately focused on batch and interactive workloads common to HPC, AI training workflows, etc. You don’t need to understand the implications of a large number of flags and options in order to integrate containers into existing workflows or run them in a complex HPC environment.
-
Our roadmap is designed to bring the benefits of OCI, while maintaining support for existing and future native Singularity images. Users will be able to use a single familiar tool, with a simple CLI interface, for both.
-
The Singularity Image Format (SIF) has unique benefits for performance and distribution, especially in the context of HPC, shared file systems, as well as air-gapped systems.
-
Singularity is the de-facto container standard in HPC. A wealth of applications have been packaged for Singularity, and Singularity is installed on a wide range of clusters. Adoption of OCI in our roadmap avoids the need for institutions and users to plan and execute a complex migration to other runtimes, or to maintain multiple installations.
This sounds like a risky change for me as a user…
We recognize that this roadmap does present quite a significant pivot for Singularity toward OCI. We’re also well aware of the difficulties that users experienced when migrating from Singularity 2.x to 3.0, and are very keen to apply what we learned from that experience.
The roadmap to, and including, 4.0 is defined by adding a new way to run OCI containers with Singularity, and the addition of a new format - SIF encapsulated OCI images. However, 4.0 will not change our default container format or remove support for how you run and build native Singularity containers today. We are committed to preserving existing ways of working, while introducing new features that users can benefit from as and when they are ready to.
How can you help?
The roadmap of any open source project is only as good as the input and feedback it generates. We really do want to hear from you, and are committed to maintaining the roadmap as a living document open to the community, not as simply a statement of Sylabs’ upcoming work.
If you’ve always been interested in getting involved in development work on Singularity, but perhaps found it too complex to understand what’s going on, then this is also a great time to jump in. The Sylabs team will be working throughout the codebase, and we would love to help you understand it so that you can make meaningful contributions of features and fixes that benefit you and the broader community of users. There are going to be big changes to the code overall, but it’s going to be achieved through PRs ranging from a few lines, up to significant rewrites. All are welcome!