Incremental builds

[D.A. P.]

I have a Singularity definition file that I am writing and a full build takes a long time. If I want to make a change to one line in the definition file, such as changing the value for an environment variable in the %environment section, it seems I have to completely rebuild the container. This makes it difficult to author anything but trivial definition files.

Is there a technique for incremental builds that would allow me to rebuild only what changed instead of always rebuilding the entire container?

[David Trudgian]

Singularity images don't have a concept of layers, which in Docker/OCI keep the filesystem changes at each step in the build separately. We don't have layers for performance and convenience reasons in HPC environments, but without layers there is no way to have increments in builds.

If you want to interactively try out commands as you write a definition file, you can build a sandbox (directory) image, and then shell into it with the `--writable` option to work out the commands you need for your def file.

If you want to be able to break up the build of big containers more generally, you may want to split a lengthy build into multiple definitions. Each definition could `Bootstrap: localimage`, `From: xxxx.sif` - where `xxx.sif` is the output of the previous definition file.

E.g....

1) Build a definition file that starts with ubuntu, adds python... build to base.sif

2) Build a definition file that bootstraps from base.sif, adds CUDA + tensorflow .... build it to tensorflow.sif

3) Build a definition file that bootstraps from tensorflow.sif, addes your specific application.

DT

[D.A. P.]

Thanks for suggestions. The --writable option is a good way to go, I just have to ensure I keep good notes on what I run. The chaining of definition files seems like a feasible improvement over my current process as well.