How to use apptainer to run in a writeable mode?

[Gary]

Hi, I am new to apptainer/singularity, so please forgive me if I am asking dumb questions...

I am running on Ubuntu 22.04 LTS and have installed apptainer-suid as I need to be able to run as a normal user.

I want to be able to run an apptainer container file (originally a docker container) in a HPC environment and I need to be able to write quite a sizeable amount (20GB) of data into the container (temporarily) whilst the job runs and then copy it out to a bound directory on the host before the job ends and the container is exited, leaving the container in it's original state before the job was run.

I can get this to work with an overlay file, but it takes some time to create the 20GB overlay file before each run, and then I need to delete it at the end of the run and create it again from scratch every time I run the container - so is there another way to run the container in a writeable mode that discards the writes when the container is exited?

FWIW - I have tried using "--writable-tmpfs" - but I get an out of space error. I presume that the 20GB of writes are filling tmpfs (which surprises me as the host has 378GB RAM) - how can I check this?

I would appreciate some pointers on how best to do this.

Gary

[Gary Mansell]

I am wondering if it is possible to extract the container.sif to a sandbox folder and then be able to write to the sandbox folder to persist the files - is this possible?

So:

apptainer build --sandbox $SANDBOX_DIR $SIF_CONTAINER

apptainer exec --contain --writeable --bind <inside/outside folders> $SANDBOX_DIR $ENTRY_SCRIPT

When, I try this, I get this error

apptainer exec --contain --writable --bind /shared/home/grma/WRF-Data/WPS_GEOG:/home/WRF_WPS/WPS_GEOG,/shared/home/grma/WRF-Data/wrfinput:/home/WRF_WPS/wrfinput,/shared/home/grma/WRF-Data/outputs:/home/WRF_WPS/WRF/run/outputs $OUT_SANDBOX_DIR /home/WRF_WPS/init_prep_azure.sh

WARNING: By using --writable, Apptainer can't create /shared destination automatically without overlay or underlay
FATAL:   container creation failed: mount hook function failure: mount /var/lib/apptainer/mnt/session/shared->/shared error: while mounting /var/lib/apptainer/mnt/session/shared: destination /shared doesn't exist in container

Why is this - it seems to not be able to bind the home dir (/shared/home/grma) to the container?

What can I do about this?

Rgds

Gary

--
You received this message because you are subscribed to a topic in the Google Groups "discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/a/apptainer.org/d/topic/discuss/YFTwoibuwRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss+u...@apptainer.org.
To view this discussion on the web visit https://groups.google.com/a/apptainer.org/d/msgid/discuss/b05e43a1-0dd9-4291-8586-6ee4c426fa76n%40apptainer.org.

[Dave Dykstra]

Hi Gary,

It should work to use a sandbox with --writable although that's generally
not recommended. The better thing to do is to leave an image read only
but bind in some writable space where you want to be able to write things.
That makes things more reproducible and doesn't require a particular
image format. The --writable-tmpfs option uses a RAM disk so yes its
size is severely limited.

Apptainer handles bind mounts before it launches the container so it can
do that even when an image is mounted read only. However when using the
--writable flag it does not automatically create a bind mount point for
you in an overlay, so you have first create those by hand in the image
before you run apptainer. The Linux kernel requires mount points to be
pre-existing for every bind mount. For the home directory in particular
I think you only need the top level to be present, in your case /shared.

Dave

> > <https://groups.google.com/a/apptainer.org/d/msgid/discuss/b05e43a1-0dd9-4291-8586-6ee4c426fa76n%40apptainer.org?utm_medium=email&utm_source=footer>
> > .
> >
>
> --
> You received this message because you are subscribed to the Google Groups "discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@apptainer.org.
> To view this discussion on the web visit https://groups.google.com/a/apptainer.org/d/msgid/discuss/CAAsSk-zRj_618f8tH6CFPByyYJRoXUYreWevJvCTTqV2eG0VWw%40mail.gmail.com.

[Gary Mansell]

Dave - thanks so much for getting back to me, it sounds like if I create the /shared mount point in the container image (you are right it does not exist) then I will be able to write to the container sandbox. Fab.

Unfortunately, the container has WRF (meteo code) and it writes to a run folder that also contains the program executables. The devs don’t seem confident to get the writes to all go to empty folder locations that we can bind from the outside.

So, if that’s what I have to work with - are there any other ways I can run it without extracting it to a sandbox directory?

Rgds

Gary

> On 29 May 2024, at 19:24, Dave Dykstra <d...@cern.ch> wrote:
>
> Hi Gary,

[Dave Dykstra]

You should be able to use a sif file if you add an --overlay option pointing to a writable directory
https://apptainer.org/docs/user/latest/persistent_overlays.html#directory-overlay
or a filesystem image
https://apptainer.org/docs/user/latest/persistent_overlays.html#filesystem-image-overlay

I would recommend doing some performance measurements, however, because I think especially the filesystem overlay could negatively affect performance.

Dave

[David Godlove]

I think you can use a directory as an overlay too, but there might be some caveats that I'm not thinking of right now.

To view this discussion on the web visit https://groups.google.com/a/apptainer.org/d/msgid/discuss/Zlephq1uAXA-avJF%40cern.ch.

[Gary Mansell]

Thanks both - the trouble that I am having is that I want to run several hundred threads of WRF, and this may involve running several hundred containers with a thread in each to get the MPI to work across node (am fine running many threads in a single container).

I can’t use a single overlay due to file locking issues between the separate container threads and I can’t use multiple overlays as this would take too long to setup and resources.

David - the directory as an overlay shared between all the apptainer instances might be worth a try - how do I do that and are they any gotchas?

Rgds

Gary

On 30 May 2024, at 01:11, David Godlove <davidg...@gmail.com> wrote:



[Gary Mansell]

Hi both, Fab - I have managed to get a local (non-nfs) overlay directory to work with the container.sif file to hold the writes, so this sounds really promising for my multi-node runs that I will want to do in future.

I can copy the SIF container file to a local (very fast striped nvme disks) /scratch folder on the node and create an overlay directory there too to write to.

Thanks for your advice - been really helpful.

Rgds

Gary