Writable overlay for /cvmfs

[Bob Dröge]

Hi all,

I'm trying to run a Singularity container with a writable overlay on a (read-only) /cvmfs mount. I'm using the new --fusemount option in Singularity 3.6 to mount the CVMFS repos inside the container, which works fine (and which is a great feature).

I then tried using a writable overlay by using the --writable option, but that does not allow me to make changed to /cvmfs ("Function not implemented" when trying to make a file).

Now I just found fuse-overlayfs:

https://github.com/containers/fuse-overlayfs

and was hoping to use it in the same way with --fusemount. This does work, in the sense that I can make new files in the overlay mountpoint. However, lots of commands don't seem to work well. For instance, doing a "mkdir -p" or even a "cd" on/to the mountpoint fails with: Operation not permitted. To rule out that it has something to do with CVMFS, I did the following simple test:

mkdir -p /tmp/overlay/{lower,upper,work,merged}

singularity shell --fusemount "container:fuse-overlayfs -o lowerdir=/tmp/overlay/lower -o upperdir=/tmp/overlay/upper -o workdir=/tmp/overlay/work /tmp/overlay/merged" fuse-overlay.sif

Singularity> cd /tmp/overlay/merged/
bash: cd: /tmp/overlay/merged/: Operation not permitted

Running this on the host itself works just fine, so it seems something weird is going on in the container. I also tried passing options like '-o allow_other', but that does not help. Is there any way to make this work?

Best regards,

Bob

[Dave Dykstra]

Hi Bob,

This sounds a lot like the problem we had with the 'overlay' feature on
top of cvmfs with singularity-2.x. The problem was that the /tmp
filesystem type (tmpfs) does not support extended attributes, and cvmfs
does. Try it with the upper layer on a full-featured filesystem like
ext3.

P.S. since you're using the singularity --fusemount option with cvmfs,
be sure checkout 'singcvmfs' from
https://github.com/cvmfs/cvmfsexec

Dave

> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/1e3da714-7dcf-4f8f-99c0-650daa2e7a5en%40lbl.gov .

[Bob Dröge]

Hi Dave,

Thanks for the quick response, and the link to cvmfsexec: that could definitely be useful!

I just tried the same thing on a new subdirectory on my root partition (which is ext4), but that did not work either. I also tried making an ext3 file system image (in the same way as described here: https://sylabs.io/guides/3.6/user-guide/persistent_overlays.html#file-system-image-overlay), mounting that on my host, and using this for the upper layer (and work directory). Unfortunately, that did not work either. As a final attempt I tried it on a VM with a separate ext3 disk, but no luck either... :-( In all cases I get this "Operation not permitted" error when I try to cd to the merged mountpoint.

Do you maybe have any other suggestions? Since you mention you had similar issues with an overlay on cvmfs with Singularity 2.x, does it mean that this should work / has worked before?

Best regards,

Bob

Op vr 28 aug. 2020 om 19:30 schreef Dave Dykstra <d...@fnal.gov>:

You received this message because you are subscribed to a topic in the Google Groups "singularity" group.
To unsubscribe from this topic, visit https://groups.google.com/a/lbl.gov/d/topic/singularity/2CobnkVUl0w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to singularity...@lbl.gov.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/20200828173022.GA82236%40fnal.gov.

--

Bob Dröge
Team High Performance Computing
Center for Information Technology, University of Groningen
P.O. Box 11044, 9700 CA  Groningen, The Netherlands
+31 50 363 9240
b.e....@rug.nl

[Kenneth Hoste]

I'm seeing similar issues when using fuse-overlay: "Permission denied" when trying to cd into a directory in the overlay, and also stuff like this (not how "ls -l" shows the right permissions, but "test -x" & co disagree):

$ mkdir -p $HOME/singularity-overlay-test/{lower,upper,work} test
$ singularity shell --fusemount "container:fuse-overlayfs -o lowerdir=$HOME/singularity-overlay-test/lower -o upperdir=$HOME/singularity-overlay-test/upper -o workdir=$HOME/singularity-overlay-test/work $PWD/test" fuse-overlay.sif

Singularity> touch test/afile
Singularity> chmod u+x test/afile
Singularity> ls -l test/afile
-rwxr--r-- 1 kehoste users 0 Aug 29 14:44 test/afile
Singularity> test -x test/afile
Singularity> echo $?
1
Singularity> test -r test/afile
Singularity> echo $?
1
Singularity> test -w test/afile
Singularity> echo $?
1
Singularity> cd test
bash: cd: test: Operation not permitted
Singularity>


My home directory in which the overlay layers are located is ext4:

$ mount
...
/dev/mapper/vg0-var on /var type ext4 (rw,relatime,data=ordered)


regards,

Kenneth

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/CA%2BBbwcEVMUysZOGAg_EcXNdkpXeY-hZJ%3D1c5znTuf2_L2VuvCA%40mail.gmail.com.

[Bob Dröge]

Today I tested this with a Debian instead of CentOS container, and then it suddenly worked fine. Debian installed a very old version of fuse-overlayfs, though. Testing different versions showed that version 0.4.1 and older seem to work fine on both CentOS and Debian, while 0.5 and newer often give these "Operation not permitted" errors. I don't know if it's something in Singularity or some bug in fuse-overlayfs (which does work fine when running without containers), but to be sure I also opened an issue on the fuse-overlayfs github page:

https://github.com/containers/fuse-overlayfs/issues/232 

Best regards,

Bob

Op zaterdag 29 augustus 2020 om 16:49:08 UTC+2 schreef Kenneth Hoste:

[David Trudgian]

FWIW - I’m not seeing this issue with fuse-overylayfs 1.1.2 on Fedora. Even 0.5 is quite old given that the fuse-overlayfs was in its early stages then, so it’s possible a lot has been changed / fixed since 0.5.

I put my notes showing it working on the GitHub issue.

If you still have problems with a newer fuse-overlays then it’s worth looking at the details of the host.What is the distro and kernel version are in use? I am also using a Fedora *host* in my test, with kernel 5.7.14-200.fc32.x86_64

DT

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/1d3ac6cb-fdbc-47d7-ae6e-8c1a7919b32cn%40lbl.gov.

[Dave Dykstra]

Bob,

What's working now in 3.x (and didn't work in 2.x) is the "use overlay"
option that can be set in /etc/singularity/singularity.conf. That uses
the kernel overlayfs to enable adding bind mountpoints at places where
directories do not exist in the underlying image, including when the
image is in cvmfs.

I went looking for the github issue describing the old problem, and I
realize now that the error message is slighly different. The previous
message was "Operation not supported", not "Operation not permitted."
https://github.com/hpcng/singularity/issues/1797
In this issue Cedric explains that in 3.x singularity uses a trick of
making the image the upper layer and puts the extra directories in a
tmpfs lower layer.

I haven't tried using fuse-overlayfs on top of cvmfs like you're trying
to do, but hopefully the primary developer of fuse-overlayfs Giuseppe
Scrivano can help you on your github issue. He is familiar with cvmfs
and has helped me a lot.

Dave

On Fri, Aug 28, 2020 at 11:06:42PM +0200, Bob Dröge wrote:
...

> +31 50 363 9240b....@rug.nl

>
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/CA%2BBbwcEVMUysZOGAg_EcXNdkpXeY-hZJ%3D1c5znTuf2_L2VuvCA%40mail.gmail.com .

[Bob Dröge]

Thanks Dave. Just a little background of what we're trying to do: we are building a large software repository on CVMFS, and we would like to have a Singularity container (which doesn't need to be stored on /cvmfs) that anyone can use for locally building/testing/installing extra software on/to the readonly /cvmfs mountpoint, ultimately without needing root privileges. This would allow us to build the software on nodes with specific architectures, and when the build/installation succeeds, we can take the installation from the overlay and copy it to a publisher node, start a transaction on the CVMFS repository, and ingest the new installation.

So, ideally, we would have a container that mounts some CVMFS repositories (this already works nicely with the --fusemount option), and adds a writable overlay on top of it. For the latter I tried to use Singularity's built-in --overlay functionality, but that does not seem to work with CVMFS (or can this be achieved in some way?): I get "Function not implemented" errors when trying to add files to /cvmfs. So then I found out about fuse-overlayfs, and really hoped that this would work. 

If you have any other ideas/suggestions for achieving this, please let me know!

Bob

Op maandag 31 augustus 2020 om 19:25:45 UTC+2 schreef Dave Dykstra:

[Dave Dykstra]

Hi Bob,

The cvmfs development team has thought a lot about similar capabilities,
but I'm not entirely sure what the state of the art on it is or the
state of the plans. I suggest bringing it up on the cvmfs...@cern.ch
mailing list.

Also, the next big singularity release 3.7 is expected to have new
capabilities for unprivileged singularity based on the Linux Kernel
Library which enables using real overlayfs in user space (and user-space
mounts) which may end up solving the problem.

Dave

[David Trudgian]

Hi Dave, Bob,

Also, the next big singularity release 3.7 is expected to have new
capabilities for unprivileged singularity based on the Linux Kernel
Library which enables using real overlayfs in user space (and user-space
mounts) which may end up solving the problem.

I’m afraid this isn’t an absolute. It was on the general roadmap prior to Greg’s departure from Sylabs, and Cedric was driving this forward, but it wasn’t added to the new collaborative roadmap document for 3.7 by him or anyone else at Ctrl-Cmd:

https://docs.google.com/document/d/1jBHjIKOpnI3HeTFMDYIBjxsSjE4PfCHm8Qugsc_HGOY/edit?usp=sharing

Cedric / Greg - are you able to comment on what the status of this work is?

Cheers,

DT