failed to resolve session directory

[Loris Bennett]

Hi,

Disclaimer: I have no previous experience of Singularity or any other
container frameworks.

I am trying to provide an image for users of an HPC system. As root on
the administration node where Singularity was installed I can create an
image from a Docker container and successfully shell into it.

However as a regular user on a different node accessing Singularity via
NFS I can create the container image

$ singularity pull docker://chloroextractorteam/org.asm_docker
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob 22dc81ace0ea done
...
Copying config faa1d033a8 done
Writing manifest to image destination
Storing signatures
2020/05/29 10:09:40 info unpack layer: sha256:22dc81ace0ea2f45ad67b790cddad29a45e206d51db0af826dc9495ba21a0b06
2020/05/29 10:09:40 warn rootless{dev/agpgart} creating empty file in place of device 10:175
...
2020/05/29 10:09:40 warn rootless{dev/zero} creating empty file in place of device 1:5
2020/05/29 10:09:44 info unpack layer: sha256:1a8b3c87dba3ed16956c881a26eb0c40502c176a35767627d87557d16ea1e0df
...
2020/05/29 10:10:08 info unpack layer: sha256:f7ab963cb320552fbc412ed86403356ec38214e73c27e46196b008f3f5a979c2
INFO: Creating SIF file...
INFO: Build complete: org.asm_docker_latest.sif

but not shell into it

$ singularity shell org.asm_docker_latest.sif
FATAL: container creation failed: failed to resolve session directory /var/singularity/mnt/session: lstat /var/singularity: no such file or directory

The administration node which exports the directory containing
Singularity via NFS read-only to all other nodes in the cluster. The
directory /var/singularity exists on the NFS server:

drwxr-xr-x 3 root root 17 May 29 07:51 /var/singularity/

What bit of the setup have I missed to allow an unprivileged user to
access the image properly?

Cheers,

Loris

--
This signature is currently under construction.

[Loris Bennett]

I'm using

singularity version 3.5.3

build as an RPM with

PREFIX=/trinity/shared/software/singularity
rpmbuild -ta --define "_prefix $PREFIX" --define "_sysconfdir $PREFIX/etc" --define "_defaultdocdir $PREFIX/share" singularity-3.5.3.tar.gz

Should the 'var' directory be set here too? The permissions will still
be wrong though.

[Loris Bennett]

Just creating the following directory allows shelling into the image:

$ ll -d /var/singularity/mnt/session/
drwxr-xr-x 2 root root 4096 May 29 13:01 /var/singularity/mnt/session/

However there will obviously a problem if an unprivileged process ever
tries to write anything there.

Although this works, I was unable to find any documentation about this
being necessary, so I assume I am doing something wrong.

Cheers,

Loris

[Thomas Hartmann]

Hi Loris,

maybe you can try instead of pulling to build a (sandbox) container
explicitly

> singularity build {--sandbox} /nfs/some/path/container.d
docker://chloroextractorteam/org.asm_docker

From my experiences, network filesystems can sometimes be a bit tricky
(nosuid-mounted - which I guess should be good practise).

I would first try a sandbox directory build, i.e., an expanded directory
tree and check, if the permissions are OK (Singularity 3 should set
permissions for a general access AFAIK). Else you can try something like
on the container directory tree [1].
The SIF container image (without --sandbox) should work as well, but for
debugging I prefer to have the directory tree at hand ;)

Cheers,
Thomas



[1]
> chmod -R a+r container.d
> find container.d -type d -exec chmod 0755 {} \;

[v]

I don't have an answer for you (too early!) but Dave just merged some really helpful admin docs that I think might be helpful - https://github.com/dctrud/singularity-admindocs/blob/e46c17102df71918a023bc785e7a61d07b732a93/installation.rst#filesystem-support--limitations. It could be just related to your filesytem (and not that you are doing something wrong!)

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/87eer30yuz.fsf%40hornfels.zedat.fu-berlin.de.

[Loris Bennett]

Thanks for the pointer, v, that was a great help.

It lead me to

https://singularity-admindoc.readthedocs.io/en/latest/admin_quickstart.html#prefix-in-special-places-localstatedir

where the use of --localstatedir is mentioned, and where it says:

In the case of cluster nodes, you will need to ensure the following
directories are created on all nodes, with root:root ownership and
0755 permissions:

${localstatedir}/singularity/mnt
${localstatedir}/singularity/mnt/container
${localstatedir}/singularity/mnt/final
${localstatedir}/singularity/mnt/overlay
${localstatedir}/singularity/mnt/session

Singularity will fail to execute without these directories. They are
normally created by the install make target; when using a local
directory for --localstatedir these will only be created on the node
make is run on.

So these directories do just all have to be added to the image for the
cluster nodes.

As it is /var would be in memory, since our nodes are nominally
diskless. However, the nodes do in fact have some small local scratch
space, which could also be used.

Is there any reason to prefer one variant over the other, perhaps in
terms of the type and amount of data which might be created?

Cheers,

Loris