Singularity within Docker: could not virtualize mount namespace

[Chris Collins]

Hi,

I've just starting playing around with Singularity as a way of potentially providing access to containerised workflows. I am however running into some issues with getting it working in my already Docker based production environment. I've got a simple example working in a similar test environment (though not exactly the same) but in principal, I know Singularity works fine from within Docker.

Anyway, the error I'm getting is:

singularity shell -C docker-anaconda27.img

ERROR  : Could not virtualize mount namespace: Operation not permitted

ABORT  : Retval = 255

I'm running as root at this stage to rule out any permission issue. A snip of the debug output doesn't make anything clearer:

VERBOSE [U=0,P=41280]      pid.c:54:singularity_ns_pid_unshare()      : Not virtualizing PID namespace by configuration

DEBUG   [U=0,P=41280]      config_parser.c:62:singularity_config_rewind(): Rewinding configuration file

DEBUG   [U=0,P=41280]      config_parser.c:107:singularity_config_get_bool(): Called singularity_config_get_bool(mount slave, 0)

DEBUG   [U=0,P=41280]      config_parser.c:80:singularity_config_get_value(): Called singularity_config_get_value(mount slave)

VERBOSE [U=0,P=41280]      config_parser.c:91:singularity_config_get_value(): Got config key mount slave (= 'no')

DEBUG   [U=0,P=41280]      config_parser.c:118:singularity_config_get_bool(): Return singularity_config_get_bool(mount slave, 0) = 0

DEBUG   [U=0,P=41280]      privilege.c:147:singularity_priv_escalate(): Running as root, not changing privileges

DEBUG   [U=0,P=41280]      mnt.c:54:singularity_ns_mnt_unshare()      : Virtualizing FS namespace

DEBUG   [U=0,P=41280]      mnt.c:61:singularity_ns_mnt_unshare()      : Virtualizing mount namespace

ERROR   [U=0,P=41280]      mnt.c:63:singularity_ns_mnt_unshare()      : Could not virtualize mount namespace: Operation not permitted

I have "allow pid ns = no" as this was also kicking up an error. I've tried setting all mount options in the conf to no.

The same command from the same installation works fine on the bare metal system.

Does anyone have any suggestions of what might be the issue or at least where to start looking?

Thanks very much,

Chris

[Gregory M. Kurtzer]

Hi Chris,

To use Singularity within a Docker container I think you must use the --privileged Docker option. 

Hope that helps!

Greg 

--


You received this message because you are subscribed to the Google Groups "singularity" group.


To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Chris Collins]

Hi Greg,

Thanks for getting back to me with that. 

That does sound right, and I'm pretty sure Docker in my production env isn't setting privileged, while journalctl is recording the launch with privileged=false.

However, unless I'm missing something, the Docker in my VM test environment (where singularity works) doesn't explicitly use the --privileged option and from journalctl it also looks like privileged is false.

Could there be anything else that would result in similar behaviour? Does anyone know if there is a way of actually testing privilege requirements from within a docker container?

Thanks,

Chris

[Chris Collins]

Hi,

Just to fill in some details on this for those interested, having resumed looking at this post-Christmas....

In my (VM) test environment, the obvious indicators are that the docker container is running non-privilege (I'm even starting docker with --privileged=false) and docker inspect is showing:

[root@centos7base ~]# docker inspect --format='{{.HostConfig.Privileged}}' test-docker

false

However, within the container, things are a bit different - using an approach suggested at http://stackoverflow.com/questions/32144575/how-to-know-if-a-docker-container-is-running-in-privileged-mode specifically the following, seems to indicate I have at least some privilege. I guess that is why singularity works here.

ip link add dummy0 type dummy >/dev/null
if [[ $? -eq 0 ]]; then
    PRIVILEGED=true
    # clean the dummy0 link
    ip link delete dummy0 >/dev/null
else
    PRIVILEGED=false
fi

On my real production servers, the docker inspect comes back the same, but the above code errors with "Operation not permitted".

I've only recently started looking at Docker and singularity, so I need to do a little more investigation i.e. is there some global Docker setting impacting here or is there something within the container that allows this? 

Either way, as things stand it is a no go in my production environment, which is a real shame.

Thanks anyway,

Chris

[Brian Bockelman]

Hi Chris,

Locally, we were able to get singularity-inside-Docker working by
adding the following to the Docker command line:

--cap-add=SYS_ADMIN

That gives the container more privilege for setuid apps, but
significantly less than adding --privileged. The full list of default
caps is here:

https://docs.docker.com/engine/reference/run/

If you drop the default caps, the other ones needed are:

DAC_OVERRIDE
SETUID
SETGID
SYS_CHROOT

That said, we still only allow this flag when the user is utilizing
sysadmin-provided images. For user-provided docker images, we drop
all caps.

HTH,

Brian

On Tue, Jan 10, 2017 at 4:40 AM, 'Chris Collins' via singularity

[Chris Collins]

Hi Brian,

Thanks very much for the suggestion on that - I'll give that a try when resource is available. There is a single top level docker image which is managed by us, so it would only be the singularity images which are user provided.

Just having a quick look with pscap and comparing docker on my real (where is doesn't work) and test (where it does work) systems the capabilities look the same, but I will give the explicit SYS_ADMIN capability a try:

Real:

6881  6882  root        docker            full

6882  25028 root        supervisord       chown, dac_override, fowner, fsetid, kill, setgid, setuid, setpcap, net_bind_service, net_raw, sys_chroot, mknod, audit_write, setfcap

Test:

1     1432  root        docker-current    full

1432  2042  root        supervisord       chown, dac_override, fowner, fsetid, kill, setgid, setuid, setpcap, net_bind_service, net_raw, sys_chroot, mknod, audit_write, setfcap

 

Thanks,

Chris