Unable to run shell subcommand with Singularity 3.0

[Trey Dockendorf]

One of my colleagues testing Singularity 3.0.0 ran into an issue where they are unable to run "singularity shell" in a way that works for others.  I'm unable to identify what about this peron's account is breaking singularity.  The same commands work just fine for me and for other staff testing one this same system.

Thanks,

- Trey

Short version:

p0070:~$ singularity shell docker://centos:7 

WARNING: passwd file doesn't exist in container, not updating

WARNING: group file doesn't exist in container, not updating

FATAL:   exec /.singularity.d/actions/shell failed: permission denied

Debug output:

p0070:~$ singularity --debug shell docker://centos:7 

DEBUG   [U=21485,P=76942]  parseURI()                    Parsing docker://centos:7 into reference

DEBUG   [U=21485,P=76942]  updateCacheSubdir()           Caching directory set to /tmp/pbstmp.15579/oci-tmp/67dad89757a55bfdfabec8abd0e22f8c7c12a1856514726470228063ed86593b

DEBUG   [U=21485,P=76942]  updateCacheSubdir()           Caching directory set to /tmp/pbstmp.15579/oci-tmp

DEBUG   [U=21485,P=76942]  updateCacheSubdir()           Caching directory set to /tmp/pbstmp.15579/oci-tmp/67dad89757a55bfdfabec8abd0e22f8c7c12a1856514726470228063ed86593b

DEBUG   [U=21485,P=76942]  updateCacheSubdir()           Caching directory set to /tmp/pbstmp.15579/oci-tmp

VERBOSE [U=0,P=76942]      singularity_message_level()               Set messagelevel to: 5

DEBUG   [U=0,P=76942]      init()                                    PIPE_EXEC_FD value: 8

VERBOSE [U=0,P=76942]      init()                                    Container runtime

VERBOSE [U=0,P=76942]      is_suid()                                 Check if we are running as setuid

DEBUG   [U=0,P=76942]      init()                                    Overlay seems supported by kernel

DEBUG   [U=0,P=76942]      init()                                    Drop privileges

DEBUG   [U=21485,P=76942]  init()                                    Read json configuration from pipe

DEBUG   [U=21485,P=76942]  init()                                    Set child signal mask

DEBUG   [U=21485,P=76942]  init()                                    Create socketpair for smaster communication channel

DEBUG   [U=21485,P=76942]  init()                                    Wait C and JSON runtime configuration from scontainer stage 1

DEBUG   [U=21485,P=76965]  set_parent_death_signal()                 Set parent death signal to 9

VERBOSE [U=21485,P=76965]  init()                                    Spawn scontainer stage 1

VERBOSE [U=21485,P=76965]  priv_escalate()                           Get root privileges

DEBUG   [U=0,P=76965]      set_parent_death_signal()                 Set parent death signal to 9

DEBUG   [U=0,P=76965]      prepare_scontainer_stage()                Entering in scontainer stage 1

DEBUG   [U=21485,P=76965]  set_parent_death_signal()                 Set parent death signal to 9

VERBOSE [U=21485,P=76965]  startup()                     Execute scontainer stage 1

DEBUG   [U=21485,P=76965]  SContainer()                  Entering scontainer stage 1

DEBUG   [U=21485,P=76965]  prepareFd()                   Open file descriptor for /fs/project

DEBUG   [U=21485,P=76965]  prepareFd()                   Open file descriptor for /fs/scratch

DEBUG   [U=21485,P=76965]  prepareFd()                   Open file descriptor for /apps/xalt/xalt

DEBUG   [U=21485,P=76965]  Init()                        Entering image format intializer

DEBUG   [U=21485,P=76965]  Init()                        Check for image format sif

DEBUG   [U=21485,P=76942]  init()                                    Receiving configuration from scontainer stage 1

DEBUG   [U=21485,P=76942]  init()                                    Wait completion of scontainer stage1

VERBOSE [U=21485,P=76942]  priv_escalate()                           Get root privileges

VERBOSE [U=0,P=76942]      create_namespace()                        Create mount namespace

DEBUG   [U=0,P=76942]      init()                                    Create RPC socketpair for communication between scontainer and RPC server

VERBOSE [U=0,P=76942]      init()                                    Spawn smaster process

DEBUG   [U=0,P=76972]      set_parent_death_signal()                 Set parent death signal to 9

VERBOSE [U=0,P=76972]      init()                                    Spawn scontainer stage 2

VERBOSE [U=0,P=76972]      create_namespace()                        Create mount namespace

VERBOSE [U=0,P=76973]      init()                                    Spawn RPC server

VERBOSE [U=21485,P=76942]  startup()                     Execute smaster process

VERBOSE [U=0,P=76973]      startup()                     Serve RPC requests

DEBUG   [U=21485,P=76942]  setupSessionLayout()          Attempting to use overlayfs (enable overlay = try)

DEBUG   [U=21485,P=76942]  setupOverlayLayout()          Creating overlay SESSIONDIR layout

DEBUG   [U=21485,P=76942]  addRootfsMount()              Mount rootfs in read-only mode

DEBUG   [U=21485,P=76942]  addRootfsMount()              Mounting block [squashfs] image: /tmp/pbstmp.15579/oci-tmp/67dad89757a55bfdfabec8abd0e22f8c7c12a1856514726470228063ed86593b/centos_7.sif

DEBUG   [U=21485,P=76942]  addKernelMount()              Checking configuration file for 'mount proc'

DEBUG   [U=21485,P=76942]  addKernelMount()              Adding proc to mount list

DEBUG   [U=21485,P=76942]  addKernelMount()              Checking configuration file for 'mount sys'

DEBUG   [U=21485,P=76942]  addKernelMount()              Adding sysfs to mount list

DEBUG   [U=21485,P=76942]  addDevMount()                 Checking configuration file for 'mount dev'

DEBUG   [U=21485,P=76942]  addDevMount()                 Adding dev to mount list

DEBUG   [U=21485,P=76942]  addHostMount()                Not mounting host file systems per configuration

VERBOSE [U=21485,P=76942]  addBindsMount()               Found 'bind path' = /etc/localtime, /etc/localtime

VERBOSE [U=21485,P=76942]  addBindsMount()               Found 'bind path' = /etc/hosts, /etc/hosts

VERBOSE [U=21485,P=76942]  addBindsMount()               Found 'bind path' = /fs/project, /fs/project

VERBOSE [U=21485,P=76942]  addBindsMount()               Found 'bind path' = /fs/scratch, /fs/scratch

VERBOSE [U=21485,P=76942]  addBindsMount()               Found 'bind path' = /apps/xalt/xalt, /apps/xalt/xalt

DEBUG   [U=21485,P=76942]  addHomeStagingDir()           Staging home directory (/users/PZS0530/dheisterberg) at /var/singularity/mnt/session/users/PZS0530/dheisterberg

DEBUG   [U=21485,P=76942]  addHomeMount()                Adding home directory mount [/var/singularity/mnt/session/users/PZS0530/dheisterberg:/users/PZS0530/dheisterberg] to list using layer: overlay

DEBUG   [U=21485,P=76942]  isLayerEnabled()              Using Layer system: overlay

DEBUG   [U=21485,P=76942]  addTmpMount()                 Checking for 'mount tmp' in configuration file

DEBUG   [U=21485,P=76942]  addScratchMount()             Not mounting scratch directory: Not requested

DEBUG   [U=21485,P=76942]  addLibsMount()                Checking for 'user bind control' in configuration file

DEBUG   [U=21485,P=76942]  addResolvConfMount()          Adding /etc/resolv.conf to mount list

DEBUG   [U=21485,P=76942]  addHostnameMount()            Skipping hostname mount, not virtualizing UTS namespace on user request

DEBUG   [U=21485,P=76942]  create()                      Mount all

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting tmpfs to /var/singularity/mnt/session

DEBUG   [U=21485,P=76942]  mountImage()                  Mounting loop device /dev/loop0 to /var/singularity/mnt/session/rootfs

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting overlay to /var/singularity/mnt/session/final

DEBUG   [U=21485,P=76942]  setSlaveMount()               Set RPC mount propagation flag to SLAVE

VERBOSE [U=21485,P=76942]  Passwd()                      Checking for template passwd file: /var/singularity/mnt/session/rootfs/etc/passwd

WARNING [U=21485,P=76942]  addIdentityMount()            passwd file doesn't exist in container, not updating

VERBOSE [U=21485,P=76942]  Group()                       Checking for template group file: /var/singularity/mnt/session/rootfs/etc/group

WARNING [U=21485,P=76942]  addIdentityMount()            group file doesn't exist in container, not updating

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting /dev to /var/singularity/mnt/session/final/dev

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting /etc/localtime to /var/singularity/mnt/session/final/etc/localtime

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting /etc/hosts to /var/singularity/mnt/session/final/etc/hosts

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting /fs/project to /var/singularity/mnt/session/final/fs/project

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting /fs/scratch to /var/singularity/mnt/session/final/fs/scratch

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting /apps/xalt/xalt to /var/singularity/mnt/session/final/apps/xalt/xalt

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting /etc/singularity/actions to /var/singularity/mnt/session/final/.singularity.d/actions

DEBUG   [U=21485,P=76942]  mountGeneric()                Remounting /var/singularity/mnt/session/final/.singularity.d/actions

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting /proc to /var/singularity/mnt/session/final/proc

DEBUG   [U=21485,P=76942]  mountGeneric()                Remounting /var/singularity/mnt/session/final/proc

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting sysfs to /var/singularity/mnt/session/final/sys

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting /users/PZS0530/dheisterberg to /var/singularity/mnt/session/users/PZS0530/dheisterberg

DEBUG   [U=21485,P=76942]  mountGeneric()                Remounting /var/singularity/mnt/session/users/PZS0530/dheisterberg

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting /var/singularity/mnt/session/users/PZS0530/dheisterberg to /var/singularity/mnt/session/final/users/PZS0530/dheisterberg

DEBUG   [U=21485,P=76942]  mountGeneric()                Remounting /var/singularity/mnt/session/final/users/PZS0530/dheisterberg

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting /tmp to /var/singularity/mnt/session/final/tmp

DEBUG   [U=21485,P=76942]  mountGeneric()                Remounting /var/singularity/mnt/session/final/tmp

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting /var/tmp to /var/singularity/mnt/session/final/var/tmp

DEBUG   [U=21485,P=76942]  mountGeneric()                Remounting /var/singularity/mnt/session/final/var/tmp

DEBUG   [U=21485,P=76942]  mountGeneric()                Skipping mount /var/singularity/mnt/session/final/users/PZS0530/dheisterberg, /var/singularity/mnt/session/final/users/PZS0530/dheisterberg already mounted

DEBUG   [U=21485,P=76942]  mountGeneric()                Mounting /var/singularity/mnt/session/etc/resolv.conf to /var/singularity/mnt/session/final/etc/resolv.conf

DEBUG   [U=21485,P=76942]  create()                      Chroot into /var/singularity/mnt/session/final

DEBUG   [U=0,P=76973]      Chroot()                      Hold reference to host / directory

DEBUG   [U=0,P=76973]      Chroot()                      Change current directory to /var/singularity/mnt/session/final

DEBUG   [U=0,P=76973]      Chroot()                      Called pivot_root on /var/singularity/mnt/session/final

DEBUG   [U=0,P=76973]      Chroot()                      Change current directory to host / directory

DEBUG   [U=0,P=76973]      Chroot()                      Apply slave mount propagation for host / directory

DEBUG   [U=0,P=76973]      Chroot()                      Called unmount(/, syscall.MNT_DETACH)

DEBUG   [U=0,P=76973]      Chroot()                      Changing directory to / to avoid getpwd issues

DEBUG   [U=21485,P=76942]  create()                      Chdir into / to avoid errors

DEBUG   [U=0,P=76972]      set_parent_death_signal()                 Set parent death signal to 9

DEBUG   [U=0,P=76972]      prepare_scontainer_stage()                Entering in scontainer stage 2

DEBUG   [U=21485,P=76972]  set_parent_death_signal()                 Set parent death signal to 9

VERBOSE [U=21485,P=76972]  startup()                     Execute scontainer stage 2

FATAL   [U=21485,P=76972]  SContainer()                  exec /.singularity.d/actions/shell failed: permission denied

DEBUG   [U=21485,P=76942]  PostStartProcess()            Post start process

DEBUG   [U=21485,P=76942]  CleanupContainer()            Cleanup container

DEBUG   [U=21485,P=76942]  SMaster()                     Child exited with exit status 255

[Justin Cook]

Trey,

How is this container built?

Justin

[Trey Dockendorf]

It's a docker image from CentOS, not sure how built.  The issue occurs for any container, not just the centos one.  The commands don't fail for myself and others, just this one person.

Thanks,

- Trey

--
You received this message because you are subscribed to a topic in the Google Groups "singularity" group.
To unsubscribe from this topic, visit https://groups.google.com/a/lbl.gov/d/topic/singularity/ccKBL6vAezE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to singularity...@lbl.gov.

[Justin Cook]

Trey,

Something odd seems to be going on with his environment.

What are the mount options for the directory they are running from?

Thanks,

Justin

[Justin Cook]

Additionally, can you just build the container by itself and shell into it?

[Trey Dockendorf]

The mount options are the same for my home directory vs the broken user's, from autofs:

* -rw,vers=4.0,rsize=65536,wsize=65536 &.OMITDOMAIN:/&

Built a demo image from my home directory and gave access to the broken account.  The definition I used is one I've used previously to verify build still works [1].

Using this new image works [2].  So it's just using docker based images that is not working it seems, and only for this one person's account.

[1]:

BootStrap: docker

From: centos:7

%post

    yum -y update

    yum -y install epel-release

    yum -y install fortune-mod cowsay

#%environment

%runscript

    fortune | cowsay

[2]:

p0065:~$ singularity run ~tdockendorf/singularity/centos7-demo.img

 ________________________________________

/ [End of diatribe. We now return you to \

| your regularly scheduled               |

| programming...]                        |

|                                        |

| -- Larry Wall in Configure from the    |

\ perl distribution                      /

 ----------------------------------------

        \   ^__^

         \  (oo)\_______

            (__)\       )\/\

                ||----w |

                ||     ||

p0065:~$ singularity shell ~tdockendorf/singularity/centos7-demo.img

Singularity centos7-demo.img:~> exit