Error using a container mounted in a Vagrant VM
595 views
Skip to first unread message
Jonathan Schulz
Dec 14, 2021, 4:53:38 PM12/14/21
to Singularity Community Edition
I've been using singularity v3.0 for 2+ years and recently upgraded to v3.7 because I started running into errors compiling containers from "Bootstrap: library"
I have a program that runs inside a Vagrant VM. It compiles containers to a mounted host directory then uses those containers to run bash scripts
In Vagrant v3.7, using the mounted containers now raises an error:
"container creation failed: mount /proc/self/fd/3->/home/vagrant/singularity-3.7_install/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to mount squashfs filesystem: input/output error"
I do not have any errors when i rsync the same container into the VM's filesystem and run the same command. Is this an expected change in behavior?
I haven't found exactly this question anywhere, but here are some logs that were requested in other similar cases:
Host OS:
macOS High Sierra (10.13.6)
uname -a inside the vagrant VM:
Linux ubuntu-bionic 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Linux ubuntu-bionic 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Logs from running with "-vvv -d" w/ a mounted container:
vagrant@ubuntu-bionic:~$ singularity -vvv -d \
> exec \
> /home/vagrant/jschulz1/containers/bowtie_1.2.3.sif \
> ls
DEBUG [U=1000,P=19046] persistentPreRun() Singularity version: 3.7.4
DEBUG [U=1000,P=19046] persistentPreRun() Parsing configuration file /home/vagrant/singularity-3.7_install/etc/singularity/singularity.conf
DEBUG [U=1000,P=19046] handleConfDir() /home/vagrant/.singularity already exists. Not creating.
DEBUG [U=1000,P=19046] execStarter() Saving umask 0002 for propagation into container
DEBUG [U=1000,P=19046] execStarter() Checking for encrypted system partition
DEBUG [U=1000,P=19046] Init() Image format detection
DEBUG [U=1000,P=19046] Init() Check for sandbox image format
DEBUG [U=1000,P=19046] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1000,P=19046] Init() Check for sif image format
DEBUG [U=1000,P=19046] Init() sif image format detected
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding LS_COLORS environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding SSH_CONNECTION environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding LESSCLOSE environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding LANG environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding DISPLAY environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding SSH_AUTH_SOCK environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding XDG_SESSION_ID environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding USER environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding PWD environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding SSH_CLIENT environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding XDG_DATA_DIRS environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding SSH_TTY environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding MAIL environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding TERM environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding SHELL environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding SHLVL environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding LOGNAME environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding XDG_RUNTIME_DIR environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding LESSOPEN environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding _ environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding OLDPWD environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding USER_PATH environment variable
VERBOSE [U=1000,P=19046] SetContainerEnv() Setting HOME=/home/vagrant
VERBOSE [U=1000,P=19046] SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG [U=1000,P=19046] init() Use starter binary /home/vagrant/singularity-3.7_install/libexec/singularity/bin/starter-suid
VERBOSE [U=0,P=19046] print() Set messagelevel to: 5
VERBOSE [U=0,P=19046] init() Starter initialization
DEBUG [U=0,P=19046] load_overlay_module() Trying to load overlay kernel module
DEBUG [U=0,P=19046] load_overlay_module() Overlay seems supported by the kernel
VERBOSE [U=0,P=19046] is_suid() Check if we are running as setuid
VERBOSE [U=0,P=19046] priv_drop() Drop root privileges
DEBUG [U=1000,P=19046] read_engine_config() Read engine configuration
DEBUG [U=1000,P=19046] init() Wait completion of stage1
VERBOSE [U=1000,P=19053] priv_drop() Drop root privileges permanently
DEBUG [U=1000,P=19053] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1000,P=19053] init() Spawn stage 1
DEBUG [U=1000,P=19053] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19053] startup() Execute stage 1
DEBUG [U=1000,P=19053] StageOne() Entering stage 1
DEBUG [U=1000,P=19053] prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG [U=1000,P=19053] prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG [U=1000,P=19053] prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG [U=1000,P=19053] prepareAutofs() Could not keep file descriptor for home directory /home/vagrant: no mount point
DEBUG [U=1000,P=19053] prepareAutofs() Could not keep file descriptor for current working directory /home/vagrant: no mount point
DEBUG [U=1000,P=19053] Init() Image format detection
DEBUG [U=1000,P=19053] Init() Check for sandbox image format
DEBUG [U=1000,P=19053] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1000,P=19053] Init() Check for sif image format
DEBUG [U=1000,P=19053] Init() sif image format detected
DEBUG [U=1000,P=19053] setSessionLayer() Overlay seems supported and allowed by kernel
DEBUG [U=1000,P=19053] setSessionLayer() Attempting to use overlayfs (enable overlay = try)
VERBOSE [U=1000,P=19046] wait_child() stage 1 exited with status 0
DEBUG [U=1000,P=19046] cleanup_fd() Close file descriptor 4
DEBUG [U=1000,P=19046] cleanup_fd() Close file descriptor 5
DEBUG [U=1000,P=19046] cleanup_fd() Close file descriptor 6
DEBUG [U=1000,P=19046] init() Set child signal mask
DEBUG [U=1000,P=19046] init() Create socketpair for master communication channel
DEBUG [U=1000,P=19046] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=1000,P=19046] priv_escalate() Get root privileges
VERBOSE [U=0,P=19046] priv_escalate() Change filesystem uid to 1000
VERBOSE [U=0,P=19046] init() Spawn master process
DEBUG [U=0,P=19058] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=19058] create_namespace() Create mount namespace
VERBOSE [U=0,P=19046] enter_namespace() Entering in mount namespace
DEBUG [U=0,P=19046] enter_namespace() Opening namespace file ns/mnt
DEBUG [U=0,P=19046] set_master_privileges() Set master privileges
DEBUG [U=0,P=19046] apply_privileges() Effective capabilities: 0x00000000000000c0
DEBUG [U=0,P=19046] apply_privileges() Permitted capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19046] apply_privileges() Bounding capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19046] apply_privileges() Inheritable capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19046] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=19046] apply_privileges() Set user ID to 1000
VERBOSE [U=0,P=19058] create_namespace() Create mount namespace
DEBUG [U=0,P=19060] set_rpc_privileges() Set RPC privileges
DEBUG [U=0,P=19060] apply_privileges() Effective capabilities: 0x0000000000200000
DEBUG [U=0,P=19060] apply_privileges() Permitted capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19060] apply_privileges() Bounding capabilities: 0x0000000008204000
DEBUG [U=0,P=19060] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=0,P=19060] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=19060] apply_privileges() Set user ID to 1000
DEBUG [U=1000,P=19060] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1000,P=19060] init() Spawn RPC server
DEBUG [U=1000,P=19046] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19046] startup() Execute master process
DEBUG [U=1000,P=19060] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19060] startup() Serve RPC requests
DEBUG [U=1000,P=19046] setupSessionLayout() Using Layer system: overlay
DEBUG [U=1000,P=19046] setupOverlayLayout() Creating overlay SESSIONDIR layout
DEBUG [U=1000,P=19046] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=1000,P=19046] addRootfsMount() Image type is 4096
DEBUG [U=1000,P=19046] addRootfsMount() Mounting block [squashfs] image: /home/vagrant/jschulz1/containers/bowtie_1.2.3.sif
DEBUG [U=1000,P=19046] addKernelMount() Checking configuration file for 'mount proc'
VERBOSE [U=1000,P=19046] addKernelMount() Skipping /proc mount
DEBUG [U=1000,P=19046] addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=1000,P=19046] addKernelMount() Adding sysfs to mount list
VERBOSE [U=1000,P=19046] addKernelMount() Default mount: /sys:/sys
DEBUG [U=1000,P=19046] addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=1000,P=19046] addDevMount() Adding dev to mount list
VERBOSE [U=1000,P=19046] addDevMount() Default mount: /dev:/dev
DEBUG [U=1000,P=19046] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=1000,P=19046] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=1000,P=19046] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=1000,P=19046] addHomeStagingDir() Staging home directory (/home/vagrant) at /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant
DEBUG [U=1000,P=19046] addHomeMount() Adding home directory mount [/home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant:/home/vagrant] to list using layer: overlay
DEBUG [U=1000,P=19046] addTmpMount() Checking for 'mount tmp' in configuration file
VERBOSE [U=1000,P=19046] addTmpMount() Default mount: /tmp:/tmp
VERBOSE [U=1000,P=19046] addTmpMount() Default mount: /var/tmp:/var/tmp
DEBUG [U=1000,P=19046] addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=1000,P=19046] addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=1000,P=19046] addFilesMount() Checking for 'user bind control' in configuration file
DEBUG [U=1000,P=19046] addResolvConfMount() Adding /etc/resolv.conf to mount list
VERBOSE [U=1000,P=19046] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG [U=1000,P=19046] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=1000,P=19046] create() Mount all
DEBUG [U=1000,P=19046] mountGeneric() Mounting tmpfs to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session
DEBUG [U=1000,P=19046] mountImage() Mounting loop device /dev/loop0 to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/rootfs of type squashfs
FATAL [U=1000,P=19046] Master() container creation failed: mount /proc/self/fd/3->/home/vagrant/singularity-3.7_install/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to mount squashfs filesystem: input/output error
vagrant@ubuntu-bionic:~$ singularity -vvv -d \
> exec \
> /home/vagrant/jschulz1/containers/bowtie_1.2.3.sif \
> ls
DEBUG [U=1000,P=19046] persistentPreRun() Singularity version: 3.7.4
DEBUG [U=1000,P=19046] persistentPreRun() Parsing configuration file /home/vagrant/singularity-3.7_install/etc/singularity/singularity.conf
DEBUG [U=1000,P=19046] handleConfDir() /home/vagrant/.singularity already exists. Not creating.
DEBUG [U=1000,P=19046] execStarter() Saving umask 0002 for propagation into container
DEBUG [U=1000,P=19046] execStarter() Checking for encrypted system partition
DEBUG [U=1000,P=19046] Init() Image format detection
DEBUG [U=1000,P=19046] Init() Check for sandbox image format
DEBUG [U=1000,P=19046] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1000,P=19046] Init() Check for sif image format
DEBUG [U=1000,P=19046] Init() sif image format detected
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding LS_COLORS environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding SSH_CONNECTION environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding LESSCLOSE environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding LANG environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding DISPLAY environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding SSH_AUTH_SOCK environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding XDG_SESSION_ID environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding USER environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding PWD environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding SSH_CLIENT environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding XDG_DATA_DIRS environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding SSH_TTY environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding MAIL environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding TERM environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding SHELL environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding SHLVL environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding LOGNAME environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding XDG_RUNTIME_DIR environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding LESSOPEN environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding _ environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding OLDPWD environment variable
DEBUG [U=1000,P=19046] SetContainerEnv() Forwarding USER_PATH environment variable
VERBOSE [U=1000,P=19046] SetContainerEnv() Setting HOME=/home/vagrant
VERBOSE [U=1000,P=19046] SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG [U=1000,P=19046] init() Use starter binary /home/vagrant/singularity-3.7_install/libexec/singularity/bin/starter-suid
VERBOSE [U=0,P=19046] print() Set messagelevel to: 5
VERBOSE [U=0,P=19046] init() Starter initialization
DEBUG [U=0,P=19046] load_overlay_module() Trying to load overlay kernel module
DEBUG [U=0,P=19046] load_overlay_module() Overlay seems supported by the kernel
VERBOSE [U=0,P=19046] is_suid() Check if we are running as setuid
VERBOSE [U=0,P=19046] priv_drop() Drop root privileges
DEBUG [U=1000,P=19046] read_engine_config() Read engine configuration
DEBUG [U=1000,P=19046] init() Wait completion of stage1
VERBOSE [U=1000,P=19053] priv_drop() Drop root privileges permanently
DEBUG [U=1000,P=19053] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1000,P=19053] init() Spawn stage 1
DEBUG [U=1000,P=19053] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19053] startup() Execute stage 1
DEBUG [U=1000,P=19053] StageOne() Entering stage 1
DEBUG [U=1000,P=19053] prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG [U=1000,P=19053] prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG [U=1000,P=19053] prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG [U=1000,P=19053] prepareAutofs() Could not keep file descriptor for home directory /home/vagrant: no mount point
DEBUG [U=1000,P=19053] prepareAutofs() Could not keep file descriptor for current working directory /home/vagrant: no mount point
DEBUG [U=1000,P=19053] Init() Image format detection
DEBUG [U=1000,P=19053] Init() Check for sandbox image format
DEBUG [U=1000,P=19053] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1000,P=19053] Init() Check for sif image format
DEBUG [U=1000,P=19053] Init() sif image format detected
DEBUG [U=1000,P=19053] setSessionLayer() Overlay seems supported and allowed by kernel
DEBUG [U=1000,P=19053] setSessionLayer() Attempting to use overlayfs (enable overlay = try)
VERBOSE [U=1000,P=19046] wait_child() stage 1 exited with status 0
DEBUG [U=1000,P=19046] cleanup_fd() Close file descriptor 4
DEBUG [U=1000,P=19046] cleanup_fd() Close file descriptor 5
DEBUG [U=1000,P=19046] cleanup_fd() Close file descriptor 6
DEBUG [U=1000,P=19046] init() Set child signal mask
DEBUG [U=1000,P=19046] init() Create socketpair for master communication channel
DEBUG [U=1000,P=19046] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=1000,P=19046] priv_escalate() Get root privileges
VERBOSE [U=0,P=19046] priv_escalate() Change filesystem uid to 1000
VERBOSE [U=0,P=19046] init() Spawn master process
DEBUG [U=0,P=19058] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=19058] create_namespace() Create mount namespace
VERBOSE [U=0,P=19046] enter_namespace() Entering in mount namespace
DEBUG [U=0,P=19046] enter_namespace() Opening namespace file ns/mnt
DEBUG [U=0,P=19046] set_master_privileges() Set master privileges
DEBUG [U=0,P=19046] apply_privileges() Effective capabilities: 0x00000000000000c0
DEBUG [U=0,P=19046] apply_privileges() Permitted capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19046] apply_privileges() Bounding capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19046] apply_privileges() Inheritable capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19046] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=19046] apply_privileges() Set user ID to 1000
VERBOSE [U=0,P=19058] create_namespace() Create mount namespace
DEBUG [U=0,P=19060] set_rpc_privileges() Set RPC privileges
DEBUG [U=0,P=19060] apply_privileges() Effective capabilities: 0x0000000000200000
DEBUG [U=0,P=19060] apply_privileges() Permitted capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19060] apply_privileges() Bounding capabilities: 0x0000000008204000
DEBUG [U=0,P=19060] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=0,P=19060] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=19060] apply_privileges() Set user ID to 1000
DEBUG [U=1000,P=19060] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1000,P=19060] init() Spawn RPC server
DEBUG [U=1000,P=19046] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19046] startup() Execute master process
DEBUG [U=1000,P=19060] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19060] startup() Serve RPC requests
DEBUG [U=1000,P=19046] setupSessionLayout() Using Layer system: overlay
DEBUG [U=1000,P=19046] setupOverlayLayout() Creating overlay SESSIONDIR layout
DEBUG [U=1000,P=19046] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=1000,P=19046] addRootfsMount() Image type is 4096
DEBUG [U=1000,P=19046] addRootfsMount() Mounting block [squashfs] image: /home/vagrant/jschulz1/containers/bowtie_1.2.3.sif
DEBUG [U=1000,P=19046] addKernelMount() Checking configuration file for 'mount proc'
VERBOSE [U=1000,P=19046] addKernelMount() Skipping /proc mount
DEBUG [U=1000,P=19046] addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=1000,P=19046] addKernelMount() Adding sysfs to mount list
VERBOSE [U=1000,P=19046] addKernelMount() Default mount: /sys:/sys
DEBUG [U=1000,P=19046] addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=1000,P=19046] addDevMount() Adding dev to mount list
VERBOSE [U=1000,P=19046] addDevMount() Default mount: /dev:/dev
DEBUG [U=1000,P=19046] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=1000,P=19046] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=1000,P=19046] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=1000,P=19046] addHomeStagingDir() Staging home directory (/home/vagrant) at /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant
DEBUG [U=1000,P=19046] addHomeMount() Adding home directory mount [/home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant:/home/vagrant] to list using layer: overlay
DEBUG [U=1000,P=19046] addTmpMount() Checking for 'mount tmp' in configuration file
VERBOSE [U=1000,P=19046] addTmpMount() Default mount: /tmp:/tmp
VERBOSE [U=1000,P=19046] addTmpMount() Default mount: /var/tmp:/var/tmp
DEBUG [U=1000,P=19046] addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=1000,P=19046] addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=1000,P=19046] addFilesMount() Checking for 'user bind control' in configuration file
DEBUG [U=1000,P=19046] addResolvConfMount() Adding /etc/resolv.conf to mount list
VERBOSE [U=1000,P=19046] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG [U=1000,P=19046] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=1000,P=19046] create() Mount all
DEBUG [U=1000,P=19046] mountGeneric() Mounting tmpfs to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session
DEBUG [U=1000,P=19046] mountImage() Mounting loop device /dev/loop0 to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/rootfs of type squashfs
FATAL [U=1000,P=19046] Master() container creation failed: mount /proc/self/fd/3->/home/vagrant/singularity-3.7_install/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to mount squashfs filesystem: input/output error
Logs from running with "-vvv -d" w/ a container inside the VM:
vagrant@ubuntu-bionic:~$ singularity -vvv -d \
> exec \
> /home/vagrant/sandbox_containers/bowtie_1.2.3.sif \
> ls
DEBUG [U=1000,P=19082] persistentPreRun() Singularity version: 3.7.4
DEBUG [U=1000,P=19082] persistentPreRun() Parsing configuration file /home/vagrant/singularity-3.7_install/etc/singularity/singularity.conf
DEBUG [U=1000,P=19082] handleConfDir() /home/vagrant/.singularity already exists. Not creating.
DEBUG [U=1000,P=19082] execStarter() Saving umask 0002 for propagation into container
DEBUG [U=1000,P=19082] execStarter() Checking for encrypted system partition
DEBUG [U=1000,P=19082] Init() Image format detection
DEBUG [U=1000,P=19082] Init() Check for sandbox image format
DEBUG [U=1000,P=19082] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1000,P=19082] Init() Check for sif image format
DEBUG [U=1000,P=19082] Init() sif image format detected
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding LS_COLORS environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding SSH_CONNECTION environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding LESSCLOSE environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding LANG environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding DISPLAY environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding SSH_AUTH_SOCK environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding XDG_SESSION_ID environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding USER environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding PWD environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding SSH_CLIENT environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding XDG_DATA_DIRS environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding SSH_TTY environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding MAIL environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding TERM environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding SHELL environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding SHLVL environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding LOGNAME environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding XDG_RUNTIME_DIR environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding LESSOPEN environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding _ environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding OLDPWD environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding USER_PATH environment variable
VERBOSE [U=1000,P=19082] SetContainerEnv() Setting HOME=/home/vagrant
VERBOSE [U=1000,P=19082] SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG [U=1000,P=19082] init() Use starter binary /home/vagrant/singularity-3.7_install/libexec/singularity/bin/starter-suid
VERBOSE [U=0,P=19082] print() Set messagelevel to: 5
VERBOSE [U=0,P=19082] init() Starter initialization
DEBUG [U=0,P=19082] load_overlay_module() Trying to load overlay kernel module
DEBUG [U=0,P=19082] load_overlay_module() Overlay seems supported by the kernel
VERBOSE [U=0,P=19082] is_suid() Check if we are running as setuid
VERBOSE [U=0,P=19082] priv_drop() Drop root privileges
DEBUG [U=1000,P=19082] read_engine_config() Read engine configuration
DEBUG [U=1000,P=19082] init() Wait completion of stage1
VERBOSE [U=1000,P=19089] priv_drop() Drop root privileges permanently
DEBUG [U=1000,P=19089] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1000,P=19089] init() Spawn stage 1
DEBUG [U=1000,P=19089] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19089] startup() Execute stage 1
DEBUG [U=1000,P=19089] StageOne() Entering stage 1
DEBUG [U=1000,P=19089] prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG [U=1000,P=19089] prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG [U=1000,P=19089] prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG [U=1000,P=19089] prepareAutofs() Could not keep file descriptor for home directory /home/vagrant: no mount point
DEBUG [U=1000,P=19089] prepareAutofs() Could not keep file descriptor for current working directory /home/vagrant: no mount point
DEBUG [U=1000,P=19089] Init() Image format detection
DEBUG [U=1000,P=19089] Init() Check for sandbox image format
DEBUG [U=1000,P=19089] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1000,P=19089] Init() Check for sif image format
DEBUG [U=1000,P=19089] Init() sif image format detected
DEBUG [U=1000,P=19089] setSessionLayer() Overlay seems supported and allowed by kernel
DEBUG [U=1000,P=19089] setSessionLayer() Attempting to use overlayfs (enable overlay = try)
VERBOSE [U=1000,P=19082] wait_child() stage 1 exited with status 0
DEBUG [U=1000,P=19082] cleanup_fd() Close file descriptor 4
DEBUG [U=1000,P=19082] cleanup_fd() Close file descriptor 5
DEBUG [U=1000,P=19082] cleanup_fd() Close file descriptor 6
DEBUG [U=1000,P=19082] init() Set child signal mask
DEBUG [U=1000,P=19082] init() Create socketpair for master communication channel
DEBUG [U=1000,P=19082] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=1000,P=19082] priv_escalate() Get root privileges
VERBOSE [U=0,P=19082] priv_escalate() Change filesystem uid to 1000
VERBOSE [U=0,P=19082] init() Spawn master process
DEBUG [U=0,P=19095] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=19095] create_namespace() Create mount namespace
VERBOSE [U=0,P=19082] enter_namespace() Entering in mount namespace
DEBUG [U=0,P=19082] enter_namespace() Opening namespace file ns/mnt
DEBUG [U=0,P=19082] set_master_privileges() Set master privileges
DEBUG [U=0,P=19082] apply_privileges() Effective capabilities: 0x00000000000000c0
DEBUG [U=0,P=19082] apply_privileges() Permitted capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19082] apply_privileges() Bounding capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19082] apply_privileges() Inheritable capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19082] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=19082] apply_privileges() Set user ID to 1000
VERBOSE [U=0,P=19095] create_namespace() Create mount namespace
DEBUG [U=0,P=19096] set_rpc_privileges() Set RPC privileges
DEBUG [U=0,P=19096] apply_privileges() Effective capabilities: 0x0000000000200000
DEBUG [U=0,P=19096] apply_privileges() Permitted capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19096] apply_privileges() Bounding capabilities: 0x0000000008204000
DEBUG [U=0,P=19096] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=0,P=19096] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=19096] apply_privileges() Set user ID to 1000
DEBUG [U=1000,P=19096] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1000,P=19096] init() Spawn RPC server
DEBUG [U=1000,P=19082] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19082] startup() Execute master process
DEBUG [U=1000,P=19082] setupSessionLayout() Using Layer system: overlay
DEBUG [U=1000,P=19082] setupOverlayLayout() Creating overlay SESSIONDIR layout
DEBUG [U=1000,P=19082] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=1000,P=19082] addRootfsMount() Image type is 4096
DEBUG [U=1000,P=19082] addRootfsMount() Mounting block [squashfs] image: /home/vagrant/sandbox_containers/bowtie_1.2.3.sif
DEBUG [U=1000,P=19082] addKernelMount() Checking configuration file for 'mount proc'
VERBOSE [U=1000,P=19082] addKernelMount() Skipping /proc mount
DEBUG [U=1000,P=19082] addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=1000,P=19082] addKernelMount() Adding sysfs to mount list
DEBUG [U=1000,P=19096] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19096] startup() Serve RPC requests
VERBOSE [U=1000,P=19082] addKernelMount() Default mount: /sys:/sys
DEBUG [U=1000,P=19082] addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=1000,P=19082] addDevMount() Adding dev to mount list
VERBOSE [U=1000,P=19082] addDevMount() Default mount: /dev:/dev
DEBUG [U=1000,P=19082] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=1000,P=19082] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=1000,P=19082] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=1000,P=19082] addHomeStagingDir() Staging home directory (/home/vagrant) at /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant
DEBUG [U=1000,P=19082] addHomeMount() Adding home directory mount [/home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant:/home/vagrant] to list using layer: overlay
DEBUG [U=1000,P=19082] addTmpMount() Checking for 'mount tmp' in configuration file
VERBOSE [U=1000,P=19082] addTmpMount() Default mount: /tmp:/tmp
VERBOSE [U=1000,P=19082] addTmpMount() Default mount: /var/tmp:/var/tmp
DEBUG [U=1000,P=19082] addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=1000,P=19082] addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=1000,P=19082] addFilesMount() Checking for 'user bind control' in configuration file
DEBUG [U=1000,P=19082] addResolvConfMount() Adding /etc/resolv.conf to mount list
VERBOSE [U=1000,P=19082] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG [U=1000,P=19082] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=1000,P=19082] create() Mount all
DEBUG [U=1000,P=19082] mountGeneric() Mounting tmpfs to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session
DEBUG [U=1000,P=19082] mountImage() Mounting loop device /dev/loop0 to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/rootfs of type squashfs
DEBUG [U=1000,P=19082] mountGeneric() Mounting overlay to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final
DEBUG [U=1000,P=19082] mountGeneric() Remounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final
DEBUG [U=1000,P=19082] setPropagationMount() Set RPC mount propagation flag to SLAVE
VERBOSE [U=1000,P=19082] Passwd() Checking for template passwd file: /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE [U=1000,P=19082] Passwd() Creating passwd content
VERBOSE [U=1000,P=19082] Passwd() Creating template passwd file and appending user data: /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/rootfs/etc/passwd
DEBUG [U=1000,P=19082] addIdentityMount() Adding /etc/passwd to mount list
VERBOSE [U=1000,P=19082] addIdentityMount() Default mount: /etc/passwd:/etc/passwd
VERBOSE [U=1000,P=19082] Group() Checking for template group file: /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/rootfs/etc/group
VERBOSE [U=1000,P=19082] Group() Creating group content
DEBUG [U=1000,P=19082] addIdentityMount() Adding /etc/group to mount list
VERBOSE [U=1000,P=19082] addIdentityMount() Default mount: /etc/group:/etc/group
DEBUG [U=1000,P=19082] mountGeneric() Mounting /dev to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/dev
DEBUG [U=1000,P=19082] mountGeneric() Mounting /etc/localtime to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/usr/share/zoneinfo/UTC
DEBUG [U=1000,P=19082] mountGeneric() Remounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/usr/share/zoneinfo/UTC
DEBUG [U=1000,P=19082] mountGeneric() Mounting /etc/hosts to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=1000,P=19082] mountGeneric() Remounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=1000,P=19082] mountGeneric() Mounting sysfs to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/sys
DEBUG [U=1000,P=19082] mountGeneric() Mounting /home/vagrant to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant
DEBUG [U=1000,P=19082] mountGeneric() Remounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant
DEBUG [U=1000,P=19082] mountGeneric() Mounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/home/vagrant
DEBUG [U=1000,P=19082] mountGeneric() Mounting /tmp to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/tmp
DEBUG [U=1000,P=19082] mountGeneric() Remounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/tmp
DEBUG [U=1000,P=19082] mountGeneric() Mounting /var/tmp to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/var/tmp
DEBUG [U=1000,P=19082] mountGeneric() Remounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/var/tmp
DEBUG [U=1000,P=19082] mountGeneric() Mounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/etc/resolv.conf to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/etc/resolv.conf
DEBUG [U=1000,P=19082] mountGeneric() Mounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/etc/passwd to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/etc/passwd
DEBUG [U=1000,P=19082] mountGeneric() Mounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/etc/group to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/etc/group
DEBUG [U=1000,P=19082] addCwdMount() Using /home/vagrant as current working directory
VERBOSE [U=1000,P=19082] addCwdMount() /home/vagrant found within container
DEBUG [U=1000,P=19082] create() Chroot into /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final
DEBUG [U=1000,P=19096] Chroot() Hold reference to host / directory
DEBUG [U=1000,P=19096] Chroot() Called pivot_root on /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final
DEBUG [U=1000,P=19096] Chroot() Change current directory to host / directory
DEBUG [U=1000,P=19096] Chroot() Apply slave mount propagation for host / directory
DEBUG [U=1000,P=19096] Chroot() Called unmount(/, syscall.MNT_DETACH)
DEBUG [U=1000,P=19096] Chroot() Changing directory to / to avoid getpwd issues
DEBUG [U=1000,P=19082] create() Chdir into / to avoid errors
VERBOSE [U=0,P=19095] wait_child() rpc server exited with status 0
DEBUG [U=0,P=19095] init() Set container privileges
DEBUG [U=0,P=19095] apply_privileges() Effective capabilities: 0x0000000000000000
DEBUG [U=0,P=19095] apply_privileges() Permitted capabilities: 0x0000000000000000
DEBUG [U=0,P=19095] apply_privileges() Bounding capabilities: 0x0000000000000000
DEBUG [U=0,P=19095] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=0,P=19095] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=19095] apply_privileges() Set user ID to 1000
DEBUG [U=1000,P=19095] set_parent_death_signal() Set parent death signal to 9
DEBUG [U=1000,P=19095] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19095] startup() Execute stage 2
DEBUG [U=1000,P=19095] StageTwo() Entering stage 2
DEBUG [U=1000,P=19095] StartProcess() Setting umask in container to 0002
DEBUG [U=1000,P=19095] sylogBuiltin() Sourcing /.singularity.d/env/01-base.sh
DEBUG [U=1000,P=19095] sylogBuiltin() Sourcing /.singularity.d/env/90-environment.sh
DEBUG [U=1000,P=19095] sylogBuiltin() Sourcing /.singularity.d/env/94-appsbase.sh
DEBUG [U=1000,P=19095] sylogBuiltin() Sourcing /.singularity.d/env/95-apps.sh
DEBUG [U=1000,P=19095] sylogBuiltin() Sourcing /.singularity.d/env/99-base.sh
DEBUG [U=1000,P=19095] sylogBuiltin() Sourcing /.singularity.d/env/99-runtimevars.sh
DEBUG [U=1000,P=19095] sylogBuiltin() Running action command exec
DEBUG [U=1000,P=19082] PostStartProcess() Post start process
go jschulz1 project_data sandbox_containers singularity singularity-3.7.4.tar.gz singularity-3.7_install
DEBUG [U=1000,P=19082] Master() Child exited with exit status 0
> exec \
> /home/vagrant/sandbox_containers/bowtie_1.2.3.sif \
> ls
DEBUG [U=1000,P=19082] persistentPreRun() Singularity version: 3.7.4
DEBUG [U=1000,P=19082] persistentPreRun() Parsing configuration file /home/vagrant/singularity-3.7_install/etc/singularity/singularity.conf
DEBUG [U=1000,P=19082] handleConfDir() /home/vagrant/.singularity already exists. Not creating.
DEBUG [U=1000,P=19082] execStarter() Saving umask 0002 for propagation into container
DEBUG [U=1000,P=19082] execStarter() Checking for encrypted system partition
DEBUG [U=1000,P=19082] Init() Image format detection
DEBUG [U=1000,P=19082] Init() Check for sandbox image format
DEBUG [U=1000,P=19082] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1000,P=19082] Init() Check for sif image format
DEBUG [U=1000,P=19082] Init() sif image format detected
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding LS_COLORS environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding SSH_CONNECTION environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding LESSCLOSE environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding LANG environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding DISPLAY environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding SSH_AUTH_SOCK environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding XDG_SESSION_ID environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding USER environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding PWD environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding SSH_CLIENT environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding XDG_DATA_DIRS environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding SSH_TTY environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding MAIL environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding TERM environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding SHELL environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding SHLVL environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding LOGNAME environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding XDG_RUNTIME_DIR environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding LESSOPEN environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding _ environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding OLDPWD environment variable
DEBUG [U=1000,P=19082] SetContainerEnv() Forwarding USER_PATH environment variable
VERBOSE [U=1000,P=19082] SetContainerEnv() Setting HOME=/home/vagrant
VERBOSE [U=1000,P=19082] SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG [U=1000,P=19082] init() Use starter binary /home/vagrant/singularity-3.7_install/libexec/singularity/bin/starter-suid
VERBOSE [U=0,P=19082] print() Set messagelevel to: 5
VERBOSE [U=0,P=19082] init() Starter initialization
DEBUG [U=0,P=19082] load_overlay_module() Trying to load overlay kernel module
DEBUG [U=0,P=19082] load_overlay_module() Overlay seems supported by the kernel
VERBOSE [U=0,P=19082] is_suid() Check if we are running as setuid
VERBOSE [U=0,P=19082] priv_drop() Drop root privileges
DEBUG [U=1000,P=19082] read_engine_config() Read engine configuration
DEBUG [U=1000,P=19082] init() Wait completion of stage1
VERBOSE [U=1000,P=19089] priv_drop() Drop root privileges permanently
DEBUG [U=1000,P=19089] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1000,P=19089] init() Spawn stage 1
DEBUG [U=1000,P=19089] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19089] startup() Execute stage 1
DEBUG [U=1000,P=19089] StageOne() Entering stage 1
DEBUG [U=1000,P=19089] prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG [U=1000,P=19089] prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG [U=1000,P=19089] prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG [U=1000,P=19089] prepareAutofs() Could not keep file descriptor for home directory /home/vagrant: no mount point
DEBUG [U=1000,P=19089] prepareAutofs() Could not keep file descriptor for current working directory /home/vagrant: no mount point
DEBUG [U=1000,P=19089] Init() Image format detection
DEBUG [U=1000,P=19089] Init() Check for sandbox image format
DEBUG [U=1000,P=19089] Init() sandbox format initializer returned: not a directory image
DEBUG [U=1000,P=19089] Init() Check for sif image format
DEBUG [U=1000,P=19089] Init() sif image format detected
DEBUG [U=1000,P=19089] setSessionLayer() Overlay seems supported and allowed by kernel
DEBUG [U=1000,P=19089] setSessionLayer() Attempting to use overlayfs (enable overlay = try)
VERBOSE [U=1000,P=19082] wait_child() stage 1 exited with status 0
DEBUG [U=1000,P=19082] cleanup_fd() Close file descriptor 4
DEBUG [U=1000,P=19082] cleanup_fd() Close file descriptor 5
DEBUG [U=1000,P=19082] cleanup_fd() Close file descriptor 6
DEBUG [U=1000,P=19082] init() Set child signal mask
DEBUG [U=1000,P=19082] init() Create socketpair for master communication channel
DEBUG [U=1000,P=19082] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=1000,P=19082] priv_escalate() Get root privileges
VERBOSE [U=0,P=19082] priv_escalate() Change filesystem uid to 1000
VERBOSE [U=0,P=19082] init() Spawn master process
DEBUG [U=0,P=19095] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=19095] create_namespace() Create mount namespace
VERBOSE [U=0,P=19082] enter_namespace() Entering in mount namespace
DEBUG [U=0,P=19082] enter_namespace() Opening namespace file ns/mnt
DEBUG [U=0,P=19082] set_master_privileges() Set master privileges
DEBUG [U=0,P=19082] apply_privileges() Effective capabilities: 0x00000000000000c0
DEBUG [U=0,P=19082] apply_privileges() Permitted capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19082] apply_privileges() Bounding capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19082] apply_privileges() Inheritable capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19082] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=19082] apply_privileges() Set user ID to 1000
VERBOSE [U=0,P=19095] create_namespace() Create mount namespace
DEBUG [U=0,P=19096] set_rpc_privileges() Set RPC privileges
DEBUG [U=0,P=19096] apply_privileges() Effective capabilities: 0x0000000000200000
DEBUG [U=0,P=19096] apply_privileges() Permitted capabilities: 0x0000003fffffffff
DEBUG [U=0,P=19096] apply_privileges() Bounding capabilities: 0x0000000008204000
DEBUG [U=0,P=19096] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=0,P=19096] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=19096] apply_privileges() Set user ID to 1000
DEBUG [U=1000,P=19096] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1000,P=19096] init() Spawn RPC server
DEBUG [U=1000,P=19082] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19082] startup() Execute master process
DEBUG [U=1000,P=19082] setupSessionLayout() Using Layer system: overlay
DEBUG [U=1000,P=19082] setupOverlayLayout() Creating overlay SESSIONDIR layout
DEBUG [U=1000,P=19082] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=1000,P=19082] addRootfsMount() Image type is 4096
DEBUG [U=1000,P=19082] addRootfsMount() Mounting block [squashfs] image: /home/vagrant/sandbox_containers/bowtie_1.2.3.sif
DEBUG [U=1000,P=19082] addKernelMount() Checking configuration file for 'mount proc'
VERBOSE [U=1000,P=19082] addKernelMount() Skipping /proc mount
DEBUG [U=1000,P=19082] addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=1000,P=19082] addKernelMount() Adding sysfs to mount list
DEBUG [U=1000,P=19096] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19096] startup() Serve RPC requests
VERBOSE [U=1000,P=19082] addKernelMount() Default mount: /sys:/sys
DEBUG [U=1000,P=19082] addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=1000,P=19082] addDevMount() Adding dev to mount list
VERBOSE [U=1000,P=19082] addDevMount() Default mount: /dev:/dev
DEBUG [U=1000,P=19082] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=1000,P=19082] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=1000,P=19082] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=1000,P=19082] addHomeStagingDir() Staging home directory (/home/vagrant) at /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant
DEBUG [U=1000,P=19082] addHomeMount() Adding home directory mount [/home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant:/home/vagrant] to list using layer: overlay
DEBUG [U=1000,P=19082] addTmpMount() Checking for 'mount tmp' in configuration file
VERBOSE [U=1000,P=19082] addTmpMount() Default mount: /tmp:/tmp
VERBOSE [U=1000,P=19082] addTmpMount() Default mount: /var/tmp:/var/tmp
DEBUG [U=1000,P=19082] addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=1000,P=19082] addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=1000,P=19082] addFilesMount() Checking for 'user bind control' in configuration file
DEBUG [U=1000,P=19082] addResolvConfMount() Adding /etc/resolv.conf to mount list
VERBOSE [U=1000,P=19082] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG [U=1000,P=19082] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=1000,P=19082] create() Mount all
DEBUG [U=1000,P=19082] mountGeneric() Mounting tmpfs to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session
DEBUG [U=1000,P=19082] mountImage() Mounting loop device /dev/loop0 to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/rootfs of type squashfs
DEBUG [U=1000,P=19082] mountGeneric() Mounting overlay to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final
DEBUG [U=1000,P=19082] mountGeneric() Remounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final
DEBUG [U=1000,P=19082] setPropagationMount() Set RPC mount propagation flag to SLAVE
VERBOSE [U=1000,P=19082] Passwd() Checking for template passwd file: /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE [U=1000,P=19082] Passwd() Creating passwd content
VERBOSE [U=1000,P=19082] Passwd() Creating template passwd file and appending user data: /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/rootfs/etc/passwd
DEBUG [U=1000,P=19082] addIdentityMount() Adding /etc/passwd to mount list
VERBOSE [U=1000,P=19082] addIdentityMount() Default mount: /etc/passwd:/etc/passwd
VERBOSE [U=1000,P=19082] Group() Checking for template group file: /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/rootfs/etc/group
VERBOSE [U=1000,P=19082] Group() Creating group content
DEBUG [U=1000,P=19082] addIdentityMount() Adding /etc/group to mount list
VERBOSE [U=1000,P=19082] addIdentityMount() Default mount: /etc/group:/etc/group
DEBUG [U=1000,P=19082] mountGeneric() Mounting /dev to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/dev
DEBUG [U=1000,P=19082] mountGeneric() Mounting /etc/localtime to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/usr/share/zoneinfo/UTC
DEBUG [U=1000,P=19082] mountGeneric() Remounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/usr/share/zoneinfo/UTC
DEBUG [U=1000,P=19082] mountGeneric() Mounting /etc/hosts to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=1000,P=19082] mountGeneric() Remounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=1000,P=19082] mountGeneric() Mounting sysfs to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/sys
DEBUG [U=1000,P=19082] mountGeneric() Mounting /home/vagrant to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant
DEBUG [U=1000,P=19082] mountGeneric() Remounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant
DEBUG [U=1000,P=19082] mountGeneric() Mounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/home/vagrant to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/home/vagrant
DEBUG [U=1000,P=19082] mountGeneric() Mounting /tmp to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/tmp
DEBUG [U=1000,P=19082] mountGeneric() Remounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/tmp
DEBUG [U=1000,P=19082] mountGeneric() Mounting /var/tmp to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/var/tmp
DEBUG [U=1000,P=19082] mountGeneric() Remounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/var/tmp
DEBUG [U=1000,P=19082] mountGeneric() Mounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/etc/resolv.conf to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/etc/resolv.conf
DEBUG [U=1000,P=19082] mountGeneric() Mounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/etc/passwd to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/etc/passwd
DEBUG [U=1000,P=19082] mountGeneric() Mounting /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/etc/group to /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final/etc/group
DEBUG [U=1000,P=19082] addCwdMount() Using /home/vagrant as current working directory
VERBOSE [U=1000,P=19082] addCwdMount() /home/vagrant found within container
DEBUG [U=1000,P=19082] create() Chroot into /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final
DEBUG [U=1000,P=19096] Chroot() Hold reference to host / directory
DEBUG [U=1000,P=19096] Chroot() Called pivot_root on /home/vagrant/singularity-3.7_install/var/singularity/mnt/session/final
DEBUG [U=1000,P=19096] Chroot() Change current directory to host / directory
DEBUG [U=1000,P=19096] Chroot() Apply slave mount propagation for host / directory
DEBUG [U=1000,P=19096] Chroot() Called unmount(/, syscall.MNT_DETACH)
DEBUG [U=1000,P=19096] Chroot() Changing directory to / to avoid getpwd issues
DEBUG [U=1000,P=19082] create() Chdir into / to avoid errors
VERBOSE [U=0,P=19095] wait_child() rpc server exited with status 0
DEBUG [U=0,P=19095] init() Set container privileges
DEBUG [U=0,P=19095] apply_privileges() Effective capabilities: 0x0000000000000000
DEBUG [U=0,P=19095] apply_privileges() Permitted capabilities: 0x0000000000000000
DEBUG [U=0,P=19095] apply_privileges() Bounding capabilities: 0x0000000000000000
DEBUG [U=0,P=19095] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=0,P=19095] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=0,P=19095] apply_privileges() Set user ID to 1000
DEBUG [U=1000,P=19095] set_parent_death_signal() Set parent death signal to 9
DEBUG [U=1000,P=19095] startup() singularity runtime engine selected
VERBOSE [U=1000,P=19095] startup() Execute stage 2
DEBUG [U=1000,P=19095] StageTwo() Entering stage 2
DEBUG [U=1000,P=19095] StartProcess() Setting umask in container to 0002
DEBUG [U=1000,P=19095] sylogBuiltin() Sourcing /.singularity.d/env/01-base.sh
DEBUG [U=1000,P=19095] sylogBuiltin() Sourcing /.singularity.d/env/90-environment.sh
DEBUG [U=1000,P=19095] sylogBuiltin() Sourcing /.singularity.d/env/94-appsbase.sh
DEBUG [U=1000,P=19095] sylogBuiltin() Sourcing /.singularity.d/env/95-apps.sh
DEBUG [U=1000,P=19095] sylogBuiltin() Sourcing /.singularity.d/env/99-base.sh
DEBUG [U=1000,P=19095] sylogBuiltin() Sourcing /.singularity.d/env/99-runtimevars.sh
DEBUG [U=1000,P=19095] sylogBuiltin() Running action command exec
DEBUG [U=1000,P=19082] PostStartProcess() Post start process
go jschulz1 project_data sandbox_containers singularity singularity-3.7.4.tar.gz singularity-3.7_install
DEBUG [U=1000,P=19082] Master() Child exited with exit status 0
Output of dmesg (only updates w/ errors in the mounted container use-case):
[347723.192540] print_req_error: 14 callbacks suppressed
[347723.192541] print_req_error: I/O error, dev loop0, sector 896768
[347723.197662] print_req_error: I/O error, dev loop0, sector 896768
[347723.199206] buffer_io_error: 8 callbacks suppressed
[347723.199207] Buffer I/O error on dev loop0, logical block 112096, async page read
[347723.201758] print_req_error: I/O error, dev loop0, sector 0
[347723.204366] SQUASHFS error: squashfs_read_data failed to read block 0x0
[347723.206270] squashfs: SQUASHFS error: unable to read squashfs_super_block
[347723.207817] print_req_error: I/O error, dev loop0, sector 896640
[347723.210262] print_req_error: I/O error, dev loop0, sector 896640
[347723.211639] Buffer I/O error on dev loop0, logical block 448320, async page read
[347723.213576] print_req_error: I/O error, dev loop0, sector 896642
[347723.214768] Buffer I/O error on dev loop0, logical block 448321, async page read
[347723.216324] print_req_error: I/O error, dev loop0, sector 896644
[347723.218011] Buffer I/O error on dev loop0, logical block 448322, async page read
[347723.219681] print_req_error: I/O error, dev loop0, sector 896646
[347723.221030] Buffer I/O error on dev loop0, logical block 448323, async page read
[347723.222519] print_req_error: I/O error, dev loop0, sector 8
[347723.223733] print_req_error: I/O error, dev loop0, sector 8
[347723.224973] Buffer I/O error on dev loop0, logical block 4, async page read
[347723.226314] Buffer I/O error on dev loop0, logical block 5, async page read
[347723.227642] Buffer I/O error on dev loop0, logical block 6, async page read
[347723.229305] Buffer I/O error on dev loop0, logical block 7, async page read
[347723.192541] print_req_error: I/O error, dev loop0, sector 896768
[347723.197662] print_req_error: I/O error, dev loop0, sector 896768
[347723.199206] buffer_io_error: 8 callbacks suppressed
[347723.199207] Buffer I/O error on dev loop0, logical block 112096, async page read
[347723.201758] print_req_error: I/O error, dev loop0, sector 0
[347723.204366] SQUASHFS error: squashfs_read_data failed to read block 0x0
[347723.206270] squashfs: SQUASHFS error: unable to read squashfs_super_block
[347723.207817] print_req_error: I/O error, dev loop0, sector 896640
[347723.210262] print_req_error: I/O error, dev loop0, sector 896640
[347723.211639] Buffer I/O error on dev loop0, logical block 448320, async page read
[347723.213576] print_req_error: I/O error, dev loop0, sector 896642
[347723.214768] Buffer I/O error on dev loop0, logical block 448321, async page read
[347723.216324] print_req_error: I/O error, dev loop0, sector 896644
[347723.218011] Buffer I/O error on dev loop0, logical block 448322, async page read
[347723.219681] print_req_error: I/O error, dev loop0, sector 896646
[347723.221030] Buffer I/O error on dev loop0, logical block 448323, async page read
[347723.222519] print_req_error: I/O error, dev loop0, sector 8
[347723.223733] print_req_error: I/O error, dev loop0, sector 8
[347723.224973] Buffer I/O error on dev loop0, logical block 4, async page read
[347723.226314] Buffer I/O error on dev loop0, logical block 5, async page read
[347723.227642] Buffer I/O error on dev loop0, logical block 6, async page read
[347723.229305] Buffer I/O error on dev loop0, logical block 7, async page read
I hope you're able to help, this workflow is very convenient for me and I've built portions of my development process around it
David Trudgian
Dec 14, 2021, 5:56:09 PM12/14/21
to Jonathan Schulz, Singularity Community Edition
Jonathan,
Thanks for the follow up info (apologies I dropped my initial response from the group by mistake)... hopefully this gets back to the thread!
I would ask that you try with an update to VirtualBox 6.1, and SingularityCE to 3.9 so that we can tackle this on currently supported versions of software. I'm afraid the policy on open source SingularityCE is that we only chase down fixes on the latest version.
VirtualBox 6.0.12 is quite old now (Sep 2019) and the 6.0 series has not been supported since July 2020. I understand it worked with an older Singularity... but we want to make sure there isn't a point in time issue that has been resolved since.
Thanks for the follow up info (apologies I dropped my initial response from the group by mistake)... hopefully this gets back to the thread!
I would ask that you try with an update to VirtualBox 6.1, and SingularityCE to 3.9 so that we can tackle this on currently supported versions of software. I'm afraid the policy on open source SingularityCE is that we only chase down fixes on the latest version.
VirtualBox 6.0.12 is quite old now (Sep 2019) and the 6.0 series has not been supported since July 2020. I understand it worked with an older Singularity... but we want to make sure there isn't a point in time issue that has been resolved since.
I will try and replicate with current versions (albeit on a newer macOS) on Thursday or Friday. I'll be out tomorrow.
Cheers,
DT
Cheers,
DT
On Tue, Dec 14, 2021 at 4:42 PM Jonathan Schulz <jschu...@gmail.com> wrote:
Hi David,Thanks for the quick response!Vagrant Version: (installed with homebrew)2.2.18Here is my vagrantfile: (I think config.vm.synced_folder + config.vm.network manage fs mounting)Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/bionic64"
config.vm.synced_folder "/Users/jschulz1", "/home/vagrant/jschulz1"
config.vm.network "forwarded_port", guest: 5011, host: 5011
config.vm.provider "virtualbox" do |v|
v.memory = 2560
end
config.ssh.forward_agent = true
config.ssh.forward_x11 = true
endVirtualBox Version:6.0.12 r133076 (Qt5.6.3)Guest Extensions:I don't think I have anything installedI hope that helpsThanks,JonathanOn Tue, Dec 14, 2021 at 4:18 PM David Trudgian <dtr...@sylabs.io> wrote:Hi Jonathan,What version of Vagrant, and VirtualBox are you using, what guest extensions do you have in the VM, and how are you configuring the shared Mac->VM filesystem?
This isn't necessarily expected.... nor is it completely unexpected. There have been issues with file access across a VirtualBox shared filesystem in the past, and matching versions of the guest additions with the host etc. can be important.DT
--
You received this message because you are subscribed to the Google Groups "Singularity Community Edition" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity-c...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/singularity-ce/e3a78ff7-9f96-4047-8eb1-f78d4061f53fn%40googlegroups.com.
--David TrudgianSylabs Inc.
David Trudgian
Sylabs Inc.
Jonathan Schulz
Dec 14, 2021, 6:06:03 PM12/14/21
to Singularity Community Edition
Hi David,
Reasonable policy! I'll give that a shot and let you know if that fixes it
Thanks,
Jonathan
Reply all
Reply to author
Forward
0 new messages