Unable to run Centos-7 container as unprivileged user on RHEL-6 host

[Chih-Song Kuo]

Hello,

This is Chih-Song, HPC Benchmark Specialist from Fujitsu.

After several days of trials and errors I realized that the OS kernel seems to have an impact on the ability to run Singularity as unprivileged user.
Centos-7 container on RHEL-6 host: I can only run with unprivileged user (root)
Centos-6 container on RHEL-6 host: I can run with privileged and unprivileged user
The sudoers file and the installation path of Singularity are the same in both settings.

Has anyone experienced the same situation?

BTW, the Q&A mentioned that "note: If you require kernel dependent features, a container platform is probably not the right solution for you.There is also some level of glibc forward compatibility that must be taken into consideration for any container system." Is this issue related to the one I am reporting?

To the developers: Does such limitation exist by intention or am I discovering a potential bug?

[Nekel-Seyew]

Hello!

Can you clarify what you meant by " I can only run with unprivileged user (root)"? Do you mean that you only run as root inside the container? Or do you mean you cannot run as root inside the container? How are you launching the container? Are you calling it as "sudo singularity cent-7.img" or "singularity cent-7.img?"

In my experience, you can only run as a privileged user inside a container if you are a privileged user inside the container. Additionally, I've had poor luck building and installing Singularity on RHEL/CENTOS systems. Instead, I build all my singularity programs on latest Ubuntu, package it as an RPM, then "sudo yum install singularity.tar.gz". Perhaps doing that will fix your issue?

[Tru Huynh]

hi,

On Wed, Feb 22, 2017 at 08:18:51AM -0800, Nekel-Seyew wrote:
> Hello!
>
...

>
> Additionally, I've had
> poor luck building and installing Singularity on RHEL/CENTOS systems.
> Instead, I build all my singularity programs on latest Ubuntu, package it
> as an RPM, then "sudo yum install singularity.tar.gz". Perhaps doing that
> will fix your issue?

oh?
until singularity is provided by some repositories:

$ sudo yum -y install rpm-build gcc wget
$ wget https://github.com/singularityware/singularity/releases/download/2.2.1/singularity-2.2.1.tar.gz
$ rpmbuild -ta singularity-2.2.1.tar.gz
$ sudo rpm -ivh ~/rpmbuild/RPMS/x86_64/singularity-2.2.1-0.1.el7.centos.x86_64.rpm

>
> On Wednesday, February 22, 2017 at 10:52:30 AM UTC-5, Chih-Song Kuo wrote:
> >
> > Hello,
> >
> > This is Chih-Song, HPC Benchmark Specialist from Fujitsu.
> >
> > After several days of trials and errors I realized that the OS kernel
> > seems to have an impact on the ability to run Singularity as unprivileged
> > user.
> > Centos-7 container on RHEL-6 host: I can only run with unprivileged user
> > (root)

How was the Centos-7 container built? from docker or bootstrap'ed with
yum ? on CentOS-7 host or CentOS-6 host? Same questions for your
CentOS-6 container.

There is not enough details to reproduce your issues.

Cheers

Tru
--
Dr Tru Huynh | http://www.pasteur.fr/research/bis
mailto:t...@pasteur.fr | tel/fax +33 1 45 68 87 37/19
Institut Pasteur, 25-28 rue du Docteur Roux, 75724 Paris CEDEX 15 France

[Chihsong]

Answers to Nekel-Seyew:

> Can you clarify what you meant by " I can only run with unprivileged user (root)"? Do you mean that you only run as root inside the container? Or do you mean you cannot run as root inside the container?

First of all, there was a bloody stupid type: It should be "run with PRIVILEGED user (root)" instead of "run with unprivileged user (root)".
By saying running with certain type of user I meant launching the container (calling singularity).

> How are you launching the container? Are you calling it as "sudo singularity cent-7.img" or "singularity cent-7.img?"

Launch as root: singularity cent-7.img

Launch as non-root: sudo singularity cent-7.img  ==> In this case I actually became root in the container

Answers to Tru

> How was the Centos-7 container built? from docker or bootstrap'ed with yum ? on CentOS-7 host or CentOS-6 host? Same questions for your CentOS-6 container.

Both containers were bootstrap'ed with yum on a RHEL6 host.

Chih-Song

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Tru Huynh]

Hi

Chihsong: I can't reproduce your claims.

1) host: centos6 x86_64
2) yum bootstrapping centos7 or centos6 containers ( just modify OSVersion in
https://github.com/singularityware/singularity/blob/master/examples/centos.def)
3) yields perfectly working setup for root and regular user with singularity 2.2.1.

Share a complete reproductible step by step procedure if you still
want help on the matter.

[Chih-Song Kuo]

Hi Tru,

After careful investigation I concluded that this problem existed only in ver. 2.2. With ver. 2.2.1 everything went smoothly!

But, now I got another problem: The host crashed and rebooted after completing bootstraping. This situation has been reproduced twice on two different hosts (both run RHEL6.7). The image file that was bootstraped was then still perfectly usable after reboot.

##As unprivileged user
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.7 (Santiago)
$ cd /home/chih/src
$ wget https://github.com/singularityware/singularity/releases/download/2.2.1/singularity-2.2.1.tar.gz
$ tar xfz singularity-2.2.1.tar.gz
$ cd singularity-2.2.1
$ ./configure --prefix=/usr/local --sysconfdir=/etc
$ make
$ su
##As privileged user
# make install
# grep secure_path /etc/sudoers
Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
# grep chih /etc/sudoers
chih    ALL=NOPASSWD:/usr/local/bin/singularity
exit

$ cd ~/containers
$ cp /home/chih/src/singularity-2.2.1/examples/centos.def ./
$ cat centos.def
# Copyright (c) 2015-2016, Gregory M. Kurtzer. All rights reserved.
#
# "Singularity" Copyright (c) 2016, The Regents of the University of California,
# through Lawrence Berkeley National Laboratory (subject to receipt of any
# required approvals from the U.S. Dept. of Energy).  All rights reserved.


BootStrap: yum
OSVersion: 7
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
Include: yum

# If you want the updates (available at the bootstrap date) to be installed
# inside the container during the bootstrap instead of the General Availability
# point release (7.x) then uncomment the following line
#UpdateURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/updates/$basearch/


%runscript
    echo "This is what happens when you run the container..."


%post
    echo "Hello from inside the container"
    yum -y install vim-minimal

$ sudo singularity create container-centos7-test.img
Creating a new image with a maximum size of 768MiB...
Executing image create helper
Formatting image with ext3 file system
Done.
$ sudo singularity bootstrap container-centos7-test.img  centos.def
$ singularity shell /home/chih/containers/container-centos6-demo.img
....
Installed:
  vim-minimal.x86_64 2:7.4.160-1.el7_3.1

Complete!
Done.
## SSH session to the host froze and then the host was rebooted. I was not able to see what went wrong on the host.

[Tru Huynh]

On Wed, Feb 22, 2017 at 03:26:27PM -0800, Chih-Song Kuo wrote:
> Hi Tru,
>
> After careful investigation I concluded that this problem existed only in
> ver. 2.2. With ver. 2.2.1 everything went smoothly!
>
> But, now I got another problem: The host crashed and rebooted after
> completing bootstraping. This situation has been reproduced twice on two
> different hosts (both run RHEL6.7). The image file that was bootstraped was
> then still perfectly usable after reboot.

oh 6.7? what kernel version?
...

> ## SSH session to the host froze and then the host was rebooted. I was not
> able to see what went wrong on the host.

:(
Are you using cgroup and NFS (v3/v4?)

You can raise a ticket to Red Hat support and add your findings/details to
https://github.com/singularityware/singularity/issues/452

OTOH, they might ask you to upgrade to 6.8 and apply all the updates...
Current kernel version is kernel-2.6.32-642.13.1.el6.x86_64.
if you can reproduce the crash, if it does crash/panic, you can raise
add your findings/details to the github issue.

Cheers