Singularity broken when installed as an environment module on an NFS server

[Ole Holm Nielsen]

I have installed Singularity 2.4 on our Linux cluster which is running CentOS 7.4.  We prefer to have our software available as environment modules, and we use Lmod and EasyBuild for this purpose.
Unfortunately, I get a failure running the test example when Singularity has been installed as a module on a central NFS server.

$ cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
$ module load Singularity
$ which singularity
/home/modules/software/Singularity/2.4-GCC-6.3.0-2.27/bin/singularity
$ singularity run docker://godlovedc/lolcow

Docker image path: index.docker.io/godlovedc/lolcow:latest
Cache folder set to /home/opt/modules/.singularity/docker
[6/6] |===================================| 100.0%
Creating container runtime...
ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT  : Retval = 255

This may be the same issue as described in https://github.com/singularityware/singularity/issues/872

When I build and install RPM packages as described in http://singularity.lbl.gov/install-linux#build-an-rpm-from-source the example above runs without errors.

I happened to talk with Gregory M. Kurtzer <gmku...@gmail.com> at the SC17 conference in Denver, and he replied to me:
"I strongly suggest to install Singularity into the operating system rather then an environment module. We should update our documents accordingly to stress this as setting it up on shared storage is prone to problems."

Conclusion: The page http://singularity.lbl.gov/install-linux should be updated with Gregory's warning about installing Singularity as an environment module, and on an NFS server.

/Ole

[Tru Huynh]

Hi,

On Fri, Nov 24, 2017 at 12:10:55AM -0800, Ole Holm Nielsen wrote:
> I have installed Singularity 2.4 on our Linux cluster which is running
> CentOS 7.4. We prefer to have our software available as environment
> modules, and we use Lmod and EasyBuild for this purpose.
> Unfortunately, I get a failure running the test example when Singularity
> has been installed as a module on a central NFS server.

maybe because your NFS share is mounted "nosuid" ?

Cheers,

Tru
--
Dr Tru Huynh | mailto:t...@pasteur.fr | tel/fax +33 1 45 68 87 37/19
https://research.pasteur.fr/en/team/structural-bioinformatics/
Institut Pasteur, 25-28 rue du Docteur Roux, 75724 Paris CEDEX 15 France

[Ole Holm Nielsen]

On 11/24/2017 09:35 AM, Tru Huynh wrote:
> Hi,
>
> On Fri, Nov 24, 2017 at 12:10:55AM -0800, Ole Holm Nielsen wrote:
>> I have installed Singularity 2.4 on our Linux cluster which is running
>> CentOS 7.4. We prefer to have our software available as environment
>> modules, and we use Lmod and EasyBuild for this purpose.
>> Unfortunately, I get a failure running the test example when Singularity
>> has been installed as a module on a central NFS server.
> maybe because your NFS share is mounted "nosuid" ?

Yes, we use "nosuid" on NFS mounts for security reasons.

But why try to figure out workarounds when the creator of Singularity
(Gregory) strongly dis-recommends placing Singularity on a shared
filesystem?

/Ole

[Oliver Freyermuth]

Hi,

if you want to run without setuid root on the singularity binary,
can you confirm you did enable user namespaces as described here:
http://opensciencegrid.github.io/docs/worker-node/install-singularity/#enabling-unprivileged-mode-for-singularity
i.e. you have both the kernel parameter and the sysctl setting changed?

Additionally, you should configure singularity to use:
allow setuid = no
enable overlay = no
Of course, this means you can not make use of "setuid root"-only features,
such as: Mounting image files, bindmounting to directories which do not exist in the container yet, etc.

@Greg:

> I happened to talk with Gregory M. Kurtzer <gmku...@gmail.com> at the SC17 conference in Denver, and he replied to me:
> "I strongly suggest to install Singularity into the operating system rather then an environment module. We should update our documents accordingly to stress this as setting it up on shared storage is prone to problems."

Could this statement be elaborated on? I think the long-term plan of WLCG is to ship Singularity on CVMFS so even sites not having it installed can make use of it.

Cheers,
Oliver

Am 24.11.2017 um 09:10 schrieb Ole Holm Nielsen:
> I have installed Singularity 2.4 on our Linux cluster which is running CentOS 7.4.  We prefer to have our software available as environment modules, and we use Lmod and EasyBuild for this purpose.
> Unfortunately, I get a failure running the test example when Singularity has been installed as a module on a central NFS server.
>
> $ cat /etc/redhat-release
> CentOS Linux release 7.4.1708 (Core)
> $ module load Singularity
> $ which singularity
> /home/modules/software/Singularity/2.4-GCC-6.3.0-2.27/bin/singularity
> $ singularity run docker://godlovedc/lolcow
>

> Docker image path: index.docker.io/godlovedc/lolcow:latest <http://index.docker.io/godlovedc/lolcow:latest>

> Cache folder set to /home/opt/modules/.singularity/docker
> [6/6] |===================================| 100.0%
> Creating container runtime...
> ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument
> ABORT  : Retval = 255
>

> This may be the same issue as described in https://github.com/singularityware/singularity/issues/872 <https://github.com/singularityware/singularity/issues/872>
>
> When I build and install RPM packages as described in http://singularity.lbl.gov/install-linux#build-an-rpm-from-source <http://singularity.lbl.gov/install-linux#build-an-rpm-from-source> the example above runs without errors.

>
> I happened to talk with Gregory M. Kurtzer <gmku...@gmail.com> at the SC17 conference in Denver, and he replied to me:
> "I strongly suggest to install Singularity into the operating system rather then an environment module. We should update our documents accordingly to stress this as setting it up on shared storage is prone to problems."
>
> Conclusion: The page http://singularity.lbl.gov/install-linux should be updated with Gregory's warning about installing Singularity as an environment module, and on an NFS server.
>
> /Ole
>

> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov <mailto:singularity...@lbl.gov>.

[Ole Holm Nielsen]

On 11/24/2017 09:46 AM, Oliver Freyermuth wrote:
> if you want to run without setuid root on the singularity binary,
> can you confirm you did enable user namespaces as described here:
> http://opensciencegrid.github.io/docs/worker-node/install-singularity/#enabling-unprivileged-mode-for-singularity
> i.e. you have both the kernel parameter and the sysctl setting changed?

I have no intention of mucking with our HPC cluster nodes' kernel setup
for the purpose of running Singularity ;-) So this is not a solution in
my book.

> Additionally, you should configure singularity to use:
> allow setuid = no
> enable overlay = no
> Of course, this means you can not make use of "setuid root"-only features,
> such as: Mounting image files, bindmounting to directories which do not exist in the container yet, etc.

Thanks, but this may only be used if you change the kernel setup first?

> @Greg:
>> I happened to talk with Gregory M. Kurtzer <gmku...@gmail.com> at the SC17 conference in Denver, and he replied to me:
>> "I strongly suggest to install Singularity into the operating system rather then an environment module. We should update our documents accordingly to stress this as setting it up on shared storage is prone to problems."
> Could this statement be elaborated on? I think the long-term plan of WLCG is to ship Singularity on CVMFS so even sites not having it installed can make use of it.

I quoted the text from Gregory's mail. He's the guru, I'm just a novice.

/Ole

> Am 24.11.2017 um 09:10 schrieb Ole Holm Nielsen:
>> I have installed Singularity 2.4 on our Linux cluster which is running CentOS 7.4.  We prefer to have our software available as environment modules, and we use Lmod and EasyBuild for this purpose.
>> Unfortunately, I get a failure running the test example when Singularity has been installed as a module on a central NFS server.
>>
>> $ cat /etc/redhat-release
>> CentOS Linux release 7.4.1708 (Core)
>> $ module load Singularity
>> $ which singularity
>> /home/modules/software/Singularity/2.4-GCC-6.3.0-2.27/bin/singularity
>> $ singularity run docker://godlovedc/lolcow
>>
>> Docker image path: index.docker.io/godlovedc/lolcow:latest <http://index.docker.io/godlovedc/lolcow:latest>
>> Cache folder set to /home/opt/modules/.singularity/docker
>> [6/6] |===================================| 100.0%
>> Creating container runtime...
>> ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument
>> ABORT  : Retval = 255
>>
>> This may be the same issue as described in https://github.com/singularityware/singularity/issues/872 <https://github.com/singularityware/singularity/issues/872>
>>
>> When I build and install RPM packages as described in http://singularity.lbl.gov/install-linux#build-an-rpm-from-source <http://singularity.lbl.gov/install-linux#build-an-rpm-from-source> the example above runs without errors.
>>
>> I happened to talk with Gregory M. Kurtzer <gmku...@gmail.com> at the SC17 conference in Denver, and he replied to me:
>> "I strongly suggest to install Singularity into the operating system rather then an environment module. We should update our documents accordingly to stress this as setting it up on shared storage is prone to problems."
>>
>> Conclusion: The page http://singularity.lbl.gov/install-linux should be updated with Gregory's warning about installing Singularity as an environment module, and on an NFS server.
>>
>> /Ole
>>
>> --
>> You received this message because you are subscribed to the Google Groups "singularity" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov <mailto:singularity...@lbl.gov>.
>

--
Ole Holm Nielsen
PhD, Manager of IT services
Department of Physics, Technical University of Denmark,
Building 307, DK-2800 Kongens Lyngby, Denmark
E-mail: Ole.H....@fysik.dtu.dk
Homepage: http://dcwww.fysik.dtu.dk/~ohnielse/
Tel: (+45) 4525 3187 / Mobile (+45) 5180 1620

[Oliver Freyermuth]

Am 24.11.2017 um 09:51 schrieb Ole Holm Nielsen:
> On 11/24/2017 09:46 AM, Oliver Freyermuth wrote:
>> if you want to run without setuid root on the singularity binary,
>> can you confirm you did enable user namespaces as described here:
>> http://opensciencegrid.github.io/docs/worker-node/install-singularity/#enabling-unprivileged-mode-for-singularity
>> i.e. you have both the kernel parameter and the sysctl setting changed?
>
> I have no intention of mucking with our HPC cluster nodes' kernel setup for the purpose of running Singularity ;-)  So this is not a solution in my book.

Ok. Of course, this means you have to install (and use) Singularity in "setuid root" mode, which means essentially that (parts of) the code will run with
full root privileges.
That's why the filesystem on which the binary is stored on must allow to set the suid bit.

>
>> Additionally, you should configure singularity to use:
>> allow setuid = no
>> enable overlay = no
>> Of course, this means you can not make use of "setuid root"-only features,
>> such as: Mounting image files, bindmounting to directories which do not exist in the container yet, etc.
>
> Thanks, but this may only be used if you change the kernel setup first?

For CentOS 7.4, yes, since RedHat decided to still disable user namespace support by default. So a Kernel cmdline parameter and a sysctl flag is needed to turn it on.
On CentOS 7.4, that's the only way to run Singularity without involving implicit root privileges (due to the Singularity binary being executed with suid root).
Each site admin has to consider the probability of potential security issues in the Kernel (enabled by activating user namespace support) versus the probability of security issues in Singularity
(which would allow to get root if the suid root status is exploited).
I think there's no definitive answer what is better, but for sure Singularity offers more functionality in suid root mode.

>
>> @Greg:
>>> I happened to talk with Gregory M. Kurtzer <gmku...@gmail.com> at the SC17 conference in Denver, and he replied to me:
>>> "I strongly suggest to install Singularity into the operating system rather then an environment module. We should update our documents accordingly to stress this as setting it up on shared storage is prone to problems."
>> Could this statement be elaborated on? I think the long-term plan of WLCG is to ship Singularity on CVMFS so even sites not having it installed can make use of it.
>
> I quoted the text from Gregory's mail.  He's the guru, I'm just a novice.

Understood ;-). I'll wait for his response on the list, then.

[Gregory M. Kurtzer]

On Fri, Nov 24, 2017 at 12:46 AM, 'Oliver Freyermuth' via singularity <singu...@lbl.gov> wrote:

@Greg:
> I happened to talk with Gregory M. Kurtzer <gmku...@gmail.com> at the SC17 conference in Denver, and he replied to me:
> "I strongly suggest to install Singularity into the operating system rather then an environment module. We should update our documents accordingly to stress this as setting it up on shared storage is prone to problems."
Could this statement be elaborated on? I think the long-term plan of WLCG is to ship Singularity on CVMFS so even sites not having it installed can make use of it.

Sure, Singularity can of course be installed to an NFS share, but it requires a few more hoops to jump through. For example, the `nosuid` mount option that was already brought up, but also the need for it to be configured with the appropriate `--localstatedir=/local/node/path` option (usually local state is within `/var`) and that directory created with appropriate permissions on all nodes.

Considering the potential for complexities and that Singularity is so easily installed properly via RPM/Deb/`make install`, and that it is a very small install, I typically recommend to just install Singularity directly onto the host operating system.

Hope that helps!

 

--

Gregory M. Kurtzer

CEO, Sylabs Inc.

[Gregory M. Kurtzer]

On Fri, Nov 24, 2017 at 12:59 AM, 'Oliver Freyermuth' via singularity <singu...@lbl.gov> wrote:

Am 24.11.2017 um 09:51 schrieb Ole Holm Nielsen:
> On 11/24/2017 09:46 AM, Oliver Freyermuth wrote:
>> if you want to run without setuid root on the singularity binary,
>> can you confirm you did enable user namespaces as described here:
>> http://opensciencegrid.github.io/docs/worker-node/install-singularity/#enabling-unprivileged-mode-for-singularity
>> i.e. you have both the kernel parameter and the sysctl setting changed?
>
> I have no intention of mucking with our HPC cluster nodes' kernel setup for the purpose of running Singularity ;-)  So this is not a solution in my book.
Ok. Of course, this means you have to install (and use) Singularity in "setuid root" mode, which means essentially that (parts of) the code will run with
full root privileges.
That's why the filesystem on which the binary is stored on must allow to set the suid bit.

Of course both Ole and Oliver are perfectly correct here. If you want to allow users to run containers, you must either use a very recent kernel that supports the user namespace and/or SetUID or file based Linux Capabilities in order to run the necessary privileged system calls. Considering most HPC systems do not support the user namespace (e.g. any supported Red Hat kernel) we must use SetUID. Additionally, one of the founding principals of Singularity is the need to support single file container images which does require more permission then the user namespace alone can provide. Of course there are pros and cons to this feature and for that reason, we support multiple different container formats.

A feature that is being actively developed by Cedric and was just merged into the development branch (which is slightly broken at the moment) uses SetUID only to bootstrap the necessary capabilities, and then drop so that the process is running via the calling user's UID. This will be part of the next release.