Singularity security problem?

[Jordi G.]

Hi all,

We are using Singularity in a project. One of the partners is reluctant to install Singularity in their premises because in their opinion it has some security flaws. In particular, they claim that:

It's "kind of" broken by design but only because it allows unprivileged users to request that the kernel interpret arbitrary data as a file system. The real underlying issue is that file system developers have depended for years on root privs being required to mount a file system. (GNOME's gVFS is a notable exception here.) Basically any file system issue that can result in a BUG or an Oops can be targeted by unprivileged users with Singularity (or any other system that allows unpriviledged users to mount file system images.) We could "check" the image before mounting but the performance penalty incurred would more or less negate any advantage that using this method brings.

Does their concern make sense?

Thanks,

[Brian Bockelman]

Hi Jordi,

Their concerns make sense, but there are reasons why one might move
forward with Singularity regardless.

A few ways to think about the issue:

0) Any time you allow arbitrary code execution on a Linux host, there
are similar concerns (i.e., this is not a new concern but just a
level-of-risk). I'd guess that there are 2-5 CVEs per year that
result in privilege escalation or BUG/Oops'ing of a kernel for folks
that have the ability to do arbitrary code execution: Singularity
images are indeed another route that this could occur (and you provide
an approximate line of reasoning to argue it may occur more often),
but it's disingenuous to say risk isn't present otherwise.
- Singularity is often used at HPC or research computing centers
where the admins and users have a pre-existing trust relationship and
have non-technical routes to stop users from being malicious. This is
a very different environment from, say, an arbitrary hosting provider.
- To give a concrete example: if a local user was able to
maliciously trigger an Oops via Singularity, they would likely get
banned from the facility.
- Malicious local users can also attack Infiniband drivers or
disrupt the cluster via clever abuse of the shared filesystem. Do you
really believe your Infiniband driver is hardened against malicious
users?
Hence, it's a measured risk; it's good to be aware of and understand
whether it is worth the benefits for your use case and community.
1) If you understand (0) but still would rather not take the risk,
alternates include:
- Disabling setuid mode and use user namespace mode (this is a
separate set of trade-offs; see
http://singularity.lbl.gov/docs-config#allow-setuid-boolean-defaultyes).
- Only allowing images owned by a certain user (i.e., those owned by
root or "admin" or whatnot; see
http://singularity.lbl.gov/docs-security#can-i-limit-usage-to-specific-users).
Even if you only allow images owned by a certain user, Singularity
is quite useful: for example, one can take an existing image and
unpack it to a directory, then invoke singularity on the directory
itself. It's one more step on the user side, but this acts more like
a traditional chroot and bypasses the concern you highlight above.

HTH,

Brian

> --
> You received this message because you are subscribed to the Google Groups
> "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to singularity...@lbl.gov.

[Gregory M. Kurtzer]

In addition to Brian's comments, we have been thinking about this already and to prelude some coming features, this should be mitigated via signed containers, and optionally enabling the system administrator to disallow containers to run which have not been signed. You could also enable Singularity to only "trust" and allow use of containers that are owned by a particular user, or residing in a particular path.

Additionally a maliciously derived container image should be caught by the kernel FS driver else that is a kernel bug... Granted we are enabling that vector, and will work with upstream kernel developers and Linux distributions to ensure that the operating systems have been fixed properly. To my knowledge this has not happened (yet) and there is plenty of precedent in distributing images (ISO, SquashFS, etc..) already.

Greg

> email to singularity+unsubscribe@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Gregory M. Kurtzer

CEO, SingularityWare, LLC.

Senior Architect, RStor

Computational Science Advisor, Lawrence Berkeley National Laboratory

[Priedhorsky, Reid]

> On Sep 13, 2017, at 2:54 PM, Gregory M. Kurtzer <gmku...@gmail.com> wrote:
>
> Additionally a maliciously derived container image should be caught by the kernel FS driver else that is a kernel bug... [...] To my knowledge this has not happened (yet) and there is plenty of precedent in distributing images (ISO, SquashFS, etc..) already.

Here’s an example of a specific kernel bug tickled by crafted filesystem image:

http://www.openwall.com/lists/oss-security/2017/02/03/1

It’s not clear to me whether one can mount arbitrary filesystem images in a user+mount namespace. For example, the following LWN article suggests that one cannot, but in the context of a proposed patch to change that:

https://lwn.net/Articles/652468/

HTH,
Reid

[Brian Bockelman]

On Wed, Sep 13, 2017 at 4:07 PM, Priedhorsky, Reid <rei...@lanl.gov> wrote:
>
>> On Sep 13, 2017, at 2:54 PM, Gregory M. Kurtzer <gmku...@gmail.com> wrote:
>>
>> Additionally a maliciously derived container image should be caught by the kernel FS driver else that is a kernel bug... [...] To my knowledge this has not happened (yet) and there is plenty of precedent in distributing images (ISO, SquashFS, etc..) already.
>
> Here’s an example of a specific kernel bug tickled by crafted filesystem image:
>
> http://www.openwall.com/lists/oss-security/2017/02/03/1

Yes -- but there's certainly plenty of CVEs (possibly many more!)
associated with kernel bugs tickled by normal, unprivileged arbitrary
user space code.

That's why I think the more pertinent question is:
- Whether the resource owner views this as a new significant risk
(particularly, if you already allow the user to submit jobs in a batch
system, have access to Infiniband, shared FS, etc).
- Whether the risk outweighs the benefits.

Certainly our local site has made one decision - and I understand and
respect that other sites may come to different conclusions. That's OK
too! To over-generalize -- if all open-science research computing
centers had to take the same security posture as the NSA, not much
science would be done!

I think the important aspect is the resource owner considers these
sorts of factors in the context of what they already allow the user to
do.

>
> It’s not clear to me whether one can mount arbitrary filesystem images in a user+mount namespace. For example, the following LWN article suggests that one cannot, but in the context of a proposed patch to change that:
>
> https://lwn.net/Articles/652468/
>

(FWIW - there's a few other *excellent* LWN articles over the last 1-2
years on the topic; short on time right now, so I don't have a link
handy...)

I don't see arbitrary FS images in user+mount namespaces in the
near-future. I *hope* that we will be able to see FUSE filesystems in
user+mount namespaces in the next year or two. After all - the Linux
devs have to consider a much broader range of use cases than I do!

The alternate approach (disable images, force users to unpack) does
help reduce the overall risk profile. Additionally, locally we're
quite pleased with the potential of user namespaces -- I always tell
folks that if your sysadmin isn't paranoid about adding a new setuid
binary, you need a new sysadmin!

Brian

[vanessa s]

There are two kinds of security checks I think about - a constant scanner (usually requiring a server) and then "on demand) (eg, like a security linter when an image is built / used, etc)

Scanner

David has been working on a Clair (security scanner) to go alongside Singularity Registry.

https://github.com/singularityhub/sregistry/issues/14

I'm not sure about the status, but it would be great to get this working for some kind of optional plugin. 

Checks

This, along with "on demand" checks:

https://github.com/singularityware/singularity/tree/development/libexec/helpers#checks

It would be great to expand the checks that are provided with Singularity - right now they are mostly basic / examples with hope that the community can contribute!

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Jordi G.]

Thanks everybody for the insightful responses!

[David Trudgian]

> *Scanner*

> David has been working on a Clair (security scanner) to go alongside
> Singularity Registry.
>
> https://github.com/singularityhub/sregistry/issues/14
>
> I'm not sure about the status, but it would be great to get this working
> for some kind of optional plugin.

I don't think scanning containers with Clair really addresses the issue in the
thread of exploiting filesystem drivers - it's just checking for CVEs present in
any apt/yum packages installed inside the container. I think it's a useful thing
though.

The current status of this work is:

* There's a working python CLI tool that will scan a Singularity image using Clair
and spit out basic JSON/text information about identified issues:

https://github.com/dctrud/clair-singularity

It requires Clair to be spun-up under docker to do this (instructions in the
README using some handy pre-populated docker hub images).


* I'm currently thinking about integration with how we plan to use sregistry
here at UTSW. We definitely want something like the quay.io security reports
visible in our internal registry web interface. I know Vanessa is keen to
keep sregistry as unencumbered as possible with extra stuff, so I'm thinking
about the idea of a plugin-able Django app, and being able to setup periodic
/ on upload scanning through handled by celery jobs with that. It's
definitely something we want to have here, so will happen - just not
entirely sure when, as I'm committed to a lot of training and things in the
next couple of months.


--
David Trudgian Ph.D.
Computational Scientist, BioHPC
UT Southwestern Medical Center
Dallas, TX 75390-9039
Tel: (214) 648-4833

[Brian Bockelman]

Hi David,

I think the really interesting exploits would happen not when the
images are created - but as they are changed by the arbitrary user
process running outside the container.

One can imagine there are undiscovered race conditions when the
filesystem image is modified while it is mounted.

I wonder if there's a new feature needed which allows one to `pull` an
image but not allow it to be modified by the unprivileged user? It'd
be a nice compromise between the default mode and limiting users to
only using image directories.

Brian

> --
> You received this message because you are subscribed to the Google Groups "singularity" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Gregory M. Kurtzer]

This will be an interesting and pretty easy configuration option to allow what kind of images to allow in the config(squashfs, ext, or directory/sandbox images). Actually I think we only really need to be worries about writable ext3 based containers. 

We are locking the development branch of Singularity soon so I will try to get that option in there before the feature freeze. 

Thanks!

[David Trudgian]

Hi Brian,

Yes, that's an interesting area - the scanner work is not concerned about those kind of exploits on the host side. I'm more concentrated on the types of scenarios where:

User is running some service inside a singularity container (maybe an http accessible jupyter notebook, nvidia digits etc.) and there is a known exploit in some package in the container (e.g. python, openssl) that would allow malicious access to the user's data that is mounted into the container. This kind of thing could allow remote attacks that grab data. No need to have a shell on the cluster - just access to any exposed network service.

If we start using containers a lot here, our Information Security folk are reasonably going to expect that the OS packages in the containers are kept up to date against critical issues, as for our host systems. That's the aim for the Clair scanning.

--
David Trudgian Ph.D.
Computational Scientist, BioHPC

Lyda Hill Department of Bioinformatics
UT Southwestern Medical Center
Dallas, TX 75390-9365

________________________________

UT Southwestern


Medical Center



The future of medicine, today.

[Dave Dykstra]

There's already a configuration option to prevent mounting arbitrary
image files for the security paranoid system administrator:
max loop devices = 0

Just make everybody use unpacked images. This fits great with grids
where we use cvmfs for software distribution, and where we want to use
unprivileged singularity on EL7 as much as possible anyway. It's more
of a challenge on supercomputers where the metadata servers on the
parallel filesystems can't handle all the small files. For those, maybe
they can give each user their own large loopback-mounted file, where the
system adminstrator controls the filesystem and users can write anything
they want but only through the operating system interface.

Dave

[Brian Bockelman]

Hi David,

Yup - that's an important (and tough!) use case, although separate from the thread at hand. Jordi was originally concerned with the case where a user outside the container manipulated the raw filesystem image in a manner that triggered an unknown security bug.

Portable applications that listen on the network are a whole other ball of yarn compared to launching batch jobs, for precisely the reasons you describe. I don't think the community has great solutions (yet!!).

Locally, we don't allow unprivileged users start network services on the cluster computing resources; if a service needs to be run within campus, we'd rather the user partner with the administrators.

Brian

Sent from my iPhone

[Dave Dykstra]

I think that doing this is a very important security fix and not just a
nice-to-have optional feature. I think it should be the default. The
feature freeze is now past but I don't see anything in the config file
about. Did you get to it, Greg?

I'm thinking that by default singularity could require image files to be
owned by root but in a directory writable by the user and not writable
by others, and that "pull" and similar commands would conform to that
when creating images. Probably it would tie in with the "limit
container owners" option.

Dave

On Fri, Sep 15, 2017 at 04:17:05AM +0000, Gregory M. Kurtzer wrote:

[Dave Dykstra]

It looks the chattr +a attribute could be a good solution. It prevents
anyone from writing to a file except in append mode. It can only be set
or cleared by a program with root privileges or CAP_LINUX_IMMUTABLE
capability. It doesn't work on all filesystems (it failed for me on an
NFS mount, for example, probably NFSv3) but it's probably good enough.
Can anyone test it on some of the parallel filesystems?

Dave

On Mon, Sep 25, 2017 at 11:02:57AM -0500, Dave Dykstra wrote:
> [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]

[Dave Dykstra]

Google searches say this should work on Lustre. I don't know what other
parallel filesystems are commonon HPC systems these days.

Dave

[Gregory M. Kurtzer]

POSIX is already bad enough on HPC! haha

On Sat, Sep 30, 2017 at 1:35 AM, Cedric Clerget <ced...@sylabs.io> wrote:

BeeGFS is growing too in HPC world and seems not supporting extended attributes

Cédric

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.