How to responding to a singularity security admin concern ?
88 views
Skip to first unread message
Samy
Feb 2, 2021, 12:35:27 PM2/2/21
to singularity
Hello everyone,
I requested an HPC cluster admins to install Singularity so i can run some containers i have. They will only allow the sandbox version to run not the sif. This is kind of what they said:
... the standard singularity installation has a security flaw. It’s theoretically possible to craft an image that could lead to an exploit and elevation of privileges. ....we can't rebuild and validate each image, the standard singularity installation will not be allowed to run on {our cluster}– only sandboxes.
Any evidence to share with them and prove this is not true or Singularity installation can really lead to security issues?
Thank you,
David Trudgian
Feb 2, 2021, 12:41:53 PM2/2/21
to singularity, Samy
Hi Samy,
It's unlikely that we can provide evidence to counter a specific security concern that we don't have detail about.
I would suggest asking the system administrators to contact us privately according to our security policy if they believe there is a security flaw in Singularity: https://sylabs.io/security-policy
We take reports seriously, and address them promptly.
We take reports seriously, and address them promptly.
Cheers,
Dave Trudgian
Dave Dykstra
Feb 2, 2021, 6:14:27 PM2/2/21
to singu...@lbl.gov, Samy
I expect they're referring to the fact that singularity uses setuid-root
to mount a squashfs filesystem from the SIF file. In theory if there's
a bug in the squashfs driver and it isn't properly validating all the
bytes somebody could through a lot of effort craft a special image that
would fool the squashfs driver into doing something it isn't supposed to
do, in the kernel space.
Nobody has demonstrated an exploit using that theoretical attack method,
but it's the reason why the kernel requires squashfs (and most
filesystem) mounts to be done by root. They're not even allowed from
the fake root inside of a user namespace, as discussed in this article:
https://lwn.net/Articles/652468
Most HPC administrators have deemed this risk to be acceptable, but
apparently some have not. Modern Linux kernels do allow fuse mounts
from unprivileged user namespaces, so at some point when singularity
supports fuse-squashfs or equivalent, it could be done at a lower risk.
It turns out that mounting SIF files in an HPC environment is a huge
performance benefit over sandboxed containers unpacked on a high-speed
file server. That's because it moves the metadata operations to the
worker node instead of sending them all to the file server. I think
this is a major reason why most HPC administrators are willing to live
with the theoretical risk.
Another way to move metadata operations to the worker node is the CernVM
Filesystem (https://cernvm.cern.ch/fs). It also works most efficiently
with sandboxed singularity containers, so it's the best of both plus it
gives instantaneous and reliable world-wide distribution. The High
Energy Physics community uses it extensively along with completely
unprivileged singularity. It can even be run as an unprivileged user
when unprivileged user namespaces are enabled:
https://github.com/cvmfs/cvmfsexec
Dave
On Tue, Feb 02, 2021 at 09:41:53AM -0800, David Trudgian wrote:
> Hi Samy,
>
> It's unlikely that we can provide evidence to counter a specific security
> concern that we don't have detail about.
>
> I would suggest asking the system administrators to contact us privately
> according to our security policy if they believe there is a security flaw
> in Singularity: https://sylabs.io/security-policy
>
> We take reports seriously, and address them promptly.
>
> Cheers,
>
> Dave Trudgian
>
> On Tuesday, February 2, 2021 at 11:35:27 AM UTC-6 Samy wrote:
>
> > Hello everyone,
> >
> > I requested an HPC cluster admins to install Singularity so i can run some
> > containers i have. They will only allow the sandbox version to run not the
> > sif. This is kind of what they said:
> >
> > *... the standard singularity installation has a security flaw. It's
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/02822d29-4491-43fd-9f8f-0417c08d1d20n%40lbl.gov .
to mount a squashfs filesystem from the SIF file. In theory if there's
a bug in the squashfs driver and it isn't properly validating all the
bytes somebody could through a lot of effort craft a special image that
would fool the squashfs driver into doing something it isn't supposed to
do, in the kernel space.
Nobody has demonstrated an exploit using that theoretical attack method,
but it's the reason why the kernel requires squashfs (and most
filesystem) mounts to be done by root. They're not even allowed from
the fake root inside of a user namespace, as discussed in this article:
https://lwn.net/Articles/652468
Most HPC administrators have deemed this risk to be acceptable, but
apparently some have not. Modern Linux kernels do allow fuse mounts
from unprivileged user namespaces, so at some point when singularity
supports fuse-squashfs or equivalent, it could be done at a lower risk.
It turns out that mounting SIF files in an HPC environment is a huge
performance benefit over sandboxed containers unpacked on a high-speed
file server. That's because it moves the metadata operations to the
worker node instead of sending them all to the file server. I think
this is a major reason why most HPC administrators are willing to live
with the theoretical risk.
Another way to move metadata operations to the worker node is the CernVM
Filesystem (https://cernvm.cern.ch/fs). It also works most efficiently
with sandboxed singularity containers, so it's the best of both plus it
gives instantaneous and reliable world-wide distribution. The High
Energy Physics community uses it extensively along with completely
unprivileged singularity. It can even be run as an unprivileged user
when unprivileged user namespaces are enabled:
https://github.com/cvmfs/cvmfsexec
Dave
On Tue, Feb 02, 2021 at 09:41:53AM -0800, David Trudgian wrote:
> Hi Samy,
>
> It's unlikely that we can provide evidence to counter a specific security
> concern that we don't have detail about.
>
> I would suggest asking the system administrators to contact us privately
> according to our security policy if they believe there is a security flaw
> in Singularity: https://sylabs.io/security-policy
>
> We take reports seriously, and address them promptly.
>
> Cheers,
>
> Dave Trudgian
>
> On Tuesday, February 2, 2021 at 11:35:27 AM UTC-6 Samy wrote:
>
> > Hello everyone,
> >
> > I requested an HPC cluster admins to install Singularity so i can run some
> > containers i have. They will only allow the sandbox version to run not the
> > sif. This is kind of what they said:
> >
> > theoretically possible to craft an image that could lead to an exploit and
> > elevation of privileges. ....we can't rebuild and validate each image, the
> > standard singularity installation will not be allowed to run on {our
> > cluster}-- only sandboxes.*
> > elevation of privileges. ....we can't rebuild and validate each image, the
> > standard singularity installation will not be allowed to run on {our
> >
> > Any evidence to share with them and prove this is not true or Singularity
> > installation can really lead to security issues?
> >
> >
> > Thank you,
> >
>
> --
> > Any evidence to share with them and prove this is not true or Singularity
> > installation can really lead to security issues?
> >
> >
> > Thank you,
> >
>
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/02822d29-4491-43fd-9f8f-0417c08d1d20n%40lbl.gov .
Reply all
Reply to author
Forward
0 new messages