Mount volumes without root permissions using fuse

[cschlick]

Hi, I'm struggling to mount a volume inside a container using programs which utilize fuse (sshfs, s3ql). My container will need to run on shared hpc resources, so having root permissions is not feasible. On my workstation (without using singularity), these mount operations do not require elevated permissions.

For example:

$ singularity shell --writable sing.img

Singularity sing.img:~/Desktop> sshfs <remotehost> <mountpoint>

fusermount: mount failed: Operation not permitted

Running the singularity shell with sudo makes everything work just fine. Any suggestions? Our group uses s3ql to provide transparent deduplication and compression independent of the storage backend, and if we can't mount inside a container then I don't think our group will be able to use containers.

[Jason Stover]

Hi,

On my system, fusermount is SUID 0.

$ ls -l `which fusermount`
-rwsr-x-r-x 1 root root 31376 Mar 7 2016 /bin/fusermount*

It _may_ not be set UID in the container, but it would still be
trying to elevate privileges with capabilities. Probably using the
overloaded CAP_SYS_ADMIN for mount operations.

NO_NEW_PRIVS blocks privilege escalation from both set UID binaries,
and capabilities. You can't raise privileges once you set it.

> Running the singularity shell with sudo makes everything work just fine.

You're running as root in this case ... so you have every privilege
and don't need to elevate.

-J

> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[cschlick]

Thanks for the reply Jason. I tried changing things with setcap and it didn't help. I also tried

$ chmod 777 /usr/bin/fusermount
$ chown user:user /usr/bin/fusermount

Just to see what would happen and it still didn't help. Though like I said before, it works fine on my machine outside of a container, where:

$ ls -l /usr/bin/fusermount

-rwsr-xr-x. 1 root root 32008 Apr 11 01:12 /usr/bin/fusermount

 which leads me to believe the problem is something about the singularity container, and not a fundamental limitation of trying to mount as a regular user. Any other ideas?

[Jason Stover]

Hi,

> which leads me to believe the problem is something about the
singularity container

Correct. You cannot elevate privileges inside of the container. So,
you can't become root for operations to run like you can outside of
the container.

It's by design, not a problem. Once the container starts up, you
can't elevate privileges to do any operations you need root privilege
for as a normal user. That's it. You're stuck at whatever privileges
you have as your user.

-J

[Dave Dykstra]

Docker behaves the same way. You can't mount fuse filesystems inside of
a container without giving the container privileged capabilities.

This changed however in Linux kernel 4.18 which was released August 12.
It supports FUSE mounts within an unprivileged user namespace.
https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.18-FUSE

Dave

[Jason Boles]

Hi Dave,

  Have you had any success on kernel 4.18 FUSE w/o privileges?

I just tested on Ubuntu 18.04.2 w/kernel 4.18.0-17, and Docker 18.09.5 (latest from Docker repo, not ubuntu’s) and sshfs, but I get the same "operation not permitted" error without --privileged or --cap-add...

I notice that /bin/fusermount is still setuid root, so perhaps that or sshfs haven't been updated to use 'unprivileged mount' that's supposed to be available in 4.18?

I will test again with libfuse3 and sshfs built from latest source, but just checking.

regards,

Jason B.

> > To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.

>
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.

[David Dykstra]

Hi Jason,

Yes, I have got it to work. I tested on a CentOS 7 system and a ELRepo
4.19 kernel. libfuse skips running /bin/fusermount when it has the
privileges. I proved it by removing it with "rpm -e --nodeps fuse".

I also found that libfuse3 has a neat feature that makes it easier to do
mounting with privileges, with older kernels. It supports pre-mounting
a filesystem and then passing through the an open fuse file descriptor
through as the mountpoint with /dev/fd/N. More details are in
https://github.com/sylabs/singularity/issues/2864

Dave

> > > On Tue, Aug 21, 2018 at 1:47 PM cschlick <csch...@iu.edu <javascript:>>

> > wrote:
> > > >
> > > > Thanks for the reply Jason. I tried changing things with setcap and it
> > didn't help. I also tried
> > > >
> > > > $ chmod 777 /usr/bin/fusermount
> > > > $ chown user:user /usr/bin/fusermount
> > > >
> > > > Just to see what would happen and it still didn't help. Though like I
> > said before, it works fine on my machine outside of a container, where:
> > > >
> > > > $ ls -l /usr/bin/fusermount
> > > > -rwsr-xr-x. 1 root root 32008 Apr 11 01:12 /usr/bin/fusermount
> > > >
> > > > which leads me to believe the problem is something about the
> > singularity container, and not a fundamental limitation of trying to mount
> > as a regular user. Any other ideas?
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > Groups "singularity" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send

> > an email to singu...@lbl.gov <javascript:>.

> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > Groups "singularity" group.
> > > To unsubscribe from this group and stop receiving emails from it, send

> > an email to singu...@lbl.gov <javascript:>.

> >
>
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Jason Boles]

Hi Dave,

  Thanks for the reply, I checked out your links, fuserpremount.c & fuse-premount.c,modified sshfs (release 3.5.2 has your changes), and read the libfuse 3.3.0 release notes that introduce /dev/fd/%u feature, but I think I still have a fundamental misunderstanding of how /dev/fd/* relates to /dev/fuse w.r.t. containers.

I've also tested on Fedora 29 host w/'fedora:latest' Docker container, passing device=/dev/fuse into the container.  When I run it without --privileged, the premount fails in the same way that fusermount3 fails, because setuid root doesn't actually become root.

--: mount failed: Operation not permitted

fuserpremount: failed to receive fd from fusermount

So, does that mean that I'd have to pre-mount (i.e. get a unique file descriptor) then assign that to --device=/dev/fd/blah as a parameter to the container at runtime, then inside the container run the fuse-command?  (If so, this is workable, but not ideal)

Ideally, I'd like to run each container with device=/dev/fuse, then inside the unprivileged container mount the fuse-based filesystem.  Even if that means that the host requires kernel 4.18+ and libfuse3.3+ and the container requires libfuse 3.3+, and the fuse filesystem inside the container must make the function calls to libfuse, rather than exec'ing fusermount3.

Inside my test container, I see this behavior:

[root@f3b78b593c16 ~]# sshfs user@host: /mnt/ssh

fusermount3: mount failed: Operation not permitted

[root@f3b78b593c16 ~]# rpm -e --nodeps fuse3

[root@f3b78b593c16 ~]# sshfs user@host: /mnt/ssh

fuse: failed to exec fusermount3: No such file or directory

[root@f3b78b593c16 ~]# fuse-premount sshfs user@host: /mnt/ssh

fuse-premount: failed to mount -o fd=3,rootmode=40000,user_id=0,group_id=0 /dev/fuse /mnt/ssh: Operation not permitted

(note above fuse-premount.c commented out "if (geteuid() != 0 || getuid() == 0) {}" test block...)

The use case is automated testing of buggy/unreliable fuse filesystems (student code) in a shared Docker environment where privileged mode is disabled.

Regards,

--Jason

> To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.

[David Dykstra]

Hi Jason,

On Tue, Apr 16, 2019 at 04:14:13PM -0700, Jason Boles wrote:
> Thanks for the reply, I checked out your links, fuserpremount.c &
> fuse-premount.c,modified sshfs (release 3.5.2 has your changes), and read
> the libfuse 3.3.0 release notes that introduce /dev/fd/%u feature, but I
> think I still have a fundamental misunderstanding of how /dev/fd/* relates
> to /dev/fuse w.r.t. containers.
>
> I've also tested on Fedora 29 host w/'fedora:latest' Docker container,
> passing device=/dev/fuse into the container. When I run it without
> --privileged, the premount fails in the same way that fusermount3 fails,
> because setuid root doesn't actually become root.
>
> --: mount failed: Operation not permitted
> fuserpremount: failed to receive fd from fusermount
>
>
> So, does that mean that I'd have to pre-mount (i.e. get a unique file
> descriptor) then assign that to --device=/dev/fd/blah as a parameter to the
> container at runtime, then inside the container run the fuse-command? (If
> so, this is workable, but not ideal)

Oh, I hadn't tried it with docker in the way yet. I tried a
fuse-premount with docker run just now, with the command to run being a
shell script that just prints the parameter and runs bash so I can do
what I want with it and the last parameter a writable mount point as
fuse-premount expects:
./fuse-premount docker run --rm -it fedora:latest /tmp/bashit $HOME/tmp/mnt
and I got an error from docker:
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:43: preparing rootfs caused \\\"Error found less than 3 fields post '-' in \\\\\\\"825 824 0:110 / /cloud/login/dwd/tmp/mnt rw,relatime - fuse rw,user_id=3382,group_id=9996\\\\\\\"\\\"\"".
ERRO[0000] error getting events from daemon: context canceled

I got around that by changing the first parameter sent to the mount command
in fuse-premount.c to "fuse", and added "-v /tmp/bashit:/tmp/bashit" to
map the script inside the container.

I then found that the file descriptor created by fuse-premount was not
passed in to the docker container. I'm quite sure that's because docker
containers aren't created as children of the docker process, they are
children of the docker-containerd daemon. So that's probably never
going to work unless there's some option to pass selected file
descriptors through (like fuserpremount does) but I don't see such an
existing option.

I haven't been able to get fuse-premount to work with singularity yet
either, because somewhere during initialization singularity tries to do
an fstat on the open file descriptor, and it hangs because there's
nothing yet implementing the fuse filesystem. It's not obvious where
this is happening. I think we'll just need to wait for the singularity
--fusecmd option, or maybe it would work in some other daemonless
container system like podman.

> Ideally, I'd like to run each container with device=/dev/fuse, then inside
> the unprivileged container mount the fuse-based filesystem. Even if that
> means that the host requires kernel 4.18+ and libfuse3.3+ and the container
> requires libfuse 3.3+, and the fuse filesystem inside the container must
> make the function calls to libfuse, rather than exec'ing fusermount3.
>
> Inside my test container, I see this behavior:
>
> [root@f3b78b593c16 ~]# sshfs user@host: /mnt/ssh
> fusermount3: mount failed: Operation not permitted
> [root@f3b78b593c16 ~]# rpm -e --nodeps fuse3
> [root@f3b78b593c16 ~]# sshfs user@host: /mnt/ssh
> fuse: failed to exec fusermount3: No such file or directory
> [root@f3b78b593c16 ~]# fuse-premount sshfs user@host: /mnt/ssh
> fuse-premount: failed to mount -o fd=3,rootmode=40000,user_id=0,group_id=0
> /dev/fuse /mnt/ssh: Operation not permitted
>
>
> (note above fuse-premount.c commented out "if (geteuid() != 0 || getuid()
> == 0) {}" test block...)
>
> The use case is automated testing of buggy/unreliable fuse filesystems
> (student code) in a shared Docker environment where privileged mode is
> disabled.

Oh, if you're willing to user linux kernel 4.18+ then you don't even
need libfuse3.

Just run "unshare -Ufirmp". That gets you a fake root unprivileged user
namespace. Then just run sshfs giving it a mount point of some empty
directory that you have write access to. I haven't tried it in
unprivileged docker yet, but it should work.

Dave

> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Jason Boles]

Oh, if you're willing to user linux kernel 4.18+ then you don't even 
need libfuse3. 

Just run "unshare -Ufirmp".  That gets you a fake root unprivileged user 
namespace.  Then just run sshfs giving it a mount point of some empty 
directory that you have write access to.  I haven't tried it in 
unprivileged docker yet, but it should work. 

Hi Dave,  I tried the "unshare -Ufirmp" but still get "Operation not permitted" errors.

(tried both outside and inside the container)

Maybe it's the same problem of container processes launching as children of containerd ?

Could a possible solution involve setns() within fuse-premount or using `nsenter` with /proc/<pid>/ns/* (where PID is that of the containerized process)?

Lastly, is it possible to pass a FD through docker run -v /some/fd:/tmp/fd:shared ?

regards,

Jason

[David Dykstra]

On Thu, Apr 18, 2019 at 09:21:52AM -0700, Jason Boles wrote:
> > Oh, if you're willing to user linux kernel 4.18+ then you don't even
> > need libfuse3.
> >
> > Just run "unshare -Ufirmp". That gets you a fake root unprivileged user
> > namespace. Then just run sshfs giving it a mount point of some empty
> > directory that you have write access to. I haven't tried it in
> > unprivileged docker yet, but it should work.
> >
>
> Hi Dave, I tried the "unshare -Ufirmp" but still get "Operation not
> permitted" errors.
> (tried both outside and inside the container)

Oh, unprivileged user namespaces are disabled by default on Debian,
maybe they are also on Ubuntu:
https://unix.stackexchange.com/questions/303213/how-to-enable-user-namespaces-in-the-kernel-for-unprivileged-unshare

The 4.19 ELRepo kernel I tried with CentOS 7.6 had it enabled by
default.

> Maybe it's the same problem of container processes launching as children of
> containerd ?

No, unshare is very simple and just invokes a few namespace-related
system calls.

> Could a possible solution involve setns() within fuse-premount or using
> `nsenter` with /proc/<pid>/ns/* (where PID is that of the containerized
> process)?

I don't know if the process started by nsenter is allowed to do mounts.
You'd have to try it. It should be easier to pass a file descriptor to
nsenter, you're right about that.

I don't recognize the usage of nsenter you're suggesting. When I use it
I pass it a pid (along with a bunch of options).

> Lastly, is it possible to pass a FD through docker run -v
> /some/fd:/tmp/fd:shared ?

No I don't think it is possible. -v just does a bind mount which acts
on files, not file descriptors. It would have to be a special option
that does something like my fuserpremount program to pass a file
descriptor from one process to another via a unix domain socket and
SCM_RIGHTS (as described in man unix).

Dave

[divyam jaiswal]

Hi Dave,

My usecase is a little different here.
I want to create a mount point and fd for /dev/fuse/ before calling fuse filesystem.
The idea is to get the ability to read/write to /dev/fuse directly without going through the kernel.

To achieve this I am using a modification of your program where I simply omit getuid and setuid part.
But I get following error on unomunting the filesystem:
"umount: /proc/self/mountinfo: parse error: ignore entry at line 24."

Regards,
Divyam

[Dave Dykstra]

Hi Divyam,

I'm sorry for the delay (sometimes I don't read the singularity mailing
list folder every day, include my email address if you want faster
response).

I'm not sure what you're trying to achieve. Are you running on a linux
kernel 4.18 or later and trying to use the unprivileged fuse mounts?
I don't know what you mean about using /dev/fuse directly without
the kernel; that device file is a connection to fuse in the kernel.

I have sometimes seen that umount warning, I didn't look into it, but I
think it is only a warning.

Dave

> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/6fd5abb1-f88d-48a4-9264-63cbb62a8ed1%40lbl.gov .

[divyam jaiswal]

Hello Dave,

Thanks for your reply.
I wanted to use your code to create a mount point before calling fuse file-system daemon.
But I am not interested in changing the privilege levels.

What I am trying to do is implement this paper https://dl.acm.org/citation.cfm?id=3217195
Since FUSE kernel module and the user daemon both read and write from /dev/fuse.
I am looking for a way to write a request to it.
Linux version is 4.26.

I found that the error occurs when a mount point is created without correct arguments.
In my case, the command was missing filesystem type.

- Divyam

> To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.

[Alan Sill]

Divyam,

We have successfully used Parrot mounts here for a similar purpose (mounting the CERN VM file system, cernvmfs) in our Singularity images. See below for a message on this from Eric Rees at TTU.

I’d appreciate any feedback from anyone who tries this.

Alan

On Feb 27, 2019, at 2:10 PM, Rees, Eric <eric...@ttu.edu> wrote:

Hi all,

 

Correct, we use a singularity container that essentially installs Parrot from ‘cctools’ (http://ccl.cse.nd.edu/software/parrot/) and then in the %environment just sets up HTTP_PROXY and these two variables:

 

export PARROT_CVMFS_REPO="<default-repositories>"

export PARROT_ALLOW_SWITCHING_CVMFS_REPOSITORIES=yes

 

Then the %runscript intervenes and pushes whatever commands they run through parrot_run or launches parrot_run interactively if the container is run that way:

 

%runscript

        #Run the commands passed by the user outside the container.

        if [ $# -eq 0 ]; then

                parrot_run bash --noprofile --norc

        else

                parrot_run "$@"

        fi

        exit

 

The final exit is there to help users not have to “exit” twice once they finish running their code.

 

This solution let us mount CVMFS via a container without having to rely on FUSE.

 

Hope this helps!

 

Sincerely,

-Eric R.

 

Eric Rees Ph.D.

Assistant Managing Director

 

Texas Tech University – High Performance Computing Center

Drane Hall 158

Lubbock, TX 79409

[Dave Dykstra]

Oh, that's an interesting concept.

Again, though, in order to mount unprivileged with kernel 4.26 you
should be able to just do "unshare -Ufirmp" as an unprivileged user and
then do the mount. Depending on where you got the kernel from you may
need to first enable unprivileged mount namespaces. After that works,
whether or not my pre-mount code will be helpful for using direct-fuse
is not clear to me (since I didn't pay for downloading the paper).

Dave

> > an email to singu...@lbl.gov <javascript:>.