[lxc-users] Using shiftfs with NFS mounted filesystem

1 view
Skip to first unread message

Michael Eager

unread,
Jul 18, 2020, 2:16:16 PM7/18/20
to LXC users mailing-list
I'm running LXD on a Centos 8 host, kernel-4.18.0. I built and
installed the shiftfs module.

I added a device to a container which mounts a local file system in the
container:
$ lxc config device add devel devel disk path=/<container-path>
source=/<host-path> shift=true
(Both container and mounted fs are named "devel".)

This appears to work with a local FS. Files which were owned by "root"
on the host system show up as owned by "root" in the container, not by
"nobody". I can create and delete files normally.

When the host filesystem is NFS mounted, it initially appears correct in
the container, but there are several odd behaviors:

Creating a file:
[eager@devel devel]$ cd tmp
[eager@devel tmp]$ ls
[eager@devel tmp]$ touch x
touch: cannot touch 'x': Permission denied
[eager@devel tmp]$ ls -l
total 512
-rw-rw-r-- 1 eager eager 0 Jul 18 17:56 x

Deleting a file:
[eager@devel tmp]$ rm x
rm: remove write-protected regular empty file 'x'? y
rm: cannot remove 'x': Permission denied
[eager@devel tmp]$ ls -l
total 512
-rw-rw-r-- 1 eager eager 0 Jul 18 17:56 x

When I remove the file 'x' on the host, it's gone in the container, but
trying to create 'x' again fails with a different error:
[eager@devel tmp]$ ls -l
total 0
[eager@devel tmp]$ touch x
touch: cannot touch 'x': Stale file handle
[eager@devel tmp]$ ls -l
total 0

But I can create a different file 'y', albeit with the permission error:
[eager@devel tmp]$ touch y
touch: cannot touch 'y': Permission denied
[eager@devel tmp]$ ls -l
total 1.0K
-rw-rw-r-- 1 eager eager 0 Jul 18 18:07 y


Something seems to be broken. Is there a way to fix this?

--
Michael Eager

Narcis Garcia

unread,
Jul 18, 2020, 3:14:37 PM7/18/20
to lxc-...@lists.linuxcontainers.org
This issue seems more apropriate here:
https://github.com/containers/fuse-overlayfs

Narcis Garcia

__________
I'm using this dedicated address because personal addresses aren't
masked enough at this mail public archive. Public archive administrator
should fix this against automated addresses collectors.
El 18/7/20 a les 20:16, Michael Eager ha escrit:

Michael Eager

unread,
Jul 18, 2020, 7:35:14 PM7/18/20
to LXC users mailing-list, Narcis Garcia
On 7/18/20 12:14 PM, Narcis Garcia wrote:
> This issue seems more apropriate here:
> https://github.com/containers/fuse-overlayfs

Why?

The shiftfs module is a kernel module, not a userspace FUSE implementation.

> _______________________________________________
> lxc-users mailing list
> lxc-...@lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users
>

Serge E. Hallyn

unread,
Jul 18, 2020, 11:37:29 PM7/18/20
to LXC users mailing-list
On Sat, Jul 18, 2020 at 11:16:16AM -0700, Michael Eager wrote:
> I'm running LXD on a Centos 8 host, kernel-4.18.0. I built and installed
> the shiftfs module.

>From what source?

Narcis Garcia

unread,
Jul 19, 2020, 3:11:10 AM7/19/20
to LXC users mailing-list
https://bugzilla.kernel.org/


Narcis Garcia

__________
I'm using this dedicated address because personal addresses aren't
masked enough at this mail public archive. Public archive administrator
should fix this against automated addresses collectors.

El 19/7/20 a les 1:35, Michael Eager ha escrit:

Michael Eager

unread,
Jul 19, 2020, 9:53:03 AM7/19/20
to LXC users mailing-list, Serge E. Hallyn

Serge E. Hallyn

unread,
Jul 19, 2020, 10:13:44 AM7/19/20
to Michael Eager, Amir Goldstein, LXC users mailing-list, Serge E. Hallyn

Interesting. Yeah the shiftfs.c looks identical to what's in
ubuntu's, but when I backported that to 4.14 I had to also take
a bunch of other patches (https://git.kernel.org/pub/scm/linux/kernel/git/sergeh/linux.git/log/?h=v4.14.180-shiftfs)
which may or may not be in 4.18. Still, I don't actually think
that's the problem. NFS has always been difficult to use
with layered filesystems.

Amir, as far as you know, does overlay on top of NFS work
these days?

-serge

Amir Goldstein

unread,
Jul 19, 2020, 12:53:22 PM7/19/20
to Serge E. Hallyn, Michael Eager, LXC users mailing-list

I know people are using it, but only as lower fs, which are files that
are not expected to be changed.
NFS is not supported as upper fs.

Thanks,
Amir.

Serge E. Hallyn

unread,
Jul 19, 2020, 5:04:20 PM7/19/20
to Amir Goldstein, Serge E. Hallyn, Michael Eager, LXC users mailing-list

Thanks Amir. I over-snipped, but Michael is having problems
when NFS is the underlay for shiftfs. If overlay works fine,
then that's an interesting complication :)

I'm hoping to have some time to try it out soon.

thanks,
-serge

Michael Eager

unread,
Jul 20, 2020, 12:21:08 PM7/20/20
to Serge E. Hallyn, Amir Goldstein, LXC users mailing-list

If you would like me to submit a bug report, just tell me where.

Serge E. Hallyn

unread,
Jul 20, 2020, 10:46:51 PM7/20/20
to Michael Eager, Christian Brauner, Serge E. Hallyn, Amir Goldstein, LXC users mailing-list

Since it's shipped with the Ubuntu kernel, you could test it in an
Ubuntu VM and then file it at
https://bugs.launchpad.net/ubuntu/+source/linux/+filebug I don't
have any better place to suggest tracking it.

Michael Eager

unread,
Jul 21, 2020, 11:42:17 AM7/21/20
to Serge E. Hallyn, Christian Brauner, Amir Goldstein, LXC users mailing-list

I'm running LXD on CentOS 8, kernel-4.18.0. I built the shiftfs module
from https://github.com/toby63/shiftfs-dkms.

I can submit an issue on GitHub or bugzilla.kernel.org. Setting up a
Ubunut VM with LXD seems like a bit of work.

Serge E. Hallyn

unread,
Jul 21, 2020, 8:20:23 PM7/21/20
to Michael Eager, Serge E. Hallyn, Christian Brauner, Amir Goldstein, LXC users mailing-list

So just to make sure, you've not reproduced this by hand doing a shiftfs
mount on top of NFS?

Michael Eager

unread,
Jul 21, 2020, 8:39:08 PM7/21/20
to Serge E. Hallyn, Christian Brauner, Amir Goldstein, LXC users mailing-list

No, I didn't try that. I only used the option with LXD.

Reply all
Reply to author
Forward
0 new messages