Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#969223: Can't rm directory on overlayfs in userns
128 views
Skip to first unread message
Shengjing Zhu
Aug 29, 2020, 10:20:03 AM8/29/20
to
Source: linux
Version: 5.7.10-1
Severity: normal
Hi,
After enabling overlayfs for userns, I find it doesn't work as expected.
$ cat /sys/module/overlay/parameters/permit_mounts_in_userns
Y
zsj@debian:~/test$ pwd
/home/zsj/test
zsj@debian:~/test$ tree
.
├── lower
│ └── a
│ └── a
├── merged
├── upper
└── work
zsj@debian:~/test$ unshare -m -U -r
root@debian:~/test# mount -t overlay -o rw,lowerdir=/home/zsj/test/lower,upperdir=/home/zsj/test/upper,workdir=/home/zsj/test/work overlay /home/zsj/test/merged
root@debian:~/test# rm -rf merged/a
rm: cannot remove 'merged/a': Input/output error
-- System Information:
Debian Release: bullseye/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 5.7.0-2-amd64 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_USER, TAINT_FIRMWARE_WORKAROUND
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
Version: 5.7.10-1
Severity: normal
Hi,
After enabling overlayfs for userns, I find it doesn't work as expected.
$ cat /sys/module/overlay/parameters/permit_mounts_in_userns
Y
zsj@debian:~/test$ pwd
/home/zsj/test
zsj@debian:~/test$ tree
.
├── lower
│ └── a
│ └── a
├── merged
├── upper
└── work
zsj@debian:~/test$ unshare -m -U -r
root@debian:~/test# mount -t overlay -o rw,lowerdir=/home/zsj/test/lower,upperdir=/home/zsj/test/upper,workdir=/home/zsj/test/work overlay /home/zsj/test/merged
root@debian:~/test# rm -rf merged/a
rm: cannot remove 'merged/a': Input/output error
-- System Information:
Debian Release: bullseye/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 5.7.0-2-amd64 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_USER, TAINT_FIRMWARE_WORKAROUND
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
Shengjing Zhu
Sep 2, 2020, 12:00:03 AM9/2/20
to
However if I boot a new debian testing VM, it seems not to work.
Both VMs are downloaded from http://cdimage.debian.org/cdimage/cloud/
What can be the difference here? I'm lost on debugging this..
--
Shengjing Zhu
Nicolas Schier
Sep 16, 2020, 4:10:03 AM9/16/20
to
On Wed, Sep 02, 2020 at 11:52:41AM +0800, Shengjing Zhu wrote:
> On Sat, Aug 29, 2020 at 10:13 PM Shengjing Zhu <zh...@debian.org> wrote:
> >
> > Source: linux
> > Version: 5.7.10-1
> > Severity: normal
> >
> > Hi,
> >
> > After enabling overlayfs for userns, I find it doesn't work as expected.
> >
> > $ cat /sys/module/overlay/parameters/permit_mounts_in_userns
> > Y
> >
> > zsj@debian:~/test$ pwd
> > /home/zsj/test
> > zsj@debian:~/test$ tree
> > .
> > ├── lower
> > │ └── a
> > │ └── a
> > ├── merged
> > ├── upper
> > └── work
> >
> > zsj@debian:~/test$ unshare -m -U -r
> > root@debian:~/test# mount -t overlay -o rw,lowerdir=/home/zsj/test/lower,upperdir=/home/zsj/test/upper,workdir=/home/zsj/test/work overlay /home/zsj/test/merged
> > root@debian:~/test# rm -rf merged/a
> > rm: cannot remove 'merged/a': Input/output error
> >
Hi,
> On Sat, Aug 29, 2020 at 10:13 PM Shengjing Zhu <zh...@debian.org> wrote:
> >
> > Source: linux
> > Version: 5.7.10-1
> > Severity: normal
> >
> > Hi,
> >
> > After enabling overlayfs for userns, I find it doesn't work as expected.
> >
> > $ cat /sys/module/overlay/parameters/permit_mounts_in_userns
> > Y
> >
> > zsj@debian:~/test$ pwd
> > /home/zsj/test
> > zsj@debian:~/test$ tree
> > .
> > ├── lower
> > │ └── a
> > │ └── a
> > ├── merged
> > ├── upper
> > └── work
> >
> > zsj@debian:~/test$ unshare -m -U -r
> > root@debian:~/test# mount -t overlay -o rw,lowerdir=/home/zsj/test/lower,upperdir=/home/zsj/test/upper,workdir=/home/zsj/test/work overlay /home/zsj/test/merged
> > root@debian:~/test# rm -rf merged/a
> > rm: cannot remove 'merged/a': Input/output error
> >
overlayfs uses filesystem xattrs to mark "whiteouts" and redirects of
directories, which are only accessable for root (CAP_SYS_ADMIN), thus,
not when overlay is mounted in a user namespace, cp. e.g. [1,2].
Ubuntu kernel "solves" this by skipping the "trusted."-xattr check, thus
allowing setting and removal of 'trusted.overlay.*' xattrs from within
user namespaces; but those are still visible in all other namespaces. A
following overlayfs mount done by the real root user will use these
modified xattrs.
To me it would seem to be more adequate if overlayfs would use
'overlay.*' instead of 'trusted.overlay.*', if it is mounted in an
unpriviledged user namespace. But this would make overlay mounts done
by root incompatible with those done in a user namespace.
Maybe you find #836211 to be related to this.
[1]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/fs/xattr.c?h=linux-5.7.y#n113
[2]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/fs/xattr.c?h=linux-5.7.y#n1049
[3]: https://kernel.ubuntu.com/git/ubuntu/ubuntu-focal.git/commit/?id=111cd1a9840ce187e28b49fe4e77b9b5e84386b1
> If I upgrade a debian10 VM to testing, it seems to work.
> However if I boot a new debian testing VM, it seems not to work.
> Both VMs are downloaded from http://cdimage.debian.org/cdimage/cloud/
> What can be the difference here? I'm lost on debugging this..
both VMs when mounting overlayfs in userns?
Kind regards,
Nicolas
--
epost: nic...@fjasle.eu irc://oftc.net/nsc
↳ gpg: 18ed 52db e34f 860e e9fb c82b 7d97 0932 55a0 ce7f
-- frykten for herren er opphav til kunnskap --
Shengjing Zhu
Sep 16, 2020, 1:00:03 PM9/16/20
to
On Wed, Sep 16, 2020 at 3:58 PM Nicolas Schier <nic...@fjasle.eu> wrote:
> > If I upgrade a debian10 VM to testing, it seems to work.
> > However if I boot a new debian testing VM, it seems not to work.
> > Both VMs are downloaded from http://cdimage.debian.org/cdimage/cloud/
> > What can be the difference here? I'm lost on debugging this..
>
> This confuses me. Are you sure, you used the same kernel version on
> both VMs when mounting overlayfs in userns?
>
I think I just mess up when debugging. It seems it never works.
> > If I upgrade a debian10 VM to testing, it seems to work.
> > However if I boot a new debian testing VM, it seems not to work.
> > Both VMs are downloaded from http://cdimage.debian.org/cdimage/cloud/
> > What can be the difference here? I'm lost on debugging this..
>
> This confuses me. Are you sure, you used the same kernel version on
> both VMs when mounting overlayfs in userns?
>
Maybe we should revert permit_mounts_in_userns? as it doesn't seem to
work. Buster is also affected.
--
Shengjing Zhu
Nicolas Schier
Sep 16, 2020, 3:00:04 PM9/16/20
to
> I think I just mess up when debugging. It seems it never works.
>
> Maybe we should revert permit_mounts_in_userns? as it doesn't seem to
> work. Buster is also affected.
Please, don't be too fast when thinking about a revert. Several of my
>
> Maybe we should revert permit_mounts_in_userns? as it doesn't seem to
> work. Buster is also affected.
colleagues (Debian users) cling to the feature since they need it for
using the company's LXC containers; if permit_mounts_in_userns is
removed again, they might be forced to switch to non-Debian kernels or
to live-patch the kernel with fragile stuff like [1], cp. #913880.
[1]: https://rocketgit.com/user/nicolas/overlay-userns-dkms
Shengjing Zhu
Sep 16, 2020, 3:10:03 PM9/16/20
to
On Thu, Sep 17, 2020 at 2:52 AM Nicolas Schier <nic...@fjasle.eu> wrote:
>
> > I think I just mess up when debugging. It seems it never works.
> >
> > Maybe we should revert permit_mounts_in_userns? as it doesn't seem to
> > work. Buster is also affected.
>
> Please, don't be too fast when thinking about a revert. Several of my
> colleagues (Debian users) cling to the feature since they need it for
> using the company's LXC containers; if permit_mounts_in_userns is
> removed again, they might be forced to switch to non-Debian kernels or
> to live-patch the kernel with fragile stuff like [1], cp. #913880.
I mean if you can't even remove a directory with files, it's too broken to use.
>
> > I think I just mess up when debugging. It seems it never works.
> >
> > Maybe we should revert permit_mounts_in_userns? as it doesn't seem to
> > work. Buster is also affected.
>
> Please, don't be too fast when thinking about a revert. Several of my
> colleagues (Debian users) cling to the feature since they need it for
> using the company's LXC containers; if permit_mounts_in_userns is
> removed again, they might be forced to switch to non-Debian kernels or
> to live-patch the kernel with fragile stuff like [1], cp. #913880.
So your colleagues find the userns overlay works?
Or you mean we should take Ubuntu's patch to fix the issue?
--
Shengjing Zhu
Nicolas Schier
Dec 11, 2020, 3:30:04 AM12/11/20
to
with overlay fs for building purposes only, thus, read-only access is
sufficient and works. (But yes, the incomplete write-support leads to
annoyance.)
Currently, there is a patch on linux-unionfs that allows using user
xattrs for overlay fs meta data [1]. If the related patchset [2] is
going to be merged, the Debian patch will become obsolete; otherwise we
could think about picking up the patch from [1].
As far as I have seen, the Ubuntu patch allows unpriviledged users to
modify 'trusted.overlay.*' xattrs, which probably has security
implications. ("Probably" as just had a superficial look at it.)
I would prefer to keep a watch on [2] and dicuss further, if it has
been merged or rejected.
Kind regards,
Nicolas
[1]: [PATCH v2 06/10] ovl: user xattr
https://lore.kernel.org/linux-unionfs/20201207163255.5...@redhat.com/
[2]: https://lore.kernel.org/linux-unionfs/CAJfpegsiuf8ib5cvVrr=zHZ+Xu7BMMTT2eY...@mail.gmail.com/T/#t
Shengjing Zhu
Mar 3, 2021, 4:40:02 AM3/3/21
to
On Wed, Mar 3, 2021 at 3:40 PM Nicolas Schier <nic...@fjasle.eu> wrote:
> > [2]: https://lore.kernel.org/linux-unionfs/CAJfpegsiuf8ib5cvVrr=zHZ+Xu7BMMTT2eY...@mail.gmail.com/T/#t
>
> The overlay fs patchset [2] has been merged and with v5.10.13 (tested
> on linux-image-5.10.0-3-arm64) the issue is no more reproducible for
> me. Might you want to re-check on your site?
>
If I understand correctly, the upstream patch is merged into the v5.11 tree.
> > [2]: https://lore.kernel.org/linux-unionfs/CAJfpegsiuf8ib5cvVrr=zHZ+Xu7BMMTT2eY...@mail.gmail.com/T/#t
>
> The overlay fs patchset [2] has been merged and with v5.10.13 (tested
> on linux-image-5.10.0-3-arm64) the issue is no more reproducible for
> me. Might you want to re-check on your site?
>
And I still can reproduce the error on the Debian v5.10 kernel.
And another thing is that the upstream patch introduces a new mount
option, userxattr, instead of module parameter.
--
Shengjing Zhu
Nicolas Schier
Mar 3, 2021, 5:40:03 AM3/3/21
to
On Wed 03 Mar 2021 17:33:16 GMT Shengjing Zhu write:
>
> On Wed, Mar 3, 2021 at 3:40 PM Nicolas Schier <nic...@fjasle.eu> wrote:
> > > [2]: https://lore.kernel.org/linux-unionfs/CAJfpegsiuf8ib5cvVrr=zHZ+Xu7BMMTT2eY...@mail.gmail.com/T/#t
> >
> > The overlay fs patchset [2] has been merged and with v5.10.13 (tested
> > on linux-image-5.10.0-3-arm64) the issue is no more reproducible for
> > me. Might you want to re-check on your site?
> >
>
> If I understand correctly, the upstream patch is merged into the v5.11 tree.
Sorry. Yes, you're right.
>
> On Wed, Mar 3, 2021 at 3:40 PM Nicolas Schier <nic...@fjasle.eu> wrote:
> > > [2]: https://lore.kernel.org/linux-unionfs/CAJfpegsiuf8ib5cvVrr=zHZ+Xu7BMMTT2eY...@mail.gmail.com/T/#t
> >
> > The overlay fs patchset [2] has been merged and with v5.10.13 (tested
> > on linux-image-5.10.0-3-arm64) the issue is no more reproducible for
> > me. Might you want to re-check on your site?
> >
>
> If I understand correctly, the upstream patch is merged into the v5.11 tree.
> And I still can reproduce the error on the Debian v5.10 kernel.
(still the 5.10.0-3-arm64 kernel):
nsc@lillesand:/tmp$ cat /sys/module/overlay/parameters/permit_mounts_in_userns
Y
nsc@lillesand:/tmp$ mkdir -p test/lower/a test/merged test/upper test/work
nsc@lillesand:/tmp$ uname -a | tee test/lower/a/a
Linux lillesand 5.10.0-3-arm64 #1 SMP Debian 5.10.13-1 (2021-02-06) aarch64 GNU/Linux
nsc@lillesand:/tmp$ unshare -m -U -r
root@lillesand:/tmp# mount -t overlay -o rw,lowerdir=/tmp/test/lower,upperdir=/tmp/test/upper,workdir=/tmp/test/work overlay /tmp/test/merged
root@lillesand:/tmp# rm -rf test/merged/a
root@lillesand:/tmp# find test -ls
1597776 4 drwxr-xr-x 6 root root 4096 mars 3 08:24 test
1973978 4 drwxr-xr-x 2 root root 4096 mars 3 08:27 test/upper
2099881 0 c--------- 1 root root 0, 0 mars 3 08:27 test/upper/a
1973978 4 drwxr-xr-x 1 root root 4096 mars 3 08:27 test/merged
1714388 4 drwxr-xr-x 3 root root 4096 mars 3 08:24 test/lower
1714389 4 drwxr-xr-x 2 root root 4096 mars 3 08:27 test/lower/a
1714393 4 -rw-r--r-- 1 root root 86 mars 3 10:48 test/lower/a/a
1973979 4 drwxr-xr-x 3 root root 4096 mars 3 10:48 test/work
2099880 4 d--------- 2 root root 4096 mars 3 10:48 test/work/work
root@lillesand:/tmp#
Do you see any kernel log message from overlay fs? Might it depend on
the underlying filesystem? Can you create a white-out char dev node
manually?
> And another thing is that the upstream patch introduces a new mount
> option, userxattr, instead of module parameter.
v5.11 as overlay fs mounts will then always be enabled in userspace
namespace.
Shengjing Zhu
Mar 3, 2021, 10:00:03 AM3/3/21
to
Y
zsj@debian:~/t$ mkdir -p test/lower/a test/merged test/upper test/work
zsj@debian:~/t$ uname -a | tee test/lower/a/a
Linux debian 5.10.0-3-amd64 #1 SMP Debian 5.10.13-1 (2021-02-06) x86_64 GNU/Linux
zsj@debian:~/t$ unshare -m -U -r
root@debian:~/t# mount -t overlay -o rw,lowerdir=./test/lower,upperdir=./test/upper,workdir=./test/work overlay ./test/merged/
root@debian:~/t# rm -rf ./test/merged/a
rm: cannot remove './test/merged/a': Input/output error
root@debian:~/t# find test -ls
7350352 4 drwxr-xr-x 6 root root 4096 Mar 3 22:44 test
7351341 4 drwxr-xr-x 3 root root 4096 Mar 3 22:44 test/lower
7353492 4 drwxr-xr-x 2 root root 4096 Mar 3 22:44 test/lower/a
7356441 4 -rw-r--r-- 1 root root 82 Mar 3 22:44 test/lower/a/a
7356069 4 drwxr-xr-x 3 root root 4096 Mar 3 22:45 test/work
7358324 4 d--------- 2 root root 4096 Mar 3 22:45 test/work/work
7358564 0 c--------- 2 root root 0, 0 Mar 3 22:45 test/work/work/#4
7354400 4 drwxr-xr-x 3 root root 4096 Mar 3 22:44 test/upper
7358563 4 drwxr-xr-x 2 root root 4096 Mar 3 22:45 test/upper/a
7358564 0 c--------- 2 root root 0, 0 Mar 3 22:45 test/upper/a/a
7354400 4 drwxr-xr-x 1 root root 4096 Mar 3 22:44 test/merged
7353492 4 drwxr-xr-x 1 root root 4096 Mar 3 22:45 test/merged/a
> Do you see any kernel log message from overlay fs? Might it depend on
> the underlying filesystem? Can you create a white-out char dev node
> manually?
>
[1215353.859841] overlayfs: overlayfs: Allowing overlay mounts in user namespaces bears security risks
[1215425.416543] overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
The underlying fs is ext4.
zsj@debian:~/t$ mount|grep nvme
/dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)
/dev/nvme0n1p4 on /home type ext4 (rw,relatime)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
I don't know how to test "create a white-out char dev node manually".
Thanks
Nicolas Schier
Mar 11, 2021, 1:20:03 AM3/11/21
to
system, but not on arm64. It's really strange, also enabling dynamic
debug output does not reveal anything useful. I guess it would be good
to dig into the related kernel code to get some deeper understanding, I
hope to get start examining a bit in the upcoming weekend.
Kind regards,
Nicolas
--
epost|xmpp: nic...@fjasle.eu irc://oftc.net/nsc
0 new messages