Bug#1002997: podman: Please provide a default /etc/containers/storage.conf

[Philip]

Package: podman
Version: 3.0.1+dfsg1-3+b2
Severity: wishlist

Dear Maintainer,

I had some problems running the dockerized version of the Unifi controller jacobalberty/unifi-docker
with podman on Debian.
On a Fedora system, starting the container only takes a few seconds.
On a Debian system, it can take about 5 minutes.

The reason is that on the Fedora system the mount-option metacopy=on
(see [1] for this mount option) is set for the container overlayfs via a default /etc/containers/storage.conf.
That makes quite the difference for this specific image because it does a
`chown unifi:unifi /usr/lib/unifi` during startup.
chown-ing these 6k files is fast with metacopy=on (on Fedora).
Without the option (on Debian), I think the files will be copied instead of only their metadata, making it rather slow.

So the solution for me was to copy /etc/containers/storage.conf from a
Fedora system. If anyone has a similar problem, the file can be extracted from the
src rpm of the containers-common package which can be downloaded at [2].

IMO it would be useful if Debian would also include a default
/etc/containers/storage.conf.
Thanks for considering this!
However I'm not sure if metacopy=on is a good idea from a security
perspective.

Best
Philip

-- System Information:
Debian Release: 11.2
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-10-amd64 (SMP w/2 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages podman depends on:
ii conmon 2.0.25+ds1-1.1
ii containernetworking-plugins 0.9.0-1+b6
ii crun 0.17+dfsg-1
ii golang-github-containers-common 0.33.4+ds1-1
ii init-system-helpers 1.60
ii iptables 1.8.7-1
ii libc6 2.31-13+deb11u2
ii libdevmapper1.02.1 2:1.02.175-2.1
ii libgpgme11 1.14.0-1+b2
ii libseccomp2 2.5.1-1+deb11u1

Versions of packages podman recommends:
ii buildah 1.19.6+dfsg1-1+b6
ii catatonit 0.1.5-2
ii fuse-overlayfs 1.4.0-1
ii golang-github-containernetworking-plugin-dnsname 1.1.1+ds1-4+b7
ii slirp4netns 1.0.1-2
ii uidmap 1:4.8.1-1

Versions of packages podman suggests:
pn containers-storage <none>
pn docker-compose <none>

-- no debconf information


[1]: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#metadata-only-copy-up
[2]: https://kojipkgs.fedoraproject.org//packages/containers-common/1/32.fc35/src/containers-common-1-32.fc35.src.rpm

[Reinhard Tartler]

Control: reassign -1 storage-common

Control: affects -1 podman

Hi Philip,

Thank you for your bug report.

The Debian equivalent to Fedora's package 'containers-common' has the same name in debian, and does ship a 'storage.conf' file in /usr/share/containers/storage.conf. This is so that the local administrator can copy it to /etc/containers/storage.conf and do local modifications. The Debian package copies the storage.conf from the upstream source verbatim. As you can see at https://github.com/containers/storage/blob/375f77c66685b14fc580daad2dc6df607fb86dee/storage.conf#L95, the mount option 'metacopy=on' is missing even upstream.

I am not sure why the Fedora package decided to patch the configuration file -- I couldn't find a comment in the .src.rpm that you linked. Also, looking at the kernel documentation you provided, it seems your concerns re: security are justified, and the option seems to have significant security implications:

Do not use metacopy=on with untrusted upper/lower directories. Otherwise it is possible that an attacker can create a handcrafted file with appropriate REDIRECT and METACOPY xattrs, and gain access to file on lower pointed by REDIRECT. This should not be possible on local system as setting “trusted.” xattrs will require CAP_SYS_ADMIN. But it should be possible for untrusted layers like from a pen drive.

I'm not sure whether enabling it by default is a good idea. I need to think more about this.

I'd also appreciate hearing additional opinions on this, and have copied some friends from podman upstream. Do you happen to know what's the background / thinking in Fedora with enabling the option metacopy=on?

Happy New Year!

-rt

--

regards,
    Reinhard

[Valentin Rothberg]

Hi folks,

Thanks for pulling me in.

On Sun, Jan 9, 2022 at 11:15 PM Reinhard Tartler <sire...@gmail.com> wrote:

Control: reassign -1 storage-common

Control: affects -1 podman

Hi Philip,

Thank you for your bug report. I'll defer to our overlay expert, Giuseppe.

The Debian equivalent to Fedora's package 'containers-common' has the same name in debian, and does ship a 'storage.conf' file in /usr/share/containers/storage.conf. This is so that the local administrator can copy it to /etc/containers/storage.conf and do local modifications. The Debian package copies the storage.conf from the upstream source verbatim. As you can see at https://github.com/containers/storage/blob/375f77c66685b14fc580daad2dc6df607fb86dee/storage.conf#L95, the mount option 'metacopy=on' is missing even upstream.

I am not sure why the Fedora package decided to patch the configuration file -- I couldn't find a comment in the .src.rpm that you linked. Also, looking at the kernel documentation you provided, it seems your concerns re: security are justified, and the option seems to have significant security implications:

Do not use metacopy=on with untrusted upper/lower directories. Otherwise it is possible that an attacker can create a handcrafted file with appropriate REDIRECT and METACOPY xattrs, and gain access to file on lower pointed by REDIRECT. This should not be possible on local system as setting “trusted.” xattrs will require CAP_SYS_ADMIN. But it should be possible for untrusted layers like from a pen drive.

I'm not sure whether enabling it by default is a good idea. I need to think more about this.

@Giuseppe Scrivano what do you think?

[Giuseppe Scrivano]

Valentin Rothberg <roth...@redhat.com> writes:

> Hi folks,
>
> Thanks for pulling me in.
>
> On Sun, Jan 9, 2022 at 11:15 PM Reinhard Tartler <sire...@gmail.com> wrote:
>
> Control: reassign -1 storage-common
> Control: affects -1 podman
>
> Hi Philip,
>
> Thank you for your bug report. I'll defer to our overlay expert, Giuseppe.
>
> The Debian equivalent to Fedora's package 'containers-common' has the same name in debian, and does ship a 'storage.conf' file in /usr/share/containers/storage.conf. This is so that the local
> administrator can copy it to /etc/containers/storage.conf and do local modifications. The Debian package copies the storage.conf from the upstream source verbatim. As you can see at
> https://github.com/containers/storage/blob/375f77c66685b14fc580daad2dc6df607fb86dee/storage.conf#L95, the mount option 'metacopy=on' is missing even upstream.
>
> I am not sure why the Fedora package decided to patch the configuration file -- I couldn't find a comment in the .src.rpm that you linked. Also, looking at the kernel documentation you provided, it
> seems your concerns re: security are justified, and the option seems to have significant security implications:
>
> Do not use metacopy=on with untrusted upper/lower directories. Otherwise it is possible that an attacker can create a handcrafted file with appropriate REDIRECT and METACOPY xattrs, and
> gain access to file on lower pointed by REDIRECT. This should not be possible on local system as setting “trusted.” xattrs will require CAP_SYS_ADMIN. But it should be possible for untrusted
> layers like from a pen drive.
>
> I'm not sure whether enabling it by default is a good idea. I need to think more about this.
>
> @Giuseppe Scrivano what do you think?

please keep in mind that unprivileged overlay mounts cannot use
metacopy. You still need root access on the host (CAP_SYS_ADMIN in
the initial user namespace) in order to use metacopy=on.

While it is safe to pull random images from the network and expect they
cannot exploit the system to gain access to files outside the image
itself, there is no guarantee when you are using a handcrafted storage
repository as you seem to imply with the pen drive example.
There are so many things that can be abused that metacopy=on is the last
I'd worry about :-) For such cases, I suggest to use rootless, and rely
on the kernel to limit what the unpriviled user can do.

Regards,
Giuseppe

[Reinhard Tartler]

On Mon, Jan 10, 2022 at 4:15 AM Giuseppe Scrivano <gscr...@redhat.com> wrote:

> @Giuseppe Scrivano what do you think?

please keep in mind that unprivileged overlay mounts cannot use
metacopy.  You still need root access on the host (CAP_SYS_ADMIN in
the initial user namespace) in order to use metacopy=on.

While it is safe to pull random images from the network and expect they
cannot exploit the system to gain access to files outside the image
itself, there is no guarantee when you are using a handcrafted storage
repository as you seem to imply with the pen drive example.
There are so many things that can be abused that metacopy=on is the last
I'd worry about :-)  For such cases, I suggest to use rootless, and rely
on the kernel to limit what the unpriviled user can do.

The quote comes directly from the kernel documentation.

So with that rationale, maybe the option 'metacopy=on' should be set upstream at https://github.com/containers/storage/blob/375f77c66685b14fc580daad2dc6df607fb86dee/storage.conf#L95 ?

That way, Debian would pick up the change on the next upstream update.

Best,
-rt

[Cian Donovan]

Hi,

This completed Pull Request upstream should solve the issue of needing
to manually add a storage.conf by adjusting the defaults to use the in-
kernel overlayfs by default instead of vfs which is slow and very heavy
on disk-space.

https://github.com/containers/storage/issues/1570
https://github.com/containers/storage/pull/1571

Is there any chance this fix could be backported/cherrypicked to Debian
Stable Bookworm? It's functionally equivalent to just setting the
storage driver in a storage.conf configuration file, but is just the
default instead.

Recently moved to Bookworm and the first thing I noticed was my disk
was completely full after only running a few containers since vfs was
the default.

Kind regards,
Cian.

[Cian Donovan]