Bug#992486: docker.io: cgroup v2 defaults break rootless container start & build

[Forest]

Package: docker.io
Version: 20.10.5+dfsg1-1+b5
Severity: important

Dear Maintainer,

After upgrading from Buster to Bullseye, rootless docker containers now fail
to build or start, with the following error message:

Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:340: applying cgroup configuration for
process caused: read unix @->/run/systemd/private: read: connection reset by peer: unknown
Error: failed to start containers: mycontainer

The failure seems related to the switch from cgroup v1 to v2 in Bullseye.
I have found two workarounds:

1. Edit ~/.config/systemd/user/docker.service (which was generated by
dockerd-rootless-setuptool.sh), adding this option to the ExecStart command:
--exec-opt native.cgroupdriver=cgroupfs

2. Boot the system with these kernel options:
systemd.unified_cgroup_hierarchy=false
systemd.legacy_systemd_cgroup_controller=false

Since there appears to be a mismatch between how Bullseye manages cgroups v2
and how docker expects them to be managed, my uninformed guess is that one of
them needs to change. Failing that, perhaps dockerd-rootless-setuptool.sh
should be updated to apply workaround #1 when generating new unit files?

(In case you wonder how rootless docker was working on Buster in the first
place, it's because I have been using the Debian Unstable docker.io package
& dependencies on my Buster system for about a year.)

Thanks for your attention.

-- System Information:
Debian Release: 11.0
APT prefers stable-security
APT policy: (500, 'stable-security'), (500, 'stable')
Architecture: arm64 (aarch64)

Kernel: Linux 5.10.0-8-arm64 (SMP w/6 CPU threads)
Kernel taint flags: TAINT_USER
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages docker.io depends on:
ii adduser 3.118
ii containerd 1.4.5~ds1-2
ii init-system-helpers 1.60
ii iptables 1.8.7-1
ii libc6 2.31-13
ii libdevmapper1.02.1 2:1.02.175-2.1
ii libsystemd0 247.3-6
ii lsb-base 11.1.0
ii runc 1.0.0~rc93+ds1-5+b2
ii tini 0.19.0-1

Versions of packages docker.io recommends:
ii apparmor 2.13.6-10
ii ca-certificates 20210119
pn cgroupfs-mount <none>
ii git 1:2.30.2-1
pn needrestart <none>
ii xz-utils 5.2.5-2

Versions of packages docker.io suggests:
pn aufs-tools <none>
pn btrfs-progs <none>
pn debootstrap <none>
pn docker-doc <none>
ii e2fsprogs 1.46.2-2
pn rinse <none>
ii rootlesskit 0.14.2-1+b3
pn xfsprogs <none>
pn zfs-fuse | zfsutils-linux <none>

-- no debconf information

[Shengjing Zhu]

Hi,

On Thu, Aug 19, 2021 at 3:27 PM Forest <fore...@sonic.net> wrote:
>
> Package: docker.io
> Version: 20.10.5+dfsg1-1+b5
> Severity: important
>
> Dear Maintainer,
>
> After upgrading from Buster to Bullseye, rootless docker containers now fail
> to build or start, with the following error message:
>
> Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:340: applying cgroup configuration for
> process caused: read unix @->/run/systemd/private: read: connection reset by peer: unknown
> Error: failed to start containers: mycontainer
>

It works for me. Here are my docker.service file and cgroup mount
info. Could you compare the output?

====>docker.service<=====
$ cat /home/zsj/.config/systemd/user/docker.service
[Unit]
Description=Docker Application Container Engine (Rootless)
Documentation=https://docs.docker.com/engine/security/rootless/

[Service]
Environment=PATH=/usr/share/docker.io/contrib:/sbin:/usr/sbin:/usr/local/bin:/usr/bin:/bin
ExecStart=/usr/share/docker.io/contrib/dockerd-rootless.sh
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
Type=simple
KillMode=mixed

[Install]
WantedBy=default.target
====>end docker.service<====

====>cgroup<====
~$ mount|grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2
(rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
====end cgroup<====

====>systemctl<====
zsj@debian:~$ systemctl --user status docker.service
● docker.service - Docker Application Container Engine (Rootless)
Loaded: loaded (/home/zsj/.config/systemd/user/docker.service;
disabled; vendor preset: enabled)
Active: active (running) since Thu 2021-08-19 22:47:03 CST; 5min ago
Docs: https://docs.docker.com/engine/security/rootless/
Main PID: 1119185 (rootlesskit)
Tasks: 40
Memory: 69.8M
CPU: 1.346s
CGroup: /user.slice/user-1000.slice/us...@1000.service/app.slice/docker.service
├─1119185 rootlesskit --net=slirp4netns --mtu=65520
--slirp4netns-sandbox=auto --slirp4netns-seccomp=auto
--disable-host-loopback --port-driver=builtin --cop>
├─1119198 /proc/self/exe --net=slirp4netns --mtu=65520
--slirp4netns-sandbox=auto --slirp4netns-seccomp=auto
--disable-host-loopback --port-driver=builtin -->
├─1119217 slirp4netns --mtu 65520 -r 3
--disable-host-loopback --enable-sandbox --enable-seccomp 1119198 tap0
├─1119224 dockerd
└─1119241 containerd --config
/run/user/1000/docker/containerd/containerd.toml --log-level info

Aug 19 22:47:04 debian dockerd-rootless.sh[1119224]:
time="2021-08-19T22:47:04.193598506+08:00" level=error msg="failed to
mount overlay: operation not permitted" storage>
Aug 19 22:47:04 debian dockerd-rootless.sh[1119224]:
time="2021-08-19T22:47:04.204710358+08:00" level=warning msg="Unable
to find cpu controller"
=====>end systemctl<=====

====>docker info<====
zsj@debian:~$ docker -c rootless info
Client:
Context: rootless
Debug Mode: false

Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 20.10.5+dfsg1
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries
splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 1.4.5~ds1-2
runc version: 1.0.0~rc93+ds1-5+b2
init version:
Security Options:
seccomp
Profile: default
rootless
cgroupns
Kernel Version: 5.10.0-7-amd64
Operating System: Debian GNU/Linux 11 (bullseye)
OSType: linux
Architecture: x86_64
====>end docker info<====

> The failure seems related to the switch from cgroup v1 to v2 in Bullseye.
> I have found two workarounds:
>
> 1. Edit ~/.config/systemd/user/docker.service (which was generated by
> dockerd-rootless-setuptool.sh), adding this option to the ExecStart command:
> --exec-opt native.cgroupdriver=cgroupfs
>
> 2. Boot the system with these kernel options:
> systemd.unified_cgroup_hierarchy=false
> systemd.legacy_systemd_cgroup_controller=false
>

> Thanks for your attention.
>
> -- System Information:
> Debian Release: 11.0
> APT prefers stable-security
> APT policy: (500, 'stable-security'), (500, 'stable')
> Architecture: arm64 (aarch64)
>
> Kernel: Linux 5.10.0-8-arm64 (SMP w/6 CPU threads)

I think the difference may be the arch, as I'm testing it on amd64.
Not sure if it's an arm64 specific kernel issue.

--
Shengjing Zhu

[Forest]

On Thu, 19 Aug 2021 22:54:41 +0800, Shengjing Zhu wrote:

>It works for me. Here are my docker.service file and cgroup mount
>info. Could you compare the output?

> ====>docker.service<=====

My ExecStart= line includes --storage-driver=overlay, which is needed to avoid
filesystem failures in my containers. (I remember using overlay2 in the past,
but after running into a bug, I reverted to the older overlay driver.)

My PATH= line is the only other difference. Using yours instead does not fix the
error. (My PATH matches the output of `systemd-path search-binaries-default`.)

Other than that, our docker.service files are the same.


> ====>cgroup<====

Our `mount|grep cgroup` output matches exactly.


> ====>systemctl<====

Our systemctl status top sections have minor differences:
- Your rootlesskit and /proc/self/exe command lines look truncated.
- My dockerd command line includes --storage-driver=overlay, of course.
- CPU time, pids, and paths are different, of course.


Our systemctl status log messages are quite different.
Starting docker without my workaround yields these messages:

level=warning msg="Unable to find cpu controller"

level=warning msg="Unable to find io controller"
level=warning msg="Unable to find cpuset controller"
level=info msg="Loading containers: start."
level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: could not insert 'br_netfilter': Operation not permitted\ninsmod /lib/modules/5.10.0-8-arm64/kernel/net/bridge/br_netfilter.ko \n, error: exit status 1"
level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
level=info msg="Loading containers: done."
level=info msg="Docker daemon" commit=363e9a8 graphdriver(s)=overlay version=20.10.5+dfsg1
level=info msg="Daemon has completed initialization"
level=info msg="API listen on /run/user/[UID]/docker.sock"

Attempting to run a container without my workaround yields these messages:

level=info msg="starting signal loop" namespace=moby path=/run/.ro138154868/user/[UID]/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/[CONTAINERID] pid=[PID]
level=info msg="shim disconnected" id=[CONTAINERID]
level=error msg="stream copy error: reading from a closed fifo"
level=error msg="stream copy error: reading from a closed fifo"
level=error msg="[CONTAINERID] cleanup: failed to delete container from containerd: no such container"
level=error msg="Handler for POST /v1.41/containers/[CONTAINERID]/start returned error: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:340: applying cgroup configuration for process caused: read unix @->/run/systemd/private: read: connection reset by peer: unknown"


> ====>docker info<====

My `docker info` output has several differences from yours:

Context: default
Images: 9
Storage Driver: overlay
Backing Filesystem: extfs
Supports d_type: true
Kernel Version: 5.10.0-8-arm64
Architecture: aarch64

Mine also includes some host details at the end (which I assume you
deliberately skipped) and these warnings:

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: Support for cgroup v2 is experimental
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
WARNING: the overlay storage-driver is deprecated, and will be removed in a future release.

> I think the difference may be the arch, as I'm testing it on amd64.
> Not sure if it's an arm64 specific kernel issue.

Maybe; I don't know much about cgroups. Do they behave differently on
arm64 vs. amd64?

I notice that you're running a kernel package one version behind mine.

[Shengjing Zhu]

On Thu, Aug 19, 2021 at 08:43:21PM -0700, Forest wrote:
> On Thu, 19 Aug 2021 22:54:41 +0800, Shengjing Zhu wrote:

[snip]

> Other than that, our docker.service files are the same.
>
> > ====>cgroup<====
>
> Our `mount|grep cgroup` output matches exactly.
>
>
> > ====>systemctl<====
>
> Our systemctl status top sections have minor differences:
> - Your rootlesskit and /proc/self/exe command lines look truncated.
> - My dockerd command line includes --storage-driver=overlay, of course.
> - CPU time, pids, and paths are different, of course.

As all outputs seem to match, so I try to search the upstream issue list.

This one looks almost same:

https://github.com/opencontainers/runc/issues/3124

Could you try command like:

busctl --user --no-pager status

Also as it's likely upstream issue, could you try Docker Inc's rootless binaries?
You can find them at https://download.docker.com/linux/static/stable/x86_64/

If the upstream's binaries are fine, then we can search potential bugfix and
backport to Debian.

[Forest]

On Fri, 20 Aug 2021 12:09:34 +0800, Shengjing Zhu wrote:

>This one looks almost same:
>
>https://github.com/opencontainers/runc/issues/3124

Hm... The conditions in that issue are different (my /proc is not mounted
with a hidepid option) but yes, the docker error message looks similar.

>Could you try command like:
>
> busctl --user --no-pager status

$ busctl --user --no-pager status
Failed to connect to bus: No such file or directory

When I run it again with strace, I see this system call failing:

connect(3, {sa_family=AF_UNIX, sun_path="/run/user/[UID]/bus"}, 21) = -1
ENOENT (No such file or directory)

The /run/user/[UID]/ directory does exist, both for the current user and
other users, but they do not contain a file named "bus". Perhaps because
this is a headless server with no desktop sessions at all?

>Also as it's likely upstream issue, could you try Docker Inc's rootless binaries?

I'm afraid I cannot. The policy for this machine is only to run binaries
from debian or built locally.

However, if runc assumes the presence of /run/user/$uid/bus, that does seem
likely to be a problem, does it not?

[Shengjing Zhu]

Yes. The default mode, which is using systemd to setup cgroup, requires dbus.
So you need to install dbus-user-session, which will provide /run/user/$uid/bus.

Or you can use cgroupfs mode, which is mentioned in your first mail,
as a workaround.

--
Shengjing Zhu

[Forest]

On Fri, 20 Aug 2021 22:48:18 +0800, Shengjing Zhu wrote:

>Yes. The default mode, which is using systemd to setup cgroup, requires dbus.
>So you need to install dbus-user-session, which will provide /run/user/$uid/bus.
>
>Or you can use cgroupfs mode, which is mentioned in your first mail,
>as a workaround.

In that case, shouldn't Debian's docker.io package document this requirement
for rootless containers? And shouldn't the runc package have Suggests:
dbus-user-session?

[Forest]

To be clear, I'm not advocating for more systemd dependencies. It's just
that as things stand today, someone installing docker.io for rootless
containers will find that they simply fail with a cryptic error message.

Maybe Debian's dockerd-rootless-setuptool.sh script should use --exec-opt
native.cgroupdriver=cgroupfs by default?

Better still, maybe this situation should be detected by the software, and
the cryptic error message replaced with an explanation of what is happening
and that it can be fixed with either native.cgroupdriver=cgroupfs or
dbus-user-session?