Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#1014631: docker.io: No networking in rootless docker with firewalld
121 views
Skip to first unread message
Tomas Janousek
Jul 9, 2022, 4:00:04 AM7/9/22
to
Package: docker.io
Version: 20.10.14+dfsg1-1+b1
Severity: normal
Dear Maintainer,
When running the docker daemon rootless, it still attempts to detect and
use firewalld. If it succeeds (and with the godbus/dbus Debian builds
against, it does; more on that later), iptables rules for NAT (necessary
for traffic to be routed out of the docker0 bridge) are set up in the
host network namespace instead of the network namespace dockerd runs in,
so networking doesn't work. This is what the traffic looks like on the
slirp4netns tap0 in the dockerd namespace:
15:30:41.600319 IP 172.17.0.2.52323 > 10.0.2.3.53: 1825+ A? deb.debian.org. (32)
15:30:41.606028 ARP, Request who-has 172.17.0.2 tell 10.0.2.2, length 28
No reply, obviously, 172.17.0.2 is connected to the bridge, it's meant
to be masqueraded when forwarded to tap0.
Running
nsenter -U --preserve-credentials -n -m -t $(cat $XDG_RUNTIME_DIR/docker.pid) /usr/sbin/iptables-save
gives no output whatsoever, because there are no rules inside the net namespace.
Now the important bit: this issue can only be reproduced with
godbus/dbus 5.0.5+ which it's built against in Debian, but docker/moby
upstream vendors 5.0.3 so the issue isn't reproducible with their
builds. The reason older godbus/dbus "works" is because it fails to
connect to dbus from inside the _user_ namespace. This is because it's
uid 0 in that namespace, it tells dbus it's uid 0 (AUTH EXTERNAL
30\r\n), and from dbus' point of view it's obviously not uid 0, so it
rejects the connection, and dockerd thinks there's no firewalld and
correctly uses iptables as it should inside a network namespace.
But this auth issue is "fixed" in godbus/dbus 5.0.5
(https://github.com/godbus/dbus/pull/265), so if docker is built with
that (it is in Debian), network doesn't work inside any containers.
I've reported the issue upstream nonetheless: https://github.com/moby/moby/issues/43781,
as I believe firewalld detection shouldn't be attempted at all when
running in rootless mode, as it makes no sense to set up iptables rules
in the host network namespace while the bridge is in another network
namespace. It will probably (understandably) be low-priority for them
since it happens to work fine with the version of godbus/dbus they
vendor in the moby repo. So it might be worth coming up with a fix in
Debian, as it's Debian packaging of docker breaking things now.
It's also worth noting that there is a workaround: when I create a shell
wrapper for dockerd which does mount --bind /dev/null /run/dbus/system_bus_socket
before invoking dockerd, networking works again, and nothing else seems
to break (I haven't done much testing though, it may break still).
-- System Information:
Debian Release: bookworm/sid
APT prefers stable-security
APT policy: (990, 'stable-security'), (990, 'testing'), (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'stable-debug'), (500, 'unstable'), (500, 'stable'), (1, 'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 5.18.0-2-amd64 (SMP w/8 CPU threads; PREEMPT)
Kernel taint flags: TAINT_USER, TAINT_WARN
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages docker.io depends on:
ii adduser 3.121
ii containerd 1.6.6~ds1-1
ii init-system-helpers 1.64
ii iptables 1.8.8-1
ii libc6 2.33-7
ii libdevmapper1.02.1 2:1.02.175-2.1
ii libsystemd0 251.2-7
ii lsb-base 11.2
ii runc 1.1.3+ds1-2
ii tini 0.19.0-1
Versions of packages docker.io recommends:
ii apparmor 3.0.4-2
ii ca-certificates 20211016
pn cgroupfs-mount <none>
ii git 1:2.35.1-1
pn needrestart <none>
ii xz-utils 5.2.5-2.1
Versions of packages docker.io suggests:
pn aufs-tools <none>
ii btrfs-progs 5.18.1-1
ii debootstrap 1.0.126+nmu1
pn docker-doc <none>
ii e2fsprogs 1.46.5-2
pn rinse <none>
ii rootlesskit 1.0.1-1
pn xfsprogs <none>
pn zfs-fuse | zfsutils-linux <none>
-- no debconf information
--
Tomáš "liskin" ("Pivník") Janoušek, https://lisk.in/
Version: 20.10.14+dfsg1-1+b1
Severity: normal
Dear Maintainer,
When running the docker daemon rootless, it still attempts to detect and
use firewalld. If it succeeds (and with the godbus/dbus Debian builds
against, it does; more on that later), iptables rules for NAT (necessary
for traffic to be routed out of the docker0 bridge) are set up in the
host network namespace instead of the network namespace dockerd runs in,
so networking doesn't work. This is what the traffic looks like on the
slirp4netns tap0 in the dockerd namespace:
15:30:41.600319 IP 172.17.0.2.52323 > 10.0.2.3.53: 1825+ A? deb.debian.org. (32)
15:30:41.606028 ARP, Request who-has 172.17.0.2 tell 10.0.2.2, length 28
No reply, obviously, 172.17.0.2 is connected to the bridge, it's meant
to be masqueraded when forwarded to tap0.
Running
nsenter -U --preserve-credentials -n -m -t $(cat $XDG_RUNTIME_DIR/docker.pid) /usr/sbin/iptables-save
gives no output whatsoever, because there are no rules inside the net namespace.
Now the important bit: this issue can only be reproduced with
godbus/dbus 5.0.5+ which it's built against in Debian, but docker/moby
upstream vendors 5.0.3 so the issue isn't reproducible with their
builds. The reason older godbus/dbus "works" is because it fails to
connect to dbus from inside the _user_ namespace. This is because it's
uid 0 in that namespace, it tells dbus it's uid 0 (AUTH EXTERNAL
30\r\n), and from dbus' point of view it's obviously not uid 0, so it
rejects the connection, and dockerd thinks there's no firewalld and
correctly uses iptables as it should inside a network namespace.
But this auth issue is "fixed" in godbus/dbus 5.0.5
(https://github.com/godbus/dbus/pull/265), so if docker is built with
that (it is in Debian), network doesn't work inside any containers.
I've reported the issue upstream nonetheless: https://github.com/moby/moby/issues/43781,
as I believe firewalld detection shouldn't be attempted at all when
running in rootless mode, as it makes no sense to set up iptables rules
in the host network namespace while the bridge is in another network
namespace. It will probably (understandably) be low-priority for them
since it happens to work fine with the version of godbus/dbus they
vendor in the moby repo. So it might be worth coming up with a fix in
Debian, as it's Debian packaging of docker breaking things now.
It's also worth noting that there is a workaround: when I create a shell
wrapper for dockerd which does mount --bind /dev/null /run/dbus/system_bus_socket
before invoking dockerd, networking works again, and nothing else seems
to break (I haven't done much testing though, it may break still).
-- System Information:
Debian Release: bookworm/sid
APT prefers stable-security
APT policy: (990, 'stable-security'), (990, 'testing'), (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'stable-debug'), (500, 'unstable'), (500, 'stable'), (1, 'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 5.18.0-2-amd64 (SMP w/8 CPU threads; PREEMPT)
Kernel taint flags: TAINT_USER, TAINT_WARN
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages docker.io depends on:
ii adduser 3.121
ii containerd 1.6.6~ds1-1
ii init-system-helpers 1.64
ii iptables 1.8.8-1
ii libc6 2.33-7
ii libdevmapper1.02.1 2:1.02.175-2.1
ii libsystemd0 251.2-7
ii lsb-base 11.2
ii runc 1.1.3+ds1-2
ii tini 0.19.0-1
Versions of packages docker.io recommends:
ii apparmor 3.0.4-2
ii ca-certificates 20211016
pn cgroupfs-mount <none>
ii git 1:2.35.1-1
pn needrestart <none>
ii xz-utils 5.2.5-2.1
Versions of packages docker.io suggests:
pn aufs-tools <none>
ii btrfs-progs 5.18.1-1
ii debootstrap 1.0.126+nmu1
pn docker-doc <none>
ii e2fsprogs 1.46.5-2
pn rinse <none>
ii rootlesskit 1.0.1-1
pn xfsprogs <none>
pn zfs-fuse | zfsutils-linux <none>
-- no debconf information
--
Tomáš "liskin" ("Pivník") Janoušek, https://lisk.in/
Shengjing Zhu
Jul 9, 2022, 5:10:04 AM7/9/22
to
Control: forwarded -1 https://github.com/moby/moby/issues/43781
Thanks for the detailed report!
I just checked that Docker 22.06 has bumped godbus to 5.0.6,
https://github.com/moby/moby/blob/22.06/vendor.mod
So it would be an issue for upstream too.
We will backport patch if they fix it in the 22.06 branch.
-
Shengjing Zhu
I just checked that Docker 22.06 has bumped godbus to 5.0.6,
https://github.com/moby/moby/blob/22.06/vendor.mod
So it would be an issue for upstream too.
We will backport patch if they fix it in the 22.06 branch.
-
Shengjing Zhu
Tomas Janousek
Jul 10, 2022, 6:30:03 AM7/10/22
to
Hi,
On Sat, Jul 09, 2022 at 05:01:55PM +0800, Shengjing Zhu wrote:
I just checked that Docker 22.06 has bumped godbus to 5.0.6,
https://github.com/moby/moby/blob/22.06/vendor.mod
So it would be an issue for upstream too.
Indeed they have. I must've been looking at an outdated vendor.conf, silly me. I'll try to reproduce the issue with moby master and update the upstream report accordingly. Thanks for pointing this out!
Tomas Janousek
Jul 18, 2022, 4:20:04 AM7/18/22
to
Hi,
On Sat, Jul 09, 2022 at 05:01:55PM +0800, Shengjing Zhu wrote:
I just checked that Docker 22.06 has bumped godbus to 5.0.6,
https://github.com/moby/moby/blob/22.06/vendor.mod
So it would be an issue for upstream too.We will backport patch if they fix it in the 22.06 branch.
Fixed in master: https://github.com/moby/moby/pull/43793
Cherry-picked into 22.06: https://github.com/moby/moby/pull/43813
--
Tomáš "liskin" ("Pivník") Janoušek, class="uri">https://lisk.in/
0 new messages