Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Bug#1033917: lxc: apparmor profile no longer allows unprivileged guest systemd-logind to start (since bookworm)

224 views
Skip to first unread message

Forest

unread,
Apr 3, 2023, 5:30:04 PM4/3/23
to
Package: lxc
Version: 1:5.0.2-1
Severity: normal
X-Debbugs-Cc: fore...@sonic.net

Dear Maintainer,

After upgrading an unprivileged container from bullseye to bookworm, LXC's
AppArmor profiles are no longer sufficient for the guest's systemd-logind.

This manifests as a 25 second hang when running certain commands (notably
sudo -i and su -) in the container. It also produces a lot of errors in the
host & guest logs.

Before the upgrade to bookworm, the hangs did not occur, and systemd-logind
started without trouble.


-- Host journal:

Apr 02 18:30:01 debtesting CRON[6361]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Apr 02 18:30:01 debtesting CRON[6362]: (root) CMD ([ -x /etc/init.d/anacron ] && if [ ! -d /run/systemd/system ]; then /usr/sbin/invoke-rc.d anacron start >/dev/null; fi)
Apr 02 18:30:01 debtesting CRON[6361]: pam_unix(cron:session): session closed for user root
Apr 02 18:30:16 debtesting audit[6365]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=6365 comm="(d-logind)" flags="rw, rslave"
Apr 02 18:30:16 debtesting kernel: kauditd_printk_skb: 13 callbacks suppressed
Apr 02 18:30:16 debtesting kernel: audit: type=1400 audit(1680485416.414:324): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=6365 comm="(d-logind)" flags="rw, rslave"
Apr 02 18:30:16 debtesting audit[6369]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=6369 comm="(d-logind)" flags="rw, rslave"
Apr 02 18:30:16 debtesting kernel: audit: type=1400 audit(1680485416.426:325): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=6369 comm="(d-logind)" flags="rw, rslave"
Apr 02 18:30:16 debtesting audit[6373]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=6373 comm="(d-logind)" flags="rw, rslave"
Apr 02 18:30:16 debtesting kernel: audit: type=1400 audit(1680485416.450:326): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=6373 comm="(d-logind)" flags="rw, rslave"
Apr 02 18:30:16 debtesting audit[6377]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=6377 comm="(d-logind)" flags="rw, rslave"
Apr 02 18:30:16 debtesting kernel: audit: type=1400 audit(1680485416.522:327): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=6377 comm="(d-logind)" flags="rw, rslave"
Apr 02 18:30:16 debtesting audit[6381]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=6381 comm="(d-logind)" flags="rw, rslave"
Apr 02 18:30:16 debtesting kernel: audit: type=1400 audit(1680485416.534:328): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=6381 comm="(d-logind)" flags="rw, rslave"


-- Guest journal:

Apr 02 18:30:16 lxbox sudo[136]: root : TTY=pts/7 ; PWD=/root ; USER=root ; COMMAND=/bin/bash
Apr 02 18:30:16 lxbox sudo[136]: pam_limits(sudo-i:session): Could not set limit for 'core' to soft=0, hard=-1: Operation not permitted; uid=0,euid=0
Apr 02 18:30:16 lxbox sudo[136]: pam_unix(sudo-i:session): session opened for user root(uid=0) by (uid=0)
Apr 02 18:30:16 lxbox dbus-daemon[97]: [system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service' requested by ':1.2' (uid=0 pid=136 comm="sudo -i")
Apr 02 18:30:16 lxbox systemd[1]: Starting modp...@drm.service - Load Kernel Module drm...
Apr 02 18:30:16 lxbox (modprobe)[137]: modp...@drm.service: Executable /sbin/modprobe missing, skipping: No such file or directory
Apr 02 18:30:16 lxbox systemd[1]: modp...@drm.service: Deactivated successfully.
Apr 02 18:30:16 lxbox systemd[1]: Finished modp...@drm.service - Load Kernel Module drm.
Apr 02 18:30:16 lxbox systemd[1]: Starting systemd-logind.service - User Login Management...
Apr 02 18:30:16 lxbox (d-logind)[138]: systemd-logind.service: Failed to set up mount namespacing: Permission denied
Apr 02 18:30:16 lxbox (d-logind)[138]: systemd-logind.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-logind: Permission denied
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Main process exited, code=exited, status=226/NAMESPACE
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Apr 02 18:30:16 lxbox systemd[1]: Failed to start systemd-logind.service - User Login Management.
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 1.
Apr 02 18:30:16 lxbox systemd[1]: Stopped systemd-logind.service - User Login Management.
Apr 02 18:30:16 lxbox systemd[1]: Starting modp...@drm.service - Load Kernel Module drm...
Apr 02 18:30:16 lxbox (modprobe)[141]: modp...@drm.service: Executable /sbin/modprobe missing, skipping: No such file or directory
Apr 02 18:30:16 lxbox systemd[1]: modp...@drm.service: Deactivated successfully.
Apr 02 18:30:16 lxbox systemd[1]: Finished modp...@drm.service - Load Kernel Module drm.
Apr 02 18:30:16 lxbox systemd[1]: Starting systemd-logind.service - User Login Management...
Apr 02 18:30:16 lxbox (d-logind)[142]: systemd-logind.service: Failed to set up mount namespacing: Permission denied
Apr 02 18:30:16 lxbox (d-logind)[142]: systemd-logind.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-logind: Permission denied
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Main process exited, code=exited, status=226/NAMESPACE
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Apr 02 18:30:16 lxbox systemd[1]: Failed to start systemd-logind.service - User Login Management.
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 2.
Apr 02 18:30:16 lxbox systemd[1]: Stopped systemd-logind.service - User Login Management.
Apr 02 18:30:16 lxbox systemd[1]: Starting modp...@drm.service - Load Kernel Module drm...
Apr 02 18:30:16 lxbox (modprobe)[145]: modp...@drm.service: Executable /sbin/modprobe missing, skipping: No such file or directory
Apr 02 18:30:16 lxbox systemd[1]: modp...@drm.service: Deactivated successfully.
Apr 02 18:30:16 lxbox systemd[1]: Finished modp...@drm.service - Load Kernel Module drm.
Apr 02 18:30:16 lxbox systemd[1]: Starting systemd-logind.service - User Login Management...
Apr 02 18:30:16 lxbox (d-logind)[146]: systemd-logind.service: Failed to set up mount namespacing: Permission denied
Apr 02 18:30:16 lxbox (d-logind)[146]: systemd-logind.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-logind: Permission denied
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Main process exited, code=exited, status=226/NAMESPACE
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Apr 02 18:30:16 lxbox systemd[1]: Failed to start systemd-logind.service - User Login Management.
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 3.
Apr 02 18:30:16 lxbox systemd[1]: Stopped systemd-logind.service - User Login Management.
Apr 02 18:30:16 lxbox (modprobe)[149]: modp...@drm.service: Executable /sbin/modprobe missing, skipping: No such file or directory
Apr 02 18:30:16 lxbox systemd[1]: Starting modp...@drm.service - Load Kernel Module drm...
Apr 02 18:30:16 lxbox systemd[1]: modp...@drm.service: Deactivated successfully.
Apr 02 18:30:16 lxbox systemd[1]: Finished modp...@drm.service - Load Kernel Module drm.
Apr 02 18:30:16 lxbox systemd[1]: Starting systemd-logind.service - User Login Management...
Apr 02 18:30:16 lxbox (d-logind)[150]: systemd-logind.service: Failed to set up mount namespacing: Permission denied
Apr 02 18:30:16 lxbox (d-logind)[150]: systemd-logind.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-logind: Permission denied
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Main process exited, code=exited, status=226/NAMESPACE
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Apr 02 18:30:16 lxbox systemd[1]: Failed to start systemd-logind.service - User Login Management.
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 4.
Apr 02 18:30:16 lxbox systemd[1]: Stopped systemd-logind.service - User Login Management.
Apr 02 18:30:16 lxbox systemd[1]: Starting modp...@drm.service - Load Kernel Module drm...
Apr 02 18:30:16 lxbox (modprobe)[153]: modp...@drm.service: Executable /sbin/modprobe missing, skipping: No such file or directory
Apr 02 18:30:16 lxbox systemd[1]: modp...@drm.service: Deactivated successfully.
Apr 02 18:30:16 lxbox systemd[1]: Finished modp...@drm.service - Load Kernel Module drm.
Apr 02 18:30:16 lxbox systemd[1]: Starting systemd-logind.service - User Login Management...
Apr 02 18:30:16 lxbox (d-logind)[154]: systemd-logind.service: Failed to set up mount namespacing: Permission denied
Apr 02 18:30:16 lxbox (d-logind)[154]: systemd-logind.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-logind: Permission denied
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Main process exited, code=exited, status=226/NAMESPACE
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Apr 02 18:30:16 lxbox systemd[1]: Failed to start systemd-logind.service - User Login Management.
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 5.
Apr 02 18:30:16 lxbox systemd[1]: Stopped systemd-logind.service - User Login Management.
Apr 02 18:30:16 lxbox systemd[1]: Starting modp...@drm.service - Load Kernel Module drm...
Apr 02 18:30:16 lxbox (modprobe)[157]: modp...@drm.service: Executable /sbin/modprobe missing, skipping: No such file or directory
Apr 02 18:30:16 lxbox systemd[1]: modp...@drm.service: Deactivated successfully.
Apr 02 18:30:16 lxbox systemd[1]: Finished modp...@drm.service - Load Kernel Module drm.
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Start request repeated too quickly.
Apr 02 18:30:16 lxbox systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Apr 02 18:30:16 lxbox systemd[1]: Failed to start systemd-logind.service - User Login Management.
Apr 02 18:30:41 lxbox dbus-daemon[97]: [system] Failed to activate service 'org.freedesktop.login1': timed out (service_start_timeout=25000ms)
Apr 02 18:30:41 lxbox sudo[136]: pam_systemd(sudo-i:session): Failed to create session: Failed to activate service 'org.freedesktop.login1': timed out (service_start_timeout=25000ms)


-- Guest busctl monitor output:

Type=method_call Endian=l Flags=0 Version=1 Cookie=1 Timestamp="Mon 2023-04-03 01:30:16.386617 UTC"
Sender=:1.2 Destination=org.freedesktop.DBus Path=/org/freedesktop/DBus Interface=org.freedesktop.DBus Member=Hello
UniqueName=:1.2
MESSAGE "" {
};

Type=method_return Endian=l Flags=1 Version=1 Cookie=1 ReplyCookie=1 Timestamp="Mon 2023-04-03 01:30:16.386790 UTC"
Sender=org.freedesktop.DBus Destination=:1.2
MESSAGE "s" {
STRING ":1.2";
};

Type=signal Endian=l Flags=1 Version=1 Cookie=5 Timestamp="Mon 2023-04-03 01:30:16.386806 UTC"
Sender=org.freedesktop.DBus Path=/org/freedesktop/DBus Interface=org.freedesktop.DBus Member=NameOwnerChanged
MESSAGE "sss" {
STRING ":1.2";
STRING "";
STRING ":1.2";
};

Type=signal Endian=l Flags=1 Version=1 Cookie=2 Timestamp="Mon 2023-04-03 01:30:16.386820 UTC"
Sender=org.freedesktop.DBus Destination=:1.2 Path=/org/freedesktop/DBus Interface=org.freedesktop.DBus Member=NameAcquired
MESSAGE "s" {
STRING ":1.2";
};

Type=signal Endian=l Flags=1 Version=1 Cookie=12 Timestamp="Mon 2023-04-03 01:30:16.392000 UTC"
Sender=org.freedesktop.DBus Destination=org.freedesktop.systemd1 Path=/org/freedesktop/DBus Interface=org.freedesktop.systemd1.Activator Member=ActivationRequest
MESSAGE "s" {
STRING "dbus-org.freedesktop.login1.service";
};

Type=method_call Endian=l Flags=0 Version=1 Cookie=2 Timestamp="Mon 2023-04-03 01:30:16.392080 UTC"
Sender=:1.2 Destination=org.freedesktop.login1 Path=/org/freedesktop/login1 Interface=org.freedesktop.login1.Manager Member=CreateSession
UniqueName=:1.2
MESSAGE "uusssssussbssa(sv)" {
UINT32 0;
UINT32 0;
STRING "sudo-i";
STRING "x11";
STRING "user";
STRING "KDE";
STRING "seat0";
UINT32 7;
STRING "pts/7";
STRING "";
BOOLEAN false;
STRING "root";
STRING "";
ARRAY "(sv)" {
};
};

Type=error Endian=l Flags=1 Version=1 Cookie=3 ReplyCookie=2 Timestamp="Mon 2023-04-03 01:30:41.416860 UTC"
Sender=org.freedesktop.DBus Destination=:1.2
ErrorName=org.freedesktop.DBus.Error.TimedOut ErrorMessage="Failed to activate service 'org.freedesktop.login1': timed out (service_start_timeout=25000ms)"
MESSAGE "s" {
STRING "Failed to activate service 'org.freedesktop.login1': timed out (service_start_timeout=25000ms)";
};

Type=signal Endian=l Flags=1 Version=1 Cookie=6 Timestamp="Mon 2023-04-03 01:30:41.417026 UTC"
Sender=org.freedesktop.DBus Destination=:1.2 Path=/org/freedesktop/DBus Interface=org.freedesktop.DBus Member=NameLost
MESSAGE "s" {
STRING ":1.2";
};

Type=signal Endian=l Flags=1 Version=1 Cookie=7 Timestamp="Mon 2023-04-03 01:30:41.417043 UTC"
Sender=org.freedesktop.DBus Path=/org/freedesktop/DBus Interface=org.freedesktop.DBus Member=NameOwnerChanged
MESSAGE "sss" {
STRING ":1.2";
STRING ":1.2";
STRING "";
};


-- System Information:
Debian Release: 12.0
APT prefers testing-security
APT policy: (500, 'testing-security'), (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 6.1.0-7-amd64 (SMP w/2 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=locale: Cannot set LC_ALL to default locale: No such file or directory
UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages lxc depends on:
ii debconf [debconf-2.0] 1.5.82
ii dnsmasq-base [dnsmasq-base] 2.89-1
ii iproute2 6.1.0-2
ii libapparmor1 3.0.8-3
ii libc6 2.36-8
ii libcap2 1:2.66-3
ii libgcc-s1 12.2.0-14
ii liblxc-common 1:5.0.2-1
ii liblxc1 1:5.0.2-1
ii libseccomp2 2.5.4-1+b3
ii libselinux1 3.4-1+b5
ii nftables 1.0.6-2
ii sysvinit-utils [lsb-base] 3.06-2

Versions of packages lxc recommends:
ii apparmor 3.0.8-3
ii debootstrap 1.0.128+nmu2
ii dirmngr 2.2.40-1.1
ii gnupg 2.2.40-1.1
ii libpam-cgfs 1:5.0.2-1
ii lxc-templates 3.0.4.48.g4765da8-1
ii lxcfs 5.0.3-1
ii openssl 3.0.8-1
ii rsync 3.2.7-1
ii uidmap 1:4.13+dfsg1-1+b1
ii wget 1.21.3-1+b2

Versions of packages lxc suggests:
pn btrfs-progs <none>
pn lvm2 <none>
pn python3-lxc <none>

-- debconf information excluded

Pierre-Elliott Bécue

unread,
Apr 4, 2023, 5:40:04 AM4/4/23
to
What's weird is that the problem was already happening in buster and
bullseye.

I guess it is plausible that /etc/lxc/default.conf has been updated in
your upgrade, resetting the lxc-apparmor-profile to something that won't
work for unprivileged containers.

The issue is "normal": the apparmor profile needed to allow
systemd-logind to work properly would allow a user in a privileged
container to escalate and become root on the host. As one can't be
certain what profile will be used, the solution lies either within LXD
(which generates custom profiles for each containers), or with creating
a dedicated apparmor profile that you use only on unprivileged
containers.

The missing lines in apparmor rules have been added in
lxc-default-with-nesting rules of apparmor for lxc 5.

See the patch below: v

From: =?utf-8?q?Pierre-Elliott_B=C3=A9cue?= <p...@debian.org>
Date: Mon, 1 Aug 2022 22:35:10 +0200
Subject: [nesting] Extend mount permissions in apparmor to allow systemd
services' restrictions to work

These options allow systemd security features to work. In particular
cases, it helps with systemd-logind and program like this

It's only added in nesting profile as it could pose security risks on
privileged containers.

mount options=(rw,rbind) -> /run/systemd/unit-root/,
mount options=(rw,rbind) -> /run/systemd/unit-root/**,
mount options=(rw,rshared) -> /,
mount options=(rw,nosuid,nodev,noexec) proc -> /run/systemd/unit-root/proc/,
---
config/apparmor/profiles/lxc-default-with-nesting | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/config/apparmor/profiles/lxc-default-with-nesting b/config/apparmor/profiles/lxc-default-with-nesting
index cd198be..01562a9 100644
--- a/config/apparmor/profiles/lxc-default-with-nesting
+++ b/config/apparmor/profiles/lxc-default-with-nesting
@@ -10,6 +10,10 @@ profile lxc-container-default-with-nesting flags=(attach_disconnected,mediate_de
mount fstype=proc -> /var/cache/lxc/**,
mount fstype=sysfs -> /var/cache/lxc/**,
mount options=(rw,bind),
+ mount options=(rw,rbind) -> /run/systemd/unit-root/,
+ mount options=(rw,rbind) -> /run/systemd/unit-root/**,
+ mount options=(rw,rshared) -> /,
+ mount options=(rw,nosuid,nodev,noexec) proc -> /run/systemd/unit-root/proc/,
mount fstype=cgroup -> /sys/fs/cgroup/**,
mount fstype=cgroup2 -> /sys/fs/cgroup/**,
}

--
PEB
signature.asc

Forest

unread,
Apr 4, 2023, 6:50:04 PM4/4/23
to
>What's weird is that the problem was already happening in buster and
>bullseye.

That doesn't seem to be true, AFAICT. Bullseye (both my usual Bullseye
guest and a freshly installed one) does not exhibit the 25 second hang. A
freshly installed Buster guest doesn't, either. Not even with the default
config instead of nesting.conf.

To be precise: Although Bullseye and Buster do generate apparmor mount
errors in the host's syslog, the 25 second hang is new with Bookworm guests.
Maybe multiple problems are in play here?

>I guess it is plausible that /etc/lxc/default.conf has been updated in
>your upgrade, resetting the lxc-apparmor-profile to something that won't
>work for unprivileged containers.

Nope. I haven't upgraded the Bullseye host machine on which I discovered the
hang, and it occurs on both that host and a newly installed Bookworm host.
Also, I checked default.conf on both hosts just now, and it matches the one
in lxc_5.0.2-1_amd64.deb.

>The missing lines in apparmor rules have been added in
>lxc-default-with-nesting rules of apparmor for lxc 5.

My fresh Bookworm VM has lxc 5, and those four additional lines are present
in /etc/apparmor.d/lxc/lxc-default-with-nesting. The contents of
/usr/share/lxc/config/nesting.conf are also identical. Even when including
it in my container config, the 25 second hang persists.

>the solution lies either within LXD
>(which generates custom profiles for each containers), or with creating
>a dedicated apparmor profile that you use only on unprivileged
>containers.

I tried LXD as a workaround. Turns out it is not a suitable replacement in
my case.

I would be happy to try a modified apparmor profile. Ideally even get it
added into Bookworm's lxc package, or accepted upstream, so Bookworm doesn't
arrive in this broken state for lxc users.

I tried modifying the apparmor profile based on the host's syslog messages.
Despite using exactly the same mount options that appeared in the logs, the
errors and the 25 second hang persisted. (And I did remember to reload the
profile with apparmor_parser -r.) I wonder if the info="failed flags match"
in those syslog messages is supposed to hint that something more is needed.

It seems like we're missing some information here.
0 new messages