Bug#993391: lxc: Unprivileged lxc example from README.Debian.gz gives AppArmor error "Failed to mount proc"

[pk1]

Package: lxc
Version: 1:4.0.6-2
Severity: important
X-Debbugs-Cc: pkoro...@gmail.com

Dear Maintainer,

On a pristine Debian 11 install, the example from "Unprivileged containers"
section of /usr/share/doc/lxc/README.Debian.gz gives "Failed to mount proc"
with an AppArmor error in dmesg, but lxc.apparmor.profile is unconfined.

reportbug said to test unstable's lxc 1:4.0.10-1, but that also fails with
a different error message.

$ cat test_config
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.mount.auto = proc:mixed sys:ro cgroup:mixed
lxc.apparmor.profile = unconfined

$ systemd-run --scope --quiet --user --property=Delegate=yes lxc-start --logfile /dev/stderr -f test_config -n machine
lxc-start machine 20210830065007.367 ERROR utils - utils.c:safe_mount:1204 - Permission denied - Failed to mount "proc" onto "/proc"
lxc-start machine 20210830065007.367 ERROR conf - conf.c:lxc_mount_auto_mounts:681 - Permission denied - Failed to mount "proc" on "/proc" with flags 14
lxc-start machine 20210830065007.367 ERROR conf - conf.c:lxc_setup:3330 - Failed to setup first automatic mounts
lxc-start machine 20210830065007.367 ERROR start - start.c:do_start:1218 - Failed to setup container "machine"
[snip]

# dmesg | tail
[snip unrelated]
[ 2127.458104] audit: type=1400 audit(1630306207.363:40): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="/usr/bin/lxc-start" name="/proc/" pid=3286 comm="lxc-start" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"

Could Debian's sysctl be related, as suggested on the LXC forum?
"At some point Debian introduced additional sysctl to restrict user namespaces
for unprivileged users, maybe they still do that and that’s what’s getting in
the way here?"
https://discuss.linuxcontainers.org/t/cannot-start-unprivileged-container-on-debian-11/12019/4

I also tried (umask 022 ; su -l non_root) per #946725 but that does not fix it.
This is also unrelated to #947863 because the config says unconfined.

-- System Information:
Debian Release: 11.0
Architecture: amd64 (x86_64)

Versions of packages lxc depends on:
ii bridge-utils 1.7-1
ii debconf [debconf-2.0] 1.5.77
ii dnsmasq-base [dnsmasq-base] 2.85-1
ii iproute2 5.10.0-4
ii iptables 1.8.7-1
ii libc6 2.31-13
ii libcap2 1:2.44-1
ii libgcc-s1 10.2.1-6
ii liblxc1 1:4.0.6-2
ii libseccomp2 2.5.1-1
ii libselinux1 3.1-3
ii lsb-base 11.1.0

Versions of packages lxc recommends:
ii apparmor 2.13.6-10
ii debootstrap 1.0.123
ii dirmngr 2.2.27-2
ii gnupg 2.2.27-2
ii libpam-cgfs 1:4.0.6-2
ii lxc-templates 3.0.4-5
ii lxcfs 4.0.7-1
ii openssl 1.1.1k-1+deb11u1
ii rsync 3.2.3-4
ii uidmap 1:4.8.1-1
ii wget 1.21-1+b1

Versions of packages lxc suggests:
ii btrfs-progs 5.10.1-2
ii lvm2 2.03.11-2.1
pn python3-lxc <none>

-- debconf information excluded

[Pierre-Elliott Bécue]

Control: tags -1 +moreinfo

I am unable to reproduce your bug on a vanilla Debian 11 or unstable
system.

Please print the output of "sysctl kernel.unprivileged_userns_clone"

Please also follow all instructions of the readme file, and give me a
feedback.

Regards,

--
Pierre-Elliott Bécue
GPG: 9AE0 4D98 6400 E3B6 7528 F493 0D44 2664 1949 74E2
It's far easier to fight for principles than to live up to them.

[pk]

Thank you for answering. kernel.unprivileged_userns_clone = 1 on my
machine and on the Live DVD. All instructions of the README.Debian.gz
were followed.

To rule out machine-specific misconfiguration, this log is from the
Live DVD, Debian 11.0 AMD64 Standard:



Warning: Permanently added '[localhost]:12346' (ECDSA) to the list of
known hosts.
user@localhost's password:
Linux debian 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
user@debian:~$ sudo su -l
root@debian:~# apt-get update ; apt-get install lxc
[snip]
root@debian:~# sysctl kernel.unprivileged_userns_clone
kernel.unprivileged_userns_clone = 1
root@debian:~# grep user /etc/subuid /etc/subgid
/etc/subuid:user:100000:65536
/etc/subgid:user:100000:65536
root@debian:~#
logout
user@debian:~$ mkdir -p .local/share/lxc
user@debian:~$ chmod +x . .local .local/share
user@debian:~$
user@debian:~$ cat > test_config

lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.mount.auto = proc:mixed sys:ro cgroup:mixed
lxc.apparmor.profile = unconfined

user@debian:~$
user@debian:~$ systemd-run --scope --quiet --user

--property=Delegate=yes lxc-start --logfile /dev/stderr -f
test_config -n machine

lxc-start machine 20210901150740.103 ERROR utils -

utils.c:safe_mount:1204 - Permission denied - Failed to mount "proc"
onto "/proc"

lxc-start machine 20210901150740.104 ERROR conf -

conf.c:lxc_mount_auto_mounts:681 - Permission denied - Failed to mount
"proc" on "/proc" with flags 14

lxc-start machine 20210901150740.104 ERROR conf -

conf.c:lxc_setup:3330 - Failed to setup first automatic mounts

lxc-start machine 20210901150740.105 ERROR start -

start.c:do_start:1218 - Failed to setup container "machine"

lxc-start machine 20210901150740.106 ERROR sync -
sync.c:__sync_wait:36 - An error occurred in another process (expected
sequence number 5)
lxc-start machine 20210901150740.106 ERROR start -
start.c:__lxc_start:1999 - Failed to spawn container "machine"
lxc-start machine 20210901150740.107 ERROR lxccontainer -
lxccontainer.c:wait_on_daemonized_start:859 - Received container state
"ABORTING" instead of "RUNNING"
lxc-start: machine: lxccontainer.c: wait_on_daemonized_start: 859
Received container state "ABORTING" instead of "RUNNING"
lxc-start machine 20210901150740.108 ERROR lxc_start -
tools/lxc_start.c:main:308 - The container failed to start
lxc-start: machine: tools/lxc_start.c: main: 308 The container failed to start
lxc-start machine 20210901150740.108 ERROR lxc_start -
tools/lxc_start.c:main:311 - To get more details, run the container in
foreground mode
lxc-start: machine: tools/lxc_start.c: main: 311 To get more details,
run the container in foreground mode
lxc-start machine 20210901150740.108 ERROR lxc_start -
tools/lxc_start.c:main:313 - Additional information can be obtained by
setting the --logfile and --logpriority options
lxc-start: machine: tools/lxc_start.c: main: 313 Additional
information can be obtained by setting the --logfile and --logpriority
options
user@debian:~$ sudo su -l
root@debian:~# dmesg | tail
[ 294.416862] audit: type=1400 audit(1630508543.972:7):
apparmor="STATUS" operation="profile_replace" info="same as current
profile, skipping" profile="unconfined" name="lsb_release" pid=2444
comm="apparmor_parser"
[ 294.526095] audit: type=1400 audit(1630508544.084:8):
apparmor="STATUS" operation="profile_load" profile="unconfined"
name="/usr/bin/man" pid=2442 comm="apparmor_parser"
[ 294.527098] audit: type=1400 audit(1630508544.084:9):
apparmor="STATUS" operation="profile_load" profile="unconfined"
name="man_filter" pid=2442 comm="apparmor_parser"
[ 294.528359] audit: type=1400 audit(1630508544.084:10):
apparmor="STATUS" operation="profile_load" profile="unconfined"
name="man_groff" pid=2442 comm="apparmor_parser"
[ 297.864908] audit: type=1400 audit(1630508547.412:11):
apparmor="STATUS" operation="profile_load" profile="unconfined"
name="lxc-container-default" pid=2618 comm="apparmor_parser"
[ 297.867516] audit: type=1400 audit(1630508547.416:12):
apparmor="STATUS" operation="profile_load" profile="unconfined"
name="lxc-container-default-cgns" pid=2618 comm="apparmor_parser"
[ 297.869845] audit: type=1400 audit(1630508547.420:13):
apparmor="STATUS" operation="profile_load" profile="unconfined"
name="lxc-container-default-with-mounting" pid=2618
comm="apparmor_parser"
[ 297.872902] audit: type=1400 audit(1630508547.420:14):
apparmor="STATUS" operation="profile_load" profile="unconfined"
name="lxc-container-default-with-nesting" pid=2618
comm="apparmor_parser"
[ 297.933031] audit: type=1400 audit(1630508547.480:15):
apparmor="STATUS" operation="profile_load" profile="unconfined"
name="/usr/bin/lxc-start" pid=2624 comm="apparmor_parser"
[ 610.653177] audit: type=1400 audit(1630508860.099:16):

apparmor="DENIED" operation="mount" info="failed flags match"

error=-13 profile="/usr/bin/lxc-start" name="/proc/" pid=3594

comm="lxc-start" fstype="proc" srcname="proc" flags="rw, nosuid,
nodev, noexec"

root@debian:~#

[Pierre-Elliott Bécue]

Control: severity -1 normal

Hi,

I don't like to make judgemental calls when I try to help our users, but
here I'll still make a guess. I guess that you actually did not read
carefully README.Debian.gz and therefore did not follow these
instructions carefully.

pk <pko...@gmail.com> writes:

> Thank you for answering. kernel.unprivileged_userns_clone = 1 on my
> machine and on the Live DVD. All instructions of the README.Debian.gz
> were followed.
>
> To rule out machine-specific misconfiguration, this log is from the
> Live DVD, Debian 11.0 AMD64 Standard:
>
>
>
> Warning: Permanently added '[localhost]:12346' (ECDSA) to the list of
> known hosts.
> user@localhost's password:
> Linux debian 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64
>
> The programs included with the Debian GNU/Linux system are free software;
> the exact distribution terms for each program are described in the
> individual files in /usr/share/doc/*/copyright.
>
> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
> permitted by applicable law.
> user@debian:~$ sudo su -l
> root@debian:~# apt-get update ; apt-get install lxc
> [snip]

What's in there apart from apt-get output?

> root@debian:~# sysctl kernel.unprivileged_userns_clone
> kernel.unprivileged_userns_clone = 1
> root@debian:~# grep user /etc/subuid /etc/subgid
> /etc/subuid:user:100000:65536
> /etc/subgid:user:100000:65536
> root@debian:~#
> logout
> user@debian:~$ mkdir -p .local/share/lxc
> user@debian:~$ chmod +x . .local .local/share
> user@debian:~$
> user@debian:~$ cat > test_config
> lxc.idmap = u 0 100000 65536
> lxc.idmap = g 0 100000 65536
> lxc.mount.auto = proc:mixed sys:ro cgroup:mixed
> lxc.apparmor.profile = unconfined

This is not in the README, and you actually don't seem to have created
any container yet. Furthermore, your configuration actually doesn't
mention any rootfs or block device to pivot on!

Here is what I get doing something like what you pasted here.

.-(0:03:50)-(~)--------------------------------------------------------------------------(peb@xxxxx)-
`--[130]-> lxc-ls -f
NAME STATE AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED
autopkgtest-lxc-xwkkud STOPPED 0 - - - true
autopkgtest-unstable STOPPED 0 - - - true

As you see I only have two containers. I'll try to start a container
named "blah" which does not exist. I wrote a blah.cfg containing roughly
the same config as you just adapted for my subuids.

.-(0:03:51)-(~)--------------------------------------------------------------------------(peb@xxxxx)-
`---> cat blah.cfg
lxc.idmap = u 0 1214112 65536
lxc.idmap = g 0 1214112 65536

lxc.mount.auto = proc:mixed sys:ro cgroup:mixed
lxc.apparmor.profile = unconfined

Here I'll use your command, but note that README.Debian.gz states we
have lxc-unpriv-start which makes things quite more elegant.

-(0:04:40)-(~)--------------------------------------------------------------------------(peb@xxxxx)-
`--[1]-> systemd-run --user --scope -p "Delegate=yes" /usr/bin/lxc-start -o /dev/stdout -f blah.cfg blah
Running scope as unit: run-r34581cfe965441428e3520ecb8c0bb7b.scope
lxc-start blah 20210901220449.759 ERROR utils - utils.c:safe_mount:1204 - Permission denied - Failed to mount "proc" onto "/proc"
lxc-start blah 20210901220449.759 ERROR conf - conf.c:lxc_mount_auto_mounts:681 - Permission denied - Failed to mount "proc" on "/proc" with flags 14
lxc-start blah 20210901220449.759 ERROR conf - conf.c:lxc_setup:3330 - Failed to setup first automatic mounts
lxc-start blah 20210901220449.759 ERROR start - start.c:do_start:1218 - Failed to setup container "blah"
lxc-start blah 20210901220449.759 ERROR sync - sync.c:__sync_wait:36 - An error occurred in another process (expected sequence number 5)
lxc-start blah 20210901220449.759 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:859 - Received container state "ABORTING" instead of "RUNNING"
lxc-start blah 20210901220449.759 ERROR start - start.c:__lxc_start:1999 - Failed to spawn container "blah"
[and it goes on]

With of course the Apparmor denial in dmesg.

I guess the reason is that lxc having no rootfs or block device to pivot
on tries to mount proc on "/proc" (maybe because it concatenates
$rootfs+"/proc", whith $rootfs being "" here?), ie on the host's /proc,
or anyway on something you don't have a right to mount on.

Of course with a created container and a real config, things are going
smoothly.

Considering what I gathered, I would recommend you take the time to
actually read the documentation properly and try to follow it.

If you fail to have a running container, please do provide a full log of
what you did step by step, and which part of README.Debian.gz it were
covered by what you did, in your opinion.

With best regards,

--
PEB

[pk]

Hello,

I copy-pasted configuration and commands from
/usr/share/doc/lxc/README.Debian.gz under "Unprivileged containers".
Are you talking about another file?
https://salsa.debian.org/lxc-team/lxc/-/blob/7d692c266c63fced9417042ae904cc2a280b96d8/debian/README.Debian

lxc.rootfs defaults to the system root / per lxc.container.conf(5).

Creation is unnecessary, it is just a convenience to avoid -f and does
not affect the container runtime. My (still privileged) lxc setup
works perfectly with -f without ever creating any containers.

I pasted full logs above. Please try to be respectful and helpful, do
not reproduce on a configured machine, and leave bug triaging to the
lxc experts.

Thanks,

[pk]

Can you post your complete config for autopkgtest-lxc-xwkkud,
autopkgtest-unstable or other working unpriv container? Your output
reads "unprivileged true".

Thanks

[Pierre-Elliott Bécue]

Because they are unprivileged which is the topic of the current
discussion.

--
PEB

[Pierre-Elliott Bécue]

Hi,

pk <pko...@gmail.com> writes:

> Hello,
>
> I copy-pasted configuration and commands from
> /usr/share/doc/lxc/README.Debian.gz under "Unprivileged containers".
> Are you talking about another file?
> https://salsa.debian.org/lxc-team/lxc/-/blob/7d692c266c63fced9417042ae904cc2a280b96d8/debian/README.Debian

The configuration in that file is

lxc.include = /etc/lxc/default.conf

lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.mount.auto = proc:mixed sys:ro cgroup:mixed
lxc.apparmor.profile = unconfined

and goes to ~/.config/lxc/default.conf

You removed at least the lxc.include statement, and actually tried
something of your own, in particular not creating a default config for
your user and a container afterwards.

> lxc.rootfs defaults to the system root / per lxc.container.conf(5).

Which is not acceptable for an *unprivileged* container, which is the
case you brought here. The reason why Apparmor intervenes instead of
letting either init crash upon startup (because not being able to
manipulate the filesystem) or things explode is because
lxc.apparmor.profile doesn't apply to lxc-start call, but to only to the
lxc child process.

> Creation is unnecessary, it is just a convenience to avoid -f and does
> not affect the container runtime. My (still privileged) lxc setup
> works perfectly with -f without ever creating any containers.

Creation is necessary as you need a valid rootfs to work, and a valid
rootfs for an unprivileged container has to fit the usernamespace which
will be created upon startup of the container. "/" is not a valid rootfs
for an unprivileged container as the uid mappings are totally out of
line. You therefore need to at least create one container using
lxc-create or manually create a rootfs using mmdebstrap or whatever fits
best.

> I pasted full logs above.

You pasted truncated logs, and actually did not follow the README.

> Please try to be respectful and helpful, do not reproduce on a
> configured machine, and leave bug triaging to the lxc experts.

Being one of the LXC maintainers, I'm totally entitled to triage your
bug report, especially since what you claim being a bug does not look
like one. I won't reply to your assumption about my expertise.

Please follow the README properly and if that fails please come back
with full logs.

[pk]

> Creation is necessary as you need a valid rootfs to work, and a valid
> rootfs for an unprivileged container has to fit the usernamespace which
> will be created upon startup of the container. "/" is not a valid rootfs
> for an unprivileged container as the uid mappings are totally out of
> line. You therefore need to at least create one container using
> lxc-create or manually create a rootfs using mmdebstrap or whatever fits
> best.

Thank you. How do I close this report?