services failing in containers, in debian buster

[Richard Hector]

Hi all,

I'm seeing things like this in my logs (in a container - buster on a
buster host):

> Feb 19 23:09:01 zabbix systemd[15524]: phpsessionclean.service: Failed to set up mount namespacing: Permission denied
> Feb 19 23:09:01 zabbix systemd[15524]: phpsessionclean.service: Failed at step NAMESPACE spawning /usr/lib/php/sessionclean: Permission denied
> Feb 19 23:09:01 zabbix systemd[1]: phpsessionclean.service: Main process exited, code=exited, status=226/NAMESPACE
> Feb 19 23:09:01 zabbix systemd[1]: phpsessionclean.service: Failed with result 'exit-code'.
> Feb 19 23:09:01 zabbix systemd[1]: Failed to start Clean php session files.
> Feb 19 23:39:01 zabbix systemd[15552]: phpsessionclean.service: Failed to set up mount namespacing: Permission denied
> Feb 19 23:39:01 zabbix systemd[15552]: phpsessionclean.service: Failed at step NAMESPACE spawning /usr/lib/php/sessionclean: Permission denied
> Feb 19 23:39:01 zabbix systemd[1]: phpsessionclean.service: Main process exited, code=exited, status=226/NAMESPACE
> Feb 19 23:39:01 zabbix systemd[1]: phpsessionclean.service: Failed with result 'exit-code'.
> Feb 19 23:39:01 zabbix systemd[1]: Failed to start Clean php session files.
> Feb 20 00:00:07 zabbix systemd[15575]: logrotate.service: Failed to set up mount namespacing: Permission denied
> Feb 20 00:00:07 zabbix systemd[15575]: logrotate.service: Failed at step NAMESPACE spawning /usr/sbin/logrotate: Permission denied
> Feb 20 00:00:07 zabbix kernel: [271414.465480] audit: type=1400 audit(1645268407.875:201): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=25925 comm="(ogrotate)" flags="rw, rslave"
> Feb 20 00:00:07 zabbix systemd[1]: logrotate.service: Main process exited, code=exited, status=226/NAMESPACE
> Feb 20 00:00:07 zabbix systemd[1]: logrotate.service: Failed with result 'exit-code'.
> Feb 20 00:00:07 zabbix systemd[1]: Failed to start Rotate log files.


This is from my logcheck email, so it only includes 'bad' lines from the
log; I could find more complete info if required, but I think this gives
the idea. I've also deleted lines that aren't relevant.

Note that the kernel apparmor line can appear on the host, or in any
container - presumably because they all use the same kernel. That's why
there's only one of them; the ones from sessionclean must have ended up
somewhere else. It seems like an information leak that the containers
can all see it; I would have hoped that the containers couldn't see
kernel messages like that at all - but that's a separate issue. It's
also annoying that the process name is truncated _backwards_ (eg
'ogrotate') - it took me ages to work out what 'ionclean' referred to.
But I think that's a kernel thing.

From what I've read so far, it's related to hardening stuff in the
systemd service definition, like this (from the logrotate service in
this case):

> # hardening options
> # details: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
> # no ProtectHome for userdir logs
> # no PrivateNetwork for mail deliviery
> # no ProtectKernelTunables for working SELinux with systemd older than 235
> # no MemoryDenyWriteExecute for gzip on i686
> PrivateDevices=true
> PrivateTmp=true
> ProtectControlGroups=true
> ProtectKernelModules=true
> ProtectSystem=full
> RestrictRealtime=true

... and because lxc already does similar stuff, and the two don't work
together.

I saw stuff about using these lines from /etc/lxc/default.conf:

> lxc.apparmor.profile = generated
> lxc.apparmor.allow_nesting = 1

(Which weren't there before because the container was built on stretch,
or possibly earlier)

I added those, but still no luck.

Do I also need this bit? :

> # Uncomment the following line to support nesting containers:
> #lxc.include = /usr/share/lxc/config/nesting.conf
> # (Be aware this has security implications)

What are those security implications? Are they inevitable if I want to
make things work in my situation?

And why does the default setup not seem to work? Won't services always
fail like this in a container? Or am I doing something odd, perhaps due
to upgrading both host and containers?

Thanks,
Richard

[Richard Hector]

On 21/02/22 08:24, Richard Hector wrote:
> Hi all,
>
> I'm seeing things like this in my logs (in a container - buster on a
> buster host):

[snip]

> And why does the default setup not seem to work? Won't services always
> fail like this in a container? Or am I doing something odd, perhaps due
> to upgrading both host and containers?

I've just installed php on a test container, also buster on buster, and
the problem doesn't show up. So I guess I'll have to figure out what the
differences are ...

Richard

[Richard Hector]

Comparing the /etc/ directories of the two hosts (actually comparing the
list of md5sums of all the files) revealed nothing. Anywhere else I
should be looking?

Oh - the test host is a KVM VM, while the real one is bare metal, but I
assume that makes no difference.

Cheers,
Richard

[Richard Hector]

No tips? Are there ways I should improve my question?

How much risk is there in removing some/all of those hardening options
from the systemd units?

Thanks,
Richard

[Richard Hector]

After an upgrade of both host and container (libexpat1 was the only
upgraded package), followed by a reboot, the problem seems to have gone
away.

Would the nesting lines in the container config require a reboot of the
host? It doesn't seem likely ...

Any ideas what happened?

Cheers,
Richard