[lxc-users] kernel messages appear in containers?

[Richard Hector]

Hi all,

I've noticed that some log messages that really belong to the host (like
those from monthly RAID checks, for example) can appear in arbitrary
containers instead - so they're spread all over the place.

Is that normal/fixable?

Cheers,
Richard

[Tomasz Chmielewski]

I'd say it's a "normal, bad default".

You can fix it by adding this to your sysctl config file:

kernel.dmesg_restrict = 1

Tomasz Chmielewski
https://lxadm.com

[Stéphane Graber]

Hi,

Yes, this is normal and as pointed out already, can be tweaked with
dmesg_restrict.

There was some interest a while back in implementing a logging
namespace, which would solve this cleanly, but it's never been enough
of a priority for anyone to actually do the kernel work for it.

Stéphane

> _______________________________________________
> lxc-users mailing list
> lxc-...@lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users

--
Stéphane

[Richard Hector]

On 3/02/19 9:47 PM, Stéphane Graber wrote:> Yes, this is normal and as

pointed out already, can be tweaked with
> dmesg_restrict.
>
> There was some interest a while back in implementing a logging
> namespace, which would solve this cleanly, but it's never been enough
> of a priority for anyone to actually do the kernel work for it.

Thanks to both.

Done; will watch for results :-)

Cheers,

Richard

[Christian Brauner]

On Sun, Feb 03, 2019 at 09:47:25AM +0100, Stéphane Graber wrote:
> Hi,
>
> Yes, this is normal and as pointed out already, can be tweaked with
> dmesg_restrict.
>
> There was some interest a while back in implementing a logging
> namespace, which would solve this cleanly, but it's never been enough
> of a priority for anyone to actually do the kernel work for it.

Well actually there has been a patchset, no? I think it's just that
noone really pushed hard enough for it (or long enough for that matter).
:)

Christian

[Richard Hector]

> On Sun, Feb 3, 2019 at 3:52 AM Richard Hector <ric...@walnut.gen.nz>
wrote:
>>
>> Hi all,
>>
>> I've noticed that some log messages that really belong to the host (like
>> those from monthly RAID checks, for example) can appear in arbitrary
>> containers instead - so they're spread all over the place.

On 3/02/19 9:47 pm, Stéphane Graber wrote:
> Hi,
>
> Yes, this is normal and as pointed out already, can be tweaked with
> dmesg_restrict.

I implemented this, and it seems to have had some effect, but not complete.

I recently had an OOM killer event, and its output appears to have been
scattered across at least several (if not all) of the containers (as
well as the host). I haven't studied the details and timestamps (I'm not
that familiar with reading these dumps), but I think I'd have to
assemble the logs from all of the containers to end up with the complete
picture, which doesn't seem ideal, even disregarding whether different
people have access to different containers, and these logs reveal info
that some people shouldn't be able to see.

Any tips on further steps?

Would I get better isolation with unprivileged containers?

I can provide the logs (possibly sanitised) if required.

Cheers,
Richard

[Richard Hector]

I think I'm beginning to understand this better.

Each of my containers runs rsyslog. The imklog module reads kernel log
messages from the kernel, to be written to the relevant log file(s).
This means that whichever container's rsyslog happens to read the kernel
log gets whatever's there at the moment. The next read might come from a
different container, or from the host, so messages could appear in any
container's log files. Does that sound right?

With that in mind, I've commented out the imklog module from all the
containers on one of my hosts, to see what happens.

Ideally though, rsyslog wouldn't be able to read those messages even if
it tried.

The suggestion above, adding "kernel.dmesg_restrict = 1" to the
sysctl.conf on the host, would be useful if the rsyslog process wasn't
running as root (privileged container, so it's real root), with both
CAP_SYS_ADMIN and CAP_SYSLOG. It appears I could configure my containers
to drop one or both of those; it's not quite so clear which I need to
drop. Do I actually need to drop both? Is that going to break other things?

Have I understood all this correctly?

Would switching to unprivileged containers take care of all this for me,
and is that easy/safe to do to an existing, running, production container?

Thanks,
Richard