locking down a daemon

[Luigi Semenzato]

I am working on locking down a new daemon, written in Rust, with
minijail0 and seccomp, and I am finding it a bit difficult.

The instructions here

https://chromium.googlesource.com/chromiumos/docs/+/master/sandboxing.md

are good, but I already knew most of the concepts. The difficult part is
debugging the minijail lockdown. I started by writing this:

F=
# F="$F -i" # fork and exit
F="$F -v" # VFS namespace
F="$F -r" # readonly /proc
# F="$F -p" # pid namespace --- loses ^Z and ^C
F="$F -l" # IPC namespace
F="$F -n" # no new privs
F="$F -d" # minimal /dev, implies -v --- loses ^Z
F="$F -b /usr/bin/memd,/usr/bin/memd" # bind mount
F="$F -P /var/empty" # pivot root
# F="$F -b /dev/chromeos-low-mem,/dev/chromeos-low-mem" # bind mount
# F="$F -b /var/log,/var/log" # bind mount

minijail0 $F -- /usr/bin/memd always-poll-fast

(most flags don't have a long version and I added comments to help)

Everything went well until I did the root pivot. At first minijail
count not find the executable (duh), so I bind-mounted it. Now memd
crashes, but quietly (normally it prints a panic message to the
console) and I only know it crashed from crash-reporter logs.

I am not sure why this happens. Shouldn't stdout and stderr survive
the pivot? Maybe I already lost them earlier, with some other flags?
Or maybe I should not bother pivotlng? The daemon only needs to
read/write a few files and I was going to bind-mount them as I
discovered them---but now they're hard to discover.

Do you have suggestions for debugging this?

Thanks!

[Mike Frysinger]

instead of doing the -b/-d/-P yourself, try using --profile minimalistic-mountns. it'll bind mount all of the rootfs which might be more than you need, but we don't currently think that's a big deal once you have a seccomp filter.

then add -b /dev/log. that should get you logger syslog access since you go through the syslog unix socket, not write directly through to /var/log.

I'm not sure how much we've tested ^Z, but ^C should still work in these cases.

-mike

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
https://groups.google.com/a/chromium.org/group/chromium-os-dev

[Luigi Semenzato]

Thank you for this advice.

The simple test below (unknown command-line argument) seems to
indicate that stderr disappears. Is there any way to keep it for
testing? That's where Rust sends its panic messages.

minijail0 --profile minimalistic-mountns -b /dev/log,/dev/log -b
/var/log,/var/log -b /dev/tty,/dev/tty -- /usr/bin/memd xxxtest
always-poll-fast

(btw, the code shows that -b <x> should be the same as -b <x>,<x> but
maybe my build (like a month old) doesn't support it yet, the error is
"Bad binding: <x> (null)")

[Mike Frysinger]

erm, the -b behavior changed landed in Jan 2018, so if your system doesn't support it, sounds like it's way more than a month behind ...

minijail doesn't mess with std{out,err}.  if you're not seeing anything, it's for a different reason.  in your case, your mount options are incorrect.  look at /var/log/messages for details as minijail sends its output to syslog.

-mike

[Luigi Semenzato]

Sorry! This image is also from January.

So here's the problem:

2018-05-21T11:31:37.132681-07:00 INFO memd[5226]: libminijail[5226]:
mount /dev/tty -> /dev/tty type ''
2018-05-21T11:31:37.134061-07:00 WARNING memd[5226]:
libminijail[5226]: creating mount target '/var/empty/var/log' failed:
Read-only file system
2018-05-21T11:31:37.145594-07:00 ERR memd[5226]: libminijail[5226]:
mount_one failed: Read-only file system

Is it the case that /var/empty is mounted read-only, and minijail
tries to create /var/empty/var, then bind /var/log to it?
So do I have to do this:

minijail0 --profile minimalistic-mountns -b /var,/var,1 -b
/var/log,/var/log,1 -- /usr/bin/memd

[Luigi Semenzato]

It doesn't look like I need /var to be writeable to mount log on it.

But I am still exposing all of /var when I only need /var/log. Should
I live with this, or is there a solution?

[Jorge Lucangeli Obes]

On Mon, May 21, 2018 at 2:44 PM Luigi Semenzato <seme...@chromium.org> wrote:

It doesn't look like I need /var to be writeable to mount log on it.

But I am still exposing all of /var when I only need /var/log.  Should
I live with this, or is there a solution?

I'd say live with this for now, even if just to unblock you.

We should make bind-mounting /var/log automatic, or part of minimalistic-mountns.

[Luigi Semenzato]

Thanks again for the assistance on this. Now I fail trying to access
a file in /sys:

minijail0 --profile minimalistic-mountns -b /dev/log -b /var -b
/var/log,/var/log/,1 -b /run -b /sys -- /usr/bin/memd always-poll-fast

thread 'main' panicked at 'trace setup error:
StringError("\"/sys/kernel/debug/tracing/current_tracer\": No such
file or directory (os error 2)")', libcore/result.rs:945:5

Is /sys turned off in minijail? I need access to /sys and /proc.

[Mike Frysinger]

why do you need /var/log ?  if you're using syslog (which i thought you were switching to), you only need /dev/log.  rsyslog takes care of updating files in /var/log.

if you need a specific dir for memd output, then mount a writable tmpfs on /var, and then do a bind mount for the subpath.

/sys/kernel/debug is a sep mount/filesystem, so you'll want to bind /sys, and then bind /sys/kernel/debug.

-mike

[Luigi Semenzato]

On Mon, May 21, 2018 at 11:01 PM, Mike Frysinger <vap...@chromium.org> wrote:
> why do you need /var/log ? if you're using syslog (which i thought you were
> switching to), you only need /dev/log. rsyslog takes care of updating files
> in /var/log.

Memd produces its output in /var/log/memd.

> if you need a specific dir for memd output, then mount a writable tmpfs on
> /var, and then do a bind mount for the subpath.

Oh I see--- but do I need the tmpfs? How about mkdir -p /tmp/var/log,
then -b /tmp/var,/var -b /var/log/memd? Or maybe the other way
around? -b /var/log/memd,/tmp/var/log/mend -b /tmp/var,/var?

> /sys/kernel/debug is a sep mount/filesystem, so you'll want to bind /sys,
> and then bind /sys/kernel/debug.

Ah thanks, I didn't realize that.

[Luigi Semenzato]

I may have the order backwards in some individual -b specifications.

[Mike Frysinger]

minijail doesn't really have support for that sort of thing.  use -k tmpfs,/var,tmpfs,0xe to get a writable but empty /var path.  then you can -b /var/log/memd,,1 to get a reduced writable view to just that path.

we haven't really looked at the resource usage of empty tmpfs mounts, but we use them heavily.  so we're assuming they're effectively free ;).

-mike

[Luigi Semenzato]

Great, thanks! That works fine.