Running coldkernel/grsecurity in Whonix on Qubes: I found a way, but...

[Reg Tiangha]

In my coldkernel/gresecurity kernel testing on Qubes, installing a stock
coldkernel will result in a Whonix VM (either ws or gw) not booting:


Begin: Loading essential drivers ... [ 1.626860] device-mapper:
uevent: version 1.0.3
[ 1.626976] device-mapper: ioctl: 4.27.0-ioctl (2013-10-30)
initialised: dm-d...@redhat.com
done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top
... /scripts/local-top/qubes_cow_setup: 44:
/scripts/local-top/qubes_cow_setup: grep: not found
Warning: dmroot not requested, probably not a Qubes VM
done.
Begin: Running /scripts/local-premount ... done.
Begin: Waiting for root file system ... Begin: Running
/scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
[ 30.884054] random: nonblocking pool is initialized
Begin: Running /scripts/local-block ... done.
done.
Gave up waiting for root device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Check root= (did the system wait for the right device?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/mapper/dmroot does not exist. Dropping to a shell!


From what I can tell, it's because 'grep' is not included in Whonix's
initramfs set of tools:

(initramfs) ls /bin
drwxr-xr-x 2 0 0 0 .
drwxr-xr-x 16 0 0 0 ..
-rwxr-xr-x 1 0 0 1752 uname
-rwxr-xr-x 1 0 0 2576 ls
-rwxr-xr-x 1 0 0 616 true
-rwxr-xr-x 1 0 0 58384 sh
-rwxr-xr-x 1 0 0 1592 insmod
-rwxr-xr-x 1 0 0 4000 dd
-rwxr-xr-x 1 0 0 1088 halt
-rwxr-xr-x 1 0 0 4792 losetup
-rwxr-xr-x 1 0 0 976 dmesg
-rwxr-xr-x 1 0 0 5160 minips
-rwxr-xr-x 1 0 0 1088 poweroff
-rwxr-xr-x 1 0 0 1088 reboot
-rwxr-xr-x 1 0 0 800 pivot_root
-rwxr-xr-x 1 0 0 976 kill
-rwxr-xr-x 1 0 0 2728 mount
-rwSr-xr-x 1 0 0 146160 ntfs-3g
-rwxr-xr-x 1 0 0 13608 ipconfig
-rwxr-xr-x 1 0 0 624 false
-rwxr-xr-x 1 0 0 313584 udevadm
-rwxr-xr-x 1 0 0 4872 run-init
-rwxr-xr-x 1 0 0 2104 mkdir
-rwxr-xr-x 1 0 0 1080 umount
-rwxr-xr-x 1 0 0 29552 gunzip
-rwxr-xr-x 1 0 0 4368 fstype
-rwxr-xr-x 1 0 0 808 sleep
-rwxr-xr-x 1 0 0 7256 nfsmount
-rwxr-xr-x 1 0 0 2792 resume
-rwxr-xr-x 1 0 0 1248 ln
-rwxr-xr-x 1 0 0 848 chroot
-rwxr-xr-x 1 0 0 158592 kmod
-rwxr-xr-x 1 0 0 1880 mknod
-rwxr-xr-x 1 0 0 1800 mkfifo
-rwxr-xr-x 1 0 0 2784 cat
-rwxr-xr-x 1 0 0 1160 readlink
-rwxr-xr-x 1 0 0 2296 mv
-rwxr-xr-x 1 0 0 5160 cpio
-rwxr-xr-x 1 0 0 1176 nuke
-rwxr-xr-x 1 0 0 624 sync
-rwxr-xr-x 1 0 0 29552 gzip


It's a check in
/usr/share/initramfs-tools/scripts/local-top/qubes_cow_setup that fails.
If I comment out this check around line 45:

if ! grep -q 'root=[^ ]*dmroot' /proc/cmdline; then
warn "dmroot not requested, probably not a Qubes VM"
exit 0
fi

and then regenerate initramfs by running sudo upgrade-initramfs -u then
the Whonix VM boots normally and the coldkernel works just as it does on
a vanilla Debian-8 template. This works for both gw and ws variants.

**Note that this behaviour is *exactly* the same when running any
pvgrub2 kernel in Whonix, even with installing the stock Debian
linux-image-amd64 package.

In other words, there's no *technical* reason as to why a
coldkernel/grsecurity-based kernel (or any locally installed kernel,
really) on a Whonix template in Qubes shouldn't work (which makes sense,
since regular Whonix works just fine with locally installed kernels on
bare-metal) and the *only* thing stopping Whonix VMs from booting with
local kernels on Qubes is that one check in qubes_cow_setup.

So what's the proper way forward, then? Is it:

a) Try to convince the Whonix project to include 'grep' in its set of
initramfs tools (I presume there might have been some security concerns
with including it and thus it was stripped out),

b) Have the Qubes project find some other way to do that check that
doesn't involve using 'grep,' or

c) Go with my cheap hack of commenting out that one single check since
everything boots fine afterwards anyway (I wouldn't recommend it though).

In the meantime, the coldkernel works fine on my Whonix VMs, and I'd
rather be running it rather than the vanilla dom0 vm kernel, so I'm
going to stick with this set up for the time being even though I don't
know if I just introduced a security hole in my VMs by commenting out
that dmroot check.

[WillyPillow]

I posted something similar in the coldkernel thread a few months ago. The fastest way is actually just to apt install busybox ;) 

--WillyPillow 
---------- 
https://blog.nerde.pw/ 
PGP fingerprint = B57E 7237 B211 419C 35C4  AF5B EB4D 3264 A318 73CB 
----------

[Reg Tiangha]

On 04/08/2017 06:42 PM, WillyPillow wrote:
>
> I posted something similar in the coldkernel thread a few months ago..

> The fastest way is actually just to apt install busybox ;)
>
> --WillyPillow

Man, how did I miss that??

Well, the only thing left is getting it to run under dom0, I guess. I
actually tried a couple of months ago, merging both the gresecurity
patches, and the patches that Qubes uses, but it kernel panicked on boot
and I didn't investigate any further than that and instead just choose
to run Kernel 4.10 on my machines. It might be worth revisiting, I
suppose. But I wonder if all those Xen patches really needed? I mean,
some of them are for XSAs that are pretty old, and while most of them
still patch in, I'd assume that the later kernel versions would have
already included them or similar mitigations?? Or if not, why haven't
they been ported to upstream by now? I mean, these coldkernels run fine
in VMs without the Qubes patches so I'm a little confused on what they
do and whether or not they're really necessary. I've just been applying
them to my dom0 4.10 kernels out of habit, but it gets tricky when
trying to apply them *after* applying the grsecurity patches since after
that, each Xen/Qubes kernel patch pretty much has to be applied manually
since a lot of the files get changed because of grsecurity.