FYI: Experimental Qubes coldkernel support now available

[Reg Tiangha]

I haven't tried it myself yet, but it looks like the coldkernel crew
pushed out experimental support for Debian templates to one of their
test branches yesterday:

https://github.com/coldhakca/coldkernel/tree/0.9a

Has anyone out there tried it yet? Thoughts, tips or tricks?

[Chris Laprise]

I look forward to building this soon. Two questions:

1. What probability it will work with Debian 9 Stretch?

2. How does this compare to using newer (4.8+) kernels and AppArmor,
which are two easy options for Qubes users?

Chris

[Reg Tiangha]

Well, I'm currently in the middle of compiling it; haven't had to
compile a kernel since my Gentoo days and I've forgotten how long it
used to take. One piece of advice at this point: If you're using a fresh
template, you'll definitely want to allocate more space to /home; the
default 2GB isn't enough. I've doubled it to 4G and am keeping an eye on
how much it grows. I can't remember how much disk space it used to take
to compile an old Gentoo kernel so I'll be babysitting this one until
it's done.

In the meantime:

1) I would assume it would work; their instructions imply Debian 7+, but
I suppose the only way to find out if it would work on a Debian 9
template under Qubes would be to compile it and find out.

2) I've never used a Gresecurity kernel before (I almost did under
Gentoo but at the time, the kernel they were using was an older one
lacking a driver I needed so I ended up going with the vanilla sources
instead), but here's how the gresecurity folks feel in terms of how they
would stack up:

https://grsecurity.net/compare.php

If you want to try a regular 4.8 kernel either in dom0 or in a vm,
there's already one (4.8.12) in the Qubes unstable repository that you
can play with right now.

[Reg Tiangha]

On 12/10/2016 05:40 PM, Reg Tiangha wrote:

> Well, I'm currently in the middle of compiling it; haven't had to
> compile a kernel since my Gentoo days and I've forgotten how long it
> used to take. One piece of advice at this point: If you're using a fresh
> template, you'll definitely want to allocate more space to /home; the
> default 2GB isn't enough. I've doubled it to 4G and am keeping an eye on
> how much it grows. I can't remember how much disk space it used to take
> to compile an old Gentoo kernel so I'll be babysitting this one until
> it's done.
>
> In the meantime:
>
> 1) I would assume it would work; their instructions imply Debian 7+, but
> I suppose the only way to find out if it would work on a Debian 9
> template under Qubes would be to compile it and find out.
>
> 2) I've never used a Gresecurity kernel before (I almost did under
> Gentoo but at the time, the kernel they were using was an older one
> lacking a driver I needed so I ended up going with the vanilla sources
> instead), but here's how the gresecurity folks feel in terms of how they
> would stack up:
>
> https://grsecurity.net/compare.php
>
> If you want to try a regular 4.8 kernel either in dom0 or in a vm,
> there's already one (4.8.12) in the Qubes unstable repository that you
> can play with right now.
>
>

Just to update: It took about 5 hours to compile on my quad core 2.2GHz
Intel i7-2720M, so your millage may vary. The coldkernel compile
directory was about 3.2GB large after compilation, so you'll want to
make sure your /home directory has at least 4.0GB free before attempting
this. I used a fresh stock but updated debian-8 template vm to compile
this on.

The machine boots, which is the good news. The bad news is that qrexec
momentarily connects and then disconnects (i.e. the light turns green in
Qubes VM Manager for about a second once it finishes booting, and then
immediately turns yellow).

However, I didn't follow the instructions exactly as git said there was
no tag marked coldkernel-0.9a-4.8.12 when I typed out the verify-tag and
checkout commands as written. So I switched to the 0.9a branch instead
using git checkout 0.9a. That said, I'm essentially a git newb. Did I do
that part right or should I have done something else?

Anyway, I don't know much about gresecurity and paxctl either, so while
I wait for someone else to post some success on this, there is lots to
read up on in the meantime.

[raah...@gmail.com]

I'e had the problem of being green for a second and immediately turning yellow with installing just a regular distro debian kernel with pvgrub. In other words the gui is not working, but you can verify the kernel is booting properly with using sudo xl console VMNAME in dom0. Marek says you have to compile the u2mfn module and make sure its installed. But I have no idea how to do this.

[raah...@gmail.com]

On Saturday, December 10, 2016 at 10:24:47 PM UTC-5, Reg Tiangha wrote:

scroll down to installing kernel in debian vm https://www.qubes-os.org/doc/managing-vm-kernel/

[Reg Tiangha]

On 12/10/2016 08:42 PM, raah...@gmail.com
wrote:

>
> I'e had the problem of being green for a second and immediately turning yellow with installing just a regular distro debian kernel with pvgrub. In other words the gui is not working, but you can verify the kernel is booting properly with using sudo xl console VMNAME in dom0. Marek says you have to compile the u2mfn module and make sure its installed. But I have no idea how to do this.
>

Ah, I see!

OK, I think I may know what *might* have happened.

I think the make script did try to do what it said in the instructions
here when it started to install the generated deb packages:

https://www.qubes-os.org/doc/managing-vm-kernel/

but I remember it throwing an error somewhere along the line saying it
couldn't find the kernel header files. But *that* was because it was
installing the kernel header file deb package afterwards; or in other
words, the gresecurity kernel header package wasn't installed yet at
that point in time. So maybe a step was missed.

That said, I now have a properly booting debian 8 template with a
gresecurity kernel. What you need to do is this:

After you follow the github instructions but before you reboot, run:

sudo dkms autoinstall -k 4.8.12-coldkernel-grsec-1
sudo update-initramfs -u
sudo update-grub2

which is essentially the final part of the "Installing kernel in Debian
VM" instructions. And then the machine should boot up fine when you
switch to the pvgrub2 kernel. Or, at least it did for me.

Thanks for the hint!!

[Reg Tiangha]

On 12/10/2016 09:10 PM, Reg Tiangha wrote:

> Ah, I see!
>
> OK, I think I may know what *might* have happened.
>
> I think the make script did try to do what it said in the instructions
> here when it started to install the generated deb packages:
>
> https://www.qubes-os.org/doc/managing-vm-kernel/
>
> but I remember it throwing an error somewhere along the line saying it
> couldn't find the kernel header files. But *that* was because it was
> installing the kernel header file deb package afterwards; or in other
> words, the gresecurity kernel header package wasn't installed yet at
> that point in time. So maybe a step was missed.
>
> That said, I now have a properly booting debian 8 template with a
> gresecurity kernel. What you need to do is this:
>
> After you follow the github instructions but before you reboot, run:
>
> sudo dkms autoinstall -k 4.8.12-coldkernel-grsec-1
> sudo update-initramfs -u
> sudo update-grub2
>
> which is essentially the final part of the "Installing kernel in Debian
> VM" instructions. And then the machine should boot up fine when you
> switch to the pvgrub2 kernel. Or, at least it did for me.
>
> Thanks for the hint!!
>

And it looks like in the last 7 hours, they've bumped the kernel up to
4.8.13, modified the Debian template instructions, and temporarily
pulled down the Fedora template instructions as well. So yeah, it's all
still in flux. Gonna go recompile that 4.8.13 kernel now...

[Reg Tiangha]

OK, the weekend is almost over and I can't spend much more time on this.
So I thought I'd just wrap up the results of my (light) testing:


GENERAL:

- You'll want at least 4GB free to build the kernel.

- I can't seem to get it to work with a DispVM. I set my dvm image to
use pvgrub2 as its kernel, but every time it launches a new DispVM, the
new machine reverts to using my default 4.8.12 kernel. Actually, it
seems to resort to using all default values for number of CPUs and RAM;
changing those values seem to have no effect on spawned machines.

- I couldn't figure out how to get the RBAC stuff to work. I wanted to
use grsecurity's gradm tool, but it would always fail at the
installation portion saying that /dev/grsec did not exist (which it
didn't). I don't know how to create that device so for now, I've
reverted to enabling Apparmor or SELinux depending on the template.
You'll need to pass those kernel instructions through the VM's grub
config file, though. You can easily do that by creating a
/etc/default/grub file and adding a GRUB_CMDLINE_LINUX="apparmor=1
security=apparmor" or "selinux=1 security=selinux" line to it (you can
also set some other grub options like GRUB_TIMEOUT=0). Those modules are
included in the coldkernel and everything seems to work fine together.


DEBIAN 8:

- Seems to work fine as per the instructions. You can save yourself a
bit of grief beforehand by editing the install-deps section of the
Makefile and swapping the order it installs the kernel header package
and the kernel image package so that the header package is installed
first. Else, follow my instructions above (substituting the correct
kernel version that becomes recent when you try this) before you shut
down the VM.


FEDORA:

- I tried it on a Fedora 24 template. The instructions were pulled down
from the GitHub readme, and for good reason: They don't work fully. The
Makefile will build a kernel image rpm and a kernel header rpm. However,
it neglected a kernel-devel package meaning that u2mfn module doesn't
get compiled at all. Personally, I would wait for the coldhak crew to
release updated Fedora instructions, but if you can't wait, here's what
you do (works as the time of this writing):

1) Install the Fedora packages as listed here (although there's probably
more than you need; I don't think you actually need the Development
Tools group, for example):

https://github.com/coldhakca/coldkernel/blob/6926736c830e994fc1e4f48df1a665ea78e29f94/README.md

2) Follow the Qubes Fedora instructions until just before the part about
making the grub config file.

3) Copy or move the linux-4.8.13 (or whatever version it is) directory
from the coldkernel directory to /lib/modules/4.8.13-coldkernel-grsec-1
and rename it to "build" as this is where the kernel source needed to
recompile the u2nfn module is expected to be found.

4) Recompile u2nfn by running:

sudo dkms autoinstall -k 4.8.13-coldkernel-grsec-1

5) Regenerate initramfs by running:

sudo dracut --regenerate-all --force

6) Create grub config file:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

7) Shutdown template


WHONIX:

- Doesn't seem to work at all by default. The light on Qubes VM Manager
stays yellow and qrexec never connects. Couldn't figure this one out
although I didn't spend much time on it.


So that's all I've been able to figure out thus far. I haven't put this
kernel through its paces yet through standard usage though; only tested
to see if machines would boot and such. grsecurity already has test
patches for 4.8.14 so there'll probably be another update soon for those
who want to hold off for a bit.

Also, for anyone out there who actually has experience with grsecurity
kernels, any hints, tips or tricks on stuff to do next would be
appreciated. Is running it on its own better protection than using a
stock kernel, or do you really need that RBAC stuff working alongside it?

Finally: What are the odds of this kernel being able to run in dom0 by
default? I wasn't brave enough to try considering I already burned an
entire weekend on this and can't afford to spend any extra time in case
something breaks. Would the procedure be the same as getting it to run
in a Fedora template? Or would something extra need to be done? I don't
know how the Qubes kernel differs from stock and coldkernel when it
comes to its patches. Otherwise, I'd be porting stock Fedora kernels to
use in dom0 for myself every time they push out a new one.

[Reg Tiangha]

Or maybe I was wrong about SELinux. I just realized that it's never been
active. Perhaps it wasn't compiled into the kernel after all. Apparmor
still works fine under Debian, though.

[raah...@gmail.com]

Thanks for all your info.

[Reg Tiangha]

On 12/11/2016 06:01 PM, raah...@gmail.com
wrote:

>
> Thanks for all your info.
>

A few last observations:

- If you run coldkernel on a NetVM or ProxyVM, *nothing* will be able to
connect behind it (which kind of sucks).
- Dropbox no longer launches and it keeps trying to download the daemon
every time you start it up. Ironically, there are no issues with
SpiderOAK or NextCloud, but those programs don't force you to download a
daemon after installation.
- coldkernel works in a usbVM with USB input proxy, however, it does
*not* work with mass storage device pass-through (which also sucks) and
it has the added effect of locking up Qubes VM Manager once you try as well.

Note that all of my sysVMs are running Fedora minimal templates; not
sure if using a Debian template would make a difference, but I would
suspect not. In the meantime, I've reverted all of my service VMs to use
normal kernels and am only running coldkernel on AppVMs.

I wonder if properly setting RBAC rules may help with some of the
issues? It'd be nice to be able to figure out how to get gradm working
in an AppVM. Does anyone know what the /dev/grsec device is or how to
create it?

[Reg Tiangha]

Looks like preliminary coldkernel support for Debian templates is now
official:

https://coldhak.ca/blog/2016/12/12/coldkernel-qubes-1.html

They fixed the makefile issue so the Debian instructions as written
should just work. They even enabled the RBAC driver in the kernel.config
file (if any of the coldhak team is out there reading this, thanks so
much! But if you really don't want it in your kernel, you can modify the
coldkernel.config file with CONFIG_GRKERNSEC_NO_RBAC=y ; and if you
really wanted SELinux in your kernel, theoretically you would add the
various SELinux kernel config options to this file as well; you can
Google for what those are although I haven't tried it myself).

The Fedora instructions are still pending for the reasons in the blog
post, but if people *really* want to try it on a FC template, I'll give
you my instructions from start to finish. I'll start with how to compile
it on an FC BuildVM, then how to install your rpms on other FC templates
without having to reinstall the entire build environment and compiling
each time. I used FC 24, but it should still work on FC 23.

First, the Build instructions:

1) On dom0:

sudo qubes-dom0-update grub2-xen

2) On FC TemplateVM (make sure /home has at least 4GB free):

a) Install support for booting from pvgrub2 kernels:

sudo dnf install qubes-kernel-vm-support

b) Install the dev environment:

sudo dnf install hmaccalc zlib-devel binutils-devel
elfutils-libelf-devel ncurses-devel gcc-plugin-devel wget git gnupg2 bc
gcc-c++ rpm-build

c) Optional: Install bison and flexx to compile gradm:

sudo dnf install bison flex


OPTIONAL: At this point, you can create an AppVM to do the actual
compiling, just make sure to save the rpms and u2mfn.ko kernel module
that you'll end up making. Otherwise, if this is the TemplateVM you
intend to also use later on in a different AppVM, then keep going.


3) Clone coldkernel from github:

wget "https://coldhak.ca/coldhak/keys/coldhak.asc" - O coldhak.asc

gpg --import coldhak.asc

git clone https://github.com/coldhakca/coldkernel

cd coldkernel

git verify-tag coldkernel-0.9a-4.8.13

git checkout tags/coldkernel-0.9a-4.8.13

4) Build coldkernel:

make qubes-guest

5) Now you'll have made two rpms. Install them:

sudo dnf install
kernel-headers-4.8.13_coldkernel_grsec_1-2.x86_64.rpm
kernel-4.8.13_coldkernel_grsec_1-2.x86_64.rpm


Now, this is the tricky part. You'll also need to compile the u2mfn.ko
kernel module, which isn't done by default because the coldkernel kernel
sources aren't installed by default. BUT a version of the kernel sources
still exists in your coldkernel directory so you can use that instead to
build it.


6) Symlink kernel source to where dkms can find it:

sudo ln -s /home/user/coldkernel/linux-4.8.13
/lib/modules/4.8.13-coldkernel-grsec-1/build

7) Build the u2mfn kernel module and rebuild initramfs:

sudo dkms autoinstall -k 4.8.13-coldkernel-grsec-1

sudo dracut --regenerate-all --force


It will compile the u2mfn kernel module and will place it in
/lib/modules/4.8.13-coldkernel-grsec-1/extra. IF YOU INTEND TO INSTALL
COLDKERNEL ON OTHER FC TEMPLATES, BACK THIS FILE UP!!

7b) Back up the u2mfn kernel module:

sudo cp /lib/modules/4.8.13-coldkernel-grsec-1/extra/u2mfn.ko
/home/user/coldkernel/


Now, you'll have your two rpms and the u2mfn kernel module in your
coldkernel directory. Save those elsewhere if you intend on installing
coldkernel on other machines (ex. Copy to another VM).


Continuing on, you'll want to install grsecurity's paxctld program:

8) Grab paxctld and verify it:

wget
https://grsecurity.net/paxctld/paxctld-systemd-1.2.1-1.x86_64.{rpm,rpm.sig}

gpg (or it might be gpg2) --homedir=.gnupg --verify
paxctld-systemd-1.2.1-1.x86_64.{rpm.sig,rpm}

9) Install and enable paxctld:

sudo dnf install paxctld-systemd-1.2.1-1.x86_64.rpm

sudo cp paxctld.conf /etc/paxctld.conf

sudo paxctld -d

sudo systemctl enable paxctld

10) Create grub config file. When you use the pvgrub2 kernel, it'll
ignore any qvm-prefs kernelopts you may have set. So you'll need to
create a /etc/default/grub file containing your kernelopts (ex.
GRUB_CMDLINE_LINUX="nopat"). Once done, run:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg


Shutdown the VM, and assuming you did this all on a Template VM, change
its kernel to pvgrub2 and start it up. If you did it on an AppVM, all of
the system changes will be erased, making it really important to have
backed up that u2mfn file.

If you want to install this on other FC templates without having to
recompile, make sure to back up the:

- kernel and kernel-header rpms

- the paxctld rpm and conf file

to another machine.


HOW TO INSTALL ON OTHER FC TEMPLATES:

1) Copy the linux and linux-headers rpm files as well as the u2mfn.ko
file to the FC template you want to install it on.

2) Install pvgrub2 support:

sudo dnf install qubes-kernel-vm-support grub2-tools (and bison and
flex if you'll be compiling gradm on it as well)

3) Install the coldkernel rpms:

sudo dnf install
kernel-headers-4.8.13_coldkernel_grsec_1-2.x86_64.rpm
kernel-4.8.13_coldkernel_grsec_1-2.x86_64.rp

4) Copy the u2mfn kernel module to the kernel module's directory:

sudo cp u2mfn.ko /lib/modules/4.8.13-coldkernel-grsec-1/extra/

5) Rebuild kernel module map:

sudo depmod -v 4.8.13-coldkernel-grsec-1

6) Regenerate initramfs:

sudo dracut --regenerate-all --force

7) Install grub (don't forget to make your /etc/default/grub file if
you need to pass additional kernel options):

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

8) Install paxctld as per the instructions above.


That should do it. Once you boot into the gresecurity environment,
you'll then be able to compile gradm and use that to profile your machine.

Unfortunately, I was hoping I could do that to profile ServiceVMs, but I
still have the same issues as before, mainly any AppVM set to use a
coldkernel-enabled Proxy or NetVM refuses to start (libxenlight throws
an error), and Qubes VM Manager crashes any time you try to attach a USB
device to another VM using a sys-usb VM with coldkernel. I don't know
how to fix this; I tried tracing things manually and using paxctl to
disable memory protections on various qubes binaries (FYI: running
paxctl -cm on the dropbox binary in your home directory allows it to
work again), but it was like trying to find a needle in a haystack. I
don't know enough about the Qubes architecture to even know where to begin.

But hopefully my instructions will allow other people to try it for
themselves and help test this stuff out. Good luck, and thanks again to
the coldhak crew for all their hard work!!

[Reg Tiangha]

Ah crap. Step 2 should read:

sudo dnf install qubes-kernel-vm-support grub2-common

Otherwise, you won't be able to install grub on the vm. Don't forget it!

[Foppe de Haan]

To clarify: this has to be done in every template in which you want to use this? Or can I just copy the whole dir after compiling (seems necessary for the make install-deb step?), install whatever packages I need to perform the post-build steps, and perform those?
(thinking of D8-template, D9-, Whonix-gw/ws)

[Reg Tiangha]

On 12/14/2016 06:03 AM, Foppe de Haan wrote:
> To clarify: this has to be done in every template in which you want to use this? Or can I just copy the whole dir after compiling (seems necessary for the make install-deb step?), install whatever packages I need to perform the post-build steps, and perform those?
> (thinking of D8-template, D9-, Whonix-gw/ws)
>

For Debian systems, you only need to follow the coldhak instructions
once to create the kernel deb packages. If you want to take your work
and install it on other Debian templates without having to set up the
dev environment again, just install the qubes-kernel-vm-support and
grub2-common packages on them (and you'll probably want paxctl too to
help with managing the pax stuff until you've figured out what you want
your pax ruleset to look like), then copy over the linux-headers and
linux-image packages that you had just made and install them in that
order (headers first, then image). Install the firmware packages only if
you need them. That seemed to work for me.

Make sure to clone the templates you want to try this on for testing
purposes if you don't want to lose your originals; for example, by
default, you won't be able to connect to a Whonix template running
coldkernel as qrexec won't start up properly (but if you switch back to
a normal kernel, it'll work fine again). And if you enable this on a
service vm like sys-net, no machine configured to use it as a net vm
will start up. I don't know how to troubleshoot this or fix this, so if
anyone out there figures that part out, please share.

Don't forget to follow the rest of the coldhak instructions to install
and configure paxctld, set up grub, and to add the relevant grsecurity
groups!

sudo groupadd -g 9001 grsecproc
sudo groupadd -g 9002 tpeuntrusted
sudo groupadd -g 9003 denysockets

[Foppe de Haan]

Thanks. When building this in d9-t, I get 'recipe failed' "Error 2" pretty soon after starting the build process. No idea why, as there is no log, and I don't see an option to provide one (or to provide verbose output).

[rtia...@gmail.com]

On Wednesday, December 14, 2016 at 2:22:57 PM UTC-7, Foppe de Haan wrote:

> Thanks. When building this in d9-t, I get 'recipe failed' "Error 2" pretty soon after starting the build process. No idea why, as there is no log, and I don't see an option to provide one (or to provide verbose output).

Hmm, I haven't had time to try it under Debian 9 yet and might not for a while.

In the meantime, peeking at their build script, what you can try to do (haven't tried it myself though) is to comment out the spinner code around the build_kernel part and the '> /dev/null 2>&1' part in the build.sh file and perhaps that would throw the verbose compile output to the terminal to give some hints on where or why it fails.

So find this section in build.sh:

start_spinner "Building coldkernel..."
build_kernel > /dev/null 2>&1
stop_spinner $?

And change it to be

build_kernel

and try to compile (make qubes-guest) again. If the output isn't enough to give any hints, you may need to run 'make clean' to get rid of anything that was pre-compiled and to start again from scratch.

[Foppe de Haan]

Thanks for that hint. Seems there was an issue with gcc not supporting plugins, even though supposedly all the required packages were installed and updated. Worked around it, it's building now.

[Reg Tiangha]

Glad to hear. For future reference, what exactly did you have to do? Was it

sudo apt install gcc-6-plugin-dev

or something else?

[Foppe de Haan]

Okay, stuck again. Downloaded a fresh d8 template, updated it and built kernel again to have a clean slate to work from. Everything builds, no error beside the expected ones afterwards. But when i switch to pvgrub2 and boot, it shuts down seconds after boot attempt. Full log attached.

.[30m.[47mWelcome to GRUB!

.[37m.[40m.[37m.[40m.[37m.[40merror: no such device: /boot/xen/pvboot-x86_64.elf.

Reading (xen/xvda/boot/grub/grub.cfg

.[H.[J.[1;1Herror: file `/boot/grub/fonts/unicode.pf2' not found.

error: no suitable video mode found.

[Reg Tiangha]

On 12/15/2016 05:00 AM, Foppe de Haan wrote:
> Okay, stuck again. Downloaded a fresh d8 template, updated it and built kernel again to have a clean slate to work from. Everything builds, no error beside the expected ones afterwards. But when i switch to pvgrub2 and boot, it shuts down seconds after boot attempt. Full log attached.
>

> ..[30m.[47mWelcome to GRUB!
>
>
> ..[37m.[40m.[37m.[40m.[37m.[40merror: no such device: /boot/xen/pvboot-x86_64.elf.
>
> Reading (xen/xvda/boot/grub/grub.cfg
>
> ..[H.[J.[1;1Herror: file `/boot/grub/fonts/unicode.pf2' not found.

>
> error: no suitable video mode found.
>

I can't replicate, but none of my systems have UEFI or if they do, I'm
using legacy boot because Qubes wouldn't install if UEFI mode was enabled.

That said, if it worked for you before, it should still work now.

Are you sure you set up grub correctly after the compile?

sudo mkdir /boot/grub
sudo update-grub2

The only other thing I can think of is to switch to a regular kernel,
try the two grub commands again and see if any errors pop up, and if
not, switch back to pvgrub2 and see if it boots again.

[Reg Tiangha]

Oh, wait. When you said you downloaded a fresh Debian 8 template and
updated, do you mean that you updated it to Debian 9 rather than just
install the regular Debian 8 updates? I'm running another test system
based on Stretch on bare metal, and I think there's something buggy in
one of the packages that was released recently (maybe X?) as it's been
locking up a lot for reason I can't figure out. I'm hoping I can just
wait it out and it'll fix itself later in another update. It could be
related to your problem if that's the case?

And I just realized there was more to the log that you attached. Still
can't figure out what caused the kernel panic from looking at it, sorry.

[Foppe de Haan]

No, I mean running 8/stable, no other repos active. Would dom0 running uefi matter to the VM

As for the rest, I've rerun every step a few times, so I can't imagine I missed anything. update-grub2 gave me the message below, but according to the qubes howto page, those errors don't matter.

Anyway, thanks for your replies / time, I guess I'll poke around the log etc a bit more.

-----
user@debian-8:/boot$ sudo update-grub2
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.8.13-coldkernel-grsec-1
Found initrd image: /boot/initrd.img-4.8.13-coldkernel-grsec-1
/usr/sbin/grub-probe: error: cannot find a GRUB drive for /dev/mapper/dmroot. Check your device.map.
/usr/sbin/grub-probe: error: cannot find a GRUB drive for /dev/mapper/dmroot. Check your device.map.
/usr/sbin/grub-probe: error: cannot find a GRUB drive for /dev/mapper/dmroot. Check your device.map.
No volume groups found
done

[Reg Tiangha]

Yeah, those particular errors don't matter.

I guess later today, I'll try reinstalling the debian-8-template from
the yum repository and try to build the kernel again from scratch, just
to make sure. What I've been doing with every Qubes machine of mine is
installing the templates from the DVD or the repository, updating it,
and then immediately cloning it and working off of the clone. That way,
all of my original template rpms are still pristine.

However, I don't think there's any difference between what's on the 3.2
iso and what's the latest debian 8 template from the repository so I
don't know if that'll do much in my testing; I think they're the same
version? I'm sure a dev will chime in if that's not the case.

Your problem is very strange, especially if coldkernel had already
worked before.

[Reg Tiangha]

The last thing I can think of off the top of my head at the moment is
that every Qubes VM by default outside of sys-net have "nopat" set as a
kernel option. If you use pvgrub2 though, it ignores any of the
kernelopts options that are set by qvm-prefs.

To make your coldkernel template vm boot with the nopat option, you have
to make an /etc/default/grub file that contains
"GRUB_CMDLINE_LINUX="nopat" (along with any other kernel options you
would normally pass through) before running update-grub2. So if you
haven't done that yet, maybe try making that file and updating grub and
try again? That's the only thing I can think of at this point.

[Colin Childs]

Hi everyone,

Sorry for not getting on this list sooner, however it looks like testing
of coldkernel on Debian is largely going well! I see the most recent
issue from foppe, and will be attempting to reproduce later this evening.

If you run into issues that require coldhak attention, please do not
hesitate to open tickets at
https://github.com/coldhakca/coldkernel/issues, or email us directly at
con...@coldhak.ca.

Thanks, and happy testing!

--
Colin Childs
Coldhak
https://coldhak.ca
Twitter: @coldhakca

[Foppe de Haan]

Thanks. If you want/need more information than that log contains, please tell me, I still have the offending templateVM :)

[Foppe de Haan]

I also built the fedora kernel according to Reg's recipe, same issue.
Comparing boot logs, the problem seems to start here:

[ 0.765662] BUG: unable to handle kernel paging request at ffff87ff95a17300
[ 0.765671] IP: [<ffffffff81316cb9>] delay_mwaitx+0x49/0x90
[ 0.765682] PGD 0
[ 0.765688] Oops: 0000 [#1] SMP
[ 0.765693] Modules linked in:
[ 0.765701] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.8.13-coldkernel-grsec-1 #1
[ 0.765709] task: ffff8800136bf540 task.stack: ffffc90000e38000
[ 0.765714] RIP: e030:[<ffffffff81316cb9>] [<ffffffff81316cb9>] delay_mwaitx+0x49/0x90
[ 0.765724] RSP: e02b:ffffc90000e3be50 EFLAGS: 00010087
[ 0.765729] RAX: ffff87ff95a17300 RBX: 0000000000000001 RCX: 0000000000000000
[ 0.765735] RDX: 0000000000000000 RSI: 00039262adda8fdb RDI: 000000000002a4e4
[ 0.765740] RBP: ffffffff81c17300 R08: 00000000ffffffff R09: 0000000000000000
[ 0.765745] R10: 0000000000000002 R11: 000000000000000f R12: 0000000000000200
[ 0.765749] R13: ffffc90000e3bea7 R14: ffffffff824055e8 R15: 607e4ce58fa6249c
[ 0.765758] FS: 0000000000000000(0000) GS:ffff880013e00000(0000) knlGS:0000000000000000
[ 0.765765] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.765770] CR2: ffff87ff95a17300 CR3: 00000000020c2000 CR4: 0000000000040660
[ 0.765775] Stack:
[ 0.765779] 0000000000000001 00000000000010d1 ffffffff814956a5 95fa1589597478a1
[ 0.765788] ffffffff814960d0 95fa1589597478a1 ffffc90000e3bec8 06937a89d974aa7d
[ 0.765797] 00000000ffffffed ffffffff8236af42 df7e4ce58fa6249c 0000000000000000
[ 0.765807] Call Trace:
[ 0.765815] [<ffffffff814956a5>] ? i8042_wait_write+0x25/0x70
[ 0.765822] [<ffffffff814960d0>] ? i8042_command+0x30/0x80
[ 0.765829] [<ffffffff8236af42>] ? i8042_init+0x606/0x6f8
[ 0.765835] [<ffffffff8236a93c>] ? i8042_probe+0xa41/0xa41
[ 0.765842] [<ffffffff8100219d>] ? do_one_initcall+0x4d/0x170
[ 0.765849] [<ffffffff822f3f0d>] ? kernel_init_freeable+0x202/0x2ff
[ 0.765856] [<ffffffff81623f65>] ? kernel_init+0x5/0x118
[ 0.765861] [<ffffffff8163c6fe>] ? ret_from_fork+0x1e/0x40
[ 0.765867] [<ffffffff81623f60>] ? rest_init+0x88/0x88
[ 0.765871] Code: 41 b8 ff ff ff ff 48 09 c6 41 ba 02 00 00 00 eb 09 48 29 c6 48 01 f7 48 89 c6 48 89 e8 65 48 03 05 25 25 cf 7e 4c 89 c9 4c 89 ca <0f> 01 fa 4c 39 c7 4c 89 c3 4c 89 d8 48 0f 46 df 4c 89 d1 0f 01
[ 0.765920] RIP [<ffffffff81316cb9>] delay_mwaitx+0x49/0x90
[ 0.765927] RSP <ffffc90000e3be50>
[ 0.765931] CR2: ffff87ff95a17300
[ 0.765939] ---[ end trace 84bc057c0ef01aab ]---
[ 0.765946] Kernel panic - not syncing: grsec: halting the system due to suspicious kernel crash caused by root

(Same error in both VMs.)

[Reg Tiangha]

Did you try passing along the "nopat" kernel option through grub to see
if that made a difference?

[Foppe de Haan]

Yes, but it didn't make a difference.

[Reg Tiangha]

Does it work when the VM has the entire compile environment? As in, if
you follow the coldhak instructions directly, does it work? Or does it
never work?

[Foppe de Haan]

Never. I compiled the d8/9 kernels in their respective (cloned) templates, but got nowhere.

[Colin Childs]

Hi everyone,

Please be aware that there are a number of issues with using coldkernel
in the Fedora templates currently. Our goal is to push out 0.9b over the
holidays to address this.

For the time being, the Fedora instructions have been entirely removed
from the master git branch, and progress will be made on the 0.9b
branch, along with Whonix support.

Happy Holidays!

[Doug Hill]

Colin Childs:

> Hi everyone,
>
> Sorry for not getting on this list sooner, however it looks like testing
> of coldkernel on Debian is largely going well! I see the most recent
> issue from foppe, and will be attempting to reproduce later this evening.
>
> If you run into issues that require coldhak attention, please do not
> hesitate to open tickets at
> https://github.com/coldhakca/coldkernel/issues, or email us directly at
> con...@coldhak.ca.
>
> Thanks, and happy testing!
>

Hi,

Running into a problem when runnning "make qubes-guest" on a stock
debian-8 template.

Below are the last few lines of the output. Thanks!

CC [M] fs/xfs/xfs_buf_item.o
CC [M] fs/xfs/xfs_extfree_item.o
CC [M] fs/xfs/xfs_icreate_item.o
CC [M] fs/xfs/xfs_inode_item.o
CC [M] fs/xfs/xfs_rmap_item.o
CC [M] fs/xfs/xfs_log_recover.o
CC [M] fs/xfs/xfs_trans_ail.o
CC [M] fs/xfs/xfs_trans_buf.o
CC [M] fs/xfs/xfs_trans_extfree.o
CC [M] fs/xfs/xfs_trans_inode.o
CC [M] fs/xfs/xfs_trans_rmap.o
CC [M] fs/xfs/xfs_sysctl.o
CC [M] fs/xfs/xfs_ioctl32.o
LD [M] fs/xfs/xfs.o
LD fs/built-in.o
scripts/package/Makefile:97: recipe for target 'bindeb-pkg' failed
make[2]: *** [bindeb-pkg] Error 2
Makefile:1317: recipe for target 'bindeb-pkg' failed
make[1]: *** [bindeb-pkg] Error 2
make[1]: Leaving directory '/home/user/coldkernel/linux-4.8.13'
Makefile:61: recipe for target 'qubes-guest' failed
make: *** [qubes-guest] Error 2
user@debian-8-coldkernel:~/coldkernel$

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Relevant error is probably earlier. I guess it's about disk space - it
require 4GB or so to build.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJYVrvMAAoJENuP0xzK19csBjYH/jajqTdkc7/eedgVVFCqT4sg
OdIkQkcJgGv2tEwMnPmMcA/U8CMc7sRN51UV/hr9b0IM2VhZy21oPgtXpf6hXdGD
1Z4ZEtl6Jo12UM07jAbKbOGGnJJMcU5Sy+PalbI2/365zefX7ALkzlOQpMA6y0Zv
e2y7zBIn5SDIusLkBah2juhv4SGVFVSvmUZ+xgmScun2qHfa2YDbLv6oS/aUAOa5
cijINdbB2flz9mIMqhaIuOty0330LbMbKJ0vWXni/TUgEY2ZCWyUN0XO2TcU8I9X
TStuyGfuE3sah2/CpO9Kx7QNNvCzVzTpfy2u/YKVEP0Mbjrudw7op6y2YqewVM8=
=UsAL
-----END PGP SIGNATURE-----

[Doug Hill]

Marek Marczykowski-Górecki:

Thanks for the tip, Marek... The process moved a bit farther along...

I believe I hit the gcc issue that was mentioned eariler; a few
meaningful-looking snippets are below. Any thoughts are
appreciated!


<snip>

CC [M] fs/ncpfs/ioctl.o
CC drivers/acpi/acpica/utobject.o
CC drivers/acpi/acpica/utosi.o
CC drivers/acpi/acpica/utownerid.o
CC [M] fs/ncpfs/mmap.o
*** WARNING *** there are active plugins, do not report this as a bug
unless you can reproduce it without enabling any plugins.
Event | Plugins
PLUGIN_FINISH_TYPE | randomize_layout_plugin constify_plugin
PLUGIN_FINISH_DECL | randomize_layout_plugin
PLUGIN_FINISH_UNIT | rap_plugin
PLUGIN_ATTRIBUTES | randomize_layout_plugin
latent_entropy_plugin size_overflow_plugin constify_plugin
PLUGIN_START_UNIT | latent_entropy_plugin
size_overflow_plugin rap_plugin constify_plugin
PLUGIN_ALL_IPA_PASSES_START | randomize_layout_plugin rap_plugin
constify_plugin
In file included from ./include/linux/seqlock.h:35:0,
from ./include/linux/time.h:5,
from ./include/linux/stat.h:18,
from fs/ncpfs/mmap.c:9:
./include/linux/spinlock.h:58:0: internal compiler error: Segmentation
fault
#include <asm/barrier.h>
^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.9/README.Bugs> for instructions.
CC drivers/acpi/acpica/utpredef.o
CC drivers/acpi/acpica/utresrc.o
The bug is not reproducible, so it is likely a hardware or OS problem.
scripts/Makefile.build:289: recipe for target 'fs/ncpfs/mmap.o' failed
make[5]: *** [fs/ncpfs/mmap.o] Error 1
scripts/Makefile.build:440: recipe for target 'fs/ncpfs' failed
make[4]: *** [fs/ncpfs] Error 2
Makefile:972: recipe for target 'fs' failed
make[3]: *** [fs] Error 2
make[3]: *** Waiting for unfinished jobs....
CC drivers/acpi/apei/apei-base.o

[Reg Tiangha]

> ../include/linux/spinlock.h:58:0: internal compiler error: Segmentation

> fault
> #include <asm/barrier.h>
> ^
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <file:///usr/share/doc/gcc-4.9/README.Bugs> for instructions.
> CC drivers/acpi/acpica/utpredef.o
> CC drivers/acpi/acpica/utresrc.o
> The bug is not reproducible, so it is likely a hardware or OS problem.
> scripts/Makefile.build:289: recipe for target 'fs/ncpfs/mmap.o' failed
> make[5]: *** [fs/ncpfs/mmap.o] Error 1
> scripts/Makefile.build:440: recipe for target 'fs/ncpfs' failed
> make[4]: *** [fs/ncpfs] Error 2
> Makefile:972: recipe for target 'fs' failed
> make[3]: *** [fs] Error 2
> make[3]: *** Waiting for unfinished jobs....
> CC drivers/acpi/apei/apei-base.o
>

Did you install the correct gcc plugin for the Debian template you're
trying to compile this on? According to the coldhak instructions, it'll
either be one of these:

sudo apt install gcc-4.9-plugin-dev (for GCC 4.9)
sudo apt install gcc-5-plugin-dev (for GCC 5.x)
sudo apt install gcc-6-plugin-dev (for GCC 6.x)

Stock Debian-8 should be using the 4.9 version unless you have
previously upgraded gcc on that template.

[Doug Hill]

Reg Tiangha:

Yes, gcc looks like the correct verion (I'm on a stock debian-8 template):

user@debian-8-coldkernel:~$ sudo apt install gcc-4.9-plugin-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
gcc-4.9-plugin-dev is already the newest version.

Thanks!

[黃于軒]

2016-12-14 23:33 GMT+08:00 Reg Tiangha <r...@reginaldtiangha.com>:

On 12/14/2016 06:03 AM, Foppe de Haan wrote:
> To clarify: this has to be done in every template in which you want to use this? Or can I just copy the whole dir after compiling (seems necessary for the make install-deb step?), install whatever packages I need to perform the post-build steps, and perform those?
> (thinking of D8-template, D9-, Whonix-gw/ws)
>
For Debian systems, you only need to follow the coldhak instructions
once to create the kernel deb packages. If you want to take your work
and install it on other Debian templates without having to set up the
dev environment again, just install the qubes-kernel-vm-support and
grub2-common packages on them (and you'll probably want paxctl too to
help with managing the pax stuff until you've figured out what you want
your pax ruleset to look like), then copy over the linux-headers and
linux-image packages that you had just made and install them in that
order (headers first, then image). Install the firmware packages only if
you need them. That seemed to work for me.

Make sure to clone the templates you want to try this on for testing
purposes if you don't want to lose your originals; for example, by
default, you won't be able to connect to a Whonix template running
coldkernel as qrexec won't start up properly (but if you switch back to
a normal kernel, it'll work fine again). And if you enable this on a
service vm like sys-net, no machine configured to use it as a net vm
will start up. I don't know how to troubleshoot this or fix this, so if
anyone out there figures that part out, please share.

​​For those of you who are trying to get the kernel working on Whonix, just install busybox and update the initramfs. Seems to be a missing dependency of qubes-kernel-vm-support.
​

 

Don't forget to follow the rest of the coldhak instructions to install
and configure paxctld, set up grub, and to add the relevant grsecurity
groups!

sudo groupadd -g 9001 grsecproc
sudo groupadd -g 9002 tpeuntrusted
sudo groupadd -g 9003 denysockets

​​

--WillyPillow

​

​

 

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Thu, Dec 15, 2016 at 03:11:29PM -0600, Colin Childs wrote:
> Hi everyone,
>
> Sorry for not getting on this list sooner, however it looks like testing
> of coldkernel on Debian is largely going well! I see the most recent
> issue from foppe, and will be attempting to reproduce later this evening.
>
> If you run into issues that require coldhak attention, please do not
> hesitate to open tickets at
> https://github.com/coldhakca/coldkernel/issues, or email us directly at
> con...@coldhak.ca.
>
> Thanks, and happy testing!

What are the plans for next stages here? I guess fixing Fedora support,
right?

What about binary packages in general: I've heard there are some
benefits from compiling the grsec-enabled kernel yourself, as some parts
are randomized compile-time. Is that true? How much benefit it gives?

Anyway, I think in the end we need some packages in the repository for
this.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJYeQymAAoJENuP0xzK19csj0cH/jh4eCtY4XoZgTd06EE+n3j6
jBi6SmvafBAewpJkTjpRM4l8OrybuBJ/7l32LkyEtquZCaZWWxZo+sRCMm5N2stc
bjYUrROaOYmXbh7T0cwH4L9uRjgZda0IUGlGcA0324TYtLR9VUds4fncH8c/h7lE
kmNB3xX8x8KyTWH1v19dtoPyay20626eJP32qzeoDptcc0cyfpOKDZR5YNmf3b/K
SZXNz2O10rbpBK+odtfY/VAHQqD3P6TKGeTKAF7WBeXHLqOjB6CBXjN/Aj9p9X94
l/xeXIW+/EWwZtCBmgcRjVcUVYXVbHdGYTrS1OArRa9KSchBY/ENBedYRCs8GBs=
=Rk3u
-----END PGP SIGNATURE-----

[Colin Childs]

On 13/01/17 11:21 AM, Marek Marczykowski-Górecki wrote:
> On Thu, Dec 15, 2016 at 03:11:29PM -0600, Colin Childs wrote:
>> Hi everyone,
>
>> Sorry for not getting on this list sooner, however it looks like testing
>> of coldkernel on Debian is largely going well! I see the most recent
>> issue from foppe, and will be attempting to reproduce later this evening.
>
>> If you run into issues that require coldhak attention, please do not
>> hesitate to open tickets at
>> https://github.com/coldhakca/coldkernel/issues, or email us directly at
>> con...@coldhak.ca.
>
>> Thanks, and happy testing!
>
> What are the plans for next stages here? I guess fixing Fedora support,
> right?
>
> What about binary packages in general: I've heard there are some
> benefits from compiling the grsec-enabled kernel yourself, as some parts
> are randomized compile-time. Is that true? How much benefit it gives?
>
> Anyway, I think in the end we need some packages in the repository for
> this.
>
>

Hi,

Please see
https://github.com/coldhakca/coldkernel/issues?q=is%3Aissue+is%3Aopen+label%3Aqubes
for the next steps along, with their planned release versions. We are
currently planning to shit 0.9b within the next week.

0.9a (the current release) was not released with Fedora support, and
this was pulled from the README before the release was cut. The 0.9b
release will be focused on Whonix as well as Fedora, however Whonix is
currently taking priority. The goal is to push out both Whonix and
Fedora support with 0.9b, however if Fedora support looks like it will
take considerably longer, it will be bumped to 0.9c.

For providing binary packages, our goal is to offer grsec enabled
binaries for Debian. Offering pre-built Fedora binaries is not currently
in the roadmap, however it could potentially be added down the line.

There are some protections that come with compiling the kernel by hand,
such as an actually random/functional GRKERNSEC_RANDSTRUCT[1].

[1]:
https://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity_and_PaX_Configuration_Options

[Colin Childs]

On 13/01/17 11:52 AM, Colin Childs wrote:
> On 13/01/17 11:21 AM, Marek Marczykowski-Górecki wrote:

>> On Thu, Dec 15, 2016 at 03:11:29PM -0600, Colin Childs wrote:
>>> Hi everyone,
>>
>>> Sorry for not getting on this list sooner, however it looks like testing
>>> of coldkernel on Debian is largely going well! I see the most recent
>>> issue from foppe, and will be attempting to reproduce later this evening.
>>
>>> If you run into issues that require coldhak attention, please do not
>>> hesitate to open tickets at
>>> https://github.com/coldhakca/coldkernel/issues, or email us directly at
>>> con...@coldhak.ca.
>>
>>> Thanks, and happy testing!
>>
>> What are the plans for next stages here? I guess fixing Fedora support,
>> right?
>>
>> What about binary packages in general: I've heard there are some
>> benefits from compiling the grsec-enabled kernel yourself, as some parts
>> are randomized compile-time. Is that true? How much benefit it gives?
>>
>> Anyway, I think in the end we need some packages in the repository for
>> this.
>>
>>

> Hi,
>
> Please see
> https://github.com/coldhakca/coldkernel/issues?q=is%3Aissue+is%3Aopen+label%3Aqubes
> for the next steps along, with their planned release versions. We are
> currently planning to shit 0.9b within the next week.
>
> 0.9a (the current release) was not released with Fedora support, and
> this was pulled from the README before the release was cut. The 0.9b
> release will be focused on Whonix as well as Fedora, however Whonix is
> currently taking priority. The goal is to push out both Whonix and
> Fedora support with 0.9b, however if Fedora support looks like it will
> take considerably longer, it will be bumped to 0.9c.
>
> For providing binary packages, our goal is to offer grsec enabled
> binaries for Debian. Offering pre-built Fedora binaries is not currently
> in the roadmap, however it could potentially be added down the line.
>
> There are some protections that come with compiling the kernel by hand,
> such as an actually random/functional GRKERNSEC_RANDSTRUCT[1].
>
> [1]:
> https://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity_and_PaX_Configuration_Options
>

Hi everyone,

Sorry about the really unfortunate typo/correction in the first
paragraph of my previous email.

I hope you all have a nice weekend!

[raah...@gmail.com]

you have to update it alot thats the thing. I think that is too much time for most repos so most people prolly always end up compiling it themselves to have the latest version.

[Антон Чехов]

Hello,

I am testing coldkernel and I have a few questions. Does or should it work with a vpn gateway? Do I have to change some config file or special permissions?
I did not use grsec much in the past so I am in the process of learning.
I could connect to my coldkernel appvm via vpn gateway after freshly compiling and starting the appvm. After reboot none of my coldkernel appvm is connecting to the internet via vpn gateway anymore but connecting to clearnet without a proxyvm.

[Reg Tiangha]

To get the coldkernel working properly with net/proxyVMs and usbVMs, you
need to add at least two more drivers to the kernel's .config file:

CONFIG_XEN_BLKDEV_BACKEND=m
CONFIG_XEN_NETDEV_BACKEND=m

The easiest way to do that is to add those two lines to the
coldkernel.config file *before* running "make qubes-guest"

[Антон Чехов]

Thanks for your answer. I added the lines to the coldkernel.config and compiled the kernel again (in the existing coldkernel-template). Don't know if that's a mistake? Upon starting an appVM with coldkernel the VM pauses (yellow) and the log is giving me loads of this:
"grsec: denied priority change of process rtkit-daemon:1115 (...)"
This applies to the templateVM as well so I am asking myself what's best practice of updating the coldkernel template?
It is obvious that I don't know my way around grsec and I have to do more research into how to do things.

[Reg Tiangha]

On 2017-01-19 9:18 AM, Антон Чехов wrote:
> On Wednesday, January 18, 2017 at 5:35:53 PM UTC+1, Reg Tiangha wrote:
>> On 2017-01-18 7:14 AM, Антон Чехов wrote:
>>> Hello,
>>>
>>> I am testing coldkernel and I have a few questions. Does or should it work with a vpn gateway? Do I have to change some config file or special permissions?

>>> I did not use grsec much in the past so I am in the process of learning..

>>> I could connect to my coldkernel appvm via vpn gateway after freshly compiling and starting the appvm. After reboot none of my coldkernel appvm is connecting to the internet via vpn gateway anymore but connecting to clearnet without a proxyvm.
>>>
>>
>> To get the coldkernel working properly with net/proxyVMs and usbVMs, you
>> need to add at least two more drivers to the kernel's .config file:
>>
>> CONFIG_XEN_BLKDEV_BACKEND=m
>> CONFIG_XEN_NETDEV_BACKEND=m
>>
>> The easiest way to do that is to add those two lines to the
>> coldkernel.config file *before* running "make qubes-guest"
>
> Thanks for your answer. I added the lines to the coldkernel.config and compiled the kernel again (in the existing coldkernel-template). Don't know if that's a mistake? Upon starting an appVM with coldkernel the VM pauses (yellow) and the log is giving me loads of this:
> "grsec: denied priority change of process rtkit-daemon:1115 (...)"
> This applies to the templateVM as well so I am asking myself what's best practice of updating the coldkernel template?
> It is obvious that I don't know my way around grsec and I have to do more research into how to do things.
>

Yeah, I get that too. I think you can ignore it in the sense that the
VMs will still work despite the error, although I don't know if there
are really any adverse effects.

I haven't tried to fix it myself, though, but I assume a PAX exception
would need to be set for the daemon in order to "fix" it (not hard to
do, but I lack the time to troubleshoot at the moment).

What the rtkit-daemon is:

"RealtimeKit is a D-Bus system service that changes the
scheduling policy of user processes/threads to SCHED_RR (i.e. realtime
scheduling mode) on request. It is intended to be used as a secure
mechanism to allow real-time scheduling to be used by normal user
processes."

I don't know how it affects the operations of VMs though and whether or
not it being denied by the grsec kernel is actually a big deal.

[Антон Чехов]

But I can't start neither the AppVm nor the template anymore so it is kind of a big deal, don't you think?
That's why I am asking how to update.
Do I have to start over from scratch or should it work from the template?
Apparently something went wrong although everything looked fine during the process of compiling.

[Reg Tiangha]

My templates and AppVMs start up just fine despite having that error.
What do you mean when you say yours don't? As in, they don't start up at
all, or they do but the indicator on Qubes Manager for that VM is yellow
instead of green and never changes even after the CPU usage drops to 0%?
And what VMs are you running the kernel on? Are they Debian based or
Fedora based? And what versions of those distros?

[Антон Чехов]

Ah, okay. Mine don't start anymore. They jump from green to yellow but the AppVM and templateVM don't start anymore. So I can't change anything in the template anymore which makes it unusable.
I did everything by the book, like described by coldhak:
https://github.com/coldhakca/coldkernel
I tried the kernel mostly on debian based VM. I built mostly debian based apps after some time. Debian 8, that is.

[Reg Tiangha]

And one more question: What do you mean when you say that you're trying
to "update?" As in, update the coldkernel to a newer version? If that's
what you're trying to do (ex. Upgrade an installed kernel from 4.8.15 to
4.8.17), you need to also a) update grub by re-running the grub command
from the coldkernel instructions and b) optionally, uninstall the older
version of the kernel. Otherwise, if you don't update grub, then the VM
will try to boot the old version of the kernel. Installing the updated
kernel does not automatically update grub like it does on a normal
distro; you have to do it manually.

[Reg Tiangha]

> Ah, okay. Mine don't start anymore. They jump from green to yellow but the AppVM and templateVM don't start anymore. So I can't change anything in the template anymore which makes it unusable.
> I did everything by the book, like described by coldhak:
> https://github.com/coldhakca/coldkernel
> I tried the kernel mostly on debian based VM. I built mostly debian based apps after some time. Debian 8, that is.
>

Hmm...do you get a qrexec error? This behaviour (in my experience)
usually means that the u2mfm kernel module didn't get compiled or
installed correctly. On Debian VMs, you need to make sure the kernel
header package is installed first before the kernel image package to
make sure that u2mfm is compiled and installed correctly and the proper
initramfs file is generated.

If you need to fix things in your template, you can switch back to a
normal Qubes kernel using Qubes Manager, rather than using pvgrub2. The
machine should come up fine after that and you can double-check your
settings that way.

So maybe retry compiling and installing the coldkernel again by pulling
a fresh copy off of github and starting from scratch, and if things
still don't work, try that again using a fresh template.

[Антон Чехов]

No qrexex error.
Re update: I meant both cases, like you assumed but primarily recompiling with changes to the coldkernel.config. I did all these steps again, like described. That includes update grub. I did have the same errors mentioned earlier in the thread. (/usr/sbin/grub-probe: error: cannot find a GRUB drive for /dev/mapper/dmroot. Check your device.map.) but other than that everything looked fine. Something must have gone wrong.
I will try again like you described. Thanks very much for your help and the tip with switching back to normal kernel...so obvious that I would have missed it.

[Reg Tiangha]

Well, keep us posted on how it goes. To be fair, I'm still running
4.8.15 of the kernel and haven't had a chance to update it to 4.8.17 so
there could very well be some issues with the latest version that I
haven't encountered yet. My plan is to attempt the upgrade this weekend
when I have more time to play around with things.

[Антон Чехов]

Will do. I used the latest kernel 4.8.17 and there were no problems until I recompiled so I guess it should work. I mostly run AppVM behind a proxyVM. I will try again and report how it went.

[Антон Чехов]

I tried again with the existing template, recompiled but the result stayed the same. So I did try from scratch, including the modified coldkernel.config. Everything is working so far, also behind a proxyVM. I did stop and restart the appVM but I did not yet reboot my system. I added the template to these groups:

sudo groupadd -g 9001 grsecproc
sudo groupadd -g 9002 tpeuntrusted

I am a little reluctant with “sudo groupadd -g 9003 denysockets” because I don't know what it does in detail (or whether it might lead to problems).

[WillyPillow]

-------- Original Message --------

It's the same as the previous two commands, creating a group with certain privileges. For that specific one, IIRC the users assigned to that group is denied the use of sockets.

--WillyPillow