Kernel 4.9 on Qubes 3.2

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi,

Have anyone tried running Kernel 4.9.x on Qubes 3.2 (both dom0 and VM)?
Any problems noticed? I'm considering pushing it to testing repository and
then to stable, but since handling kernel downgrades (if something goes
wrong) with dnf is tricky, I'd like to collect some feedback first.
There have been a lot of changes since the current 4.4.x...

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJY9mqKAAoJENuP0xzK19csm7IH/3GCQZYPUxCiSECFRaUePbY2
9b8GF1b93nPAd0ZN4Get4TJH8fDKvjzpUQXNC9a63qPpokgFvpunFVwNl62HxAAf
b9qK47t1+r5IpZzWOWUsTcjwIgk13N3+B4oWtN89MsPyDQAoyT5EdixadYB8k80t
LLPZgRaDvgew9Pg9r8CMpg8KSdveN9Z4KbFCnt28E+WFIZqeJdiOtSrCc347nPPL
g+7QQxz7niDU8hgUkD21HED4Fd7pFxrTK/E4BdRgXajDAeAkvO2wuYI6iCkparun
1cpTARVyb4gBhcH4qzmadVK8fAZcDT1+TQSqpKazbGguoKrcrIrum2ebfa8MTIM=
=/aZv
-----END PGP SIGNATURE-----

[Reg Tiangha]

I can only speak from my experience, but I briefly ran 4.9 in dom0
before switching full time to 4.10 (both dom0 and vm). I didn't
encounter any issues on my hardware that I've noticed (I initially
worked off the stock Qubes 4.4 configuration (actually, it might have
been from your devel-4.8 branch) but have since customized to my
hardware by cutting out a lot of stuff), although I only have Intel
hardware so I can't test what the experience is like on an AMD system.

I also wasn't sure what options to enable in upgrading from 4.8-4.9 (and
then to 4.10); I think I took most of the defaults, said 'no' to
anything exotic that probably didn't apply to Qubes on x86_64, enabled
anything new that looked like it had to do with kernel protection
(mostly stuff with randomization and sanitation), and made modules out
of any new USB or network drivers that might work under Qubes (and
netfilter as well), but it would have been nice to have some guidance on
the Qubes philosophy (if there is one) on what to enable in the kernel
and what not to, at least from a security standpoint.

The coldkernel project has switched to 4.9 since some time last year as
Grsecurity only provides free patches for that branch now, so I assume
that anyone that runs a coldkernel in their VM is using that. I did
notice one person having an issue with installing it in a Qubes Debian
VM, but he was running an AMD machine, and there's not enough sample
cases for me to see if it's a 4.9 coldkernel on AMD (on Qubes?) issue,
or a general 4.9 issue on AMD with Xen (I don't think it's an AMD CPU
issue specifically since other distros are running 4.9 kernels so it's
probably something to do with the combination of things). So regardless,
more feedback from people with AMD cpus would be useful.

For anyone that wants to test, I've got 4.9 and 4.10 branches on my GitHub:

https://github.com/rtiangha/qubes-linux-kernel

Feel free to edit the config files to suit your hardware/needs,
especially if there are any drivers that are missing.

[0spin...@gmail.com]

Another anecdote here, from (that) someone running an AMD cpu. 4.9 and 4.10 (custom and vanilla) builds both booted fine, and all the obvious stuff is working. Was running your (Marek's) 4.8.12 kernel prior to this.

[Reg Tiangha]

On 04/18/2017 02:05 PM,

> --
> You received this message because you are subscribed to the Google
> Groups "qubes-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to
> qubes-devel...@googlegroups.com
> <mailto:qubes-devel...@googlegroups.com>.
> To post to this group, send email to
> qubes...@googlegroups.com
> <mailto:qubes...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/qubes-devel/d3ca4d87-3270-4756-8e3d-888fc7941b6d%40googlegroups.com
> <https://groups.google.com/d/msgid/qubes-devel/d3ca4d87-3270-4756-8e3d-888fc7941b6d%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.


After the latest round of Qubes stable updates I can confirm that kernel
4.9 does *not* work properly in a PVGRUB VM.

Discussion here (first reported on April 7 by Patrick Schleizer, and I
wonder if he was on the current-testing branch when he first reported
it; I definitely was not at the time):

https://groups.google.com/forum/#!topic/qubes-users/2X8wi5XebJc

But I've been running a 4.9 coldkernel since December with absolutely no
issues, and now neither it nor the stock Debian 4.9 kernel works
correctly. I'm not sure what to downgrade in order to confirm that it is
one of the updated Qubes packages that's incompatible, but it's hard for
me to test because my VMs keep freezing, exactly like in this issue here:

https://github.com/QubesOS/updates-status/issues/18

I've haven't yet tried to downgrade a VM (I primarily use Debian 8) to
gui-agent-linux-3.2.8 yet to see if the problem goes away, however. But
I'm starting to get desperate so I might try it soon.

[Reg Tiangha]

On 04/19/2017 12:02 AM, Reg Tiangha wrote:
>
> After the latest round of Qubes stable updates I can confirm that kernel
> 4.9 does *not* work properly in a PVGRUB VM.
>
> Discussion here (first reported on April 7 by Patrick Schleizer, and I
> wonder if he was on the current-testing branch when he first reported
> it; I definitely was not at the time):
>
> https://groups.google.com/forum/#!topic/qubes-users/2X8wi5XebJc
>
> But I've been running a 4.9 coldkernel since December with absolutely no
> issues, and now neither it nor the stock Debian 4.9 kernel works
> correctly. I'm not sure what to downgrade in order to confirm that it is
> one of the updated Qubes packages that's incompatible, but it's hard for
> me to test because my VMs keep freezing, exactly like in this issue here:
>
> https://github.com/QubesOS/updates-status/issues/18
>
> I've haven't yet tried to downgrade a VM (I primarily use Debian 8) to
> gui-agent-linux-3.2.8 yet to see if the problem goes away, however. But
> I'm starting to get desperate so I might try it soon.
>
>

I just realized that Patrick was only relaying the message from someone
on the Whonix forums, which explains why the person who originally
reported it hasn't responded, lol.

But to add a bit more info from my previous report, running the stock
Debian 9 4.9 kernel from jessie-backports in an up-to-date VM on the
stable track throws the same BadAccess error in the guid log as the 4.9
coldkernel does, so anyone who wants to test that can easily replicate
it by spinning off a new Debian 8 kernel and installing
linux-image-amd64 from jessie-backports; no need to compile and install
a 4.9 coldkernel to test for the failure.

[Reg Tiangha]

I'm racing the VM freezing, hence the typos (it's also late in my part
of the world). Here's what I previously wrote, without the typos. Sorry
about the spam; ideas to solve my freezing issues (Qubes VM Manager also
freezes, although I don't know if it's related to the freezing VMs or
something different and sometimes killing it makes the VM responsive
again) are certainly welcome:

I just realized that Patrick was only relaying the message from someone
on the Whonix forums, which explains why the person who originally
reported it hasn't responded, lol.

But to add a bit more info from my previous report, running the stock

Debian 8 4.9 kernel from jessie-backports in an up-to-date VM on the

stable track throws the same BadAccess error in the guid log as the 4.9
coldkernel does, so anyone who wants to test that can easily replicate

it by spinning off a new Debian 8 vm and installing

linux-image-amd64 from jessie-backports; no need to compile and install

a 4.9 coldkernel to replicate the failure.

[Reg Tiangha]

On 04/19/2017 12:33 AM, Reg Tiangha wrote:
> I'm racing the VM freezing, hence the typos (it's also late in my part
> of the world). Here's what I previously wrote, without the typos. Sorry
> about the spam; ideas to solve my freezing issues (Qubes VM Manager also
> freezes, although I don't know if it's related to the freezing VMs or
> something different and sometimes killing it makes the VM responsive
> again) are certainly welcome:
>
> I just realized that Patrick was only relaying the message from someone
> on the Whonix forums, which explains why the person who originally
> reported it hasn't responded, lol.
>
> But to add a bit more info from my previous report, running the stock
> Debian 8 4.9 kernel from jessie-backports in an up-to-date VM on the
> stable track throws the same BadAccess error in the guid log as the 4.9
> coldkernel does, so anyone who wants to test that can easily replicate
> it by spinning off a new Debian 8 vm and installing
> linux-image-amd64 from jessie-backports; no need to compile and install
> a 4.9 coldkernel to replicate the failure.
>
>
>

OK, I may have spoken too soon about 4.9 not working on PVGRUB VMs. In
my case, it really was dkms not compiling the u2mfn module properly, and
that's why I wasn't getting the green light. My post mortem is here:

https://github.com/QubesOS/qubes-issues/issues/2762

So I go back to my original opinion that there's no technical reason
that Qubes couldn't run a 4.9 kernel in dom0 or in VMs.

However, I do think it should be an opt-in thing that users can do with
a meta-package that can help manage the upgrade (similar to how Ubuntu
users can choose to opt into newer kernels in LTS releases by explicitly
installing an Hardware Enablement package).

I really do think that Qubes needs to do it by metapackage because the
version of u2mfn that ships with the original R3.2 iso does *not*
support kernels higher than 4.8. And in order for things to work
correctly, u2mfn 3.2.4 must be installed first so that dkms doesn't fail
to compile that module when the kernel is upgraded.

So a metapackage could help manage that. It could ensure that a stock
R3.2 system has all the prerequisites like an update u2mfn.c file and
the appropriate kernel headers installed first before upgrading the
kernel, and can also ensure that dkms recompiles all modules for all
kernels that may be installed on the system.

Plus there's no guarantee that a user coming from a fresh install from
the original R3.2 iso will have a completely updated system before
running a 4.9 kernel upgrade. After a fresh install, the first thing
that person could choose to do is to first upgrade the kernel to 4.9 to
fix some communicability issues (for example, garbled video displays
that may be fixed with newer drivers) before proceeding to upgrade
everything else. So having a metapackage that could ensure all 4.9
prerequisites are met could help ease the transition.

A metapackage probably wouldn't be needed if the shipped version of
u2mfn.c worked with kernels 4.9+, but it is what it is.

Anyways, those are my thoughts. But it would be nice to hear from more
people who are running 4.9 and/or 4.10 kernels on different hardware
configurations to see what their experiences are like.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, Apr 19, 2017 at 03:20:59PM -0600, Reg Tiangha wrote:
> OK, I may have spoken too soon about 4.9 not working on PVGRUB VMs. In
> my case, it really was dkms not compiling the u2mfn module properly, and
> that's why I wasn't getting the green light. My post mortem is here:
>
> https://github.com/QubesOS/qubes-issues/issues/2762
>
> So I go back to my original opinion that there's no technical reason
> that Qubes couldn't run a 4.9 kernel in dom0 or in VMs.
>
> However, I do think it should be an opt-in thing that users can do with
> a meta-package that can help manage the upgrade (similar to how Ubuntu
> users can choose to opt into newer kernels in LTS releases by explicitly
> installing an Hardware Enablement package).
>
> I really do think that Qubes needs to do it by metapackage because the
> version of u2mfn that ships with the original R3.2 iso does *not*
> support kernels higher than 4.8. And in order for things to work
> correctly, u2mfn 3.2.4 must be installed first so that dkms doesn't fail
> to compile that module when the kernel is upgraded.

This particular thing is not a problem for having kernel 4.9 in Qubes
repository by default, because it already contains u2mfn module in it -
you don't need to compile anything when installing it.
Are there any other reasons why this should be opt-in? I'm asking
because currently we don't have a mechanism for doing it. Only an
opt-out (you can change kernel version in VM properties, and also choose
different default option for dom0 kernel in bootloader settings). So, if
new kernel is for example incompatible with your hardware, you always
have option to rollback. And I think (until convinced otherwise)
inventing/implementing opt-in mechanism for this isn't justified.

> Anyways, those are my thoughts. But it would be nice to hear from more
> people who are running 4.9 and/or 4.10 kernels on different hardware
> configurations to see what their experiences are like.

This was my intention of starting this thread initially :)

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJY9/FMAAoJENuP0xzK19cs3GgIAILtCH0O/azBNyAsSWbZf4nQ
KPyyhqLiksvckyK2sYJjP5SCEZHtgBPJJor8sYyQTDjIYcjkKBTLJg6/VTdP7F6E
Qcu3htMv6vg1eM11XC3mvFGjmOBmCg6Gn8TMfEWI/OoTrxyj+7D0AvwWvDhiZFPu
OOU2DE1o8Nca2FW+VKfsGuohCTK1dmv1pbQuGUCyLGOir7HwP2+LnunxkSCjvhWZ
gIjQKiEHE0BN3i/q8NRn3ZolwnsYBPlgYQbpoWFCy3sY/WIe+4SbW4uSPya5obPI
wg/MDaKSZxAWDsGbf5gZDJaBcYrZavuWRchSBW8sAEgbQPlQbvoKZZtjToGKMY0=
=JDnD
-----END PGP SIGNATURE-----

[Reg Tiangha]

> -----END PGP SIGNATURE-----
>
Well, if it's all bundled in, then I guess that's fine. My biggest
concern was ensuring that the u2mfn module was compiled and included
since that's the main pain point I've encountered in my adventures with
the kernel stuff. But if the package scripts do that at build time and
not at install time in dom0, then I don't foresee any issues.

So yeah, if the upgrade is smooth then I see no reason for it to be
opt-in either and it could probably be safely pushed out.

[Joonas Lehtonen]

Marek Marczykowski-Górecki:

> Hi,
>
> Have anyone tried running Kernel 4.9.x on Qubes 3.2 (both dom0 and VM)?
> Any problems noticed? I'm considering pushing it to testing repository and
> then to stable, but since handling kernel downgrades (if something goes
> wrong) with dnf is tricky, I'd like to collect some feedback first.
> There have been a lot of changes since the current 4.4.x...

I'm running kernel 4.9.x (debian package) in a Debian 9 VM via pvgrub.
No issues so far (apart from the longer boot times).