grsecurity kernel 4.9.20 not working - Qubes ErrorHandler: BadAccess MIT-SHM

[Patrick Schleizer]

Proxying a message from torjunkie at Whonix forums here due to google
group vs Tor spam false positive issues. Source:

https://forums.whonix.org/t/long-wiki-edits-thread/3477/55?

#####

Greetings,

I am currently using Qubes 3.2 and have had success to date with
running the 4.8 grsec kernel series (coldkernel) with Debian-8 AppVMs
following the steps / advice outlined on the coldhak blog and github
account.

I have recently tried to apply the 4.9.20 upgraded kernel to the
Debian-8 TemplateVM and hit some problems.

I have followed the advice to install the latest
qubes-kernel-vm-support package from the Qubes testing repository (for
the Debian-8 TemplateVM) and avoided the error messages around "Bad
return status for module build."
[https://github.com/coldhakca/coldkernel/issues/55]
[https://github.com/QubesOS/qubes-issues/issues/2691]

The upgraded kernel successfully builds and the TemplateVM boots.
However, the TemplateVM state light soon shifts from green to yellow.
The qrexec.log and console.log look okay (no obvious error messages),
but the guid.log shows a new cryptic error message I've never seen before:

ErrorHandler: BadAccess (attempt to access private resource denied)
Major opcode: 130 (MIT-SHM)
Minor opcode: 1 (X_ShmAttach)
ResourceID: 0x219
Failed serial number: 49
Current serial number: 50

Any attempts to run applications fail e.g. terminal. So, grsec
groups can't be set, paxtest can't be run, and obviously it's not
functional, so there is no point creating a new AppVM based on it.

Can anyone who has the 4.9 grsec kernel up and running provide any
advice on how to resolve this problem?

Regards

[Reg Tiangha]

I've been running the 4.9 series for a while and have had no issues. The
one time I noticed something similar (indicator turning green briefly
before turning yellow, although unfortunately I didn't look at the logs
at the time to find out if it was the same issue as yours) was when I
compiled a coldkernel on a Debian 8 template and then installed the .deb
package on a Debian 9 template. I "fixed" the issue by compiling a
version of the kernel on a Debian 9 template and installing that .deb
package instead.

I guess the first thing I would look at is your sysctl.conf file. Do you
have any special grsecurity kernel parameters set in there? If so, try
disabling all of them and see if the issue persists. If if it does not,
then it might be one of those settings that's the problem.

[Reg Tiangha]

Well I just hit this exact same issue today in my Debian templates after
updating my system yesterday. I never noticed until I restarted some VMs
this afternoon. I've also installed all packages in current-testing in
both dom0 and vms but it did not help. I too get the yellow light, and
in my TemplateVMs for these AppVMs running a dom0 kernel, sometimes
things freeze up. Right-clicking on the VM in Qubes Manager and
launching an xterm or something seems to help a little; as soon as the
window pops up, everything else unfreezes. But it's unreliable.

As a last resort, I used Qubes Builder to rebuild all Qubes-specific
Debian packages and installed all the ones my VM had, but that didn't
help either.

I know that it's not linked to coldkernel 4.9.20 specifically because
everything was working fine before I updated the various Qubes packages
that got pushed out recently, and up until this morning, I was running a
grsecurity kernel based on 4.9.22 (upgraded to 4.9.23 but that didn't
help either).

It doesn't happen running a dom0 kernel, so I think it only shows up
when using PVGRUB kernels. The u2mfn version is 3.2.4 and I've been
using that fix since last December with no issues so that package isn't
the issue. I don't know how to fix this. Patrick, is this issue still
happening to you?

[Reg Tiangha]

Well, I think I can rule out it being a PVGRUB issue alone, at least in
this case. I just created a Debian 8 template with the stock Debian 3.16
kernel and that booted up fine.

I don't think it's a coldkernel issue specifically, or at least one tied
to a certain version. Like I said, I've been running all the coldkernels
in the 4.9 series since they switched from 4.8, and use their build
scripts to compile newer ones as they come out. Until today and prior to
the recent Qubes package updates, I was running 4.9.22 with no issues.
It wasn't until after the Qubes updates that I encountered the BadAccess
errors, and updating to 4.9.23 has no effect.

So my guess is (at least until other people test some other PVGRUB
kernels to see how they behave; it could be a kernel 4.x or 4.9 issue
for all I know right now) that the newest version of whatever just got
pushed out on the Qubes end is doing something that the older versions
didn't and that the grsecurity stuff doesn't like, and this is shutting
it down. The problem is I don't know what package it might have been,
and there is absolutely *nothing* in dmesg that would give a hint.
Usually grsecurity spits out some kind of error when it kills
qubes-guid, which is why I'm still not convinced it's completely a
grsecurity/coldkernel problem at the moment. The next thing to try is a
stock 4.9 in PVGRUB mode to see if the issue persists.

[Reg Tiangha]

It's a Kernel 4.9 issue with the latest Qubes updates. jessie-backports
has a stock 4.9 kernel so I installed that, and I'm getting the *exact*
same behaviour as with the coldkernel 4.9 version.

I know it's *not* necessarily a 4.9 issue alone because I've been
running a 4.9 coldkernel in my VMs since at least December. So something
in the latest Qubes updates breaks compatibility with kernel 4.9. Not
sure how to fix that, but I'll report it.

[Reg Tiangha]

On 04/18/2017 11:56 PM, Reg Tiangha wrote:
> It's a Kernel 4.9 issue with the latest Qubes updates. jessie-backports
> has a stock 4.9 kernel so I installed that, and I'm getting the *exact*
> same behaviour as with the coldkernel 4.9 version.
>
> I know it's *not* necessarily a 4.9 issue alone because I've been
> running a 4.9 coldkernel in my VMs since at least December. So something
> in the latest Qubes updates breaks compatibility with kernel 4.9. Not
> sure how to fix that, but I'll report it.
>
>

For anyone following the thread, this is just a follow up, but it turned
out *not* to be a 4.9 on PVGRUB problem, but the typical dkms not
compiling the u2mfn module on upgrade issue that sometimes happens. I
wrote a post mortem here:

https://github.com/QubesOS/qubes-issues/issues/2762

But I wonder if that ErrorHandler: BadAccess message that appears in
guid.conf is indicative of this issue. I haven't kept an eye on it long
enough to be able to say either way, but it's definitely something to
keep in mind if that kind of error pops up in the future.


For me, the fix was to manually make dkms recompile the u2mfn module. In
Debian, run:

ls /var/lib/initramfs-tools | sudo xargs -n1
/usr/lib/dkms/dkms_autoinstaller start

and what that will do is force dkms to recompile all modules for any and
all locally installed kernels in the VM template. Then shutdown the
TemplateVM and then reboot your AppVMs and they should hopefully come
back up normally.

[cooloutac]

Is this related to starting a vm with custom kernel would start but then go yellow and no gui? I could remote in and boot screen would show all loaded ok but I couldn't load nothing. I have to try those commands tks.