HCL - Lenovo A485
208 views
Skip to first unread message
panina
Jul 14, 2019, 5:20:50 PM7/14/19
to qubes-users mailing list
So, I decided to get Qubes on a AMD Ryzen machine. It's been
interesting. Fair warning: this is a bit of a novel.
BIOS/uefi
---------
Firstly, Qubes installation disk will not start in legacy BIOS mode. I'm
not sure why, but X does not start. I'm not that interested in legacy
BIOS anyway, so I did not investigate much. I simply installed in uefi mode.
Secondly, sys-net crashes on installation. Instructions on how to get it
working follows further down.
Ryzen gpu & linux kernel < 4.17
-------------------------------
Proper support for AMD Ryzen needs Linux kernel at least 4.17, so for
Qubes 4.0, that means I had to enable dom0 testing repo. With the older,
standard, kernel, the system needs to be booted with "nomodeset" kernel
parameters. This has to be done on first boot. If the system reboots,
after the first boot, without this parameter, it will not boot properly.
So, on first boot, add "nomodeset" to /boot/efi/EFI/qubes/xen.cfg, last
in the very long "kernel=..." line.
To get graphics to work (backlight, gpu etc), we need to enable the
testing repos in /etc/yum.repos.d/qubes-dom0.repo. Find the testing
post, and change the "enabled=0" to "enabled=1".
Then, we need to update and upgrade dom0. Sadly, though, sys-net doesn't
work out of the box.
sys-net
-------
Enabling networking is a bit complicated, though, because AMD has rather
bad iommu support. The hardware is grouped in rather large groups, and
the network cards cannot be added to sys-net without some extra pci
hardware.
The network cards on this machine are on pci addresses 1:00:0, 3:00:0
and 4:00:0 . But with only these PCI devices, sys-net cannot boot,
because the 3:00:0 network card is grouped together with USB ports and a
few other devices. These devices cannot be split between several
machines. To get sys-net to boot, we need to edit it's Devices, and add
everything with 3:00:x, or remove the 3:00:0 network card.
After that, networking works fine (except occasionally the WiFi hangs,
and needs to be dis- & reconnected. Probably about once a day or so).
sys-usb
-------
Getting sys-usb to work (this will probably have to be on the sys-net
machine) is something I still haven't managed.
If the system is booted with the rd.qubes.hide_all_usb kernel parameter,
the graphics drivers crash, and the system cannot boot. The only way to
get a stable system is to remove that parameter, and then sys-usb
doesn't work as intended. USB devices get attached straight into dom0.
This isn't terribly acceptable to me. I'm currently using udev to
whitelist USB devices, everything not on the whitelist doesn't get
activated. This gives some protection, but it's not quite good enough.
I think if I dig into the iommu groups, or possibly blacklists some
devices like camera, I might get around this. But so far, sys-usb isn't
working.
However, if the kernel is up to date, and the hide_all_usb parameter is
removed from /boot/efi/EFI/qubes/xen.cfg, we can activate the gpu. I
removed "nomodeset" and added "iommu=1 iommu=pt". I honestly don't
remember if the iommu parts are needed or not.
AEM vs TPM2 TOTP
----------------
To my great disappointment, AEM does not work. It needs legacy BIOS
mode. Also, it might not work with this machine's rather splendid TPM2.0
from AMD. It seems it needs Intel's TXT engine, and I'm not sure this
machine could work with it.
I did, however, find an alternate solution that I'm quite happy with.
First, I use secure boot, to sign my kernel. The, once the system is
booted, I use TPM2 TOTP to verify the integrity of the BIOS & firmware.
I'd rather get this done during boot, but I haven't quite figured out
how to get dracut & plymouth to cooperate. But it's no big deal to me -
I will find out if the firmware has been compromised, just a little
later than I'd like.
This solution, however, does not need a USB devices attached to dom0. It
works with my TOTP app in my phone, which does not need to be attached.
It would be fantastic if Qubes could package tpm2-totp and tpm2-tss
(and, preferrably, tpm2-tools) in a good way. To get this to work, I had
to build the packages myself, and then copy them into dom0. I'm not
happy about this, but feel the gains outweigh the cost, security-wise.
Later versions of fedora does have these packages, so it'll sort itself
out later on.
I believe this is all of it. It's taken about a month of tinkering, but
now I have a stable system that I'm happy with. And without the random
never-ending Intel security holes...
If anyone has ideas on the sys-usb things, please do let me know. And if
anyone tries to follow in my wobbly footsteps: I've likely missed some
step somewhere. Get in touch in that case, I'll gladly help others.
<3
/panina
interesting. Fair warning: this is a bit of a novel.
BIOS/uefi
---------
Firstly, Qubes installation disk will not start in legacy BIOS mode. I'm
not sure why, but X does not start. I'm not that interested in legacy
BIOS anyway, so I did not investigate much. I simply installed in uefi mode.
Secondly, sys-net crashes on installation. Instructions on how to get it
working follows further down.
Ryzen gpu & linux kernel < 4.17
-------------------------------
Proper support for AMD Ryzen needs Linux kernel at least 4.17, so for
Qubes 4.0, that means I had to enable dom0 testing repo. With the older,
standard, kernel, the system needs to be booted with "nomodeset" kernel
parameters. This has to be done on first boot. If the system reboots,
after the first boot, without this parameter, it will not boot properly.
So, on first boot, add "nomodeset" to /boot/efi/EFI/qubes/xen.cfg, last
in the very long "kernel=..." line.
To get graphics to work (backlight, gpu etc), we need to enable the
testing repos in /etc/yum.repos.d/qubes-dom0.repo. Find the testing
post, and change the "enabled=0" to "enabled=1".
Then, we need to update and upgrade dom0. Sadly, though, sys-net doesn't
work out of the box.
sys-net
-------
Enabling networking is a bit complicated, though, because AMD has rather
bad iommu support. The hardware is grouped in rather large groups, and
the network cards cannot be added to sys-net without some extra pci
hardware.
The network cards on this machine are on pci addresses 1:00:0, 3:00:0
and 4:00:0 . But with only these PCI devices, sys-net cannot boot,
because the 3:00:0 network card is grouped together with USB ports and a
few other devices. These devices cannot be split between several
machines. To get sys-net to boot, we need to edit it's Devices, and add
everything with 3:00:x, or remove the 3:00:0 network card.
After that, networking works fine (except occasionally the WiFi hangs,
and needs to be dis- & reconnected. Probably about once a day or so).
sys-usb
-------
Getting sys-usb to work (this will probably have to be on the sys-net
machine) is something I still haven't managed.
If the system is booted with the rd.qubes.hide_all_usb kernel parameter,
the graphics drivers crash, and the system cannot boot. The only way to
get a stable system is to remove that parameter, and then sys-usb
doesn't work as intended. USB devices get attached straight into dom0.
This isn't terribly acceptable to me. I'm currently using udev to
whitelist USB devices, everything not on the whitelist doesn't get
activated. This gives some protection, but it's not quite good enough.
I think if I dig into the iommu groups, or possibly blacklists some
devices like camera, I might get around this. But so far, sys-usb isn't
working.
However, if the kernel is up to date, and the hide_all_usb parameter is
removed from /boot/efi/EFI/qubes/xen.cfg, we can activate the gpu. I
removed "nomodeset" and added "iommu=1 iommu=pt". I honestly don't
remember if the iommu parts are needed or not.
AEM vs TPM2 TOTP
----------------
To my great disappointment, AEM does not work. It needs legacy BIOS
mode. Also, it might not work with this machine's rather splendid TPM2.0
from AMD. It seems it needs Intel's TXT engine, and I'm not sure this
machine could work with it.
I did, however, find an alternate solution that I'm quite happy with.
First, I use secure boot, to sign my kernel. The, once the system is
booted, I use TPM2 TOTP to verify the integrity of the BIOS & firmware.
I'd rather get this done during boot, but I haven't quite figured out
how to get dracut & plymouth to cooperate. But it's no big deal to me -
I will find out if the firmware has been compromised, just a little
later than I'd like.
This solution, however, does not need a USB devices attached to dom0. It
works with my TOTP app in my phone, which does not need to be attached.
It would be fantastic if Qubes could package tpm2-totp and tpm2-tss
(and, preferrably, tpm2-tools) in a good way. To get this to work, I had
to build the packages myself, and then copy them into dom0. I'm not
happy about this, but feel the gains outweigh the cost, security-wise.
Later versions of fedora does have these packages, so it'll sort itself
out later on.
I believe this is all of it. It's taken about a month of tinkering, but
now I have a stable system that I'm happy with. And without the random
never-ending Intel security holes...
If anyone has ideas on the sys-usb things, please do let me know. And if
anyone tries to follow in my wobbly footsteps: I've likely missed some
step somewhere. Get in touch in that case, I'll gladly help others.
<3
/panina
0 new messages