Installation, no AMD-vi, interrupt mapping, etc.

[naas...@gmail.com]

My first attempt at installing Qubes brought up a dialog indicating AMD-Vi and interrupt remapping wasn't available. I know my hardware supports the necessary virtualization extensions (AMD FX-8120 on a 990FX mobo), so further investigation suggested that the bios was probably buggy.

I updated the bios but still no luck, so I tried a manual procedure as described at [1]: I ran a recent live linux distro from a USB key and confirmed that interrupt remapping was disabled by default due to this BIOS bug. I then figured out the IOMMU and SMBus addresses using the described procedure and managed to get the live linux to boot with interrupt remapping and virtualization enabled as reported by dmesg.

I tried the same ivrs_ioapic mapping procedure for the Qubes installer, but it still raises that dialog saying no interrupt remapping, AMD-Vi, etc. Running dmesg in the Qubes installer console reports only that IOMMUv2 is not available, which the live linux also reported, but no other errors. The dmesg output is a little different though, so perhaps I missed something.

Any suggestions on what to do next?

[1] https://superuser.com/questions/1052023/ioapic0-not-in-ivrs-table

[Jonathan Seefelder]

maybe a dump Question, but did you check if its enabled in the BIOS ? ;)


cheers

--
Kind Regards
Jonathan Seefelder
CryptoGS IT-Security Solutions
Hofmark 43b
D-84564 Oberbergkirchen
Phone: +49 8637-7505
Fax: +49 8637-7506
Mail: in...@cryptogs.de
www.cryptogs.de

[naas...@gmail.com]

Yes, SVM and IOMMU are enabled and show as enabled when booting raw Linux when I pass in explicit ivrs_ioapic parameters at boot.

I just tried again passing in iommu=debug and loglvl=all as described at [1], and xl dmesg just reports:

(Xen) IVHD Error: Invalid IO-APIC 0
(Xen) AMD-Vi: Error initialization
(Xen) I/O virtualization disabled

No further information as to what's causing the issue seems to be available. I checked the xl dmesg also for a boot without explicitly passing in ivrs_ioapic and this output looks the same, so perhaps my explicit settings are ignored.

If that's the case, then I already know my BIOS doesn't provide the proper mappings which is why I try to do so explicitly. That might explain why virtualization appears disabled. If anyone knows this for sure and knows how I can specify the IOMMU/SMBus mappings explicitly, I'd appreciate it!

Sandro

[1] https://groups.google.com/forum/#!searchin/qubes-users/ivrs_ioapic|sort:date/qubes-users/GESQugF4eIo/wyUZyDubAgAJ

[Sergio Matta]

> I updated the bios but still no luck, so I tried a manual procedure as described at [1]: I ran a recent live linux distro from a USB key and confirmed that interrupt remapping was disabled by default due to this BIOS bug. I then figured out the IOMMU and SMBus addresses using the described procedure and managed to get the live linux to boot with interrupt remapping and virtualization enabled as reported by dmesg.
>
> I tried the same ivrs_ioapic mapping procedure for the Qubes installer, but it still raises that dialog saying no interrupt remapping, AMD-Vi, etc. Running dmesg in the Qubes installer console reports only that IOMMUv2 is not available, which the live linux also reported, but no other errors. The dmesg output is a little different though, so perhaps I missed something.
>
> Any suggestions on what to do next?
>
> [1] https://superuser.com/questions/1052023/ioapic0-not-in-ivrs-table

To fix the bug I inserted some parms on the xen command line (grub boot) and after I did "sudo grub2-mkconfig -o /boot/grub2/grub.cfg"
But at that time there was a qubes-hcl-report bug not recognizing Remapping:
HVM: Active
I/O MMU: Active
HAP/SLAT Yes
Remapping: no

xl dmesg works fine:
(XEN) AMD-Vi: IOMMU 0 Enabled.
(XEN) I/O virtualisation enabled
(XEN) – Dom0: mode: Relaxed
(XEN) Interrupt remapping enabled

see: https://ubuntuforums.org/showthread.php?t=2254677

[naas...@gmail.com]

Yes, I read that thread as well. The last post in that thread did not enable IOMMU on Linux, the second to last did and it's the same procedure I linked in my first post on superuser.com.

However, adding these parameters while booting the Qubes installer doesn't seem to have an effect. Are you suggesting I just proceed with the installation regardless and try to set these parameters in the grub booting config after I get up and running?

Sandro

[Sergio Matta]

Are you suggesting I just proceed with the installation regardless and try to set these parameters in the grub booting config after I get up and running?
>
> Sandro
>

Yes! And even if iommu does not works, you will be able to use Qubes 4 VM as PV. Networking will need to set up by hand and save those commands in files to start it automatically.
You will have time to solve it or buy another motherboard. Mine is a US$150 used sabertooth 990fx rev.2

[naas...@gmail.com]

Installation went fine except for a libxenlight config error of some kind. I still can't enable IOMMU using either of the approaches described in that Ubuntu thread, even though it successfully worked with raw Linux.

What boot parameters did you add? I have the earlier rev.1 Sabertooth 990FX mobo that you have.

I'm not sure there's much advantage in sticking to Qubes without the hardware acceleration over a Linux distro I'm more familiar with that has virtualization working. I can just use accelerated VirtualBox instead of accepting the PV overheads for the workflows I have in mind, but I'd definitely like to use Qubes if I can get this working.

Sandro

[naas...@gmail.com]

I meant qemu-kvm not VirtualBox. I now see this table [1] that claims that Xen doesn't support IOMMU on rev.1 of our mobo have, but does on rev.2. I'll probably go with the alternative I have in mind then since, unless someone else has any ideas.

If not, thanks for the suggestions everyone!

Sandro

[1] https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware#Motherboards_2

[Sergio Matta]

Em segunda-feira, 1 de outubro de 2018 16:04:34 UTC-3, naas...@gmail.com escreveu:
> Installation went fine except for a libxenlight config error of some kind. I still can't enable IOMMU using either of the approaches described in that Ubuntu thread, even though it successfully worked with raw Linux.
>
> What boot parameters did you add? I have the earlier rev.1 Sabertooth 990FX mobo that you have.

My mobo is rev.2, firmware 2901
I used (ivrs_ioapic[7]=00:14.0 ivrs_ioapic[8]=00:00.2). I am not using anymore and my qubes 4.0 is working fine.

But ubuntu forum has a solved solution with different ioapic:
Quick solution for Sabertooth 990FX (R1.0):
Edit file /etc/default/grub, find line "GRUB_CMDLINE_LINUX_DEFAULT=", edit it to look like:
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash ivrs_ioapic[7]=00:14.0 ivrs_ioapic[8]=00:00.1"

There are iommu info here too:
from Xen https://wiki.xen.org/wiki/VTd_HowTo

If you can not solve the iommu problem, change all vms to PV. Maybe this is the cause of libxenlight error. Change all vms to PV, including sys-net.
Later I will send you the commands to start networking.

[Sergio Matta]

My cpu is a AMD 1100T and PVH is not much more fast then PV.

If you want to test it without iommu:
Change the VMs to PV, including sys-net and sys-firewall (qvm-prefs yourvmname virt_mode PV)
Using sys-firewall terminal do:
sudo cp /etc/resolv2.conf /etc/resolv.conf (resolv2.conf has your preferred nameservers)
ping -c 2 10.137.0.8 (to create vif interface)
sudo ip link set vif3.0 up
sudo ip addr add 10.137.0.4//255.255.255.255 dev vif3.0
sudo ip route add 10.137.0.8/255.255.255.255 dev vif3.0
- Save the commands above in /rw/config/rc.local and make it executable (chmod +x /rw/config/rc.local):
Using sys-net terminal do:
Save then in /rw/config/rc.local and make it executable:
ip link set vif2.0 up
ip addr add 10.137.0.3/255.255.255.255 dev vif2.0
ip route add 10.137.0.4 dev vif2.0
It should works.

[Sergio Matta]

PS: in the command "sudo ip route add 10.137.0.8/255.255.255.255 dev vif3.0", change the 10.137.0.8 to your correct VM IP

[Tai...@gmx.com]

Most consumer mobos have broken IOMMU, I suggest instead of wasting your
time trying to make it work you simply buy a KCMA-D8 or KGPE-D16 plus
used cpu and install coreboot-libre.

Without HVM/IOMMU your security will suck.