Help regarding IOMMU and HVM

[Ana Z]

Hi.

First of all, I want to mention that I'm not really experienced with Linux or Xen but I tried my best for the past 3 days to make my Qubes installation work. I need help with enabling hardware-assisted virtual machines.

I was trying to install Qubes 4.0.1 but I ran into some issues. While installing, I was first greeted with "Missing features: IOMMU/VT-d/AMD-Vi, Interrupt Remapping. Without these feature, Qubes OS will not function normally.". I checked my BIOS and IOMMU was enabled (motherboard is ASUS A88XM-A), but my BIOS version was old. I updated the BIOS to the latest version and the error from the installation disappeared, but then, after the install, on setup I got another message saying:

[“/usr/bin/qvm-start’,’sys-firewall’] failed:
stdout: ""
stderr: “Start failed: internal error: libxenlight failed to create new domain ‘sys-
firewall’, see /var/log/libvirt/libxl/libxl-driver.log for details
""

I searched qubes-users for help and I came across this post: Bug in libxenlight

This indeed helped me finish the setup without errors, but all of my qubes are paravirtualized and when I try to change, for example, my sys-usb to HVM I get a timeout error and that I should check /var/log/xen/console/quest-sys-usb.log. I have searched for some explanation online but to no avail. I've put all the important logs (I hope) in attachments and I'd appreciate if someone could help me.

Ana.

[qubes123]

Maybe if you post the content of /var/log/libvirt/libxl/libxl-driver.log file also, we can see what the problem is. Otherwise the logs seem OK for me.


If you installed a std R4.0.1 and let the setup configure the base qubes, then sys-net should already be a HVM with all your Net cards passed through to this HVM.

For USB devices, when passed to a HVM, it could require, that you set "strict reset" in "Devices" tab for the USB devices of that qubes, otherwise it will not boot. Because of this, I recommend to try to start a hvm with only Net devices passed through to it, or even without any passed devices to narrow down the problem to HVMs in general or passthrough issues.

One tip: if the HVMs problem still persists, change the Virt mode to PV temporarily for the sys-net (make sure, that a Net card is passed to it), and if networking is OK, then update dom0 to the latest version.

Then switch it back to HVM and try to boot.  PV mode is not recommended (larger attack surface) on long term, but R4.0.1 is and old version...

[qubes123]

...correction: R4.0.1 is not old (this is the current released version), but the iso build is from January...

[Ana Z]

Hi.

I sure can! I uploaded the libxl-driver.log as well.

As of your second part of the message, I didn't mention that after doing all the steps suggested in linked "Bug in libxenlight" topic I encountered that all assigned HVMs (bullet 8) wouldn't load (such as sys-usb and sys-net) and dom0 was updated before that change (bullet 7), and yes, now that I think of it, it gave me a different error and suggestion that I read the /var/log/libvirt/libxl/libxl-driver.log you asked for! So I hope you could tell me from it what's the matter. I then, changed back all the VMs to PV mode and I was setting up new templates, even got my USB WIFI dongle to work, but as you mentioned it's not as secure as HVM is so I really want to make it work

[qubes123]

The libxl-driver.log shows symptoms, when indeed the USB device cannot be reseted (FLR functionality missing).

As I wrote, you can disable strictreset for HVM passed through devices:

- go for the "Qube Settings" of one of the HVMs (sys-usb), where the USB devices (PCI: 00:10.0, 00:10.1, 00:12.0, 00:13.0 in your case) are passed through

- under tab "Devices", you should see in the right side box ("Selected") all the USB device I noted above

- there is a horizontal bar in the bottom "Configure Strict Reset for PCI Devices" --> push this, then a selection box comes up

- make sure you highlighted all visible devices by clicking on each

- then OK --> (Apply) --> OK

This should disable Strict Reset, and - if no other problems are there - the HVM should start

One note: I also have and AMD based PC (laptop).  For me, not all USB devices should be passed through to sys-usb, because some of the devices (eg. camera, wifi-card usb BT module and somehow the internal keyboard) are internally connected to an USB port, and passing those through sys-usb qube results sometimes in weird problems like not having WIFI or not being able to type etc.  So I only pass through one USB device, which I know don't have any internal connections. 

Lastly, you might need to experiment, which devices and ports would work for you (passed through to eg. sys-usb), but make sure, that PCI devices with multiple functions (00:10.0 and 00:10.1) are "moving" together either way.

[Ana Z]

Hi.

I tried everything you said (in both sys-usb and sys-net). I even tried to not assign any pass through devices and I still get the same output in /var/log/xen/console/guest-sys-net.log.

But not assigning any device, or giving all devices strictreset as you mention gives me this output in /var/log/libvirt/libxl/libxl-driver.log:

libxl_linux.c:155:libxl_loopdev_cleanup: unable to release device /dev/loop0: No such device or address

I guess there are some errors in deeper level or my installation/BIOS is just bugged with IOMMU.

[qubes123]

The log file you provided as xl_log.txt is actually the Xen hypervisor log. Yes, there are some warnings - which show that there are some bugs in the BIOS, but overall, the virtualization and all the required features needed for IOMMU are there and enabled. So this is strange.

Maybe - and I just now noticed - you use kernel version 5.x from kernel latest in sys-usb - that is still a development version, might not work.  I'd switch that back to the stable 4.19.x version in the VM settings for sys-usb.

Do you have logs for the stubdoms? (eg. dom0: /var/log/xen/console/guest-sys-usb-dm.log or similar log for sys-net). It might show what is wrong, as the stuboms provide the device model (emulated pci devices) needed for HVMs.

[Ana Z]

Hi.

I'm using the 5.3.11.1 kernel because I tried to install usb wifi drivers on 4.19 but I kept getting errors in dkms build. I'm using these drivers here: https://github.com/AstroDrabb/rtl8812au and even though it says it's for 4.19 I can't make it work.

I also reinstalled Qubes this morning, but I had to go through pretty much everything as on the last attempt, and I checked the dmesg, xl dmesg logs and they are the same as before, even the errors. Tried HVM with strictreset, it again hangs.

Here are the latest sys.usb.log (I think it's the same as before) and stubdoms.

Ana.

[qubes123]

When I compared your logs to what I have, it came to my mind that some years ago I had similar problems - because of processor microcode which was too old. I had to update that to get hvms working.

I couldn't find which microcode version you have, it is somehow not shown in the cpu log you uploaded (cat /proc/cpuinfo should show that).

But, the solution is not easy, because processor microcode is usually stored in the BIOS statically (in non-UEFI bioses), and I don't know if you can change that dynamically somehow with your UEFI BIOS or while the kernel is booting.  I use a special BIOS (coreboot), where I can statically change the microcode to a newer one, but it needs external flashing with the right set of tools (SOIC clip, flash programmer etc). Unfortunately coreboot is not supported on your MB.

So try to find a solution on the net, how to update the microcode for your APU - this works with Intel processors even with kernel parameters (eg. ucode=scan in the Qubes cmdline is just enabling that functionality using the latest signed processor microcodes available in mainstream Linux distros) AFAIK, but success can dependend on your UEFI BIOS....

If I'm not mistaken, the latest microcode for A8-6600K is from March 2018 (https://github.com/platomav/CPUMicrocodes/blob/master/AMD/cpu00610F01_ver0600111F_2018-03-05_AC55EB96.bin) - but this have to be verified knowing your exact CPU infos.  This microcode version already has all the needed mitigations for Spectre vulnerabilities, therefore strongly recommended...

[Ana Z]

!!!

I have some, hopefully, great news.

I was messing around searching some help online and I got an idea to try installing Win7 StandaloneVM following the https://www.qubes-os.org/doc/windows-tools/ guide.

I kid you not, it worked on first try! HVM mode, even got the Windows Tools working.

Is there some logs I could extract from this new VM that would help find the error in Linux VMs?

Thank you very much for your help so far.

[Ana Z]

I found a solution to my problem.

The culprit was my overclock. I don't know why I didn't reset my BIOS settings in start of troubleshooting, but today I reset it and everything works...

Interesting note though, naturally I tried to pinpoint what part of overclock is messing up the start of HVM VM's and it's the CPU multiplier. My CPU is AMD A8-6600K so I assumed everything above its maximum turbo frequency would cause the error (x42) but last multiplier before VM's wouldn't load is 43. Vcore didn't affect loading.