[4.0] Intel Wi-Fi 6 AX200 adapter

[Vít Šesták]

Hello,

on a new laptop, I am trying to setup Qubes OS 4.0. I have installed the latest point release (i.e., 4.0.3). I have bunch of issues, the most important one is that I cannot connect to any network. I have made some progress, but now, I am asking for some assistance.

The command lspci shows the adapter as Intel Corporation Wi-Fi 6 AX200 (rev 1a).

1. Problem: Domain sys-net did not boot at all because of issues with attaching ethernet PCI device.

Workaround: I don't care much about ethernet, so I removed the PCI device. Not optimal, but easy and good enough for now.

2. Domain sys-net does not even load iwlwifi kernel module.

Cause: I have found that I need kernel 5.1 or 5.2 (depending on the source).

Solution: Use a backup of up-to-date Fedora 30 VM. Configure the mode to HVM and the kernel to “(none)”. This will use up-to-date kernel from Fedora, which is new-enough.

Relevant link: https://www.reddit.com/r/ManjaroLinux/comments/bx0skx/iwlwifi_driver_not_loading_for_intel_ax200_wifi/

Relevant link: https://www.intel.com/content/www/us/en/support/articles/000005511/network-and-i-o/wireless-networking.html

3. Domain sys-net loads iwlwifi kernel module, but the Wi-Fi fails immediately with “Microcode SW error detected. Restarting 0x2000000”. It seems that Fedora automatically retries again and again and spams the log.

This is the issue I am struggling with.

What I have tried:

* Use Fedora 31. I just cloned Fedora 30 on other laptop, created backup and transferred to the other laptop. See https://github.com/QubesOS/qubes-issues/issues/5289#issuecomment-582442319

* Download ucode from https://www.intel.com/content/www/us/en/support/articles/000005511/network-and-i-o/wireless-networking.html

* Replace /usr/lib/firmware/iwlwifi-cc-a0-48.ucode by /usr/lib/firmware/iwlwifi-cc-a0-46.ucode if `sudo dmesg | grep 'firmware version'` suggests that the 48 is being loaded. I did it in the TemplateVM, rebooted sys-net and ensured that 46 is being loaded. According to the https://www.intel.com/content/www/us/en/support/articles/000005511/network-and-i-o/wireless-networking.html , the version 46 is for AX200 and 48 is for AX201.

No combination of those has helped, I am still getting the errors.

Logs (Fedora 31, ucode from Intel):

* Log sample (truncated, because it was too large, but it seems that messages are repeating): https://gist.github.com/v6ak/1cb7b172f63f8c20d7c4004f7996de39

* Just unique messages (just proves there is nothing more interesting, computed before the truncation): https://gist.github.com/v6ak/f429774566cca097f51445ed93305136

What I haven't tried:

* Updating the dom0 – a bit chicken-egg problem, since I don't have Internet connection on this laptop. I probably could transfer updates from the other laptop, but I don't think this would make a difference.

* Use Debian. Due to the kernel version, it does not seem to be worth trying.

Any ideas?

Regards,

Vít Šesták 'v6ak'

[Vít Šesták]

Hello,

I have some interesting updates. I have tried to:

a. Boot Fedora 31 on the laptop (Live version from USB drive) – adapter is detected and finds Wi-Fi networks. It just works.

b. Boot Fedora 31 Live (from the same USB drive) in a HVM with attached Wi-Fi card. It had 2000MiB of RAM. It fails in the same way as my previous attempts, not sure why.

This looks like the AppVM is fine, but there is some glitch in the PCI handling. It might be related to Xen or to the DM, not sure.

HVM: https://gist.github.com/v6ak/76f2c089c63b1fe184f3717d5bd5254e

sys-net with Fedora 31: https://gist.github.com/v6ak/30ecc502d1ce7508953eb3d505564668

I have also resolved the chicken-egg problem – I can connect to the Internet via USB. This is not a permanent solution, but it was good enough for updating dom0. However, the update (+ subsequent reboot) has not changed anything.

Regards,

Vít Šesták 'v6ak'

[Ilpo Järvinen]

On Thu, 19 Mar 2020, Vít Šesták wrote:

> Hello,
> I have some interesting updates. I have tried to:
>
> a. Boot Fedora 31 on the laptop (Live version from USB drive) – adapter is
> detected and finds Wi-Fi networks. It just works.
> b. Boot Fedora 31 Live (from the same USB drive) in a HVM with attached
> Wi-Fi card. It had 2000MiB of RAM. It fails in the same way as my previous
> attempts, not sure why.
>
> This looks like the AppVM is fine, but there is some glitch in the PCI
> handling. It might be related to Xen or to the DM, not sure.
>
> HVM: https://gist.github.com/v6ak/76f2c089c63b1fe184f3717d5bd5254e
> sys-net with Fedora 31:
> https://gist.github.com/v6ak/30ecc502d1ce7508953eb3d505564668
>
> I have also resolved the chicken-egg problem – I can connect to the Internet
> via USB. This is not a permanent solution, but it was good enough for
> updating dom0. However, the update (+ subsequent reboot) has not changed
> anything.

One option would be compile a kernel with CONFIG_IWLWIFI_TRACING (or
something like that) and try to provide the trace log to iwlwifi devs.
...It might not help though if it's not a HW/driver issue but xen/dm/pci
related thing.

--
i.

[Vít Šesták]

Well, maybe it would be better to just compile some newer StubDom.

Also, I have realized that there is some similar discussion on Github: https://github.com/QubesOS/qubes-issues/issues/5615

Regards,

Vít Šesták 'v6ak'

[Ilpo Järvinen]

On Thu, 19 Mar 2020, Vít Šesták wrote:

> Well, maybe it would be better to just compile some newer StubDom.
>
> Also, I have realized that there is some similar discussion on Github:
> https://github.com/QubesOS/qubes-issues/issues/5615

Yes, multiple person have reported this same issue also on this list.
I even saw one report in some in some external bug tracker. In that case
it was mixed up with some other issue within the same issue. The Intel
devs chose to prioritize the other issue over the one that seems to occur
in case of Qubes. It is very obvious that this HW was out when the driver
still had very subpar quality (Intel claimed support but many found out
the device just doesn't work, this wasn't at all Qubes specific at start)
so I wouldn't be surprised if it has something to do with the driver.

--
i.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I've seen very similar thing recently, not sure if exactly the same, but
it's very likely.
Sad news is neither me nor Paweł managed to fix it yet.
Things we've tried:
- various kernel versions (including 5.5 and 5.4)
- different firmware versions (apparently the driver tries to load
versions that new that are nowhere to be found yet)
- various options like permissive mode

I didn't spot VT-d errors, but I'm not entirely sure if I've checked.
If they are there, this is something definitely worth looking into and
most likely an issue within iwlwifi driver (or the firmware). It could
be also worth trying booting Fedora 31 Live directly, but add
intel_iommu=on kernel option. If that would break it, it's a clear
indication that the issue is somewhere between firmware and the driver.

>> 1. Problem: Domain sys-net did not boot at all because of issues with attaching ethernet PCI device.

Is it a Realtek card? I don't remember exactly what helped, but
something helped here. Paweł, can you help?
It was either attaching SD card reader (which is another function on the
same PCI device) to the sys-net, or enabling no-strict-reset option (or
maybe permissive?).

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl50AUwACgkQ24/THMrX
1yzCQwf/RHg7jCK7CS0ut98MoI2oDvRf6SJc6oVTNbbklovmmZcRwj9SrXcEw7j9
KQ0X/i7HpEr03MmMxOlQO8R4BUdXqZ5iyDWLnPLNRZimH2ftA55ndOaOqecaQZOc
nzxpeUyEHbO8D/ZwodRoTBF9Tl+e4lI7wOz/O6Ruy604++z3P5gzTuYVp390CiYU
Jt5suLKxoIuICO8EaRBT/5KGDM4BsuW9pfe2YDBs4USWg75D9C86KvgSGZhD6xd/
GwfPd3KXyiPTYHWT5fymupatiPnMVPKjpMQDyOvPbHJqUoUJ+owO/nqfSquWV8Mz
h4M2DffaBN/i8zxxtIWwh0nGLbX/kA==
=c9Wb
-----END PGP SIGNATURE-----

[Paweł Marczewski]

On 2020-03-20 00:33, Marek Marczykowski-Górecki wrote:
> Is it a Realtek card? I don't remember exactly what helped, but
> something helped here. Paweł, can you help?
> It was either attaching SD card reader (which is another function on the
> same PCI device) to the sys-net, or enabling no-strict-reset option (or
> maybe permissive?).

It was *not* attaching the SD card reader (that caused sys-net to crash
with Xen errors) and enabling no-strict-reset. I think the permissive
option turned out to be unnecessary.

Also, it didn't work at first and we had to reboot the computer, but
that might have been because we were trying other things as well.

The devices in question:

26:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd.
RTL8411B PCI Express Card Reader (rev 01)
26:00.1 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 12)


--
Paweł Marczewski
Invisible Things Lab

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Mar 20, 2020 at 01:05:02AM +0100, Vít Šesták wrote:
> Hello,

>
> On March 20, 2020 12:33:31 AM GMT+01:00, "Marek Marczykowski-Górecki" <marm...@invisiblethingslab.com> wrote:
> >I didn't spot VT-d errors, but I'm not entirely sure if I've checked.
> >If they are there, this is something definitely worth looking into and
> >most likely an issue within iwlwifi driver (or the firmware). It could
> >be also worth trying booting Fedora 31 Live directly, but add
> >intel_iommu=on kernel option. If that would break it, it's a clear
> >indication that the issue is somewhere between firmware and the driver.
>

> I have tried to add this option, but it remained to work. Does it mean that the driver itself is OK and the issue is in Xen or stubdom?

Likely, but not surely.

Some more ideas what could be wrong:
1. Some config space access is filtered and driver doesn't cope with it.
2. Some extended PCIe feature that driver/firmware assumes present is
not implemented in PCI passthrough.
3. Bug in handling any of supported PCIe features.
4. And there is still a possibility for a bug in the driver/firmware.

For the first hypothesis, I'd try enabling permissive option (AFAIR
didn't helped in itself), but then enable verbose logging in pciback
driver:
echo 1 > /sys/module/xen_pciback/parameters/verbose_request
before starting sys-net.

You'll get quite a lot of logs this way, and for understanding them
fully, PCI spec would be handy... But maybe there will be some less
obscure clues, like messages about explicitly failing requests?

For the second hypothesis, I'd take lspci -vv of the device from both
sys-net and dom0 (preferably in exact the same time, during enabling the
interface, but that's unrealistic). And compare. There will be definitely
some differences (more features visible in dom0), but what would be
valuable is:
- comparing configuration of features visible in both places
- correlating missing features with iwlwifi driver
It may be also useful to increase iwlwifi log level (I see 'debug'
module option, seems to be a bitmask).

For the third hypothesis, enable iwlwifi debugging and hope for more
details. Decoding that firmware error would also be useful, but unlikely
without firmware documentation or source code.

If everything above fails, I would thoroughly compare driver behavior on
bare metal and in the VM. Start with the driver debug output and if
still no clues, then log hardware interactions (may require modifying
the driver) and compare them.

Some of the above ideas are quite extreme, and tedious to execute...

PS adding the list back.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl50F2QACgkQ24/THMrX
1ywLuAgAjd/zJfxu9cHAo4h7vEib+0LWOTx55/hv8cGP9CsKNcpIdY6SJfg2Smy1
nhzh7BwxTkwnzGjYsmLzN+NX8uaTOXiLLdoEhoaOGbLekdsfJnXKLyXdu2AEhFHj
ejKYfWTRNLnqDWNViNNhFgx9vbOsasyiQWh0tMJm5cwUkXMz82DTjVqDXBBtgog1
86hMdLcSxRYVNw0q+KL3zmlagpbxDcmwnf0cV6NjyckQ1LWQ+pr/zx3FS74WatJP
D7k7dJ1M6rEdEDJaZ+K+XiXkLzJUufuwwKMlN4VL4SFvwmD5aglZm2B0rdcBBF9a
7pGs+ORAnWH4T4gnpiPi5OcfZWjiAQ==
=SAV0
-----END PGP SIGNATURE-----