HCL - Gigabyte B650E Aorus Master - AMD Ryzen 9 7950X

[Edwin Török]

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  32
  On-line CPU(s) list:   0-31
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 9 7950X 16-Core Processor
    CPU family:          25
    Model:               97
    Thread(s) per core:  2
    Core(s) per socket:  16
    Socket(s):           1
    Stepping:            2
    Frequency boost:     enabled
    CPU(s) scaling MHz:  52%
    CPU max MHz:         5879.8818
    CPU min MHz:         3000.0000
    BogoMIPS:            8999.60
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep
mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx
mmxext f
                         xsr_opt pdpe1gb rdtscp lm constant_tsc
rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl
pni pclmul
                         qdq monitor ssse3 fma cx16 sse4_1 sse4_2
x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm
extapic cr
                         8_legacy abm sse4a misalignsse 3dnowprefetch
osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext
perfctr_llc m
                         waitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba
perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2
smep bmi2
                          erms invpcid cqm rdt_a avx512f avx512dq
rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw
avx512vl xs
                         aveopt xsavec xgetbv1 xsaves cqm_llc
cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf
xsaveerptr rdpr
                         u wbnoinvd cppc arat npt lbrv svm_lock
nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter
pfthreshold
                         avic v_vmsave_vmload vgif x2avic v_spec_ctrl
vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq
avx512_vnn
                         i avx512_bitalg avx512_vpopcntdq rdpid
overflow_recov succor smca fsrm flush_l1d
Virtualization features:
  Virtualization:        AMD-V
Caches (sum of all):    
  L1d:                   512 KiB (16 instances)
  L1i:                   512 KiB (16 instances)
  L2:                    16 MiB (16 instances)
  L3:                    64 MiB (2 instances)
NUMA:                   
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-31
Vulnerabilities:        
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled
via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and
__user pointer sanitization
  Spectre v2:            Mitigation; Enhanced / Automatic IBRS, IBPB
conditional, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected
edwin ~ [4.14.1] ❯ cat /mnt/home/edwin/Qubes-HCL-
Gigabyte_Technology_Co___Ltd_-B650E_AORUS_MASTER-20230807-162029.yml
---
layout:
  'hcl'
type:
  'Desktop'
hvm:
  'yes'
iommu:
  'yes'
slat:
  'yes'
tpm:
  '2.0'
remap:
  'yes'
brand: |
  Gigabyte Technology Co., Ltd.
model: |
  B650E AORUS MASTER
bios: |
  F3c
cpu: |
  AMD Ryzen 9 7950X 16-Core Processor
cpu-short: |
  FIXME
chipset: |
  Advanced Micro Devices, Inc. [AMD] Device [1022:14d8]
chipset-short: |
  FIXME
gpu: |
  Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600
OEM/5600 XT / 5700/5700 XT] [1002:731f] (rev c0) (prog-if 00 [VGA
controller])
  Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c1)
(prog-if 00 [VGA controller])
gpu-short: |
  FIXME
network: |
  Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 01)
  MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
[14c3:0616]
  0616]
memory: |
  64663
scsi: |
  SanDisk SDSSDXPS Rev: 00RL
  TOSHIBA MG09ACA1 Rev: 0104
  TOSHIBA HDWD130  Rev: ACF0
usb: |
  4
certified:
  'no'
versions:
  - works:
      'partial'
    qubes: |
      4.2.0-alpha
    xen: |
      4.17.1
    kernel: |
      6.1.35-1
    remark: |
      Must create a GPT partition table on the boot media with 'gdisk',
see https://github.com/QubesOS/qubes-issues/issues/8395
      Must ensure that installation target is NOT 4Kn, had to reformat
NVME namespace to 512 byte, see
https://github.com/QubesOS/qubes-issues/issues/7398#issuecomment-1668545707
      Must enable IOMMU in BIOS (default is Auto which doesn't work.
Enable it under 'Misc settings and AMD CBS -> NBIO settings').
      Must NOT use a USB qube. Using a USB qube results in an instant
host reboot as soon as the USB qube is booted (same issue happen on KVM
when attempting to pass through any USB controller, even if the
controller is in an IOMMU group of its own). There is a newer BIOS with
newer AGESA available, I'll have to retry using that. Using permissive
mode or non-strict PCI reset doesn't help:
   13:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device
15b8 (prog-if 30 [XHCI])
   Passthrough of network to sys-net does work.
   I have 4K screens, had to adjust the Font DPI in the appearance
settings to 192 DPI, although that doesn't seem to propagate to qubes
(https://github.com/QubesOS/qubes-issues/issues/1951)
   Also tested Qubes 4.1.2 with kernel-latest, but it just results in a
black screen, so 4.2.0-RC1 seems to be the 1st version that works on
this HW.
   I'm not submitting this report from within Qubes itself due to the
lack of USB (lack of working sys-usb) or U2F passthrough (see
https://github.com/QubesOS/qubes-issues/issues/8298) which would be
required to unlock my password manager.
    credit: |
      FIXAUTHOR
    link: |
      FIXLINK

[Sven Semmler]

Thank you Edwin for your HCL report, which is online now:

https://www.qubes-os.org/hcl/#gigabyte_b650e-aorus-master_ryzen-9-7950x_integrated-graphics-amd-radeon-rx-navi-10_edwin-t%C3%B6r%C3%B6k_4-2-0-alpha

/Sven

--
https://keys.openpgp.org/vks/v1/by-fingerprint/DA5975C9ABC40C833B2F620B2A632C537D744BC7

[Edwin Török]

On Mon, 2023-08-07 at 21:45 +0100, 'Edwin Török' via qubes-users wrote:
>
>       Must NOT use a USB qube. Using a USB qube results in an instant
> host reboot as soon as the USB qube is booted (same issue happen on
> KVM
> when attempting to pass through any USB controller, even if the
> controller is in an IOMMU group of its own). There is a newer BIOS
> with
> newer AGESA available, I'll have to retry using that. Using
> permissive
> mode or non-strict PCI reset doesn't help:

New BIOS doesn't help, but I found one source of the host reboot: bus
reset of the USB3 controller PCI device.

/sys/bus/pci/devices/0000:13:00.0/reset_method has 'bus',
and its parent /sys/bus/pci/devices/0000:00:08.3/reset_method has 'pm'.
No FLR on these (which is confirmed by 'lspci' it says FLReset-).

Changing 'bus' to 'pm', and performing an unbind from
/sys/bus/pci/drivers/xhci_hcd, and then echoing '1' to 'reset' seems to
survive and it no longer causes the system to power off.
(now I don't know whether the problem is with reseting the device, or
resetting the parent's secondary bus, IIUC even if parent device's
reset method is pm, secondary bus will still be reset if that is what
the child does).

Note that all the other USB controller's reset_method is already 'pm'.

All but one USB controller is in D3hot (which is expected), the other
is D0. Is a bus reset of a device in D3hot even valid?

Although changing the 'reset_method' still doesn't make device pass-
through work: attempting to pass it through still causes a host reboot
(although more like a poweroff followed by a powerup than a warm
reboot, I tried disabling various options in the BIOS but it still
powers off directly, e.g. the PSP/SMU debug mode appeared promising but
didn't help).

Best regards,
--Edwin

[Edwin Török]

I tried plugging a USB device in the controller so it went to D0 and
changes reset method to pm on its own, but that didn't help.

However I found out that I *can* pass-through 2/4 USB controllers,
these ones are safe to pass through:
```
+-02.1-[06-10]----00.0-[07-10]--+-00.0-[08]--
| +-0c.0-[0f]----00.0
Advanced Micro Devices, Inc. [AMD] Device [1022:43f7]
+-08.1-[12]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Raphael
[1002:164e]
| +-00.3 Advanced Micro Devices, Inc. [AMD]
Device [1022:15b6]
```

These ones are NOT safe to pass through, they result in instant power
off of the host, followed by a poweron couple of seconds later:
```
+-08.1-[12]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Raphael
[1002:164e]
| +-00.4 Advanced Micro Devices, Inc. [AMD]
Device [1022:15b7]
+-08.3-[13]----00.0 Advanced Micro Devices, Inc. [AMD] Device
[1022:15b8]
```

This is very weird because 12.03 and 12.04 are quite similar from a
topology point of view.

I'm not sure what Qubes could do to detect or avoid this situation, but
perhaps just hiding the USB controllers that are in D3hot instead of
passing them through is the safer option?
But even that becomes unsafe if I plug a USB device into 12.04.

Anyway I have a workaround now to at least pass some of my USB devices
through, and I have my Yubikey working in my 'vault' VM.

Best regards,
--Edwin

[Ulrich Windl (Google)]

Edwin,

your findings are interesting, but hard to follow, because you didn't write which devices are connected to which USB port; or did I miss it?

Kind regards,
Ulrich

[51l...@ileg.al]

Its interesting, since my amd legion is testing machine, i never use
sys-usb, and yeah i found that usb passthrough lead system to reboot.
can you provide a more detail about step by step what have you done ?

Thanks

[Edwin Török]

At install time untick the 'sysusb' creation, and then create sysusb by
hand and attach just 1 USB controller at a time (following the
instructions here
https://www.qubes-os.org/doc/usb-qubes/#how-to-create-a-usb-qube-for-use-with-a-usb-keyboard)

Another way to find out which device is causing the system to poweroff
is to use any other VM (e.g. 'personal'), change it to HVM, and go to
the devices tab in settings and try passing through one controller at a
time, and make a note which one worked and which one didn't.
If it works, power off the VM, remove the device from it and try the
other ones in turn.
If the machine gets instantly powered off instead then make a note
about the broken device.

I'll write some step-by-step examples once I got it working a bit more
reliably, but for now the next device that is misbehaving sometimes is
the network card:

0e:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express
Wireless Network Adapter

Sometimes it just keeps printing messages that it is waiting after FLR,
and the boot doesn't actually finish (or maybe I didn't wait long
enough, I gave up after half a minute after it has printed a few retry
messages about FLR).
Other times it works reliably (it seems to misbehave if I boot into
Qubes directly after the machine has been powered off, but seems to
work if I first boot into native Fedora and then reboot into Qubes).

Best regards,
--Edwin