| PCI passthrough working in Qubes R4.0? | mossy-nw | 17/01/18 15:54 | Dear Qubes community,
Has anyone using Qubes R4.0 been able to resolve any issues (e.g. with wifi) by setting PCI permissive mode? (as described in:) https://www.qubes-os.org/doc/assigning-devices/#pci-passthrough-issues I successfully used this in Qubes R3.2 to get wifi working for dell xps 13 9350 (bcm4350) but am having no luck using this method in R4.0_rc3. This situation seems identical (driver errors that persist but nonetheless wifi works using permissive mode) resolved by in a previous (R3.2) qubes-users thread by miaoski -- https://groups.google.com/forum/#!msg/qubes-users/m4GzpfOVBiQ/bf5XCEc-DQAJ I'm hesitant to give up and go back to R3.2 on this machine, so I don't have access to the R3.2 dom0 dmesg output for PCI spermissive mode, but I recall before setting permissive mode dmesg suggested doing so. This is not the case in R4.0_rc3 however. Some dmesg output, if this gives anyone any ideas: [dom0 ~] lspci | grep -i network 3a:00.0 Network controller: Broadcom Limited BCM4350 802.11ac Wireless Network Adapter (rev 08) Before setting permissive mode: [dom0 ~]$ sudo dmesg | grep [ 7.222306] pci 0000:3a:00.0: [14e4:43a3] type 00 class 0x028000 [ 7.222348] pci 0000:3a:00.0: reg 0x10: [mem 0xdc400000-0xdc407fff 64bit] [ 7.222379] pci 0000:3a:00.0: reg 0x18: [mem 0xdc000000-0xdc3fffff 64bit] [ 7.222618] pci 0000:3a:00.0: supports D1 D2 [ 7.222619] pci 0000:3a:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 7.222766] pci 0000:3a:00.0: System wakeup disabled by ACPI [ 8.053279] pciback 0000:3a:00.0: seizing device [ 68.211970] xen_pciback: vpci: 0000:3a:00.0: assign to virtual slot 0 [ 68.212362] pciback 0000:3a:00.0: registering for 4 [ 68.217475] pciback 0000:3a:00.0: enabling device (0000 -> 0002) After setting permissive mode (and rebooting): [dom0 ~]$ sudo dmesg | grep 3a:00 [ 7.256107] pci 0000:3a:00.0: [14e4:43a3] type 00 class 0x028000 [ 7.256150] pci 0000:3a:00.0: reg 0x10: [mem 0xdc400000-0xdc407fff 64bit] [ 7.256181] pci 0000:3a:00.0: reg 0x18: [mem 0xdc000000-0xdc3fffff 64bit] [ 7.256419] pci 0000:3a:00.0: supports D1 D2 [ 7.256421] pci 0000:3a:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 7.256568] pci 0000:3a:00.0: System wakeup disabled by ACPI [ 8.099473] pciback 0000:3a:00.0: seizing device [ 57.896166] pciback 0000:3a:00.0: enabling permissive mode configuration space accesses! [ 57.896168] pciback 0000:3a:00.0: permissive mode is potentially unsafe! [ 72.466856] xen_pciback: vpci: 0000:3a:00.0: assign to virtual slot 0 [ 72.467231] pciback 0000:3a:00.0: registering for 2 [ 72.472685] pciback 0000:3a:00.0: enabling device (0000 -> 0002) could it be anything to do with assign to virtual slot 0, or qvm-pci? Many thanks for anyone's input, -m0ssy |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | Marek Marczykowski-Górecki | 17/01/18 16:46 | -----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256 Did you get anything after that message? Permissive mode helps mostly with cases reported in dom0 dmesg with message like this: pciback 0000:03:00.0: Driver tried to write to a read-only configuration space field at offset 0x110, size 4. This may be harmless, but if you have problems with your device: 1) see permissive attribute in sysfs 2) report problems to the xen-devel mailing list along with details of your device obtained from lspci. Interesting would be to see messages in sys-net about that device driver. You may also try updated kernel for VM, there is kernel-qubes-vm-4.14.13-1 package in current-testing repository: https://www.qubes-os.org/doc/software-update-dom0/#testing-repositories - -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlpf7kIACgkQ24/THMrX 1yzK7Qf/bu/GFioi0K4hWy5L5DgzI08Q1CcIGoupw3gXXMhOxzecyy/Ak+W1xPYs m1PBk7EAaCLa2Rr9iUCZiP2vXKiMEdKT6NvxDF7GU/c4YlLRiAHcOXtKnPQjy/GT TH29dN4oXNLzQhgcXxVg7qKvigE3BeDY5a1EYbByW+2gacwRqjPxnDvTjnMx/G4r h/GhdjOs4qhQYQ04OdbIBxmx4Yec+SWznS6xDCzaO3lA4f6/LiyW9+UiXpNb1o92 4yVTZEdEu086eurndITYnS2lEI7h3r7OP8YySFFKlZlaNMOkqZa+eOyGQLhUw+Jz jNIEyqwwgJVj+ggKXd9Y9vt6Rki1dA== =o9wL -----END PGP SIGNATURE----- |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | mossy-nw | 17/01/18 17:06 | Marek Marczykowski-Górecki:
Thanks, Marek--I will try the updated kernel (though sys-net has the appropriate driver binary file, and this was working under R3.2 for many months i.e. with older kernels). Here are relevant messages from sys-net: [user@sys-net ~]$ lspci -k 00:05.0 Network controller: Broadcom Limited BCM4350 802.11ac Wireless Network Adapter (rev 08) Subsystem: Dell Device 0023 Kernel modules: brcmfmac [user@sys-net ~]$ sudo dmesg | grep brcmfmac [ 1.703756] usbcore: registered new interface driver brcmfmac [ 1.970170] brcmfmac: brcmf_chip_recognition: SB chip is not supported [ 1.970185] brcmfmac: brcmf_pcie_probe: failed 14e4:43a3 [user@sys-net ~]$ modinfo brcmfmac | grep -E '^(description|author|depends):' description: Broadcom 802.11 wireless LAN fullmac driver. author: Broadcom Corporation depends: mmc_core,brcmutil,cfg80211 Trying advice from - https://www.qubes-os.org/doc/wireless-troubleshooting - I tried disabling/reenabling brcmfmac and some combinations of those others (mmc_core,brcmutil,cfg80211; reversing remove/add order), but no luck. I also tried using as the sys-net template qubes-template-debian-9 with broadcom-sta-dkms and others (which I recall got this working for running non-Qubes debian on this laptop), to no avail. Thanks again, -m0ssy |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | Marek Marczykowski-Górecki | 17/01/18 17:17 | -----BEGIN PGP SIGNED MESSAGE-----Do you see other relevant kernel messages about that time (not necessary containing brcmfmac)? I assume this is after enabling permissive mode, right? Simon, is setting permissive mode in dom0 enough? Or maybe qemu in stubdomain perform some additional filtering?
- --iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlpf9Z0ACgkQ24/THMrX 1yz77gf+JxndavVwAAduAI/vwF6dHWtP9nrTCEESUh/4zna2lWZzWnhbvq8lufOe NmXQW67PqlytfKDr2s2dPVsslb9/HN1d3LCKvB3nyS5qOT986BQX8yAwNlUoObFk iFG1gmzVepvxWlPgKJ9AYpnyeXwyBxytymebwFrD21WdxnHo6ZGjzWw86Dfu6u5d KRXTcZms4l3COGqEjVlWU87fAqWJj3BmpeYt7zWXuD9mXf73GB8Pzk3gmSEtJZSG CnbBTskFqqtBMWJ63amTqoG9DjJUMsRK2dZMwbXvVxgWge1AasnPIBGT49FgLPxH ZiRdONBI33y9rBe6ky0UJBnAegG8ew== =41YY -----END PGP SIGNATURE----- |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | awokd | 18/01/18 06:12 | On Thu, January 18, 2018 1:17 am, Marek Marczykowski-Górecki wrote: >>> On Wed, Jan 17, 2018 at 11:54:23PM +0000, mossy-nw wrote: >>>> Has anyone using Qubes R4.0 been able to resolve any issues (e.g. > I assume this is after enabling permissive mode, right?Per device permissive mode is broken in 4.0. I was trying to figure out where exactly while troubleshooting my Atheros card. I found where the init script in the stubdomain reads the device values from xenstore, but I couldn't find anywhere that actually added per device permissive entries to xenstore (whereas I did find where msitranslate and power_mgmt got set). For my troubleshooting, I set the init script to default to permissive=true which results in every device being added as permissive but it didn't help my Atheros case anyways. |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | Marek Marczykowski-Górecki | 18/01/18 06:26 | -----BEGIN PGP SIGNED MESSAGE-----According to logs provided by mossy-nw permissive mode is correctly enabled in xen-pciback in dom0 for this device. The question here is what else is needed for HVM (using qemu in stubdomain). iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlpgrokACgkQ24/THMrX 1yx8tQf9GA3s9nLdFQjLeQ6SRcsQGUVd/BSUkz/n+ssrcJHOm3xmcbi7bxE5/Pyd iubkwqZwxCkNOLD1SAVVGEXmcPCh1mLvxCNhms2s16J0YQSBYTYEEgytgc3Maf6N SZ80cCSnj4wM/OOc3xG7dbjq7cB827OyE4l3wxTzjnCKubRK6BKLMrJp9S/mG0O6 q45Hr1q75kRwZwRg77DAwq7wkVt9CSk8+S9tltlA6Cj0zK6wnw+4Enp21QHCgEim z78RvpHZEjQefQAxIX9omIR1QsDvWj4uLSdQTisJZUseOU23rSqxtNU1MNisjKNn 88zC1Z4pJP6MX8SmfRlFR6RdkWi3lg== =Nn3O -----END PGP SIGNATURE----- |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | awokd | 18/01/18 06:50 | On Thu, January 18, 2018 2:26 pm, "Marek Marczykowski-Górecki" wrote:My dom0 log shows that too, but the guest-dm/debug log showed the PCI device always getting added with permissive=false. Maybe that's what you are saying, qemu isn't passing the value along? |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | Simon Gaiser | 18/01/18 07:06 | awokd:
Yes that's the problem. The guide sets permissive mode in pciback and not via libxl. So the stubdomain never learns about it. Will take a look today. |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | Marek Marczykowski-Górecki | 18/01/18 07:26 | -----BEGIN PGP SIGNED MESSAGE-----If that's the case, it looks we need another option for PCI device, like this: qvm-pci attach sys-net dom0:xx_yy.zz -o permissive=True iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlpgvHYACgkQ24/THMrX 1ywg3Qf/c/gV4AycMBti0O65ZP8s2tROvOBH8JkAZlNkO988GeQrPc4/O6fW4wR8 sfLLPzfylFTqcYm/gX8/UmNY01b/v167zUf2SLdqvCEqt08hykJ+fVII0tLuPz6c fxvCyjrF3Q4zRdUaT3irL20TYoVPYZrbJwvtVk0gtHdmSMcmQIVQ4gU3isf3hYu9 RdRJ58Qulom9fmAlTu1LeQWdnzFLFW7ryIMtM7xaYC1RawLlEuxeIUjOCHxZP0Uw zfexZkFyHHFTpGSE0tJT47jhK8JiHOIj+YEFASeaNu9DTTsGPTZxmnDD0aAh0FvT OFXRpDPqIh31xQlnoHI7fGokETOTDg== =tSUs -----END PGP SIGNATURE----- |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | Simon Gaiser | 18/01/18 07:35 | Marek Marczykowski-Górecki:> On Thu, Jan 18, 2018 at 03:06:00PM +0000, Simon Gaiser wrote:Yes, exactly. And if I didn't miss something we first need to teach libvirt about the permissive option. |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | awokd | 18/01/18 07:43 | Was trying to beat you to it but got lost in all the layers! Does anyone
know of a flowchart somewhere of what happens when you enter "qvm-start vm" in 4.0? I gathered: qvm-start qubesadmin -> QubesDB salt libxl calls Xen code-> stubdomain with qemu qemu uses libxl calling Xen code to start vm |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | Marek Marczykowski-Górecki | 18/01/18 08:16 | -----BEGIN PGP SIGNED MESSAGE-----Not exactly... 1. qvm-start sends a request to qubesd, using Admin API 2. qubesd starts required netvm (recursively), if needed 3. qubesd request qmemman to allocate needed memory for new VM (according to VM's 'memory' property) 4. qubesd calls into appropriate storage pool driver to prepare for VM startup (create copy-on-write layers etc) 5. qubesd gathers needed VM properties etc and builds libvirt VM configuration (XML format, can be seen using `virsh dumpxml`) 6. qubesd calls into libvirt to start the VM (but in paused mode) 7. libvirt setup the VM using libxl, this include starting stubdomain if needed 8. qubesd start auxiliary processes, including: - qrexec-daemon - qubesdb-daemon (and fill its content) 9. libvirt unpause the VM 10. qvm-start-gui process (running separately from qubesd, as part of dom0 user GUI session) starts gui daemon iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlpgyFIACgkQ24/THMrX 1yyEEwf+K1bF8s/zig+WmCFKfVrF+G9kFTC0l4FWejwbNBd3pslgz2ED6fu9S3S/ j/5OpEj40DF404FT7i1c9R6J0+hWvrIcuizZUJ3sY0Ip1na12sivxInhesLsOzMk Qn/tz5Tob1HuowJEZG5CwBo/WkVZvkDmzn9hBFDg3hOoRPBg/kdFNJduQM+/zYGq 0SPfG9dUDKlDsi41hV4/ciZJiMRwOJIhaPj8nioyBUxrZSrhy8V1Z4/q8sCXxPu3 h8akFQxqyb27D7FjCgZALiISR0Er5pm3vsVhaNamEMV98A16BbmjHJAAbtTFGMUL aFqppX3FK0v/g+RogLvyoBPxHYKkkA== =eXzz -----END PGP SIGNATURE----- |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | awokd | 18/01/18 08:45 | On Thu, January 18, 2018 4:16 pm, "Marek Marczykowski-Górecki" wrote:Thank you, sir. No wonder I was having trouble following the control flow. |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | Marek Marczykowski-Górecki | 18/01/18 08:48 | -----BEGIN PGP SIGNED MESSAGE-----For the record, most of the above is in "start" method: https://dev.qubes-os.org/projects/core-admin/en/latest/qubes-vm/qubesvm.html#qubes.vm.qubesvm.QubesVM.start (there is "source" link at the right side) iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlpgz6YACgkQ24/THMrX 1ywAeQgAiNAlBk75F0ezCZHOgPed8Rd3qc8/pP7zRTYeWUOUCiDxSky+FSompw/k 1JS/ROgisCcma+16W2EPmGPDkxUjkJJcL+OJj/jZGZolxnFKoY3Atbu42qEckAlG br11XSXtdVuV3IPR3nHQhdCn5BltwAgw0QE80ziekvb+9Fh+bIdzH8S9qyk/sVbf WML6a/Gn81Nh4dWyktLL5V0onZmpn0wbRWhyXFGi9HUOJUgnl7WImLgjGD+3YBch iCVwULNxewDA0Nkfq2UARnsODjP23M/9CNplUAq1oPumtQYIcsc4wGGDcZH/xXo4 aAhtZskNopUgBeCvBWGr7sywGSs1LA== =G7vM -----END PGP SIGNATURE----- |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | mossy-nw | 18/01/18 08:50 | Marek Marczykowski-Górecki: Sounds like you all are way ahead of me on this thread now (great!)--but yes, this is after permissive mode enabled. I want to add upon first booting single clicking the dock icon for sys-net reads "device not managed" but after sleeping and waking there is no longer any entry at all. I don't feel confident to judge for sure what's relevant, but here are some things that might be. [ 1.979068] FUJITSU Extended Socket Network Device Driver - version 1.1 - Copyright (c) 2015 FUJITSU LIMITED [ 1.980511] piix4_smbus 0000:00:01.3: SMBus Host Controller not enabled! [ 1.981868] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.982959] ehci-pci: EHCI PCI platform driver [ 1.998059] uhci_hcd: USB Universal Host Controller Interface driver [ 1.999999] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 2.000540] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 2.000693] uhci_hcd 0000:00:01.2: detected 2 ports [ 2.001491] uhci_hcd 0000:00:01.2: irq 23, io base 0x0000c200 [ 2.002251] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 2.002286] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 2.002304] usb usb1: Product: UHCI Host Controller [ 2.002316] usb usb1: Manufacturer: Linux 4.9.56-21.pvops.qubes.x86_64 uhci_hcd [ 2.002332] usb usb1: SerialNumber: 0000:00:01.2 [ 2.002720] hub 1-0:1.0: USB hub found [ 2.002736] hub 1-0:1.0: 2 ports detected [ 2.006457] FDC 0 is a S82078B [ 2.023658] [drm] Initialized [ 2.058903] [drm] Found bochs VGA, ID 0xb0c0. [ 2.058917] [drm] Framebuffer size 16384 kB @ 0xf1000000, mmio @ 0xf242c000. [ 2.066469] [TTM] Zone kernel: Available graphics memory: 177706 kiB [ 2.066491] [TTM] Initializing pool allocator [ 2.066509] [TTM] Initializing DMA pool allocator [ 2.152836] ppdev: user-space parallel port driver [ 2.189430] intel_rapl: Found RAPL domain package [ 2.189437] intel_rapl: Found RAPL domain core [ 2.189444] intel_rapl: Found RAPL domain uncore [ 2.189450] intel_rapl: Found RAPL domain dram [ 2.202672] usbcore: registered new interface driver brcmfmac [ 2.354325] fbcon: bochsdrmfb (fb0) is primary device [ 2.425230] Console: switching to colour frame buffer device 128x48 [ 2.456353] bochs-drm 0000:00:03.0: fb0: bochsdrmfb frame buffer device [ 2.458278] usb 1-1: new full-speed USB device number 2 using uhci_hcd [ 2.465078] [drm] Initialized bochs-drm 1.0.0 20130925 for 0000:00:03.0 on minor 0 [ 2.465584] xen: --> pirq=16 -> irq=36 (gsi=36) [ 2.466104] brcmfmac: brcmf_chip_recognition: SB chip is not supported [ 2.466121] brcmfmac: brcmf_pcie_probe: failed 14e4:43a3 I can DM you full logs if you like. Finally, I am able to get networking working with a usb wired ethernet adapter, then assigning the USB device to sys-net. If you want logs for this, LMK. I'm curious--would it be better for security to avoid PCI permissive mode and instead use a USB network adapter for wifi? With IOMMU I don't know how much permissive mode weakens isolation (at all?) Thanks again, you all! -m0ssy |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | Marek Marczykowski-Górecki | 18/01/18 09:34 | -----BEGIN PGP SIGNED MESSAGE-----I don't see anything else related to this device. So until Simon gets permissive mode sorted out, it look like you have to use that usb ethernet. In theory it should be safe (IOMMU with HVM as used in Qubes 4.0). In practice it largely depends on the device, chipset and such. There are some devices that could crash the system after specific config space modification (normally blocked in non-permissive mode). I'm not aware of any combination allowing privilege escalation (IMO that would be considered a IOMMU bug). But, as we all know, hardware bugs happens... iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlpg2o4ACgkQ24/THMrX 1yxEWwf/cED94BTrpq3YmDQO3Pp4EpNB4vdwmTSCuIZpaN6ZZ7VtDB6POxpeB6cw Vfz6mmuetSMnrMy8jizh81roxAN3MuF6QQyNDYu4nsZpJa7q4FaVu0s1E7WIGyjB cmCf5gEUfd4z1oNfXgC5q1nq/7Bn4ITtApdKEQwS0+ntiSt+Un/RollCJNfa9/w5 R3/i3/B1Fc1lrzTksG/J28eJp6JlRkPwldFU43T8p27KF6wN6PXuu3Kezh8uFXLT TgRSzIk8xXOoh81X1/RU8lKMR1DMxYz2hcLbgQWwUZ7OS2AVCSmWQR81RQGYeBqw QOMtlPhERRbGBxXWSnz92+z8k9pUPA== =lEQR -----END PGP SIGNATURE----- |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | Marek Marczykowski-Górecki | 18/01/18 11:29 | On Thu, Jan 18, 2018 at 07:21:14PM +0000, mossy-nw wrote:
> OK thanks, Marek. I'll try to help get that backup bug reproduced > then... should I file a permissive mode bug report based on this > discussion, or do you feel like all of the relevant folks are looped in > on this thread already? Even if relevant people are aware of it, a github issue would be useful for tracking purpose (for example to easily check what package version contains a fix). |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | mossy-nw | 18/01/18 16:05 | Marek Marczykowski-Górecki:
> On Thu, Jan 18, 2018 at 07:21:14PM +0000, mossy-nw wrote:Great -- please see issue #3476 <https://github.com/QubesOS/qubes-issues/issues/3476> -- I linked this thread there and pasted what I see as most relevant comments from you all here, feel free to crticize/clarify. thx, -m0ssy |
| Re: [qubes-users] PCI passthrough working in Qubes R4.0? | Simon Gaiser | 18/01/18 20:10 | Marek Marczykowski-Górecki:
[...] > I don't see anything else related to this device. So until Simon getsFYI: https://github.com/QubesOS/qubes-core-admin/pull/184 |