lemur7 Unable to reset PCI device

[pixel fairy]

used a working desktop to install and update to qubes-unstable. this time the lemur was able to boot, but sys-net could not run. heres an hcl-report followed by a typescript session showing pci devices and the error. adding pci devices to sys-net in the qubes-manager gave the same result.

---
layout:
'hcl'
type:
'notebook'
hvm:
'yes'
iommu:
'yes'
slat:
'yes'
tpm:
'unknown'
brand: |
System76, Inc
model: |
Lemur
bios: |
5.12
cpu: |
Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
cpu-short: |
FIXME
chipset: |
Intel Corporation Device [8086:5904] (rev 02)
chipset-short: |
FIXME
gpu: |
Intel Corporation Device [8086:5916] (rev 02) (prog-if 00 [VGA controller])
gpu-short: |
FIXME
network: |
Intel Corporation Wireless 8260 (rev 3a)
Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 12)
memory: |
32655
scsi: |
Samsung SSD 850 Rev: 1B6Q
CT750MX300SSD1 Rev: 0100

versions:

- works:
'FIXME:yes|no|partial'
qubes: |
R3.2
xen: |
4.6.3
kernel: |
4.8.12-12
remark: |
FIXME
credit: |
FIXAUTHOR
link: |
FIXLINK

---

Script started on Mon 02 Jan 2017 09:33:34 PM PST
]0;user@dom0:~ [user@dom0 ~]$ lspci
00:00.0 Host bridge: Intel Corporation Device 5904 (rev 02)
00:02.0 VGA compatible controller: Intel Corporation Device 5916 (rev 02)
00:08.0 System peripheral: Intel Corporation Skylake Gaussian Mixture Model
00:14.0 USB controller: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller (rev 21)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-LP Thermal subsystem (rev 21)
00:16.0 Communication controller: Intel Corporation Sunrise Point-LP CSME HECI #1 (rev 21)
00:17.0 SATA controller: Intel Corporation Sunrise Point-LP SATA Controller [AHCI mode] (rev 21)
00:1c.0 PCI bridge: Intel Corporation Device 9d10 (rev f1)
00:1c.2 PCI bridge: Intel Corporation Device 9d12 (rev f1)
00:1c.5 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #6 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Device 9d58 (rev 21)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-LP PMC (rev 21)
00:1f.3 Audio device: Intel Corporation Device 9d71 (rev 21)
00:1f.4 SMBus: Intel Corporation Sunrise Point-LP SMBus (rev 21)
02:00.0 Network controller: Intel Corporation Wireless 8260 (rev 3a)
03:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTL8411B PCI Express Card Reader (rev 01)
03:00.1 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 12)
]0;user@dom0:~ [user@dom0 ~]$ qvm-start sys-net
--> Creating volatile image: /var/lib/qubes/servicevms/sys-net/volatile.img...
--> Loading the VM (type = NetVM)...
Traceback (most recent call last):
File "/usr/bin/qvm-start", line 136, in <module>
main()
File "/usr/bin/qvm-start", line 120, in main
xid = vm.start(verbose=options.verbose, preparing_dvm=options.preparing_dvm, start_guid=not options.noguid, notify_function=tray_notify_generic if options.tray else None)
File "/usr/lib64/python2.7/site-packages/qubes/modules/005QubesNetVm.py", line 122, in start
xid=super(QubesNetVm, self).start(**kwargs)
File "/usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py", line 1966, in start
self.libvirt_domain.createWithFlags(libvirt.VIR_DOMAIN_START_PAUSED)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1059, in createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirt.libvirtError: internal error: Unable to reset PCI device 0000:03:00.1: internal error: Active 0000:03:00.0 devices on bus with 0000:03:00.1, not doing bus reset
]0;user@dom0:~ [user@dom0 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
]0;user@dom0:~ [user@dom0 ~]$ ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
]0;user@dom0:~ [user@dom0 ~]$ exit
exit

Script done on Mon 02 Jan 2017 09:34:09 PM PST

[Jon Nash]

I have a similar issue on a modern laptop with a workaround for you which may help. My acer f5 573 55lv will also only work properly with either the wireless in the devices for sys-net or the wired (collectively or separately - as you have the same bridging as i do with a card reader plus ethernet arrangement - that i use so infrequently i forget if have just the ethernet & not the reader is ok). It's all down to the conflicts caused by the necessary evil of bridging & the demands of dealing with how xen currently copes (due to current capabilities & security). Give a try like this - boot laptop up, shutdown sys vms if they ever got started & then enter the device list of qubes vm qui specific to the sys-net & make sure that only either bdf 03:00.1 is listed or both 03 devices if you want wired access to work, bdf 02:00.0 if you want wireless to be possible. Then try to start sys-net & if it starts ok then try sys-firewall. I have noticed also issues where if you could get both wired & wireless devices assigned (different hardware) the nm solution seemed to only cope with the wireless from the gui if you disabled the wired from the sys-net devicelist (or i am misremembering something).

[Andrew David Wong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 2017-01-02 22:27, pixel fairy wrote:
> used a working desktop to install and update to qubes-unstable.
> this time the lemur was able to boot, but sys-net could not run.
> heres an hcl-report followed by a typescript session showing pci
> devices and the error. adding pci devices to sys-net in the
> qubes-manager gave the same result.
>

> [...]

Sounds like you might have the same device assigned to more than one
VM. When you try to start sys-net, is there already a VM running
(e.g., autostarting) with one or more of the same devices as those
assigned to sys-net?

- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJYa7dIAAoJENtN07w5UDAwWjsP/Av7m3ts3Bb7LMcuancJptZC
WOm2hHlDY52dxjVtVc3OC8dEtmNCUTaVnEpf6RmDu0WUdT3ejMIduMe7+jOnp9dK
ee9R7kuqlBJPBRJvvHSLteWPk/+SSZ+ZzsH1LeSYdg/F3v6Yzy9KoaE5Pia5+aUI
4MKBrpt9TFXvPmFGEyHnDWxx1SxG/AORPTFf/GQVIJI2/NMkPYKeP9uc8lwbnZRc
tPBznNWYbTtxlSTFOfXgAgMET7coqNYhO53F3rzSccViGoj9p/M4nIX+D9tzNWX+
0S/qzGveMoagmgt97rFjf+h/rkEinEHBravkiW54Nu8PHUwGbgIBwQknTt3PoAPU
IwEGwPCMoinZAOwTAe5WLuKoBjZKORlm7rqYmty0zJGeFXcXMhS9E8IYHojetcwy
Z+99ULnsH1ZbwP+Bc/neuhnE1PHvzZr8b/YKZ2+jU5DG/DQuQSyq4K4ujGhI/Ur2
hjABiuIN6DlpOj+dQO+CuvcVv1CYtUTSzwhsqTlNsbPyKfojcQmrjbfUTy/8ScYu
wqmVUtNuMNsiqn9HCj+fNc5TyX8jRlMhfrR1WIZWxe8VAQwzc6RjyVdG3W3I3Jhh
NtkixpjjH5LvLJqgG1zSRks+TWMfo+5NnfxhNcpQq6SAwJKhMLhwQFOy7CA++BrZ
x7zjbQ3uBZwQXqalNGVn
=9gJh
-----END PGP SIGNATURE-----

[Sean]

For Wifi, in sys-net keep 02:00.0, but remove (03:00.0 and) 03:00.1.

If you need Ethernet, good luck. I have been unable to get the Realtek Ethernet devices to function reliably -- barely at all -- on my Lemur, but Wifi works great.

For the error that you're getting, you need to add 03:00.0 to sys-net. However, even with that change, I was only sometimes able to get Ethernet connectivity.

Similar:
https://groups.google.com/forum/#!topic/qubes-users/Fs94QAc3vQI
https://github.com/QubesOS/qubes-issues/issues/1393

I played around with permissive settings but ultimately it is unneeded for Wifi and did not seem to impact the reliability of the Ethernet.

My experience:
Sometimes Ethernet would work only to not work upon reboot, regardless of the kernel version or configurations I attempted; although, my Linux troubleshooting skills are minimal so maybe I missed something in the kernel/xen and other logs. Yet, it seemed strange that it would work during one session but not after a VM and/or system reboot. Something about the 03:00.0/1 device does not play well with Xen/Qubes. So, unless you really need Ethernet, I recommend only connecting your Wifi device to sys-net. The 8260 works great when on its own.

[pixel fairy]

On Tuesday, January 3, 2017 at 11:36:23 PM UTC-8, Sean wrote:
> For Wifi, in sys-net keep 02:00.0, but remove (03:00.0 and) 03:00.1.
>
> If you need Ethernet, good luck. I have been unable to get the Realtek Ethernet devices to function reliably -- barely at all -- on my Lemur, but Wifi works great.

that worked. first machine ive had to manually assign the net device too. mine also didnt work when both devices were assigned to it. ill revisit this at the next stable release.

[spfmc...@gmail.com]

I finally took some time to troubleshoot the Ethernet issue. To get the Ethernet adapter to work, you need to blacklist the PCI Express Card Reader (RTL8411B) [03:00:0] module (rtsx_pci).

Add "modprobe.blacklist=rtsx_pci" to your <sys-net> kernelopts.

In dom0, you can determine your current kernelopts using qvm-prefs; replace <sys-net> with the name of your sys-net qvm.
[user@dom0 ~]$ qvm-prefs <sys-net> kernelopts

Once you have your current kernelopts, you can just append the modprobe command to it. For example, the following uses values from my system; replace <sys-net> with your sys-net qvm.
[user@dom0 ~]$ qvm-prefs -s <sys-net> kernelopts "nopat iommu=soft swiotlb=8192 modprobe.blacklist=rtsx_pci"

You should now be able to use your Ethernet port in that qvm. Just make sure that both [03:00:0] and [03:00:1] are attached. It will also work with your wireless adapter.

One additional note is that -- at least, on my system -- the qvm takes a bit longer than 20 seconds to shutdown, so you may see one warning about it taking too long. Just select "Wait another 20 seconds" and it should be good.

Ref: https://groups.google.com/forum/#!msg/qubes-devel/logslMTHyW4/khnEzBjAAwAJ

=================================Extra Info

You should be seeing an error in your Xen console qvm log when your kernel attempts to configure the 'rtsx_pci' module.

[ 2.357934] rtsx_pci 0000:00:00.0: Xen PCI mapped GSI16 to IRQ24
[ 2.358347] rtsx_pci 0000:00:00.0: rtsx_pci_acquire_irq: pcr->msi_en = 1, pci->irq = 25
[ 2.358473] BUG: unable to handle kernel paging request at ffffc900002f6010

[tenc...@gmail.com]

For Qubes-4.0 and the latest Lemur Laptop,
Blacklist the rtsx_pci module AND add the device to sys-net lets ethernet come up.