ERROR: PCI device <qubes.ext.pci.PCIDevice object at 0x...> does not exist

[Eric]

Hello, new user here.  Impressed with the design and philosophy of Qubes!

Everything seems to work as described excepting one problem. When I first installed R4.0 from the current iso on a mini PC (Celeron 3215U) it came up with the error message above in a dialog and no VMs were running. Worked out it was sys-net having a broken device attached, being the first of 3 network controllers found (2 * Realtek NICs + Qualcomm WiFi controller). Everything runs if I move that device out of the selected list for sys-net in Qubes manager.  Brought dom0 and all templates up to date and still have the problem.  If I assign that device to a standalone HVM I get the same error upon attempting a start, works fine with the other NIC.

The only PCI error I can find in the journal while booting dom0 is:
"dom0 libvirtd[2788]: 2018-09-20 19:01:02.052+0000: 2829: error : virPCIGetHeaderType:3129 : internal error: Unknown PCI header type '127'"

"qvm-pci list" does not show that NIC (understandable) however it is in the Qubes manager device selection list as:
"dom0:01_00.0   Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev ff)"
The NIC that is OK (device dom0:03_00.0) is the same except it has "(rev 07)" at the end.

I do need this ethernet port operational - any ideas?

Many thanks,
Eric

[awokd]

Eric:

> The only PCI error I can find in the journal while booting dom0 is:
> "dom0 libvirtd[2788]: 2018-09-20 19:01:02.052+0000: 2829: error :
> virPCIGetHeaderType:3129 : internal error: Unknown PCI header type '127'"
>
> "qvm-pci list" does not show that NIC (understandable) however it is in the
> Qubes manager device selection list as:
> "dom0:01_00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev ff)"
> The NIC that is OK (device dom0:03_00.0) is the same except it has "(rev 07)" at
> the end.

Your troubleshooting steps are sound. Test that "rev ff" device under a
different OS to rule out a hardware problem, and in dom0 do a "sudo
lspci -vvv" and compare the two Realtek entries against each other.
Check to see if there's a firmware update for your system, sometime
those include NIC firmware.

[Eric]

Your troubleshooting steps are sound. Test that "rev ff" device under a different OS to rule out a hardware problem, and in dom0 do a "sudo lspci -vvv" and compare the two Realtek entries against each other. Check to see if there's a firmware update for your system, sometime those include NIC firmware.

Thanks, the "lspci -vvv" gave:
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev ff) (prog-if ff)
    !!! Unknown header type 7f
    Kernel driver in use: pciback
    Kernel modules: r8169

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07)
    Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
+whole page of capabilities +

So it looks like the problem is current in the running kernel.  I have not brought up another OS fully as this PC is a tiny fanless box with only an SSD and I don't have another similar so I would have to trash Qubes.  I did start an instal of Centos7 from the current iso and Anaconda correctly detected both NICs and could detect the switch either one was plugged into correctly and after manually configuring enp1s0 connected to the LAN fine, so it looks like the hardware is OK.  It only just arrived from the manufacturer so the firmware should be up to date.

[Eric Mills]

Just adding a quick note that I did complete Centos7 instal onto a slow external USB drive and updated it fine through the problem NIC and still working after, then comfirmed still broken in Qubes (same result to lspci).

[Eric]

awokd,

 is there anything else you want me to do before I wipe Qubes off this box?

I put up a Centos7.4/Xen4.6 Dom0 (Kernel 4.9.127 which is way earlier than qubes 4.14.67) and everything seems to be running OK, which does seem a bit odd. Since this is a server application, and not a workstation, Qubes is a bit of an overkill. A centos7 in HVM will do here.

Perhaps there may be 3 tickets here: the PCI drivers, need to fix half alive devices - multiple inconsistent copies of state? and that error message in the subject here is not very informative for the novice user.

Thanks, Eric

[awokd]

Eric wrote on 9/30/18 10:35 PM:

When running Centos, does lspci -vvv give the same unusual output for
that problem device? Don't know what else to suggest; I have a Realtek
dual port RTL8111/8168/8411 PCI Express card in one of my systems and it
works fine under Qubes 4.0.

[Eric]

When running Centos, does lspci -vvv give the same unusual output for that problem device? Don't know what else to suggest; I have a Realtek dual port RTL8111/8168/8411 PCI Express card in one of my systems and it works fine under Qubes 4.0.

No "lspci -vvv" outputs a complete description of both controllers as expected and 01:00.0's first line ends with "(rev 07)" as it should.

[awokd]

Eric wrote on 10/1/18 10:29 AM:

Strange, sounds like Qubes is initializing just that one device wrong
then. I had a similar issue with an old PCIe v1.1 wireless NIC. What
version is your NIC? I never got mine working, but you might try adding
"xen-pciback.hide=(01:00.0)" to your boot options in xen.cfg and seeing
if it will let you assign it to sys-net then.

[Eric]

With boot option "xen-pciback.hide=(01:00.0)" (on the options= line in default config in /boot/efi/EFI/qubes/xen.cfg) everything comes up exactly the same. Getting the same error on PCI while booting and same error attempting to start sys-net with 01:00.0 attached. That option appears to be having no effect.

The hardware is new as mentioned previously - there are 3 PCI bridges (ports 3,4&5) same as:
"00:1c.0 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #3 (rev e3) (prog-if 00 [Normal decode])"

This is the bad lspci output for NIC 1:

"01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev ff) (prog-if ff)
    !!! Unknown header type 7f
    Kernel driver in use: pciback
    Kernel modules: r8169"

Compared to NIC 2:

"03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07)
    Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 16
    Region 0: I/O ports at d000 [size=256]
    Region 2: Memory at f7000000 (64-bit, non-prefetchable) [size=4K]
    Region 4: Memory at f0000000 (64-bit, prefetchable) [size=16K]"
...lots of capabilities...

   "Kernel driver in use: pciback
    Kernel modules: r8169"

Only other thing to note am getting "pcilib:sysfs_read_vpd: read failed: Input/output error" on stderr when running lspci for the above output in the good NIC capabilities section, however Centos gets two of these messages, one in the same place for each NIC so I am presuming it is not related to the problem I have with Qubes.