It's my first time installing Qubes and I need some help with my network adapters.
I am using Qubes 4.0.1
The hardware is Fitlet2 with Intel Celeron J3455 (supports both VT-x and VT-d).
It comes with 4 Gigabit Ethernet ports using Intel i211.
Installation was successful but EFI boot entry could not be added. Editing EFI entries using tty2 (Ctrl-Alt-F2) or 'Rescue a Qubes' mode both throw errors. Eventually solved by booting into Live Ubuntu and editing EFI entries from there.
Successfully booted into Qubes but couldn't get network working.
1. Physically, port LEDs go off and start blinking after awhile. Keeps repeating.
2. Network icon on top right is red and shows 'loading animation'
3. Network adapter can negotiate speed and duplex but cannot get IP address from DHCP server
4. Changing of network cable and connecting to another switch gives the same results
5. dmesg shows adapter in reset cycle
6. Network adapter works perfect when boot into Live Ubuntu using same hardware
dmesg from Qubes
[user@sys-net ~]$ sudo dmesg | grep igb
[ 4.296718] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 4.296783] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 4.7426971 igb 0000:00:06.0: added PHC on eth0
[ 4.742724] igb 0000:00:06.0: Intel(R) Gigabit Ethernet Network Connection
[ 4.742758] igb 0000:00:06.0: eth0: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:66:d2
[ 4.742798] igb 0000:00:06.0: eth0: PBA No: FFFFFF-0FF
[ 4.742826] igb 0000:00:06.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
[ 5.034919] igb 0000:00:07.0: added PHC on eth1
[ 5.634921] igb 0000:00:07.0: Intel(R) Gigabit Ethernet Network Connection
[ 5.034924] igb 0000:00:07.0: eth1: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:46:54
[ 5.034927] igb 0000:00:07.0: eth1: PBA No: FFFFFF-0FF
[ 5.034929] igb 0000:00:07.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
[ 5.281971] igb 0000:00:08.0: added PHC on eth2
[ 5.281973] igb 0000:00:08.0: Intel(R) Gigabit Ethernet Network Connection
[ 5.281976] igb 0000:00:08.0: eth2: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:46:55
[ 5.281979] igb 0000:00:08.0: eth2: PBA No: FFFFFF-0FF
[ 5.281981] igb 0000:00:08.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
[ 5.564667] igb 0000:00:09.0: added PHC on eth3
[ 5.564697] igb 0000:00:09.0: Intel(R) Gigabit Ethernet Network Connection
[ 5.564735] igb 0000:00:09.0: eth3: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:66:d1
[ 5.564780] igb 0000:00:09.0: eth3: PBA No: FFFFFF-0FF
[ 5.564810] igb 0000:00:09.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
[ 5.614700] igb 0000:00:08.0: ens8: renamed from eth2
[ 5.639207] igb 0000:00:07.0: ens7: renamed from eth1
[ 5.949373] igb 0000:00:09.0: ens9: renamed from eth3
[ 5.962094] igb 0000:00:06.0: ens6: renamed from eth0
[11.744508] igb 0000:00:06.0: igb: ens6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[13.728120] igb 0000:00:06.0: Reset adapter
[17.761975] igb 0000:00:06.0: igb: ens6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[23.007124] NETDEV WATCHDOG: ens6 (igb): transmit queue 0 timed out
[23.067303] Modules linked in: ip6table_raw iptable_raw nft_reject_ipv4 nft_reject xen_netback nf_conntrack_ipv6 nf_tables_ipv6 nf_tables_ipv4 nf_tables nfnetlink xt_REDIRECT nf_nat_redirect ip6table_filter ip6_tables xt_conntrack ipt_v4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nt_nat_ipv4 nf_conntrack libcrc32c joydev intel_rapl crctl0dif_p ghash_clmulni_intel intel_rapl_perf ttm drm_kms_helper igb ptp pps_core serio_raw drm i2c_algo_bit ehci_pci pcspkr dca i2c_piix4 floppy u2mfn(0) xen_gntdev xen_gntalloc xen_blkback xenfs xen_privcmd xen_evtchn xen_blkfront
[23.009259] igb 0000:00:06.0: Reset adapter
[27.744442] igb 0000:00:06.0: igb: ens6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
dmesg from Ubuntu
ubuntu@ubuntu:~$ sudo dmesg | grep igb
[13.454671] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[13.454673] igb: Copyright (c) 2007-2014 Intel Corporation.
[13.454862] igb 0000:01:00.0: enabling device (0000 -> 0002)
[13.700328] igb 0000:01:00.0: added PHC on eth0
[13.700330] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[13.700332] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:46:54
[13.700335] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF
[13.700337] igb 0000:01:00.0: Using MSI interrupts. 2 rx queue(s), 2 tx queue(s)
[13.700418] igb 0000:02:00.0: enabling device (0000 -> 0002)
[13.944973] igb 0000:02:00.0: added PHC on eth1
[13.944974] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
[13.944977] igb 0000:02:00.0: eth1: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:66:d1
[I3.944979] igb 0000:02:00.0: eth1: PBA No: FFFFFF-0FF
[13.944981] igb 0000:02:00.0: Using MSI interrupts. 2 rx queue(s), 2 tx queue(s)
[13.945065] igb 0000:03:00.0: enabling device (0000 -> 0002)
[14.186215] igb 0000:03:00.0: added PHC on eth2
[14.186217] igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network Connection
[14.186219] igb 0000:03:00.0: eth2: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:66:d2
[14.186222] igb 0000:03:00.0: eth2: PBA No: FFFFFF-0FF
[14.186224] igb 0000:03:00.0: Using MSI interrupts. 2 rx queue(s), 2 tx queue(s)
[14.186316] igb 0000:04:00.0: enabling device (0000 -> 0002)
[14.427243] igb 0000:04:00.0: added PHC on eth3
[14.427245] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[14.427247] igb 0000:04:00.0: eth3: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:46:55
[14.427249] igb 0000:04:00.0: eth3: PBA No: FFFFFF-0FF
[14.427251] igb 0000:04:00.0: Using MSI interrupts. 2 rx queue(s), 2 tx queue(s)
[14.428748] igb 0000:02:00.0 enp2s0: renamed from eth1
[14.448177] igb 0000:01:00.0 enp1s0: renamed from eth0
[14.464176] igb 0000:03:00.0 eno1: renamed from eth2
[14.480334] igb 0000:04:00.0 enp4s0: renamed from eth3
[33.272690] igb 0000:03:00.0 eno1: igb: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
I tried the Debian template but it still doesn't work. Same symptoms
> https://www.qubes-os.org/doc/pci-devices/#pci-passthrough-issues attach
> options.
I need some help here. Couldn't find the configuration file mentioned (/var/lib/qubes/servicevms/ is empty) to insert pci_permissive=1. I am not very good with Linux.
The default was Fedora-29.
And some observations..
When I shutdown sys-net, the physical LED of the network adapter is still going through the blinking and off cycle. This probably means that it has nothing to do with the vm but with xen itself.
(~_^) I am a professional Googler and I might have found a solution.
Rmb the dmesg logs above where qubes show
[ 4.742826] igb 0000:00:06.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
while Ubuntu show
[13.700337] igb 0000:01:00.0: Using MSI interrupts. 2 rx queue(s), 2 tx queue(s)
Apparently since 2013, the Intel note for igb drivers contains a troubleshooting section. The latest note:
> Some systems have trouble supporting MSI and/or MSI-X interrupts. If your
> system needs to disable this style of interrupt, the driver can be built and
> installed with the command:
> make CFLAGS_EXTRA=-DDISABLE_PCI_MSI install
> Normally the driver will generate an interrupt every two seconds. If you're no
> longer getting interrupts in cat /proc/interrupts for the ethX igb device,
> then this workaround may be necessary.
Couldn't figure out how to build the drivers (no network to install kernel headers) so more Googling pointed to a kernel option to disable msi.
I used to command 'qvm-prefs sys-net kernelopts "pci=nomsi"' in dom0 and rebooted sys-net. Lo and behold, the interface has gotten an IP address!!! Ping is good and so far no network drop.
Kinda lost track of how many changes I have made so I will do a fresh install and confirm again that disabling msi does the trick.
> I'd find new
> hardware if it was me.
I would have did that if Qubes had some certified hardware. But I also wanted something small just for secure work and leave my larger desktop PC for more resource intensive apps and Windows stuff. This thing is literally the size of my hand.
Confirmed. I tweaked the command a bit because it removed the default kernel options.
> qvm-prefs sys-net kernelopts "nopat iommu=soft swiotlb=8192 pci=nomsi"
Any idea if this would introduce any security vulnerabilities?
Thanks for the reminder!
> For a canonical answer, you might try the qubes-devel mailing list since
> they get more in-depth.
Will try to ask over there.
Thanks much!
Case closed