Unable to get network adapter working

[Chris]

Hi all!

It's my first time installing Qubes and I need some help with my network adapters.
I am using Qubes 4.0.1
The hardware is Fitlet2 with Intel Celeron J3455 (supports both VT-x and VT-d).
It comes with 4 Gigabit Ethernet ports using Intel i211.

Installation was successful but EFI boot entry could not be added. Editing EFI entries using tty2 (Ctrl-Alt-F2) or 'Rescue a Qubes' mode both throw errors. Eventually solved by booting into Live Ubuntu and editing EFI entries from there.

Successfully booted into Qubes but couldn't get network working.

1. Physically, port LEDs go off and start blinking after awhile. Keeps repeating.
2. Network icon on top right is red and shows 'loading animation'
3. Network adapter can negotiate speed and duplex but cannot get IP address from DHCP server
4. Changing of network cable and connecting to another switch gives the same results
5. dmesg shows adapter in reset cycle
6. Network adapter works perfect when boot into Live Ubuntu using same hardware

dmesg from Qubes

[user@sys-net ~]$ sudo dmesg | grep igb
[ 4.296718] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 4.296783] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 4.7426971 igb 0000:00:06.0: added PHC on eth0
[ 4.742724] igb 0000:00:06.0: Intel(R) Gigabit Ethernet Network Connection
[ 4.742758] igb 0000:00:06.0: eth0: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:66:d2
[ 4.742798] igb 0000:00:06.0: eth0: PBA No: FFFFFF-0FF
[ 4.742826] igb 0000:00:06.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
[ 5.034919] igb 0000:00:07.0: added PHC on eth1
[ 5.634921] igb 0000:00:07.0: Intel(R) Gigabit Ethernet Network Connection
[ 5.034924] igb 0000:00:07.0: eth1: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:46:54
[ 5.034927] igb 0000:00:07.0: eth1: PBA No: FFFFFF-0FF
[ 5.034929] igb 0000:00:07.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
[ 5.281971] igb 0000:00:08.0: added PHC on eth2
[ 5.281973] igb 0000:00:08.0: Intel(R) Gigabit Ethernet Network Connection
[ 5.281976] igb 0000:00:08.0: eth2: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:46:55
[ 5.281979] igb 0000:00:08.0: eth2: PBA No: FFFFFF-0FF
[ 5.281981] igb 0000:00:08.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
[ 5.564667] igb 0000:00:09.0: added PHC on eth3
[ 5.564697] igb 0000:00:09.0: Intel(R) Gigabit Ethernet Network Connection
[ 5.564735] igb 0000:00:09.0: eth3: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:66:d1
[ 5.564780] igb 0000:00:09.0: eth3: PBA No: FFFFFF-0FF
[ 5.564810] igb 0000:00:09.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
[ 5.614700] igb 0000:00:08.0: ens8: renamed from eth2
[ 5.639207] igb 0000:00:07.0: ens7: renamed from eth1
[ 5.949373] igb 0000:00:09.0: ens9: renamed from eth3
[ 5.962094] igb 0000:00:06.0: ens6: renamed from eth0
[11.744508] igb 0000:00:06.0: igb: ens6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[13.728120] igb 0000:00:06.0: Reset adapter
[17.761975] igb 0000:00:06.0: igb: ens6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[23.007124] NETDEV WATCHDOG: ens6 (igb): transmit queue 0 timed out
[23.067303] Modules linked in: ip6table_raw iptable_raw nft_reject_ipv4 nft_reject xen_netback nf_conntrack_ipv6 nf_tables_ipv6 nf_tables_ipv4 nf_tables nfnetlink xt_REDIRECT nf_nat_redirect ip6table_filter ip6_tables xt_conntrack ipt_v4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nt_nat_ipv4 nf_conntrack libcrc32c joydev intel_rapl crctl0dif_p ghash_clmulni_intel intel_rapl_perf ttm drm_kms_helper igb ptp pps_core serio_raw drm i2c_algo_bit ehci_pci pcspkr dca i2c_piix4 floppy u2mfn(0) xen_gntdev xen_gntalloc xen_blkback xenfs xen_privcmd xen_evtchn xen_blkfront
[23.009259] igb 0000:00:06.0: Reset adapter
[27.744442] igb 0000:00:06.0: igb: ens6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

dmesg from Ubuntu

ubuntu@ubuntu:~$ sudo dmesg | grep igb
[13.454671] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[13.454673] igb: Copyright (c) 2007-2014 Intel Corporation.
[13.454862] igb 0000:01:00.0: enabling device (0000 -> 0002)
[13.700328] igb 0000:01:00.0: added PHC on eth0
[13.700330] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[13.700332] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:46:54
[13.700335] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF
[13.700337] igb 0000:01:00.0: Using MSI interrupts. 2 rx queue(s), 2 tx queue(s)
[13.700418] igb 0000:02:00.0: enabling device (0000 -> 0002)
[13.944973] igb 0000:02:00.0: added PHC on eth1
[13.944974] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
[13.944977] igb 0000:02:00.0: eth1: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:66:d1
[I3.944979] igb 0000:02:00.0: eth1: PBA No: FFFFFF-0FF
[13.944981] igb 0000:02:00.0: Using MSI interrupts. 2 rx queue(s), 2 tx queue(s)
[13.945065] igb 0000:03:00.0: enabling device (0000 -> 0002)
[14.186215] igb 0000:03:00.0: added PHC on eth2
[14.186217] igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network Connection
[14.186219] igb 0000:03:00.0: eth2: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:66:d2
[14.186222] igb 0000:03:00.0: eth2: PBA No: FFFFFF-0FF
[14.186224] igb 0000:03:00.0: Using MSI interrupts. 2 rx queue(s), 2 tx queue(s)
[14.186316] igb 0000:04:00.0: enabling device (0000 -> 0002)
[14.427243] igb 0000:04:00.0: added PHC on eth3
[14.427245] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[14.427247] igb 0000:04:00.0: eth3: (PCIe:2.5Gb/s:Width xl) 00:01:c0:15:46:55
[14.427249] igb 0000:04:00.0: eth3: PBA No: FFFFFF-0FF
[14.427251] igb 0000:04:00.0: Using MSI interrupts. 2 rx queue(s), 2 tx queue(s)
[14.428748] igb 0000:02:00.0 enp2s0: renamed from eth1
[14.448177] igb 0000:01:00.0 enp1s0: renamed from eth0
[14.464176] igb 0000:03:00.0 eno1: renamed from eth2
[14.480334] igb 0000:04:00.0 enp4s0: renamed from eth3
[33.272690] igb 0000:03:00.0 eno1: igb: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

[awokd]

Chris:
> Hi all!

Welcome!

> Successfully booted into Qubes but couldn't get network working.
>
> 1. Physically, port LEDs go off and start blinking after awhile. Keeps repeating.
> 2. Network icon on top right is red and shows 'loading animation'
> 3. Network adapter can negotiate speed and duplex but cannot get IP address from DHCP server
> 4. Changing of network cable and connecting to another switch gives the same results
> 5. dmesg shows adapter in reset cycle
> 6. Network adapter works perfect when boot into Live Ubuntu using same hardware

Quickest thing to try is to switch sys-net's template to Debian. Also,
experiment with
https://www.qubes-os.org/doc/pci-devices/#pci-passthrough-issues attach
options.

[Chris Laprise]

Debian can be a good sys-net distro, but unfortunately the template
doesn't have all the wifi drivers pre-installed... I have to manually
install 'firmware-iwlwifi' to get Intel cards working.

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Jon deps]

On 6/26/19 1:33 PM, Chris Laprise wrote:
> On 6/26/19 7:56 AM, 'awokd' via qubes-users wrote:
>> Chris:
>>> Hi all!
>>
>> Welcome!
>>
>>> Successfully booted into Qubes but couldn't get network working.
>>>
>>> 1. Physically, port LEDs go off and start blinking after awhile.
>>> Keeps repeating.
>>> 2. Network icon on top right is red and shows 'loading animation'
>>> 3. Network adapter can negotiate speed and duplex but cannot get IP
>>> address from DHCP server
>>> 4. Changing of network cable and connecting to another switch gives
>>> the same results
>>> 5. dmesg shows adapter in reset cycle
>>> 6. Network adapter works perfect when boot into Live Ubuntu using
>>> same hardware
>>
>> Quickest thing to try is to switch sys-net's template to Debian. Also,
>> experiment with
>> https://www.qubes-os.org/doc/pci-devices/#pci-passthrough-issues
>> attach options.
>>
>
> Debian can be a good sys-net distro, but unfortunately the template
> doesn't have all the wifi drivers pre-installed... I have to manually
> install 'firmware-iwlwifi' to get Intel cards working.
>

I don't think you said which template you are using for sys-net did you?

IMO use Fedora-30 as last I checked using Debian-9 is likely to mess
up your dom0 clock , which matters for updates

[Chris]

> Quickest thing to try is to switch sys-net's template to Debian. Also,
> experiment with

I tried the Debian template but it still doesn't work. Same symptoms

> https://www.qubes-os.org/doc/pci-devices/#pci-passthrough-issues attach
> options.

I need some help here. Couldn't find the configuration file mentioned (/var/lib/qubes/servicevms/ is empty) to insert pci_permissive=1. I am not very good with Linux.

[Chris]

> I don't think you said which template you are using for sys-net did you?

The default was Fedora-29.

And some observations..

When I shutdown sys-net, the physical LED of the network adapter is still going through the blinking and off cycle. This probably means that it has nothing to do with the vm but with xen itself.

[Jon deps]

fedora-30 is the newer qubes template , maybe it will have some support
for your device, that older OS doesn't , seems logical to me

in dom0:

$ sudo qubes-dom0-update qubes-template-fedora-30


IMO: you shouldn't have to get into the config files, I'd find new
hardware if it was me.

[awokd]

Jon deps:

> On 6/27/19 1:10 AM, Chris wrote:

>> I tried the Debian template but it still doesn't work. Same symptoms
>>
>>> https://www.qubes-os.org/doc/pci-devices/#pci-passthrough-issues attach
>>> options.
>>
>> I need some help here. Couldn't find the configuration file mentioned
>> (/var/lib/qubes/servicevms/ is empty) to insert pci_permissive=1. I am
>> not very good with Linux.

You fooled me with the cogent problem description and troubleshooting
approach. There's no config. file. Look below "Additional Attach
Options" in the link I sent. Use qvm-pci options when you re-attach your
NIC to sys-net. However:

> I'd find new
> hardware if it was me.
>

With the additional info you provided in your other reply of the NIC
repeatedly resetting even without sys-net running, I second this. Single
port NICs are cheap. If you still want to troubleshoot further, check
journalctl and xl dmesg in a dom0 terminal to hopefully see why it's
resetting.

[Chris]

> You fooled me with the cogent problem description and troubleshooting
> approach.

(~_^) I am a professional Googler and I might have found a solution.

Rmb the dmesg logs above where qubes show

[ 4.742826] igb 0000:00:06.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)

while Ubuntu show

[13.700337] igb 0000:01:00.0: Using MSI interrupts. 2 rx queue(s), 2 tx queue(s)

Apparently since 2013, the Intel note for igb drivers contains a troubleshooting section. The latest note:

> Some systems have trouble supporting MSI and/or MSI-X interrupts. If your
> system needs to disable this style of interrupt, the driver can be built and
> installed with the command:
> make CFLAGS_EXTRA=-DDISABLE_PCI_MSI install

> Normally the driver will generate an interrupt every two seconds. If you're no
> longer getting interrupts in cat /proc/interrupts for the ethX igb device,
> then this workaround may be necessary.

Couldn't figure out how to build the drivers (no network to install kernel headers) so more Googling pointed to a kernel option to disable msi.

I used to command 'qvm-prefs sys-net kernelopts "pci=nomsi"' in dom0 and rebooted sys-net. Lo and behold, the interface has gotten an IP address!!! Ping is good and so far no network drop.

Kinda lost track of how many changes I have made so I will do a fresh install and confirm again that disabling msi does the trick.

> I'd find new
> hardware if it was me.

I would have did that if Qubes had some certified hardware. But I also wanted something small just for secure work and leave my larger desktop PC for more resource intensive apps and Windows stuff. This thing is literally the size of my hand.

[Chris]

> I will do a fresh install and confirm again that disabling msi does the trick.

Confirmed. I tweaked the command a bit because it removed the default kernel options.

> qvm-prefs sys-net kernelopts "nopat iommu=soft swiotlb=8192 pci=nomsi"

Any idea if this would introduce any security vulnerabilities?

[awokd]

Chris:

>> I will do a fresh install and confirm again that disabling msi does the trick.
>
> Confirmed. I tweaked the command a bit because it removed the default kernel options.

Depending on hardware, getting Qubes up and running can sometimes be the
hardest part of using it. In comparison, the rest should be smooth
sailing. :) Do remember to run backups, though. EXT3 on thin LVM is not
as resilient as NTFS, for example.

>> qvm-prefs sys-net kernelopts "nopat iommu=soft swiotlb=8192 pci=nomsi"
>
> Any idea if this would introduce any security vulnerabilities?
>

I've looked at the Xen/Qubes PCI virtualization code when
troubleshooting a similar issue with interrupts on one of my systems.
IIRC, MSI and standard interrupts get handled the same way and processed
through the same code base, so I don't see any difference in exposure.
For a canonical answer, you might try the qubes-devel mailing list since
they get more in-depth.

[Chris]

> Do remember to run backups, though. EXT3 on thin LVM is not
> as resilient as NTFS, for example.

Thanks for the reminder!

> For a canonical answer, you might try the qubes-devel mailing list since
> they get more in-depth.

Will try to ask over there.

Thanks much!
Case closed