XEN domU: Guest Rx stalled, unreachable

[Felix Odenkirchen]

Dear all,

I'm running a stock Debian SID vm on xen, hostname "vm-sid".
Dom0 is on stock Debian Bullseye, hostname "bigiron-one".

apt updating vm-sid packages on 2022-01-13 08:11:00 UTC rendered vm-sid
unaccessible over network upon reboot.
Updated packages:
bsdutils bsdextrautils dbus eject initramfs-tools libblkid1 libc-bin
libfdisk1 libnss-systemd libpam-systemd libpython3.9-minimal
libpython3.9-stdlib libsmartcols1 libsystemd0 libudev1 libuuid1 mailcap
man-db mount python3-pkg-resources python3.9 python3.9-minimal systemd
systemd-sysv systemd-timesyncd udev util-linux

log messages on dom0:
    12:36:35 felix: /etc/xen/scripts/vif-bridge: online type_if=vif
XENBUS_PATH=backend/vif/18/0
    12:36:35 kernel: [2348158.091840] xenbr0: port 4(vif18.0) entered
blocking state
    12:36:35 kernel: [2348158.091846] xenbr0: port 4(vif18.0) entered
disabled state
    12:36:35 kernel: [2348158.091991] device vif18.0 entered
promiscuous mode
    12:36:35 felix: /etc/xen/scripts/vif-bridge: Successful vif-bridge
online for vif18.0, bridge xenbr0.11:58:39     12:37:14 kernel:
[2348197.078134] vif vif-18-0 vif18.0: Guest Rx ready
    12:37:14 kernel: [2348197.078164] IPv6: ADDRCONF(NETDEV_CHANGE):
vif18.0: link becomes ready
    12:37:14 kernel: [2348197.078235] xenbr0: port 4(vif18.0) entered
blocking state
    12:37:14 kernel: [2348197.078238] xenbr0: port 4(vif18.0) entered
forwarding state
    12:38:49 kernel: [2348292.051684] vif vif-18-0 vif18.0: Guest Rx
stalled
    12:38:49 kernel: [2348292.051759] xenbr0: port 4(vif18.0) entered
disabled state

WORKAROUND:

1. Logging in on virtual console
    felix@bigiron-one:~$ sudo xl console 18

2. bringing interface up manually
    felix@vm-sid:~# sudo ip a
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: enX0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
        link/ether 00:16:3e:14:07:93 brd ff:ff:ff:ff:ff:ff
    felix@vm-sid:~# sudo ip link set enX0 up
    felix@vm-sid:~# sudo dhclient

3. log on dom0
    12:41:24 kernel: [2348446.972241] vif vif-18-0 vif18.0: Guest Rx ready
    12:41:24 kernel: [2348446.972304] xenbr0: port 4(vif18.0) entered
blocking state
    12:41:24 kernel: [2348446.972308] xenbr0: port 4(vif18.0) entered
forwarding state

4. network running and stable


I fail to see why SID domU stopped bringing up its network interface.
Worked perfectly before.
Another apt update && apt upgrade didn't remove the problem, so no
package fix yet  :-/
All the other domU (Bullseye and Bookworm vm's) are not affected and
running fine,
How can I determine the source of the problem on vm-sid, or the relevant
package?

Regards,
Felix

[Chuck Zmudzinski]

Hi Felix,

It looks to me like a change in one of the systemd or udev packages in
vm-sid is causing the trouble here. In my experience with Debian, the
networking startup stuff is not so straightforward, but I do know that
systemd and udev are the software components responsible for setting up
the network and responding to networking events such as plugging in an
ethernet cable or turning on wifi. So I would look in the changelogs of
the systemd and udev packages that were updated and see if you can get a
clue on what might have changed to break your networking startup
configuration. For example, if your network is setup using
/etc/network/interfaces and ifup at startup and the systemd
configuration changed so that the ifup script that reads
/etc/network/interfaces are no longer called correctly at startup, then
your network interface would not be brought up automatically at startup.
In my experience, once when I changed some packages the network
interface name changed and that caused my network interface to not come
up properly at startup. To fix it, I had to tweak the systemd configuration.

All the best,

Chuck

[Felix Odenkirchen]

Hi Felix,

It looks to me like a change in one of the systemd or udev packages in vm-sid is causing the trouble here. In my experience with Debian, the networking startup stuff is not so straightforward, but I do know that systemd and udev are the software components responsible for setting up the network and responding to networking events such as plugging in an ethernet cable or turning on wifi. So I would look in the changelogs of the systemd and udev packages that were updated and see if you can get a clue on what might have changed to break your networking startup configuration. For example, if your network is setup using /etc/network/interfaces and ifup at startup and the systemd configuration changed so that the ifup script that reads /etc/network/interfaces are no longer called correctly at startup, then your network interface would not be brought up automatically at startup. In my experience, once when I changed some packages the network interface name changed and that caused my network interface to not come up properly at startup. To fix it, I had to tweak the systemd configuration.

All the best,

Chuck

Hi Chuck,

pointing me at /etc/network/interfaces guided me exactly in the right direction.
A udev update changed the interface name for virtual network devices from "eth0" to "enX0" (using persistent block device naming scheme), but /etc/network/interfaces still contained an entry to bring eth0 up only, which subsequently failed. Changing /etc/network/interfaces to bring up "enX0" instead solved the problem.
Bug is reported against udev package, suggesting postinst script should change /etc/network/interfaces entries.

Thanks a lot for your kind reply, which was exactly on point!

Best regards,
Felix

[Greg Wooledge]

On Sun, Jan 16, 2022 at 10:11:30PM +0100, Felix Odenkirchen wrote:
> A udev update changed the interface name for virtual network devices
> from "eth0" to "enX0" (using persistent block device naming scheme),

https://wiki.debian.org/NewInBuster#Network_interface_name_migration

Which version of Debian is that? If it's buster or later, this should
not be a surprise, because it's in the release notes.

[Andy Smith]

Hello,

It's only in the udev that's in sid. udev learned about xen virtual
interfaces after all these years, and now wants them as enX0 whereas
up until now it did not know anything about them and left them as
eth0.

Cheers,
Andy

--
https://bitfolk.com/ -- No-nonsense VPS hosting