Bug#1042842: network interface names wrong in domU (>10 interfaces)

[Valentin Kleibel]

Package: xen-utils-4.17
Version: 4.17.1+2-gb773c48e36-1
Severity: important

Dear Maintainers,

On one of our domUs we discovered that the network interface names were
wrongly assigned since recreating the domU after an upgrade to bookworm.

If over 10 network interfaces are configured the mapping (dom0) vifX.10
<-> eth10 (domU) does not apply anymore.
Instead the interfaces on dom0 are sorted primarily by the leftmost
digit. so for 11 interfaces we will end up with:
vifX.0 <> eth0
vifX.1 <> eth1
vifX.10 <> eth2
vifX.2 <> eth3
vifX.3 <> eth4
....

This was observed with linux-kernel versions 5.10.179-3 and 6.1.38-2
(all combinations of domU and dom0) and xen 4.17.1+2-gb773c48e36-1.
You can find a config snippet and "xl network-list" + "ip a" command
output demonstrating the issue below.
Booting the host with Xen 4.14.5+94-ge49571868d-1 restored the expected
behaviour.

Looking for relevant changes i found commit fce6999 [0] which changes
the way libxl__device_list works, but i'm not sure that's the cause of
this issue.

Thanks for your help,
Valentin

[0] http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=fce6999

Sample vif configuration (ascending MACs):
vif = [
'mac=00:16:3e:fd:83:2f,bridge=lanbr',
'mac=00:16:3e:fd:83:30,bridge=lanbr',
'mac=00:16:3e:fd:83:31,bridge=lanbr',
'mac=00:16:3e:fd:83:32,bridge=lanbr',
'mac=00:16:3e:fd:83:33,bridge=lanbr',
'mac=00:16:3e:fd:83:34,bridge=lanbr',
'mac=00:16:3e:fd:83:35,bridge=lanbr',
'mac=00:16:3e:fd:83:36,bridge=lanbr',
'mac=00:16:3e:fd:83:37,bridge=lanbr',
'mac=00:16:3e:fd:83:38,bridge=lanbr',
'mac=00:16:3e:fd:83:39,bridge=lanbr',
]

dom0# xl network-list 3
Idx BE Mac Addr. handle state evt-ch tx-/rx-ring-ref BE-path

0 0 00:16:3e:fd:83:2f 0 4 -1 -1/-1
/local/domain/0/backend/vif/3/0
1 0 00:16:3e:fd:83:30 1 4 -1 -1/-1
/local/domain/0/backend/vif/3/1
10 0 00:16:3e:fd:83:39 10 4 -1 -1/-1
/local/domain/0/backend/vif/3/10
2 0 00:16:3e:fd:83:31 2 4 -1 -1/-1
/local/domain/0/backend/vif/3/2
3 0 00:16:3e:fd:83:32 3 4 -1 -1/-1
/local/domain/0/backend/vif/3/3
4 0 00:16:3e:fd:83:33 4 4 -1 -1/-1
/local/domain/0/backend/vif/3/4
5 0 00:16:3e:fd:83:34 5 4 -1 -1/-1
/local/domain/0/backend/vif/3/5
6 0 00:16:3e:fd:83:35 6 4 -1 -1/-1
/local/domain/0/backend/vif/3/6
7 0 00:16:3e:fd:83:36 7 4 -1 -1/-1
/local/domain/0/backend/vif/3/7
8 0 00:16:3e:fd:83:37 8 4 -1 -1/-1
/local/domain/0/backend/vif/3/8
9 0 00:16:3e:fd:83:38 9 4 -1 -1/-1
/local/domain/0/backend/vif/3/9

domU# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
group default qlen 1000
link/ether 00:16:3e:fd:83:2f brd ff:ff:ff:ff:ff:ff
inet X.X.X.X/16 brd X.X.X.X scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fefd:832f/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 00:16:3e:fd:83:30 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 00:16:3e:fd:83:39 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 00:16:3e:fd:83:31 brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 00:16:3e:fd:83:32 brd ff:ff:ff:ff:ff:ff
7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 00:16:3e:fd:83:33 brd ff:ff:ff:ff:ff:ff
8: eth6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 00:16:3e:fd:83:34 brd ff:ff:ff:ff:ff:ff
9: eth7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 00:16:3e:fd:83:35 brd ff:ff:ff:ff:ff:ff
10: eth8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 00:16:3e:fd:83:36 brd ff:ff:ff:ff:ff:ff
11: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 00:16:3e:fd:83:37 brd ff:ff:ff:ff:ff:ff
12: eth10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 00:16:3e:fd:83:38 brd ff:ff:ff:ff:ff:ff

[zithro]

Hello,

the bug has been mentionned on #xen-devel, will keep you posted.

Meanwhile, you may try to force the domU vif names with a letter, like :

vif = [
'mac=00:16:3e:fd:83:2f,bridge=lanbr,vifname=domu-a',
'mac=00:16:3e:fd:83:30,bridge=lanbr,vifname=domu-b',
'mac=00:16:3e:fd:83:31,bridge=lanbr,vifname=domu-c',
...
]

Note it's just a workaround, and I've not tested it.
I only guess letters would be sorted correctly.
If you test this, can you report back please ?


--
zithro / Cyril

[Valentin Kleibel]

Hi,

> the bug has been mentionned on #xen-devel, will keep you posted.

Thanks. I wasn't able to find such a report, could you link the archive
or post the threads subject so i can find it?

> Meanwhile, you may try to force the domU vif names with a letter

The sorting with letters doesn't work out as renaming the interface is a
secondary step.

...
[53408.899507] vif vif-5-0 sort-a: renamed from vif5.0
...

Our current workaround is to edit the interface names in the domUs
config to match the wrong sorting. And be extra careful that the domUs
MACs match the ones we expect on that network.

Valentin

[zithro]

On 02 Aug 2023 10:22, Valentin Kleibel wrote:
> Hi,
>
>> the bug has been mentionned on #xen-devel, will keep you posted.
>
> Thanks. I wasn't able to find such a report, could you link the archive
> or post the threads subject so i can find it?

#xen-devel is the IRC Xen channel. I just pinged them, I'll wait.
Depending on their answers, I'll post on the xen-devel mailing list.

>> Meanwhile, you may try to force the domU vif names with a letter
>
> The sorting with letters doesn't work out as renaming the interface is a
> secondary step.
> ...
> [53408.899507] vif vif-5-0 sort-a: renamed from vif5.0

Yeah I just tried with vifnames, no more luck, sorry.
Running "xenstore-ls /local/domain/DOMU_ID/device/vif" shows that vif10+
are sorted before vif10- (ie. vif1, vif10, vif11, vif2, ...).

> Our current workaround is to edit the interface names in the domUs
> config to match the wrong sorting. And be extra careful that the domUs
> MACs match the ones we expect on that network.

Via udev (MAC matching) or /etc/network/interfaces ?
I ask because it may help others, while this gets resolved.


--
zithro / Cyril

[Valentin Kleibel]

> #xen-devel is the IRC Xen channel. I just pinged them, I'll wait.
> Depending on their answers, I'll post on the xen-devel mailing list.

thanks for the clarification, looking forward to an answer.

>> Our current workaround is to edit the interface names in the domUs
>> config to match the wrong sorting. And be extra careful that the domUs
>> MACs match the ones we expect on that network.
>
> Via udev (MAC matching) or /etc/network/interfaces ?
> I ask because it may help others, while this gets resolved.

We just edited /etc/network/interfaces, as it only affects a few of our
domUs.
i think udev rules matching the MAC would be a better solution. I just
didn't take the time to write them and went for the quick and dirty
solution.

Valentin

[zithro]

On 02 Aug 2023 18:09, Valentin Kleibel wrote:
>> #xen-devel is the IRC Xen channel. I just pinged them, I'll wait.
>> Depending on their answers, I'll post on the xen-devel mailing list.
>
> thanks for the clarification, looking forward to an answer.

I posted on xen-devel, you can follow from :
https://lists.xenproject.org/archives/html/xen-devel/2023-08/msg00244.html
(Unfortunately, the formatting is weird via html, split the IRC part on
"- ").

Note that, at first sight, I was told this seems "not-a-Xen" bug (read
the IRC excerpts).

>>> Our current workaround is to edit the interface names in the domUs
>>> config to match the wrong sorting. And be extra careful that the
>>> domUs MACs match the ones we expect on that network.
>>
>> Via udev (MAC matching) or /etc/network/interfaces ?
>> I ask because it may help others, while this gets resolved.
>
> We just edited /etc/network/interfaces, as it only affects a few of our
> domUs.
> i think udev rules matching the MAC would be a better solution. I just
> didn't take the time to write them and went for the quick and dirty
> solution.

Till it works, "whatever the bottle, till we have the poison" ;)

This link may be useful: https://wiki.debian.org/NetworkInterfaceNames

[Valentin Kleibel]

> I posted on xen-devel, you can follow from :
> https://lists.xenproject.org/archives/html/xen-devel/2023-08/msg00244.html
> (Unfortunately, the formatting is weird via html, split the IRC part on
> "- ").

Thank you for posting upstream.

> Note that, at first sight, I was told this seems "not-a-Xen" bug (read
> the IRC excerpts).

All documentation i found found on the Xen wiki suggests that interfaces
are connected vifX.Y <-> ethY. [0] [1]
The only other way i know of for identifying the interfaces are MAC
Addresses which can be randomly assigned if you don't configure them.

I also checked which net_ids udev knows about and the only things that
pop up are:
ID_NET_NAMING_SCHEME=v247
ID_NET_NAME_MAC=enx00163efd832b
ID_OUI_FROM_DATABASE=Xensource, Inc.

Either i am missing the way you're supposed to do this, or there is a
bug somewhere in the toolchain.
Unfortunately i'm not able to pinpoint the source of the issue, any help
would be appreciated.

Valentin

[0]
https://wiki.xenproject.org/wiki/Xen_Networking#Paravirtualised_Network_Devices
[1]
https://wiki.xenproject.org/wiki/Network_Throughput_and_Performance_Guide#Technical_Overview

[zithro]

On 08 Aug 2023 12:08, Valentin Kleibel wrote:
>> I posted on xen-devel, you can follow from :
>> https://lists.xenproject.org/archives/html/xen-devel/2023-08/msg00244.html
>> (Unfortunately, the formatting is weird via html, split the IRC part
>> on "- ").
>
> Thank you for posting upstream.

No prob, although if that very answer does not answer your question, I
guess you'd be better off replying on xen-devel ML (reply to my post or
at least reference it).

> All documentation i found found on the Xen wiki suggests that interfaces
> are connected vifX.Y <-> ethY. [0] [1]
> The only other way i know of for identifying the interfaces are MAC
> Addresses which can be randomly assigned if you don't configure them.

On [0], you can read "In both cases the device naming is subject to the
usual guest or backend domain facilities for renaming network devices".
It says "naming/renaming", but you can assume "detecting".

> I also checked which net_ids udev knows about and the only things that
> pop up are:
> ID_NET_NAMING_SCHEME=v247
> ID_NET_NAME_MAC=enx00163efd832b
> ID_OUI_FROM_DATABASE=Xensource, Inc.

Is it from dom0 or domU ?
Are you using "net.ifnames=0" on the domU kernel command line ?
"v247" looks like systemd "predictive naming scheme" (eth -> enX).
From bookworm on, domUs vifs get named enXN (enX0, enX1, ...).
Read on :
https://www.debian.org/releases/stable/i386/release-notes/ch-information.en.html#xen-network

> Either i am missing the way you're supposed to do this, or there is a
> bug somewhere in the toolchain.
> Unfortunately i'm not able to pinpoint the source of the issue, any help
> would be appreciated.

I made some tests with a domU using many interfaces, like :

[...]
vif = [ 'bridge=xbr-tst,mac=00:16:3e:de:bd:00,type=vif,vifname=domu-a' ,
'bridge=xbr-tst,mac=00:16:3e:de:bd:01,type=vif,vifname=domu-b' ,
'bridge=xbr-tst,mac=00:16:3e:de:bd:02,type=vif,vifname=domu-c' ,
'bridge=xbr-tst,mac=00:16:3e:de:bd:03,type=vif,vifname=domu-d' ,
'bridge=xbr-tst,mac=00:16:3e:de:bd:04,type=vif,vifname=domu-e' ,
'bridge=xbr-tst,mac=00:16:3e:de:bd:05,type=vif,vifname=domu-f' ,
'bridge=xbr-tst,mac=00:16:3e:de:bd:06,type=vif,vifname=domu-g' ,
'bridge=xbr-tst,mac=00:16:3e:de:bd:07,type=vif,vifname=domu-h' ,
'bridge=xbr-tst,mac=00:16:3e:de:bd:08,type=vif,vifname=domu-i' ,
'bridge=xbr-tst,mac=00:16:3e:de:bd:09,type=vif,vifname=domu-j' ,
'bridge=xbr-tst,mac=00:16:3e:de:bd:10,type=vif,vifname=domu-k' ,
]
[...]

-----
This is dom0's corresponding dmesg:

[...]
xbr-tst: port 3(domu-b) entered blocking state
xbr-tst: port 3(domu-b) entered disabled state
device domu-b entered promiscuous mode
xbr-tst: port 4(domu-i) entered blocking state
xbr-tst: port 4(domu-i) entered disabled state
device domu-i entered promiscuous mode
[...]

Here you can see :
port 3 <-> domu-b
port 4 <-> domu-i

We learn here that dom0 did not detect vifs serially.

-----
In the domU, "ip link" shows :

[...]
eth0
link/ether 00:16:3e:de:bd:00
altname enX0
eth1
link/ether 00:16:3e:de:bd:01
altname enX1
eth2
link/ether 00:16:3e:de:bd:10
altname enX10
eth3
link/ether 00:16:3e:de:bd:02
altname enX2
[...]

See how ethN interfaces get messed up, like in your setup, but
predictable names would work, as you can see in "altname enXN" :
eth1 (:01) -> enX1
eth2 (:10) -> enX10
eth3 (:02) -> enX2

So, my answer does not tell you if something changed in Xen itself, only
in Debian.
But I guess it relates to what Xen devs told us : vifs detection order
cannot be relied upon, that's why "predictable names" were invented.
The vif detection part is related to the domains kernels, not Xen itself
(at least that's what I understood).

Using eth0 nowadays is a bit like using /dev/sda for hard drives, it's
considered legacy as it may create problems in some setups, like yours
(ie. for disks, it's recommended to use UUIDs or /dev/disk/by-*).

I hope this answers your question.

[Valentin Kleibel]

> On [0], you can read "In both cases the device naming is subject to the
> usual guest or backend domain facilities for renaming network devices".
> It says "naming/renaming", but you can assume "detecting".
>
>> I also checked which net_ids udev knows about and the only things that
>> pop up are:
>> ID_NET_NAMING_SCHEME=v247
>> ID_NET_NAME_MAC=enx00163efd832b
>> ID_OUI_FROM_DATABASE=Xensource, Inc.
>
> Is it from dom0 or domU ?
> Are you using "net.ifnames=0" on the domU kernel command line ?
> "v247" looks like systemd "predictive naming scheme" (eth -> enX).
> From bookworm on, domUs vifs get named enXN (enX0, enX1, ...).
> Read on :
> https://www.debian.org/releases/stable/i386/release-notes/ch-information.en.html#xen-network

This is from the domU, running bullseye with a bookworm dom0.

> See how ethN interfaces get messed up, like in your setup, but
> predictable names would work, as you can see in "altname enXN" :
> eth1 (:01) -> enX1
> eth2 (:10) -> enX10
> eth3 (:02) -> enX2

I could not get our bullseye domU to show the "predictable names" even
though i tried installing the bullseye-backports kernel 6.1.
After you wrote this i installed udev 252.5 from backports and it now
uses the correct enXn interface names, even with kernel 5.10.

> So, my answer does not tell you if something changed in Xen itself, only
> in Debian.
> But I guess it relates to what Xen devs told us : vifs detection order
> cannot be relied upon, that's why "predictable names" were invented.
> The vif detection part is related to the domains kernels, not Xen itself
> (at least that's what I understood).
>
> Using eth0 nowadays is a bit like using /dev/sda for hard drives, it's
> considered legacy as it may create problems in some setups, like yours
> (ie. for disks, it's recommended to use UUIDs or /dev/disk/by-*).
>
> I hope this answers your question.

Thank you, yes it does.

In our case the dom0 was updated to bookworm while the domU is still
running bullseye.
-> updated Xen so the vif detection order changed (which we relied on)
-> the predictable network names for Xen don't work with bullseye

So my new resolution for bullseye domUs on a bookworm dom0 is to install
udev from backports and change the domUs network config to use the new
enXn naming scheme instead of ethn.

[Hans van Kranenburg]

Hi,

On 8/8/23 15:22, Valentin Kleibel wrote:
>> On [0], you can read "In both cases the device naming is subject to the
>> usual guest or backend domain facilities for renaming network devices".
>> It says "naming/renaming", but you can assume "detecting".
>>
>>> I also checked which net_ids udev knows about and the only things that
>>> pop up are:
>>> ID_NET_NAMING_SCHEME=v247
>>> ID_NET_NAME_MAC=enx00163efd832b
>>> ID_OUI_FROM_DATABASE=Xensource, Inc.

What I do is stuff like this:

-$ cat /etc/udev/rules.d/70-persistent-vifname.rules

SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/0",
NAME="vlan2"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/1",
NAME="vlan3"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/2",
NAME="vlan4"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/3",
NAME="vlan6"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/4",
NAME="vlan9"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/5",
NAME="vlan10"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/6",
NAME="vlan11"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/7",
NAME="vlan12"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/8",
NAME="vlan13"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/9",
NAME="vlan14"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/10",
NAME="vlan15"
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{nodename}=="device/vif/11",
NAME="vlan16"

The vif/X always matches the order in which you define the interfaces
inside the guest config file.

After starting to build router VMs (well before the whole interface
naming madness was a thing), it took only the first time when we wanted
to throw away a vlan, to realize that all the ethX numbers would shift 1
up, and from then on, I've always been using this so set my own style
predictable names (whenever there's more than one, otherwise it's just
eth0).

>> Is it from dom0 or domU ?
>> Are you using "net.ifnames=0" on the domU kernel command line ?
>> "v247" looks like systemd "predictive naming scheme" (eth -> enX).
>> From bookworm on, domUs vifs get named enXN (enX0, enX1, ...).
>> Read on :
>> https://www.debian.org/releases/stable/i386/release-notes/ch-information.en.html#xen-network
>
> This is from the domU, running bullseye with a bookworm dom0.
>
>> See how ethN interfaces get messed up, like in your setup, but
>> predictable names would work, as you can see in "altname enXN" :
>> eth1 (:01) -> enX1
>> eth2 (:10) -> enX10
>> eth3 (:02) -> enX2

But yeah, so, even while not depending on whatever order it gets
initialized, and still having it function correctly, this is still just
pretty annoying... If I'm doing stuff around here, and just quickly want
to look up things (e.g. messing around with vlan15 settings), and
quickly type ip a instead of having to spend more time typing ip a show
dev vlan15 jadijadi, I still every time get this short "WTF huh, argh",
raises arms, does table flip, grmbl grbml feeling for a split second.

2: vlan2: <BROADCAST,MULT[...]
[...]
altname enX0
3: vlan3: <BROADCAST,MULT[...]
[...]
altname enX1
4: vlan15: <BROADCAST,MULT[...]
[...]
altname enX10
5: vlan16: <BROADCAST,MULT[...]
[...]
altname enX11
6: vlan4: <BROADCAST,MULT[...]
[...]
altname enX2
7: vlan6: <BROADCAST,MULT[...]
[...]
altname enX3
8: vlan9: <BROADCAST,MULT[...]
[...]
altname enX4
9: vlan10: <BROADCAST,MULT[...]
[...]
altname enX5
10: vlan11: <BROADCAST,MULT[...]
[...]
altname enX6
11: vlan12: <BROADCAST,MULT[...]
[...]
altname enX7
12: vlan13: <BROADCAST,MULT[...]
[...]
altname enX8
13: vlan14: <BROADCAST,MULT[...]
[...]
altname enX9

First, there's the split second start of "wait, what, I just added an IP
address, where did the whole interface go?!" and then the "Oh, right, of
course, it's the silly thing again.", and then it costs an extra second
looking at the screen like a confused Jeffrey Lebowski to find the right
line.

> I could not get our bullseye domU to show the "predictable names" even
> though i tried installing the bullseye-backports kernel 6.1.
> After you wrote this i installed udev 252.5 from backports and it now
> uses the correct enXn interface names, even with kernel 5.10.
>
>> So, my answer does not tell you if something changed in Xen itself, only
>> in Debian.
>> But I guess it relates to what Xen devs told us : vifs detection order
>> cannot be relied upon, that's why "predictable names" were invented.
>> The vif detection part is related to the domains kernels, not Xen itself
>> (at least that's what I understood).
>>
>> Using eth0 nowadays is a bit like using /dev/sda for hard drives, it's
>> considered legacy as it may create problems in some setups, like yours
>> (ie. for disks, it's recommended to use UUIDs or /dev/disk/by-*).
>>
>> I hope this answers your question.
>
> Thank you, yes it does.
>
> In our case the dom0 was updated to bookworm while the domU is still
> running bullseye.
> -> updated Xen so the vif detection order changed (which we relied on)

I didn't read the other mailthread on the xen list fully yet. But, I
think it's shouldn't be very hard to find the code changes and see if
it's deterministic and can just be fixed. Simply just to decrease the
totally unnecessary amount of silliness.

> -> the predictable network names for Xen don't work with bullseye
>
> So my new resolution for bullseye domUs on a bookworm dom0 is to install
> udev from backports and change the domUs network config to use the new
> enXn naming scheme instead of ethn.

Or the "device/vif/X" way...

So, anyway, did someone already did some test "just because we can" to
see how much network interfaces you can get added for fun, and if the
pattern keeps looking the same, also with enX4 enX40 .. enX49 enX5 etc?
:D enX1 enX10 enX100 .. enX109 enX11 enX110 argh o_O

Have fun,
Hans

[zithro]

On 08 Aug 2023 16:59, Hans van Kranenburg wrote:
> I didn't read the other mailthread on the xen list fully yet.

You gave me the idea to post the IRC digest, so the report here is more
complete, and people not tracking the xen-devel ML can read it nicely.
For those who do, the mail is dated "02 Aug 2023 18:19", and titled
"Network interfaces naming changes in domUs with >10 vifs (Debian bug
1042842)".
The ML post got no answer yet.

[--- IRC ---]
- AFAIK, there is no sorting in Xenstored. And you should not expect
that even if libxl sorted properly it will be seen in the same order on
the other end.
- is the ethN number in domU related to vif number in xenstore, or to
device detection order?
- there's no order to eth names at all. they're allocated
first-come-first-serve, so it entirely depends on how parallel the
probing of nic drivers are. even if netfront is serialised around
xenstore accesses, it probably allocates in the order that XS_DIRECTORY
comes back with
- from simple tests, it looks like VIFs are created in Xenstore in the
order of the config file, but if you "xenstore-ls /[...]/vif", you can
see vifs are ordered like vif1,vif10,vif11,vif2,etc
- the order is different between Xen 4.14 and 4.17 (ie. the "expected"
order works on 4.14, not 4.17)
- But really, Debian should have never relied on how the nodes are
ordered. This is not something we guarantee in the Xenstored API
- the last big batch of XSA content for the xenstoreds did some major
rearranging of oxenstored. We dropped a NIH second garbage collector,
and a NIH weakref system IIRC. I could entirely believe that the
apparent sort order changed as a result
- generally, I think Linux world established quite some time ago that
ethN names are not stable
- It's definitely a complicated issue. Perhaps best to post to
xen-devel so we can have a discussion. I expect the answer is not-a-Xen
bug, but I don't think we have a clear understanding of the problem yet
[--- /IRC ---]

I'll report back when having tested the 111 vifs domU ... if my system
agrees o_O
As it requires a script to populate the cfg, one could also enhance it
to try how dynamically adding/removing vifs is handled.

(BTW, before this report I thought Xen had a hard limit of 8 vifs per
domU. Or was that only on FreeBSD domUs ? Can't remember).