Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
184 views
Skip to first unread message
Luca Boccassi
Oct 27, 2023, 5:30:05 AM10/27/23
to
Control: tags -1 upstream
On Fri, 27 Oct 2023 at 09:18, <peter.ga...@orange.com> wrote:
>
> Package: iproute2
> Version: 4.20.0-2+deb10u1
> Problem: External MAC@ reaches max Linux bridge, but not net namespace linked via veth pair to it, based on very minimal config
>
> Hello Debian team,
>
> I would like to report problem which possibly has to do with IPROUTE2 package, I’m experiencing it both Debian 10 (this) and 12 (6.1.0-3). I really did my best reviewing at least 7 stack-exchange (and like) stories and I’m at my wit’s end, wondering why this is possibly not fixed in 2023 seeing debates go back into like 2014..
>
> So it’s plain simple to want to make multiple namespaces able to communicate via common host bridge to external network. VETH tech is all time documented as solution to this. The problem on given path in subject is this:
>
> NS veth IP@ = .251 , 0e:61:72:97:aa:ff
> (Bridge) veth IP@ = .44 , ce:18:16:4b:0c:c2
> Bridge IP@ = .254 , 00:50:56:01:01:02
> External IP@ = .other
>
> 1) When I initially set up plain “veth port --> NS veth port”, with IP@ at each end, it’s all seamless, ARP and pings..
> >== 2) Once I enslave veth port to bridge, I can’t reach external network <===
> 3) Veth also does not work on IP level anymore, all the time with ICMP echo from NS space it runs ARP first, though both “ip nei” are populated with mutual MAC records. The following goes in loop..
> peterg@debian:~$ sudo tcpdump -ni vinet-brp
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vinet-brp, link-type EN10MB (Ethernet), capture size 262144 bytes
> 11:18:12.966955 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2333, seq 0, length 64
> 11:18:12.966984 ARP, Request who-has 70.0.0.251 tell 70.0.0.44, length 28
> 11:18:12.966989 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length 28
> 11:18:13.967994 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2333, seq 1, length
> 4) Once I configure bridge iface with some IP address of same subnet /24 like veth and NS veth (also externals) use → the NS nei can show changing MAC address for bridge veth iface
> 11:30:27.860907 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length 28
> 11:30:28.848251 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2352, seq 14, length 64
> 11:30:28.884892 ARP, Request who-has 70.0.0.251 tell 70.0.0.44, length 28
> 11:30:28.884908 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length 28
> 11:30:28.980890 ARP, Request who-has 70.0.0.44 tell 70.0.0.251, length 28
> 11:30:28.980909 ARP, Reply 70.0.0.44 is-at 00:50:56:01:01:02, length 28 <---
>
> inet_bash >> ip nei
> 70.0.0.1 dev vinet FAILED
> 70.0.0.44 dev vinet lladdr ce:18:16:4b:0c:c2 DELAY <---
>
> 5) The bridge vs NS veth pinging works
> 6) The bridge relays ARP into external network and back (checked on Cisco switch), learns of external MAC@s
> ===> 7) External MAC@ does not make it to NS space by ARP <===
> 8) I don’t aim to deploy IP@ with bridge and bridge veth ifaces → this is just to check how it works
> 9) This blog was quite surprising stating that bridge without IP@ can affect routing in global namespace, few /sys kernel tweaks → no help
> https://unix.stackexchange.com/questions/655602/why-two-bridged-veth-cannot-ping-each-other/674621#674621
> 10) Even tried to stop default MAC learning on bridge veth iface by “ip link set dev vinet-brp type bridge_slave learning off” ⇒ did not work, neigh flushed and pinging .251 vs .254 just worked.
>
> So I believe that bridge veth iface is broken in its essential functionality using default “learning/flooding on” settings.
>
> Thanks for your time to look at this and give hope or deny this so I need to create extra ports in my host to get what I want to.
> BR
> Peter
You need to report this upstream, nobody here is going to look at
something like this
On Fri, 27 Oct 2023 at 09:18, <peter.ga...@orange.com> wrote:
>
> Package: iproute2
> Version: 4.20.0-2+deb10u1
> Problem: External MAC@ reaches max Linux bridge, but not net namespace linked via veth pair to it, based on very minimal config
>
> Hello Debian team,
>
> I would like to report problem which possibly has to do with IPROUTE2 package, I’m experiencing it both Debian 10 (this) and 12 (6.1.0-3). I really did my best reviewing at least 7 stack-exchange (and like) stories and I’m at my wit’s end, wondering why this is possibly not fixed in 2023 seeing debates go back into like 2014..
>
> So it’s plain simple to want to make multiple namespaces able to communicate via common host bridge to external network. VETH tech is all time documented as solution to this. The problem on given path in subject is this:
>
> NS veth IP@ = .251 , 0e:61:72:97:aa:ff
> (Bridge) veth IP@ = .44 , ce:18:16:4b:0c:c2
> Bridge IP@ = .254 , 00:50:56:01:01:02
> External IP@ = .other
>
> 1) When I initially set up plain “veth port --> NS veth port”, with IP@ at each end, it’s all seamless, ARP and pings..
> >== 2) Once I enslave veth port to bridge, I can’t reach external network <===
> 3) Veth also does not work on IP level anymore, all the time with ICMP echo from NS space it runs ARP first, though both “ip nei” are populated with mutual MAC records. The following goes in loop..
> peterg@debian:~$ sudo tcpdump -ni vinet-brp
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vinet-brp, link-type EN10MB (Ethernet), capture size 262144 bytes
> 11:18:12.966955 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2333, seq 0, length 64
> 11:18:12.966984 ARP, Request who-has 70.0.0.251 tell 70.0.0.44, length 28
> 11:18:12.966989 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length 28
> 11:18:13.967994 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2333, seq 1, length
> 4) Once I configure bridge iface with some IP address of same subnet /24 like veth and NS veth (also externals) use → the NS nei can show changing MAC address for bridge veth iface
> 11:30:27.860907 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length 28
> 11:30:28.848251 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2352, seq 14, length 64
> 11:30:28.884892 ARP, Request who-has 70.0.0.251 tell 70.0.0.44, length 28
> 11:30:28.884908 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length 28
> 11:30:28.980890 ARP, Request who-has 70.0.0.44 tell 70.0.0.251, length 28
> 11:30:28.980909 ARP, Reply 70.0.0.44 is-at 00:50:56:01:01:02, length 28 <---
>
> inet_bash >> ip nei
> 70.0.0.1 dev vinet FAILED
> 70.0.0.44 dev vinet lladdr ce:18:16:4b:0c:c2 DELAY <---
>
> 5) The bridge vs NS veth pinging works
> 6) The bridge relays ARP into external network and back (checked on Cisco switch), learns of external MAC@s
> ===> 7) External MAC@ does not make it to NS space by ARP <===
> 8) I don’t aim to deploy IP@ with bridge and bridge veth ifaces → this is just to check how it works
> 9) This blog was quite surprising stating that bridge without IP@ can affect routing in global namespace, few /sys kernel tweaks → no help
> https://unix.stackexchange.com/questions/655602/why-two-bridged-veth-cannot-ping-each-other/674621#674621
> 10) Even tried to stop default MAC learning on bridge veth iface by “ip link set dev vinet-brp type bridge_slave learning off” ⇒ did not work, neigh flushed and pinging .251 vs .254 just worked.
>
> So I believe that bridge veth iface is broken in its essential functionality using default “learning/flooding on” settings.
>
> Thanks for your time to look at this and give hope or deny this so I need to create extra ports in my host to get what I want to.
> BR
> Peter
You need to report this upstream, nobody here is going to look at
something like this
Daniel Gröber
Oct 29, 2023, 6:50:05 AM10/29/23
to
Hi Peter,
On Fri, Oct 27, 2023 at 09:29:25AM +0000, peter.ga...@orange.com wrote:
> No attempt at all? Then it's against your own rules I've read before
> submitting this.
I think Luca was a bit harsh here, I'd be happy to help debug this. From a
first look it seems unlikely this is related to iproute2, smells more like
a kernel issue to me, but either way we need a reproducer.
So first step to move this forward is to put together a self contained set
of instructions to reproduce the problem. Your original report is a bit
sparse on context and details.
If you don't feel up to compiling the reproducer script yourself you could
start by showing us your system state using
$ ip -d addr show # on the host and inside the NS if you could
$ bridge -d link; bridge fdb
snippets from /etc/network/interfaces or similar relevant config would help
too.
--Daniel
On Fri, Oct 27, 2023 at 09:29:25AM +0000, peter.ga...@orange.com wrote:
> No attempt at all? Then it's against your own rules I've read before
> submitting this.
I think Luca was a bit harsh here, I'd be happy to help debug this. From a
first look it seems unlikely this is related to iproute2, smells more like
a kernel issue to me, but either way we need a reproducer.
So first step to move this forward is to put together a self contained set
of instructions to reproduce the problem. Your original report is a bit
sparse on context and details.
If you don't feel up to compiling the reproducer script yourself you could
start by showing us your system state using
$ ip -d addr show # on the host and inside the NS if you could
$ bridge -d link; bridge fdb
snippets from /etc/network/interfaces or similar relevant config would help
too.
--Daniel
Daniel Gröber
Oct 30, 2023, 8:10:05 AM10/30/23
to
Hi Peter,
On Mon, Oct 30, 2023 at 10:43:39AM +0000, peter.ga...@orange.com wrote:
> Would it be possible to join a Webex session setup by me to check this
> out quickly? It's all lab environment.
I don't think that would help with reproducing your environment in this
case, besides I only offer synchronous debugging sessions for paid
consulting engagements.
> If not I will proceed per your instructions
Please do.
--Daniel
On Mon, Oct 30, 2023 at 10:43:39AM +0000, peter.ga...@orange.com wrote:
> Would it be possible to join a Webex session setup by me to check this
> out quickly? It's all lab environment.
I don't think that would help with reproducing your environment in this
case, besides I only offer synchronous debugging sessions for paid
consulting engagements.
> If not I will proceed per your instructions
Please do.
--Daniel
Daniel Gröber
Nov 17, 2023, 9:50:05 PM11/17/23
to
Hi Peter,
On Mon, Nov 13, 2023 at 09:40:46AM +0000, peter.ga...@orange.com wrote:
> In the meantime, I was stubborn to find a solution to what I need in
> order to progress and MACVLAN tech actually delivered it (private mode
> enough),
I used to love macvlan too but now I do L3 instead ;P
> something newer than VETH tech what I could read about, and it's
> just perfect avoiding bridge itself. So no problem to cooperate on this
> fix, I will be glad, just it can get lower priority on your side if you
> even attributed it some 😊
I'd be happy to still track this bug down but I need you to investigate the
behaviour in your environment. If you've torn down the lab already we can
also just call it quits.
If you do want to continue some questions are still pending, see quoted below.
> On Mon, Oct 30, 2023 at 07:25:38PM +0000, peter.ga...@orange.com wrote:
> > 1) As was reported, foreign external world MAC@ does not pass into
> > network namespace, just external border point "vlan199"
>
> How did you check this?
>
> > 2) now collecting data for you, honestly I don’t see external mac
> > address on "inet-br" object, so my previous statement was incorrect..
> > {ossibly I might mixed this up with another "labinet-br" (working in
> > its limited
> > scope) which is IP-defined, while "inet-br" in question is not.
> >
> > 3) so question is, if the MACs learnt via vlan199 are supposed to be
> > paired (displayed) with "inet-br" object and all way up into NS....
>
> I want to make sure we're on the same page, how do you check if the MAC is reaching into the NS? I assume using `ip neigh`? I'd have a look at tcpdump this will tell you whether ARP is even reaching the NS or not.
>
> Something simple like
>
> $ tcpdump -enli $IFACE 'arp or icmp'
>
> optionally you can filter by MAC (`ether host` in pcap-filter speak):
>
> $ tcpdump -enli $IFACE ('arp or icmp) and ether host aa:00:00:00:00:01
>
> Oh and one last thing: please double and tripple check that a firewall isn't interfering.
--Daniel
On Mon, Nov 13, 2023 at 09:40:46AM +0000, peter.ga...@orange.com wrote:
> In the meantime, I was stubborn to find a solution to what I need in
> order to progress and MACVLAN tech actually delivered it (private mode
> enough),
I used to love macvlan too but now I do L3 instead ;P
> something newer than VETH tech what I could read about, and it's
> just perfect avoiding bridge itself. So no problem to cooperate on this
> fix, I will be glad, just it can get lower priority on your side if you
> even attributed it some 😊
I'd be happy to still track this bug down but I need you to investigate the
behaviour in your environment. If you've torn down the lab already we can
also just call it quits.
If you do want to continue some questions are still pending, see quoted below.
> On Mon, Oct 30, 2023 at 07:25:38PM +0000, peter.ga...@orange.com wrote:
> > 1) As was reported, foreign external world MAC@ does not pass into
> > network namespace, just external border point "vlan199"
>
> How did you check this?
>
> > 2) now collecting data for you, honestly I don’t see external mac
> > address on "inet-br" object, so my previous statement was incorrect..
> > {ossibly I might mixed this up with another "labinet-br" (working in
> > its limited
> > scope) which is IP-defined, while "inet-br" in question is not.
> >
> > 3) so question is, if the MACs learnt via vlan199 are supposed to be
> > paired (displayed) with "inet-br" object and all way up into NS....
>
> I want to make sure we're on the same page, how do you check if the MAC is reaching into the NS? I assume using `ip neigh`? I'd have a look at tcpdump this will tell you whether ARP is even reaching the NS or not.
>
> Something simple like
>
> $ tcpdump -enli $IFACE 'arp or icmp'
>
> optionally you can filter by MAC (`ether host` in pcap-filter speak):
>
> $ tcpdump -enli $IFACE ('arp or icmp) and ether host aa:00:00:00:00:01
>
> Oh and one last thing: please double and tripple check that a firewall isn't interfering.
--Daniel
0 new messages