iocage (or more likely vnet) networking issues

Joseph Ward

unread,

May 23, 2019, 9:34:01 PM5/23/19

to iocage

Hi,

I'm struggling to get a jail using vnet to communicate with the lan.

I'm running:

# iocage -vVersion 1.1 RELEASE 2019/01

on

# uname -m -r -s -v
FreeBSD 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 GENERIC amd64

The jail I created:

iocage create -r 12.0-RELEASE vnet=on defaultrouter=192.168.12.1 ip4_addr="vnet0|192.168.12.50" boot=on children_max=10 jail_zfs=on allow_mount_devfs=1 allow_mount_procfs=1 allow_mount_nullfs=1 allow_raw_sockets=1 allow_socket_af=1 allow_sysvipc=1 allow_chflags=1 securelevevl=0 allow_mount_tmpfs=1 mount_linprocfs=1 -n jail

My host network settings:

# ifconfigre0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE> ether 40:8d:5c:c4:cc:55 inet 192.168.6.30 netmask 0xffffff00 broadcast 192.168.6.255 inet 192.168.6.52 netmask 0xffffffff broadcast 192.168.6.52 inet 192.168.6.53 netmask 0xffffffff broadcast 192.168.6.53 inet 192.168.6.47 netmask 0xffffffff broadcast 192.168.6.47 inet 192.168.6.44 netmask 0xffffffff broadcast 192.168.6.44 inet 192.168.6.42 netmask 0xffffffff broadcast 192.168.6.42 inet 192.168.6.51 netmask 0xffffffff broadcast 192.168.6.51 inet 192.168.6.45 netmask 0xffffffff broadcast 192.168.6.45 inet 192.168.6.49 netmask 0xffffffff broadcast 192.168.6.49 inet 192.168.6.43 netmask 0xffffffff broadcast 192.168.6.43 inet 192.168.6.48 netmask 0xffffffff broadcast 192.168.6.48 inet 192.168.6.46 netmask 0xffffffff broadcast 192.168.6.46 inet 192.168.6.31 netmask 0xffffffff broadcast 192.168.6.31 inet 192.168.6.36 netmask 0xffffffff broadcast 192.168.6.36 inet 192.168.6.35 netmask 0xffffffff broadcast 192.168.6.35 inet 192.168.6.33 netmask 0xffffffff broadcast 192.168.6.33 inet 192.168.6.38 netmask 0xffffffff broadcast 192.168.6.38 inet 192.168.6.39 netmask 0xffffffff broadcast 192.168.6.39 inet 192.168.6.34 netmask 0xffffffff broadcast 192.168.6.34 inet 192.168.6.37 netmask 0xffffffff broadcast 192.168.6.37 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet 127.0.1.52 netmask 0xffffffff inet 127.0.1.53 netmask 0xffffffff inet 127.0.1.51 netmask 0xffffffff inet 127.0.1.1 netmask 0xffffffff inet 127.0.1.6 netmask 0xffffffff inet 127.0.1.5 netmask 0xffffffff inet 127.0.1.3 netmask 0xffffffff inet 127.0.1.8 netmask 0xffffffff inet 127.0.1.9 netmask 0xffffffff inet 127.0.1.4 netmask 0xffffffff inet 127.0.1.7 netmask 0xffffffff groups: lo nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>bridge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:66:3a:87:e6:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.84 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 6 priority 128 path cost 2000 member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 1 priority 128 path cost 55 groups: bridge nd6 options=9<PERFORMNUD,IFDISABLED>ue0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE> ether d8:eb:97:bd:2d:dd inet 10.43.0.53 netmask 0xffffffff broadcast 10.43.0.53 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>vnet0.84: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: jail as nic: epair0b options=8<VLAN_MTU> ether 02:ff:60:89:12:77 hwaddr 02:8b:c6:9f:47:0a inet6 fe80::ff:60ff:fe89:1277%vnet0.84 prefixlen 64 scopeid 0x6 groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

(I've also tried to use the same host subnet with the jail interface; what happens is identical to the following when I have the 192.168.6.0/24 network shared with the host.)

My sysctls regarding the bridge are:

# sysctl -a |grep net.link.bridge
net.link.bridge.ipfw: 0
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 0

# sysctl -a | grep net.inet.ip.forwarnet.inet.ip.forwarding: 1

When I ping the gateway from the jail, I get the following activity:

# ping 192.168.12.1PING 192.168.12.1 (192.168.12.1): 56 data bytesping: sendto: Host is downping: sendto: Host is downping: sendto: Host is downping: sendto: Host is downping: sendto: Host is downping: sendto: Host is downping: sendto: Host is downping: sendto: Host is down^C--- 192.168.12.1 ping statistics ---13 packets transmitted, 0 packets received, 100.0% packet lossroot@poudriere-20190427:~ # ping 192.168.12.1PING 192.168.12.1 (192.168.12.1): 56 data bytesping: sendto: Host is down

# tcpdump -ni vnet0.84 host 192.168.12.50tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on vnet0.84, link-type EN10MB (Ethernet), capture size 262144 bytes21:11:27.240246 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:11:28.243038 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:11:29.245525 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:11:30.246126 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:11:31.250830 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:11:32.254789 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 28

tcpdump -ni re0 host 192.168.12.50tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on re0, link-type EN10MB (Ethernet), capture size 262144 bytes21:11:27.240250 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:11:27.240675 ARP, Reply 192.168.12.1 is-at 00:0d:b9:34:d5:51, length 4621:11:28.243061 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:11:28.243277 ARP, Reply 192.168.12.1 is-at 00:0d:b9:34:d5:51, length 4621:11:29.245530 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:11:29.245675 ARP, Reply 192.168.12.1 is-at 00:0d:b9:34:d5:51, length 4621:11:30.246130 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:11:30.246345 ARP, Reply 192.168.12.1 is-at 00:0d:b9:34:d5:51, length 4621:11:31.250834 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:11:31.251189 ARP, Reply 192.168.12.1 is-at 00:0d:b9:34:d5:51, length 4621:11:32.254811 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:11:32.255075 ARP, Reply 192.168.12.1 is-at 00:0d:b9:34:d5:51, length 46

As you can see, the LAN interface sends out the initial ARP request, and recieves the ARP reply. However, that's not passed to the VNET interface (and the tcpdump for the bridge looks identical to the vnet interface). Once the initial ping attempt is made, the router has the MAC because of the ARP request. When I try to use nc to hit port 80, the SYN gets through, but there can't be a response because the ARP reply never gets back through!

nc -vvv 192.168.12.50 80

tcpdump -ni re0 host 192.168.12.50tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on re0, link-type EN10MB (Ethernet), capture size 262144 bytes21:26:53.051528 IP 192.168.12.1.16905 > 192.168.12.50.80: Flags [S], seq 3027475721, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1574444148 ecr 0], length 021:26:53.051579 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:26:53.051734 ARP, Reply 192.168.12.1 is-at 00:0d:b9:34:d5:51, length 4621:26:56.051279 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:26:56.051460 ARP, Reply 192.168.12.1 is-at 00:0d:b9:34:d5:51, length 4621:26:59.251163 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:26:59.251452 ARP, Reply 192.168.12.1 is-at 00:0d:b9:34:d5:51, length 4621:27:02.451157 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:27:02.451442 ARP, Reply 192.168.12.1 is-at 00:0d:b9:34:d5:51, length 46

tcpdump -ni vnet0.84 host 192.168.12.50tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on vnet0.85, link-type EN10MB (Ethernet), capture size 262144 bytes21:26:53.051549 IP 192.168.12.1.16905 > 192.168.12.50.80: Flags [S], seq 3027475721, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1574444148 ecr 0], length 021:26:53.051576 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:26:56.051274 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:26:59.251150 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 2821:27:02.451144 ARP, Request who-has 192.168.12.1 tell 192.168.12.50, length 28

Does anyone have any ideas? I'm totally stumped.

Joseph Ward

unread,

May 26, 2019, 4:53:46 PM5/26/19

to ioc...@googlegroups.com

Hi everyone;

This is now "SOLVED".

For posterity: apparently there's some sort of issue between virtualbox and if_bridge. Stopping all vbox VMs allows the ARP requests to make it to the bridge just fine and everything now works appropriately.

--
You received this message because you are subscribed to the Google Groups "iocage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iocage+un...@googlegroups.com.
To post to this group, send email to ioc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/iocage/6503379d-c2d0-b8e8-92e5-bf38ceef4e6d%40hilltopgroup.com.
For more options, visit https://groups.google.com/d/optout.

Brandon Schneider

unread,

May 26, 2019, 4:56:11 PM5/26/19

to Joseph Ward, ioc...@googlegroups.com

Glad to hear Joseph! I was waiting until I was off vacation to help you troubleshoot as it looked very complicated since everything looked correct!

Thanks for putting your solution here :)

Brandon

To view this discussion on the web visit https://groups.google.com/d/msgid/iocage/b2216088-ef2c-15ba-6978-b5d95ae264be%40hilltopgroup.com.

Joseph Ward

unread,

May 26, 2019, 5:13:09 PM5/26/19

to ioc...@googlegroups.com

No worries at all, I assumed everyone was out on the long weekend. I really appreciate everything you've done to put iocage together, and the help you're always willing to give.

You're right that it was complicated; I finally gave up, reinstalled all of the jail configs on new hardware where it just worked. The ONLY networking difference i saw was that the new hardware nic was igb instead of re, so I then assumed that was the issue. I found an igb card, installed it in the problematic server, reconfigured the jails to use it, and then again, everything worked! That is, until I reconfigured the VMs to use the igb nic, and then it all quit working again. A couple of tests, and I'd proven that everything works when the VMs are stopped. So.. now I have to mitigate that, but otherwise I'm all good.

Thanks!

-Joseph

To view this discussion on the web visit https://groups.google.com/d/msgid/iocage/df7a2849-e858-4a21-b080-1917c12e5159%40www.fastmail.com.

Brandon Schneider

unread,

May 26, 2019, 5:18:43 PM5/26/19

to Joseph Ward, ioc...@googlegroups.com

No problem at all, thanks for the kind words :)

Thats frustrating to say the least. Makes you think we chose the wrong hobby sometimes ;)

Brandon

To view this discussion on the web visit https://groups.google.com/d/msgid/iocage/6dec303a-34f2-3697-2a8f-210d306daa30%40hilltopgroup.com.

Reply all

Reply to author

Forward