Hi, thanks for the clues.
We have not updated the emulab software much. We updated some drivers in the MFSes and may have patched some scripts, but no proper update. We have downloaded and updated the OS images, though. When I run 'wap image_import -g -r' on the FBSD112-64-STD image, it returns very quickly. I assume that means we have the latest version?
I'll stick to describing the problem with delay node functionality. Your tips really helped narrow it down. In both the 10.2 and 11.2 images we get what appears to be a properly configured bridge with two members. Also ipfw is configured the same. In both versions /var/emulab/boot/tmcc/ifconfig is the same, except for different MAC addresses. The difference is that in 11.2, /var/emulab/boot/rc.ifc is missing. I copied the rc.ifc script from the 10.2 delay node to the 11.2 one, modified for different interface numbers, and executed it. Now the 11.2 delay node works!
I see that /usr/local/etc/emulab/
libsetup.pm has changed a lot between images. Is it likely that our old installation is no longer compatible with some of the scripts in the new FBSD image?
I'm not sure how to invoke the functions of
libsetup.pm to figure out what is going wrong there, or where to look for errors that may have come up during swap-in or boot. I looked in the normal log files, but could not find anything.
In case it's informative, here are the relevant configurations from the two images used as a delay node in identical single-link experiments:
10.2:
============================
[brenton@tbdelay0 ~]$ ifconfig
[...]
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=403fb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,POLLING,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether 00:1b:21:6b:cc:30
inet6 fe80::21b:21ff:fe6b:cc30%igb2 prefixlen 64 tentative scopeid 0x3
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 1000baseT <full-duplex>
status: active
[...]
igb4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=403fb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,POLLING,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether 00:1b:21:6b:cc:32
inet6 fe80::21b:21ff:fe6b:cc32%igb4 prefixlen 64 tentative scopeid 0x5
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 1000baseT <full-duplex>
status: active
[...]
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:a0:1d:88:ee:01
nd6 options=1<PERFORMNUD>
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igb4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 55
member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 3 priority 128 path cost 55
[brenton@tbdelay0 ~]$ sudo ipfw list
60110 pipe 60110 ip from any to any out recv igb2
60120 pipe 60120 ip from any to any out recv igb4
65534 allow ip from any to any
65535 deny ip from any to any
[brenton@tbdelay0 ~]$ sudo ipfw pipe 60110 show
60110: 10.000 Mbit/s 0 ms burst 0
q191182 50 sl. 0 flows (1 buckets) sched 125646 weight 0 lmax 0 pri 0 droptail
sched 125646 type FIFO flags 0x0 0 buckets 0 active
[brenton@tbdelay0 ~]$ sudo ipfw pipe 60120 show
60120: 10.000 Mbit/s 0 ms burst 0
q191192 50 sl. 0 flows (1 buckets) sched 125656 weight 0 lmax 0 pri 0 droptail
sched 125656 type FIFO flags 0x0 0 buckets 0 active
[brenton@tbdelay0 /var/emulab/boot/tmcc]$ cat ifconfig
INTERFACE IFACETYPE=igb INET= MASK= MAC=001b216bcc30 SPEED=1000Mbps DUPLEX=full IFACE= RTABID= LAN=
INTERFACE IFACETYPE=igb INET= MASK= MAC=001b216bcc32 SPEED=1000Mbps DUPLEX=full IFACE= RTABID= LAN=
11.2
============================
[brenton@tbdelay0 ~]$ ifconfig
[...]
igb0: flags=8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403fb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,POLLING,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether b4:96:91:43:de:2c
hwaddr b4:96:91:43:de:2c
inet6 fe80::b696:91ff:fe43:de2c%igb0 prefixlen 64 tentative scopeid 0x3
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
[...]
igb2: flags=8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403fb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,POLLING,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether b4:96:91:43:de:2e
hwaddr b4:96:91:43:de:2e
inet6 fe80::b696:91ff:fe43:de2e%igb2 prefixlen 64 tentative scopeid 0x5
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
[...]
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:a0:1d:88:ee:01
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000000
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 3 priority 128 path cost 2000000
[brenton@tbdelay0 ~]$ sudo ipfw list
60110 pipe 60110 ip from any to any out recv igb0
60120 pipe 60120 ip from any to any out recv igb2
65534 allow ip from any to any
65535 deny ip from any to any
[brenton@tbdelay0 ~]$ sudo ipfw pipe 60110 show
60110: 10.000 Mbit/s 0 ms burst 0
q191182 50 sl. 0 flows (1 buckets) sched 125646 weight 0 lmax 0 pri 0 droptail
sched 125646 type FIFO flags 0x0 0 buckets 0 active
[brenton@tbdelay0 ~]$ sudo ipfw pipe 60120 show
60120: 10.000 Mbit/s 0 ms burst 0
q191192 50 sl. 0 flows (1 buckets) sched 125656 weight 0 lmax 0 pri 0 droptail
sched 125656 type FIFO flags 0x0 0 buckets 0 active
[brenton@tbdelay0 /var/emulab/boot/tmcc]$ cat ifconfig
INTERFACE IFACETYPE=igb INET= MASK= MAC=b4969143de2c SPEED=1000Mbps DUPLEX=full IFACE= RTABID= LAN=
INTERFACE IFACETYPE=igb INET= MASK= MAC=b4969143de2e SPEED=1000Mbps DUPLEX=full IFACE= RTABID= LAN=
Thanks,
Brenton