Interfaces not set up with FBSD112-64-STD image

[Brenton Walker]

Dear emulab-admins,

     We have a somewhat old emulab installation (emulab-devel d28d8354d8e7031b4616cd482f059156e39c6087 Tue Nov 15 14:40:02 2016) with boss and ops running FreeBSD 10.2.

    We have been having a problem with FBSD112-64-STD.  When we use the image on a named node in an experiment, the node comes up with unconfigured interfaces.  If we manually give the interfaces IP addresses, the links work.  All the vlans are being set up fine.  When the image is used as a delay node image, it just does not work. I'm not sure how to configure that manually, but was not able to see any incoming packets using tcpdump (I know packets were coming to the delay node interface because I watched the outgoing packet counter on the experimental switch).  I can't find any relevant error message in the swap-in log or in the files in /var/log/ on the experimental node.

    This hasn't been much of a problem because the FBSD images only get used on delay nodes here, and the FBSD102-64-STD image does the job on our older nodes.  But as our older nodes get replaced by newer ones that don't work with FBSD 10.2, we are running out of nodes that can act as delay nodes.

    I have checked that the MAC addresses of the nodes match what is in the database.

    The experimental interfaces are igb type.  On the older nodes the control net is igb, and on the newer nodes the control net is ixl.

    Is this a case where we just need to start by upgrading the emulab?  Is there a reasonable upgrade path from what we have?

Thanks,

Brenton

[Mike Hibler]

So you have not updated the Emulab software recently either? Or are you
just running old boss/ops OSes? What about the FBSDxxx-64-STD images,
have you updated them at all? It would certainly be good to update your
install, but let's put that aside for the moment.

So if you use FBSD112-64-STD as either a regular node image or a delay
node, it does not configure the interfaces? Note that for a delay node,
there are no IP addresses assigned, but the interfaces should get put in
a bridge. If you see no bridge device with ifconfig, or if it contains
no interfaces, something is wrong.

In either case you can look at /var/emulab/boot/rc.ifc on the node to
see what it wants to do to configure the interfaces. This is a script we
generate at boot time based on information returned from the boss DB.
The raw info should be in /var/emulab/boot/tmcc/ifconfig (I think).

Also look on the boss node at /usr/testbed/log/tmcd.log and see if it is
complaining about the client version being newer than the server. That
can happen if you are using an image that is newer than your server side.

> --
> You received this message because you are subscribed to the Google Groups
> "emulab-admins" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to emulab-admin...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/
> emulab-admins/fae1dbb6-0e4c-45d3-82d0-63dcfe5ad1f8o%40googlegroups.com.

[Brenton Walker]

Hi, thanks for the clues.

We have not updated the emulab software much.  We updated some drivers in the MFSes and may have patched some scripts, but no proper update.  We have downloaded and updated the OS images, though.  When I run 'wap image_import -g -r' on the FBSD112-64-STD image, it returns very quickly.  I assume that means we have the latest version?

I'll stick to describing the problem with delay node functionality.  Your tips really helped narrow it down.  In both the 10.2 and 11.2 images we get what appears to be a properly configured bridge with two members.  Also ipfw is configured the same.  In both versions /var/emulab/boot/tmcc/ifconfig is the same, except for different MAC addresses.  The difference is that in 11.2, /var/emulab/boot/rc.ifc is missing.  I copied the rc.ifc script from the 10.2 delay node to the 11.2 one, modified for different interface numbers, and executed it.  Now the 11.2 delay node works!

I see that /usr/local/etc/emulab/libsetup.pm has changed a lot between images.  Is it likely that our old installation is no longer compatible with some of the scripts in the new FBSD image?

I'm not sure how to invoke the functions of libsetup.pm to figure out what is going wrong there, or where to look for errors that may have come up during swap-in or boot.  I looked in the normal log files, but could not find anything.

In case it's informative, here are the relevant configurations from the two images used as a delay node in identical single-link experiments:

10.2:

============================

[brenton@tbdelay0 ~]$ ifconfig

[...]

igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

options=403fb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,POLLING,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>

ether 00:1b:21:6b:cc:30

inet6 fe80::21b:21ff:fe6b:cc30%igb2 prefixlen 64 tentative scopeid 0x3 

nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

media: Ethernet 1000baseT <full-duplex>

status: active

[...]

igb4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

options=403fb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,POLLING,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>

ether 00:1b:21:6b:cc:32

inet6 fe80::21b:21ff:fe6b:cc32%igb4 prefixlen 64 tentative scopeid 0x5 

nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

media: Ethernet 1000baseT <full-duplex>

status: active

[...]

bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

ether 02:a0:1d:88:ee:01

nd6 options=1<PERFORMNUD>

id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15

maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200

root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0

member: igb4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

        ifmaxaddr 0 port 5 priority 128 path cost 55

member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

        ifmaxaddr 0 port 3 priority 128 path cost 55

[brenton@tbdelay0 ~]$ sudo ipfw list

60110 pipe 60110 ip from any to any out recv igb2

60120 pipe 60120 ip from any to any out recv igb4

65534 allow ip from any to any

65535 deny ip from any to any

[brenton@tbdelay0 ~]$ sudo ipfw pipe 60110 show

60110:  10.000 Mbit/s    0 ms burst 0 

q191182  50 sl. 0 flows (1 buckets) sched 125646 weight 0 lmax 0 pri 0 droptail

 sched 125646 type FIFO flags 0x0 0 buckets 0 active

[brenton@tbdelay0 ~]$ sudo ipfw pipe 60120 show

60120:  10.000 Mbit/s    0 ms burst 0 

q191192  50 sl. 0 flows (1 buckets) sched 125656 weight 0 lmax 0 pri 0 droptail

 sched 125656 type FIFO flags 0x0 0 buckets 0 active

[brenton@tbdelay0 /var/emulab/boot/tmcc]$ cat ifconfig 

INTERFACE IFACETYPE=igb INET= MASK= MAC=001b216bcc30 SPEED=1000Mbps DUPLEX=full IFACE= RTABID= LAN=

INTERFACE IFACETYPE=igb INET= MASK= MAC=001b216bcc32 SPEED=1000Mbps DUPLEX=full IFACE= RTABID= LAN=

11.2

============================

[brenton@tbdelay0 ~]$ ifconfig

[...]

igb0: flags=8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500

options=6403fb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,POLLING,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>

ether b4:96:91:43:de:2c

hwaddr b4:96:91:43:de:2c

inet6 fe80::b696:91ff:fe43:de2c%igb0 prefixlen 64 tentative scopeid 0x3 

nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

media: Ethernet autoselect (1000baseT <full-duplex>)

status: active

[...]

igb2: flags=8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500

options=6403fb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,POLLING,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>

ether b4:96:91:43:de:2e

hwaddr b4:96:91:43:de:2e

inet6 fe80::b696:91ff:fe43:de2e%igb2 prefixlen 64 tentative scopeid 0x5 

nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

media: Ethernet autoselect (1000baseT <full-duplex>)

status: active

[...]

bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

ether 02:a0:1d:88:ee:01

nd6 options=1<PERFORMNUD>

groups: bridge 

id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15

maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200

root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0

member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

        ifmaxaddr 0 port 5 priority 128 path cost 2000000

member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

        ifmaxaddr 0 port 3 priority 128 path cost 2000000

[brenton@tbdelay0 ~]$ sudo ipfw list

60110 pipe 60110 ip from any to any out recv igb0

60120 pipe 60120 ip from any to any out recv igb2

65534 allow ip from any to any

65535 deny ip from any to any

[brenton@tbdelay0 ~]$ sudo ipfw pipe 60110 show

60110:  10.000 Mbit/s    0 ms burst 0 

q191182  50 sl. 0 flows (1 buckets) sched 125646 weight 0 lmax 0 pri 0 droptail

 sched 125646 type FIFO flags 0x0 0 buckets 0 active

[brenton@tbdelay0 ~]$ sudo ipfw pipe 60120 show

60120:  10.000 Mbit/s    0 ms burst 0 

q191192  50 sl. 0 flows (1 buckets) sched 125656 weight 0 lmax 0 pri 0 droptail

 sched 125656 type FIFO flags 0x0 0 buckets 0 active

[brenton@tbdelay0 /var/emulab/boot/tmcc]$ cat ifconfig 

INTERFACE IFACETYPE=igb INET= MASK= MAC=b4969143de2c SPEED=1000Mbps DUPLEX=full IFACE= RTABID= LAN=

INTERFACE IFACETYPE=igb INET= MASK= MAC=b4969143de2e SPEED=1000Mbps DUPLEX=full IFACE= RTABID= LAN=

Thanks,

Brenton

[Leigh Stoller]

at 1:12 AM, Brenton Walker <brenton....@gmail.com> wrote:

> I see that /usr/local/etc/emulab/libsetup.pm has changed a lot between images. Is it likely that our old installation is no longer compatible with some of the scripts in the new FBSD image?

Just to clarify. We try very hard to make sure that old images continue to
work with new versions of Emulab software. We do not try as hard to ensure
new images against old versions. :-(

Leigh

[Mike Hibler]

On Fri, Oct 23, 2020 at 01:12:15AM -0700, Brenton Walker wrote:
> Hi, thanks for the clues.
>
> We have not updated the emulab software much. We updated some drivers in the
> MFSes and may have patched some scripts, but no proper update. We have
> downloaded and updated the OS images, though. When I run 'wap image_import -g
> -r' on the FBSD112-64-STD image, it returns very quickly. I assume that means
> we have the latest version?
>

Yes. The FreeBSD 11.2 image is not updated any longer. We have moved on to
11.3 and shortly 11.4. But for you, the older client side if probably
beneficial given your older boss/ops install.

> I'll stick to describing the problem with delay node functionality. Your tips
> really helped narrow it down. In both the 10.2 and 11.2 images we get what
> appears to be a properly configured bridge with two members. Also ipfw is
> configured the same. In both versions /var/emulab/boot/tmcc/ifconfig is the
> same, except for different MAC addresses. The difference is that in 11.2, /var
> /emulab/boot/rc.ifc is missing. I copied the rc.ifc script from the 10.2 delay
> node to the 11.2 one, modified for different interface numbers, and executed
> it. Now the 11.2 delay node works!
>
> I see that /usr/local/etc/emulab/libsetup.pm has changed a lot between images.
> Is it likely that our old installation is no longer compatible with some of the
> scripts in the new FBSD image?
>

As Leigh said, we do our best to ensure older clients (images) work with
current server-side software but not vice versa. We do check on the server
side and log a warning if a newer client connects to the "tmcd" server.
That is why I wanted you to look at /usr/testbed/log/tmcd.log on boss.

> I'm not sure how to invoke the functions of libsetup.pm to figure out what is
> going wrong there, or where to look for errors that may have come up during
> swap-in or boot. I looked in the normal log files, but could not find
> anything.
>

I will see if I can determine why rc.ifc is not being produced. The client
side is a bit odd. There is a script /usr/local/etc/emulab/rc/rc.ifconfig
which makes the call to the tmcd server to get ifconfig info. That is cached
in the /var/emulab/boot/tmcc/ifconfig file. That script then generates and
runs the /var/emulab/boo/rc.ifc script. The only thing that script is going
to do on a delay node is bring the involved interfaces up. You will see in
the 11.2 ifconfig info below that the flags do not include "UP".

> emulab-admins/7764adbc-22e7-42b6-8ca1-32ecda7178d0n%40googlegroups.com.

[Mike Hibler]

On a fresh delay node (one in which you did not copy over an rc.ifc),
log in and run the setup by hand:

sudo /usr/local/etc/emulab/rc/rc.ifconfig boot

and see if that generates any errors or produces the rc.ifc file.

> > > ???????? We have a somewhat old emulab installation (emulab-devel

> > > d28d8354d8e7031b4616cd482f059156e39c6087 Tue Nov 15 14:40:02 2016) with
> > boss
> > > and ops running FreeBSD 10.2.
> > >

> > > ?????? We have been having a problem with FBSD112-64-STD.?? When we use the

> > image
> > > on a named node in an experiment, the node comes up with unconfigured

> > > interfaces.?? If we manually give the interfaces IP addresses, the links
> > work.??
> > > All the vlans are being set up fine.?? When the image is used as a delay

> > node
> > > image, it just does not work. I'm not sure how to configure that
> > manually, but
> > > was not able to see any incoming packets using tcpdump (I know packets
> > were
> > > coming to the delay node interface because I watched the outgoing packet

> > > counter on the experimental switch).?? I can't find any relevant error

> > message
> > > in the swap-in log or in the files in /var/log/ on the experimental node.
> > >

> > > ?????? This hasn't been much of a problem because the FBSD images only get

> > used on
> > > delay nodes here, and the FBSD102-64-STD image does the job on our older

> > > nodes.?? But as our older nodes get replaced by newer ones that don't work

> > with
> > > FBSD 10.2, we are running out of nodes that can act as delay nodes.
> > >

> > > ?????? I have checked that the MAC addresses of the nodes match what is in
> > the
> > > database.
> > >
> > > ?????? The experimental interfaces are igb type.?? On the older nodes the

> > control
> > > net is igb, and on the newer nodes the control net is ixl.
> > >

> > > ?????? Is this a case where we just need to start by upgrading the emulab???

> > Is
> > > there a reasonable upgrade path from what we have?
> > >
> > > Thanks,
> > > Brenton
> > >
> > >
> > >
> > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "emulab-admins" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > email
> > > to emulab-admin...@googlegroups.com.
> > > To view this discussion on the web visit https://groups.google.com/d/
> > msgid/
> > > emulab-admins/fae1dbb6-0e4c-45d3-82d0-63dcfe5ad1f8o%40googlegroups.com.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "emulab-admins" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email
> > to emulab-admin...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/
> > emulab-admins/7764adbc-22e7-42b6-8ca1-32ecda7178d0n%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups "emulab-admins" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to emulab-admin...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/emulab-admins/20201023171213.GF75197%40flux.utah.edu.

[Brenton Walker]

Thanks.  I get the following error:

[brenton@tbdelay0 /var/emulab/boot]$ sudo /usr/local/etc/emulab/rc/rc.ifconfig boot

Checking Testbed interface configuration ... 

*** WARNING: Bad ifconfig line: INTERFACE IFACETYPE=igb INET= MASK= MAC=001b216bcbd1 SPEED=1000Mbps DUPLEX=full IFACE= RTABID= LAN=

*** WARNING: Bad ifconfig line: INTERFACE IFACETYPE=igb INET= MASK= MAC=001b216bcbd3 SPEED=1000Mbps DUPLEX=full IFACE= RTABID= LAN=

In this case the file /var/emulab/boot/tmcc/ifconfig looks like this:

[brenton@tbdelay0 /var/emulab/boot/tmcc]$ cat ifconfig 

INTERFACE IFACETYPE=igb INET= MASK= MAC=001b216bcbd1 SPEED=1000Mbps DUPLEX=full IFACE= RTABID= LAN=

INTERFACE IFACETYPE=igb INET= MASK= MAC=001b216bcbd3 SPEED=1000Mbps DUPLEX=full IFACE= RTABID= LAN=

Thanks,

Brenton

[Brenton Walker]

I looked through/usr/local/etc/emulab/libsetup.pm for the error message. It seems that it fails because the /var/emulab/boot/tmcc/ifconfig lines are missing the "MTU=" field at the end.  When I add that, it runs with no errors and rc.ifc gets creted.

To answer your question that i missed before, yes, /usr/testbed/log/tmcd.log contains lots of messages about TMCD version mismatch.

Oct 27 13:05:45 boss tmcd[1394]: version skew on request from XXX.XXX.XXX.223: server=40, request=42, old TMCD installed?
Oct 27 13:05:47 boss tmcd[1393]: version skew on request from YYY.YYY.YYY.219: server=40, request=42, old TMCD installed?

thanks,

Brenton

You received this message because you are subscribed to a topic in the Google Groups "emulab-admins" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/emulab-admins/aU2yt6wpZTs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to emulab-admin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emulab-admins/7b7fb06d-d314-4178-92bb-4702b3a19248n%40googlegroups.com.

[Mike Hibler]

This is a case where we don't provide "forward compatibility". Newer clients
expect the MTU to be passed, but your server version is from before we added
that parameter. There is no easy way out of this situation.

You could make your own custom version of newer images that have older
client-side scripts, but that is not a sustainable path.

Updating the server side (tmcd) pretty much requires updating all of the
Emulab software and that will likely require updating your server OS.

So you are stuck between an unsustainable hack and a painful full upgrade.
I would recommend the latter. There are some guidelines in the current
Emulab repo in the install subdir: README-upgrade-X-Y.txt so you can get
a sense of what is required.

Or there is a third option of a from-scratch reinstall of the system.
But that would wipe all your existing experiments, projects and users.

> > > emulab-admins/7764adbc-22e7-42b6-8ca1-32ecda7178d0n%
> 40googlegroups.com.

> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "emulab-admins" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send an email to emulab-admin...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/

> msgid/emulab-admins/20201023171213.GF75197%40flux.utah.edu.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "emulab-admins" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/
> emulab-admins/aU2yt6wpZTs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> emulab-admin...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/
> emulab-admins/7b7fb06d-d314-4178-92bb-4702b3a19248n%40googlegroups.com.
>

> --
> You received this message because you are subscribed to the Google Groups
> "emulab-admins" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to emulab-admin...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/
> emulab-admins/

> CAAmnW%2BqZ44Eov9bgzB8OkNpisbpkOhm_3htOFL6bgnkQoQ%2BH%3Dg%40mail.gmail.com.