no address and/or no carrier on some NICs after swap-in

[burnettb317]

I've got a user that is swapping in a medium sized experiment and once swapped he checks the various NICs configurations and some of them never got configured, in some cases the switches are not getting configured for 1 or 2 specific ports not getting enabled so the node says "no carrier".   It happens with LAN and link connections.  I think it might only be for links/lans that need a vlan but I'm not 100% on that.  In the past I've had similar issues and it turned out to be trunk connections in the wires table got outdated, I checked to make sure wires matched reality and it did, but I didn't have TrunkID values set for several wires, so I fixed that using old/unused TrunkID numbers for wires that had been deleted (would that cause any problems?  would TrunkID = 0 for multiple wires affect anything??).

The experiment is also doing some installs after the swap-in and those are getting cut-off early in some cases (not necessarily the same nodes), it seems to be an rpm hang-up but we are using dkms.  We were going to remove rpm from our Debian image.  Either way, that doesn't seem to be connected to the NIC issues.

I've checked the script and it all looks good, I've verified the ports and wires and they look good.  

Any ideas on what to check next?

Thanks,

--Ben

[Mike Hibler]

So you are saying that some switch ports are not getting configured with the
correct VLAN and/or are not being enabled? The node is reporting that there
is no carrier, as opposed to just not being able to get traffic through it?

On your boss node you can run the SNMP tool manually for the experiment:

wap snmpit -r <pid> <eid> # tear down vlans
wap snmpit -t <pid> <eid> # setup vlans

and see if you see any warnings or errors that you might have missed in the
um..."loquacious" logfile for swapin.

I don't think the trunkid is used for interswitch links, but I will defer
to Leigh on that one!

> --
> You received this message because you are subscribed to the Google Groups
> "emulab-admins" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to emulab-admin...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/
> emulab-admins/5cdd4217-ebcc-459a-ac21-f6ce842076ee%40googlegroups.com.

[Mike Hibler]

I missed the "no address" part. That is typically caused when the interface
MACs either don't match what is in the DB or if an interface fails to show
up under the OS. The latter can be caused by a custom kernel that lacks the
proper driver or version of a driver. Or the NIC hardware getting into a
goofy state.

On Tue, Apr 21, 2020 at 08:24:43PM -0600, Mike Hibler wrote:
> So you are saying that some switch ports are not getting configured with the
> correct VLAN and/or are not being enabled? The node is reporting that there
> is no carrier, as opposed to just not being able to get traffic through it?
>
> On your boss node you can run the SNMP tool manually for the experiment:
>
> wap snmpit -r <pid> <eid> # tear down vlans
> wap snmpit -t <pid> <eid> # setup vlans
>
> and see if you see any warnings or errors that you might have missed in the
> um..."loquacious" logfile for swapin.
>
> I don't think the trunkid is used for interswitch links, but I will defer
> to Leigh on that one!
>
> On Tue, Apr 21, 2020 at 02:13:16PM -0700, burnettb317 wrote:
> > I've got a user that is swapping in a medium sized experiment and once swapped
> > he checks the various NICs configurations and some of them never got
> > configured, in some cases the switches are not getting configured for 1 or 2

> > specific ports not getting enabled so the node says "no carrier".?? ??It happens
> > with LAN and link connections.?? I think it might only be for links/lans that
> > need a vlan but I'm not 100% on that.?? In the past I've had similar issues and

> > it turned out to be trunk connections in the wires table got outdated, I
> > checked to make sure wires matched reality and it did, but I didn't have
> > TrunkID values set for several wires, so I fixed that using old/unused TrunkID

> > numbers for wires that had been deleted (would that cause any problems??? would

> > TrunkID = 0 for multiple wires affect anything??).
> >
> > The experiment is also doing some installs after the swap-in and those are
> > getting cut-off early in some cases (not necessarily the same nodes), it seems

> > to be an rpm hang-up but we are using dkms.?? We were going to remove rpm from
> > our Debian image.?? Either way, that doesn't seem to be connected to the NIC

> > issues.
> >
> > I've checked the script and it all looks good, I've verified the ports and

> > wires and they look good.????

> >
> > Any ideas on what to check next?
> >
> > Thanks,
> > --Ben
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "emulab-admins" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email
> > to emulab-admin...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/
> > emulab-admins/5cdd4217-ebcc-459a-ac21-f6ce842076ee%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups "emulab-admins" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to emulab-admin...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/emulab-admins/20200422022443.GF83711%40flux.utah.edu.

[Ben Burnett]

Thanks,  I'll check out both of those.

-Ben

Ben Burnett <burne...@gmail.com>          

  R&D Software Engineer / Computer Scientist
  ATC (Architecture Technology Corporation,  http://www.atcorp.com/)             
  wk: (952) 829-5864 x167     

  http://ben.burnett.ws/  

                       
  

To view this discussion on the web visit https://groups.google.com/d/msgid/emulab-admins/20200422022808.GG83711%40flux.utah.edu.

[burnettb317]

Mike,

     I haven't had any luck getting this experiment to swap in cleanly, tried several times, does not appear to have any consistency with specific nodes or specific switches or specific ports, but it does consistently have problems (some nodes either show "No-Carrier" or have no IP config).

I tried the "snmpit -r" then "snmpit -t" technique a few times, mostly nothing, but then if I do an "edit experiment" and reboot all nodes after the snmpit stuff, then it "fixed" the problems in one case but created new problems on different nodes/NICs/ports...  then another reboot of the problem nodes fixed one but not the other.

Any other ideas? I've checked the MAC addresses and compared those to what is expected from the swap logs and details (I have not gone back to the database yet on all the MACs, but it will work fine on node x/port y on one swap in and then fail on another for that same node and port...)

I'm checking other things as well (OS/Images, setup scripts, etc.)

-Ben

[burnettb317]

Mike, or anyone,

      ok, I've been doing some more experimenting and the nodes in question seem to be fine when using on original image we got from you. but we needed debian 9, so we took the debian 8 image abd upgraded it, and it had been working fine, but since the debian 8 swaps fine and so does an ubuntu, I'd say it looks like something in our customized image....  that being said, do you happen to have any newer images for download?  I'd like a debian 9 and some newer ubuntu images if possible, or if you don't have time to make them is there a way we can do a fresh install of an OS and then add the emulab magic to make our own?

Thanks,

--Ben

[Leigh Stoller]

at 11:00 PM, burnettb317 <burne...@gmail.com> wrote:

> Mike, or anyone,
> ok, I've been doing some more experimenting and the nodes in question seem to be fine when using on original image we got from you. but we needed debian 9, so we took the debian 8 image abd upgraded it, and it had been working fine, but since the debian 8 swaps fine and so does an ubuntu, I'd say it looks like something in our customized image.... that being said, do you happen to have any newer images for download? I'd like a debian 9 and some newer ubuntu images if possible, or if you don't have time to make them is there a way we can do a fresh install of an OS and then add the emulab magic to make our own?

Hi. Utah does not maintain a Debian image, sorry. The Debian 8 image we had
was provided by a third party, but we have never updated or supported it.

Leigh

[Ben Burnett]

Is there a URL or link to describe the process of creating a new Emulab image?  We are more than willing to make our own, but I've never known the process, or what needs to be added?

Ben Burnett <burne...@gmail.com>          

  R&D Software Engineer / Computer Scientist
  ATC (Architecture Technology Corporation,  http://www.atcorp.com/)             
  wk: (952) 829-5864 x167     

  http://ben.burnett.ws/  

                       
  

--
You received this message because you are subscribed to the Google Groups "emulab-admins" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emulab-admin...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emulab-admins/35B9BF04-03A9-40EA-BEEA-CAF47D33A98F%40gmail.com.

[Mike Hibler]

Can you send me the NS file (experiment description)?

> --
> You received this message because you are subscribed to the Google Groups
> "emulab-admins" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to emulab-admin...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/

> emulab-admins/99835c91-c205-4862-8acc-e0ff6f0c2bb2%40googlegroups.com.

[burnettb317]

5/1/20 update:  We have several Debian 9 images, that all seem to have the problems.  We have used those images in the past fine though.  The debian 9 image was derived from the BETA Debian 8.0 image we got from you guys.  It did swap in with the Debian 8.0 image.   so far It has worked with UBUNTU-16-STD and I'm trying 18 now..... 

5/7/20 update:  it worked with UBUNTU-18-STD last week, so we moved from debian to Ubuntu, we increased the number of nodes for the experiment, and it had been fine.  Today we got the same (or similar problem):  all the links are there except one, but only on one side of the link. modeme2 doesn't have a link to router2, but router2 does have an interface the other way, although on router2 it lists the interface as UNKNOWN instead of UP.

I ran the suggested "snmpit -r" and "snmpit -t" command you suggested last week, and the teardoen had no errors but the rebuild did:

*** reserved vlan tag 49 for vlan 47514 already in use by vlan 29614

*** ERROR: snmpit_test: Failed to create VLAN with id 47514

ERROR: VLAN 47514 not found on switch! at /usr/testbed/lib/snmpit_test/snmpit_stack.pm line 813, <DATA> line 6010.

*** Could not remove failed vlan: 47514

I tried it again and same thing...  does this mean maybe a switch is bad, it doesn't say which switch...

when we checked the experiment links, it looks like that process lost an additional link as well, so now modemd1-router1 link is gone. Both sides have an interface configured, but says state UNKNOWN instead of UP, and no connectivity

elsewhere in the output it has these lines:

% wap snmpit -r TSCANR multicoi

...

getTrunksForVlan: 47514: cisco17

...

getExperimentTrunksForVlan: 47514: cisco13

...

% wap snmpit -t TSCANR multicoi

...

getTrunksForVlan: 47514: cisco17

...

*** reserved vlan tag 49 for vlan 47514 already in use by vlan 29614

*** ERROR: snmpit_test: Failed to create VLAN with id 47514

ERROR: VLAN 47514 not found on switch! at /usr/testbed/lib/snmpit_test/snmpit_stack.pm line 813, <DA                TA> line 6010.

*** Could not remove failed vlan: 47514

...

not sure where to go from here....

-Ben

On Friday, May 1, 2020 at 9:58:41 AM UTC-5, Mike Hibler wrote:

Can you send me the NS file (experiment description)?

On Thu, Apr 30, 2020 at 03:39:34PM -0700, burnettb317 wrote:
> Mike,
>      I haven't had any luck getting this experiment to swap in cleanly, tried
> several times, does not appear to have any consistency with specific nodes or
> specific switches or specific ports, but it does consistently have problems
> (some nodes either show "No-Carrier" or have no IP config).
>
> I tried the "snmpit -r" then "snmpit -t" technique a few times, mostly nothing,
> but then if I do an "edit experiment" and reboot all nodes after the snmpit
> stuff, then it "fixed" the problems in one case but created new problems on
> different nodes/NICs/ports...  then another reboot of the problem nodes fixed
> one but not the other.
>
> Any other ideas? I've checked the MAC addresses and compared those to what is
> expected from the swap logs and details (I have not gone back to the database
> yet on all the MACs, but it will work fine on node x/port y on one swap in and
> then fail on another for that same node and port...)
>
> I'm checking other things as well (OS/Images, setup scripts, etc.)
>
> -Ben
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "emulab-admins" group.
> To unsubscribe from this group and stop receiving emails from it, send an email

> to emulab...@googlegroups.com.

[Leigh Stoller]

at 9:23 AM, burnettb317 <burne...@gmail.com> wrote:

> I ran the suggested "snmpit -r" and "snmpit -t" command you suggested last week, and the teardoen had no errors but the rebuild did:
>
> *** reserved vlan tag 49 for vlan 47514 already in use by vlan 29614
> *** ERROR: snmpit_test: Failed to create VLAN with id 47514
> ERROR: VLAN 47514 not found on switch! at /usr/testbed/lib/snmpit_test/snmpit_stack.pm line 813, <DATA> line 6010.
> *** Could not remove failed vlan: 47514

Terminate the experiment (if not already terminated) and do this:

boss> wap snmpit --impotent --prunestalevlans

Lets see what it says.

[Ben Burnett]

do I have to terminate?  can I just swap out?

Ben Burnett <burne...@gmail.com>          

  R&D Software Engineer / Computer Scientist
  ATC (Architecture Technology Corporation,  http://www.atcorp.com/)             
  wk: (952) 829-5864 x167     

  http://ben.burnett.ws/  

                       
  

--
You received this message because you are subscribed to the Google Groups "emulab-admins" group.

To unsubscribe from this group and stop receiving emails from it, send an email to emulab-admin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emulab-admins/D8E634A3-FBE1-47C8-9979-012CF70BB7CC%40gmail.com.

[Leigh Stoller]

at 10:00 AM, Ben Burnett <burne...@gmail.com> wrote:

> do I have to terminate? can I just swap out?

Swap out is fine.

[Ben Burnett]

ran it and got:

root@boss:~ # su elabman
% wap snmpit --impotent --prunestalevlans
*** WARNING: snmpit_test:
***   SNMP GET failed - In snmpit_test
***   SNMPIT get failed for device cisco14 (try 30 of 30)
***   Variable was [sysDescr,0,(undefined),(undefined)]
***   Returned (undefined), ErrorNum was -12
***   Error string is: Failure in sendto
RPC::Async::Server: Disconnecting client for error: SNMP GET failed - In snmpit_test
SNMPIT get failed for device cisco14 (try 30 of 30)
Variable was [sysDescr,0,(undefined),(undefined)]
Returned (undefined), ErrorNum was -12
Error string is: Failure in sendto


*** ERROR: snmpit_test: RPC::Async::Client: server disconnected

the problem is that cisco14 was removed long ago and replaced...... so I assume there is some remnant of a vlan from that old switch someplace that needs to get cleaned up.  where should I look?

would it be prudent to run that command nightly in a cron job?

Ben Burnett <burne...@gmail.com>          

  R&D Software Engineer / Computer Scientist
  ATC (Architecture Technology Corporation,  http://www.atcorp.com/)             
  wk: (952) 829-5864 x167     

  http://ben.burnett.ws/  

                       
  

--
You received this message because you are subscribed to the Google Groups "emulab-admins" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emulab-admin...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emulab-admins/13FFD1F4-FBCB-4E60-9C18-54A9FCF63F01%40gmail.com.

[Leigh Stoller]

at 11:10 AM, Ben Burnett <burne...@gmail.com> wrote:

> he problem is that cisco14 was removed long ago and replaced...... so I assume there is some remnant of a vlan from that old switch someplace that needs to get cleaned up. where should I look?

boss> was addswitch -r cisco14

The -r says to remove.

> would it be prudent to run that command nightly in a cron job?

No, it is more of a diagnostic tool.

Leigh

[Leigh Stoller]

at 11:24 AM, Leigh Stoller <lbst...@gmail.com> wrote:

> boss> was addswitch -r cisco14

Gack, autocorrect. That should be “wap”.

[burnettb317]

should I have addswitch?

% wap addswitch -r cisco14

env: addswitch: No such file or directory

% which addswitch

addswitch: Command not found.

root@boss:~ # find /usr/ -name addswitch

root@boss:~ #

is addswitch new?

[Leigh Stoller]

at 2:34 PM, burnettb317 <burne...@gmail.com> wrote:

> should I have addswitch?
>
> % wap addswitch -r cisco14
> env: addswitch: No such file or directory

It was added two years ago. :-)

Try this instead.

boss> wap deletenode -f -F cisco14
boss> mysql tbdb
mysql> delete from switch_stacks where node_id=‘cisco14’;

[Ben Burnett]

Thanks.  Yeah I know we need an update.

ran those, tried  wap snmpit --impotent --prunestalevlans again

su elabman

 wap snmpit --impotent --prunestalevlans

yielded same error but cisco9.  cisco9 exists and I can ping it and telnet to it, should I restart cisco9?  does this error indicate that boss can't administrate cisco9 somehow?

also ran it as root and got the same error... (before I got different errors if I ran as elabman or as root)

trying to swap again...

-Ben

Ben Burnett <burne...@gmail.com>          

  R&D Software Engineer / Computer Scientist
  ATC (Architecture Technology Corporation,  http://www.atcorp.com/)             
  wk: (952) 829-5864 x167     

  http://ben.burnett.ws/  

                       
  

--
You received this message because you are subscribed to the Google Groups "emulab-admins" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emulab-admin...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emulab-admins/7D8852C7-0495-4686-B98D-4700906DF3FD%40gmail.com.

[burnettb317]

Leigh, Mike,

   This problem is happening again, swapped in with many nodes and links and 2 don't work, in one case, both are configured correctly, in another, one side has no IPv4 configuration.  I haven't checked the switches, but my question is, is there any reason why I can't just manually configure the IP addresses and the vlans on the switches when this happens?  will it hurt anything, is there something special I would need to do to make sure boss can cleanup everything?  Where do I get the vlan #'s to configure specific links, are there boss commands to do config on a node, or should I just ssh to the node and do it?

Thanks,

--Ben

To unsubscribe from this group and stop receiving emails from it, send an email to emulab-admins+unsubscribe@googlegroups.com.

[Mike Hibler]

I may be repeating myself here, but I am not in a position to go refresh
myself on the thread right now...

When you say "no IP configuration" do you mean that the node scripts did not
configure the appropriate interface in Linux? Or are you saying that the
interfaces are configured, but traffic does not flow because the VLAN does
not exist on one or more switches?

If it was just the latter, then what does

wap snmpit -r <pid> <eid>

wap snmpit -t <pid> <eid>

show? That will remake the switch VLANs. If you are having issues with
interfaces getting configured under Linux, not sure about that. Are you
using vlan devices on the nodes (i.e., multiplexed links)? If not, then
you don't have to worry about vlans on the nodes. You also don't need to
worry about IP addresses on the switches.

Your software is old, and we have fixed assort client-side issues with newer
versions of CentOS and Ubuntu, so it is hard to say what might be wrong on
the nodes.

To answer your questions, you can manually configure the IPs on the nodes.
The worst that will happen is that you make them not work in a different
way :-) and you have to reboot the node. I would not try to manually
manipulate the switch other than to use the snmpit -r/-t commands above.

> an email to emulab-admin...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/
> msgid/emulab-admins/7D8852C7-0495-4686-B98D-4700906DF3FD%40gmail.com.
>

> --
> You received this message because you are subscribed to the Google Groups
> "emulab-admins" group.
> To unsubscribe from this group and stop receiving emails from it, send an email

> to emulab-admin...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/

> emulab-admins/9626148a-f2c7-479c-8d8e-8eeab3daa381o%40googlegroups.com.

[Ben Burnett]

I was afraid you'd say something like that about upgrading... we will
but not before July 15th. (Since we are so old, should we upgrade? or
just do a clean install, we'd like to save old/current projects and
experiments)

I tried these last night:

wap snmpit -r <pid> <eid>
wap snmpit -t <pid> <eid>

didn't fix it.

This morning I checked out the vlan for the first link from cisco8 to
cisco18 and the trunk ports, and everything checked, then I checked ping
again and the link was fine.... (not sure what "woke" it up)

The second link vlan was also good (all on one switch - cisco17), when
we fixed the IP config for the correct interface on the node that link
worked (emulab missed that one somehow)

I assume Emulab creates the hosts file on each node? is it the same file
on every node? I assume it is, does emulab also use boss as a DNS, or
just setup the hosts file? is there a way to make it more consistent?

In some cases our "routerX" nodes are assigned to the routers "host"
interface (IP with "0" in third octet), and in other cases "routerX" is
assigned to the "modem" interface (IP with "100,101,102" in third octet).

summary:
1. I think it is working and pinging through now with some manual
interventions...

2. We need to upgrade - what is the easiest method to save past projects
and experiments? do we need a 2-step (or more) update to latest version?
(see version below)

3. is there a way to easily make hosts file consistent? (other than us
editing and copying around to the nodes)


FYI version is:
buildinfo: 04/07/2016
commithash: 82a3cbbdfbf9a436f808d0013f63eb076d1b4690
dbrev: 4.497
install: 5.46
needreboot: 0
OS Version: FreeBSD 10.0-RELEASE-p18
Perl Version: 5.014004


Thanks,
--Ben

--
Ben Burnett
Principal Investigator / R&D Software Engineer / Computer Scientist
ATC (Architecture Technology Corporation, www.atcorp.com)
wk: (952) 829-5864 x167
bbur...@atcorp.com

[Mike Hibler]

We are mulling over the issue of upgrading. It will hurt however you do it!
I am not sure we have a "reinstall from scratch, put back the DB state" path.
The problem with the upgrade from FreeBSD 10.0 to 11.3 is that it involves
significant upgrades to just about every major package: perl, mysql, apache,
php, ...

To the issue at hand, you say: "we fixed the IP config for the correct
interface on the node". Did Emulab not configure any interface or did it
configure the wrong one? The latter would just indicate a mixup in the DB.

The /etc/hosts file is generated on each node and is intended only for
the experimental interfaces (non-fully-qualified names for hosts). Boss
is the DNS for resolving control net addresses (FQDNs) and their aliases.

So from a node "A" (on pc1) which has an experiment link called "link"
to node "B" (on pc2):

"pc2", "pc2.emulab.net" and "B.eid.pid.emulab.net"
would resolve to the control net address via DNS on boss.
"B" and "B-link"
would resolve to the experiment net address via /etc/hosts.

And then it gets worse. If there are multiple paths from A to B, say
there are two nodes C and D in between such that you could do A->C->B
or A->D->B, we arbitrarily pick one of B's IP addresses to be what we
call "B". This may be what you are running into.

Can you show us a hosts file of both types you mention below?

There should be aliases in the hosts file like "routerX-host" and
"routerX-modem" that unambiguously tell you which interface you are
talking to on "routerX". You should use those if you want to reach a
node via a specific interface.

> >> ?? ??This problem is happening again, swapped in with many nodes and links and 2

> >> don't work, in one case, both are configured correctly, in another, one side

> >> has no IPv4 configuration.?? I haven't checked the switches, but my question is,

> >> is there any reason why I can't just manually configure the IP addresses and

> >> the vlans on the switches when this happens??? will it hurt anything, is there

> >> something special I would need to do to make sure boss can cleanup everything?
> >> Where do I get the vlan #'s to configure specific links, are there boss
> >> commands to do config on a node, or should I just ssh to the node and do it?
> >>
> >> Thanks,
> >> --Ben
> >>
> >>
> >>
> >> On Thursday, May 7, 2020 at 4:59:15 PM UTC-5, burnettb317 wrote:
> >>

> >> Thanks.?? Yeah I know we need an update.
> >>
> >> ran those, tried?? wap snmpit --impotent --prunestalevlans??again
> >>
> >> su elabman
> >> ??wap snmpit --impotent --prunestalevlans
> >>
> >> yielded??same error but cisco9.?? cisco9 exists and I can ping it and telnet
> >> to it, should I restart cisco9??? does this error indicate that boss can't

> >> administrate cisco9 somehow?
> >>
> >> also ran it as root and got the same error... (before I got different
> >> errors if I ran as elabman or as root)
> >>
> >> trying to swap again...
> >> -Ben
> >>
> >>
> >> Ben Burnett <burne...@gmail.com>

> >> ?? R&D Software Engineer / Computer Scientist
> >> ?? ATC (Architecture Technology Corporation,?? http://www.atcorp.com/)
> >>
> >> ?? wk: (952) 829-5864 x167
> >> ????http://ben.burnett.ws/

> >>
> >>
> >>
> >>
> >> On Thu, May 7, 2020 at 4:39 PM Leigh Stoller <lbst...@gmail.com> wrote:
> >>
> >> at 2:34 PM, burnettb317 <burne...@gmail.com> wrote:
> >>
> >> > should I have addswitch?
> >> >
> >> > % wap addswitch -r cisco14
> >> > env: addswitch: No such file or directory
> >>
> >> It was added two years ago. :-)
> >>
> >> Try this instead.
> >>
> >> boss> wap deletenode -f -F cisco14
> >> boss> mysql tbdb

> >> mysql> delete from switch_stacks where node_id=???cisco14???;

> >>
> >>
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> >> Groups "emulab-admins" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> >> an email to emulab-admin...@googlegroups.com.
> >> To view this discussion on the web visit https://groups.google.com/d/
> >> msgid/emulab-admins/7D8852C7-0495-4686-B98D-4700906DF3FD%40gmail.com.
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "emulab-admins" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an email
> >> to emulab-admin...@googlegroups.com.
> >> To view this discussion on the web visit https://groups.google.com/d/msgid/
> >> emulab-admins/9626148a-f2c7-479c-8d8e-8eeab3daa381o%40googlegroups.com.
> >
>
> --
> Ben Burnett
> Principal Investigator / R&D Software Engineer / Computer Scientist
> ATC (Architecture Technology Corporation, www.atcorp.com)
> wk: (952) 829-5864 x167
> bbur...@atcorp.com
>
> --
> You received this message because you are subscribed to the Google Groups "emulab-admins" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to emulab-admin...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/emulab-admins/57153280-d3d0-5272-6259-352d7a6f7e1e%40gmail.com.