Virtual node experiment that created multiple links failed

36 views
Skip to first unread message

eric

unread,
Mar 7, 2019, 3:53:22 AM3/7/19
to emulab-admins

I was successful in the experiment of creating two virtual nodes and one link, but when I created three virtual nodes and two links experiments, only the node connecting the two links could be successfully started. The other two nodes are always blocked in the state of TBSETUP. This is the NS script,log,and the screenshot of pcvm.
I hope to get some useful suggestions, thank you!


NS script:

set ns [new Simulator]
source tb_compat.tcl

# Nodes
set node0 [$ns node]
set node1 [$ns node]
set node2 [$ns node]

tb-set-vlink-emulation vlan

tb-set-node-os $node0 UBUNTU14-64-STD
tb-set-node-os $node1 FBSD102-64-STD
tb-set-node-os $node2 CENTOS71-64-STD

tb-set-hardware $node0 pcvm
tb-set-hardware $node1 pcvm
tb-set-hardware $node2 pcvm

tb-set-node-failure-action $node0 "nonfatal"
tb-set-node-failure-action $node1 "nonfatal"
tb-set-node-failure-action $node2 "nonfatal"

set link0 [$ns duplex-link $node0 $node1 100000.0kb 0.0ms DropTail]
set link1 [$ns duplex-link $node0 $node2 100000.0kb 0.0ms DropTail]

$ns rtproto Static
$ns run


log:

Running 'tbprerun  -e 153 xen.ns'
Beginning pre run for testbed/xen. 07:53:35:119614
Running parser ... 07:53:35:299260
Parser done! 07:53:38:106349
Precomputing visualization ...
Image rendering proceeding in background mode ...
Setting up static routes (if requested) ... 
Generating topomap ...
Doing a pre-assign: '/usr/testbed/bin/vtopgen -p testbed xen' ...
Checking for feature LibVtopTest.
Minimum nodes   = 0
Maximum nodes   = 0
Virtual nodes     = 3
Writing environment strings ...
Setting up additional program agent support ...
Setting up additional network agent support ...
Writing program agent info ...
Pre run finished. 07:53:39:997825
Running 'tbswap in  testbed xen'
Beginning swap-in for testbed/xen (153). 03/07/2019 07:53:40
TIMESTAMP: 07:53:40:452393 tbswap in started
Checking for feature SyncVlans.
Checking with Admission Control ...
Checking for feature ExternalNodeManagement.
Allocating public addresses...
Mapping to physical reality ...
TIMESTAMP: 07:53:40:834374 mapper wrapper started
Checking for feature OldMapper.
Checking for feature NewAssign.
Checking for feature ControlNetVlans.
Checking for feature LibVtopTest.
Starting the new and improved mapper wrapper.
Minimum nodes   = 0
Maximum nodes   = 0
Virtual nodes     = 3
Clearing physical state before updating.
Mapper loop 1
ptopargs: '-p testbed -e xen -v '
assign command: 'assign -uod -c .75 -H 3 -P -f text testbed-xen-2787.ptop testbed-xen-2787.vtop'
Reading assign results.
pc1
TIMESTAMP: 07:53:44:706591 mapper wrapper finished
Mapped to physical reality!
Fetching tarballs and RPMs (if any) ...
TIMESTAMP: 07:53:44:711472 tarfiles_setup started
TIMESTAMP: 07:53:45:094412 tarfiles_setup finished
Creating dynamic blobs.
Locking down arp entries
TIMESTAMP: 07:53:45:163461 arplockdown started
TIMESTAMP: 07:53:45:344630 arplockdown finished
Setting up mountpoints.
TIMESTAMP: 07:53:45:346258 mountpoints started
TIMESTAMP: 07:53:47:809865 mountpoints finished
TIMESTAMP: 07:53:47:811002 named started
Setting up named maps.
TIMESTAMP: 07:53:48:168703 named finished
TIMESTAMP: 07:53:48:170786 gentopofile started
Generating ltmap (again) ...
TIMESTAMP: 07:53:48:528793 gentopofile finished
snmpit: testbed/xen has no VLANs to create, skipping
Resetting OS and rebooting.
TIMESTAMP: 07:53:49:220143 launching os_setup
Checking for feature OldOsSetup.
Setting up VLANs.
TIMESTAMP: 07:53:49:223782 snmpit started
TIMESTAMP: 07:53:49:617712 os_setup started
Adding pcvm1-1 to type object pcvm
pcvm1-1 - [OS 10015:0 emulab-ops,UBUNTU14-64-STD]
osload: Using [Image 10015:0 emulab-ops,UBUNTU14-64-STD] for pcvm1-1
osload (pcvm1-1): Changing default OS to [OS 10015:0 emulab-ops,UBUNTU14-64-STD]
Setting up reload for pcvm1-1 (mode: Frisbee)
TIMESTAMP: 07:53:49:696156 osload: database setup done
osload: Done! There were 0 failures.
Adding pcvm1-2 to type object pcvm
pcvm1-2 - [OS 10018:0 emulab-ops,FBSD102-64-STD]
osload: Using [Image 10018:0 emulab-ops,FBSD102-64-STD] for pcvm1-2
osload (pcvm1-2): Changing default OS to [OS 10018:0 emulab-ops,FBSD102-64-STD]
Setting up reload for pcvm1-2 (mode: Frisbee)
TIMESTAMP: 07:53:49:714599 osload: database setup done
osload: Done! There were 0 failures.
Adding pcvm1-3 to type object pcvm
pcvm1-3 - [OS 10016:0 emulab-ops,CENTOS71-64-STD]
osload: Using [Image 10016:0 emulab-ops,CENTOS71-64-STD] for pcvm1-3
osload (pcvm1-3): Changing default OS to [OS 10016:0 emulab-ops,CENTOS71-64-STD]
Setting up reload for pcvm1-3 (mode: Frisbee)
TIMESTAMP: 07:53:49:732624 osload: database setup done
osload: Done! There were 0 failures.
Adding pc1 to type object pc
pc1 - [OS 10021:0 emulab-ops,XEN44-64-STD]
Asking [pcvm] for volunteers
Skipping pcvm1-1 this time around
Skipping pcvm1-3 this time around
Skipping pcvm1-2 this time around
Asking [pc] for volunteers
[pc] returns volunteers: [Node: pc1]
Lighting up nodes in parallel ...
snmpit: testbed/xen has no VLANs to create, skipping
TIMESTAMP: 07:53:49:920577 snmpit finished
Setting up email lists.
TIMESTAMP: 07:53:49:926349 genelists started
TIMESTAMP: 07:53:50:167573 genelists finished
Clearing port counters.
TIMESTAMP: 07:53:50:169285 portstats started
TIMESTAMP: 07:53:50:747006 portstats finished
Lighting up nodes: [Node: pc1]
pc1 will be reloaded with [Image 10021:0 emulab-ops,XEN44-64-STD]
osload: Using [Image 10021:0 emulab-ops,XEN44-64-STD] for pc1
osload (pc1): Changing default OS to [OS 10021:0 emulab-ops,XEN44-64-STD]
Setting up reload for pc1 (mode: Frisbee)
TIMESTAMP: 07:53:50:900324 osload: database setup done
osload: Issuing reboot for pc1 and then waiting ...
TIMESTAMP: 07:53:50:902664 osload: event handler enabled
reboot: no privs; invoking real nodereboot script!
reboot: Running in pipemode (27).
BATCH: rebooting pc1
reboot (pc1): Attempting to reboot ...
pc1: in PXEWAIT, sending PXEWAKEUP
reboot (pc1): Successful!
reboot: Done. There were 0 failures.
reboot (pc1): child returned 0 status.
Waiting for pc1 to finish reloading
Thu Mar  7 07:53:51 CST 2019
TIMESTAMP: 07:53:51:232382 osload: starting reload-done wait
osload: eventhandler: pc1 => RELOADSETUP @ 1551916456
got 1 events
osload: eventhandler: pc1 => RELOADING @ 1551916461
got 1 events
osload (pc1): still waiting; it has been 1 minute(s)
osload (pc1): still waiting; it has been 2 minute(s)
osload (pc1): left reloading mode at Thu Mar  7 07:56:12 CST 2019
osload: Done! There were 0 failures.
reload (pc1): child returned 0 status.
Waiting for nodes ...
*** os_setup: Still waiting for pc1 (BOOTING) - it's been 1 minute(s).
pc1 has reported state ISUP
pc1 is alive and well
Asking [pcvm] for volunteers
pcvm1-1 will boot up with pc1
pcvm1-3 will boot up with pc1
pcvm1-2 will boot up with pc1
[pcvm] returns volunteers: [Node: pcvm1-1] [Node: pcvm1-3] [Node: pcvm1-2]
Asking [pc] for volunteers
Lighting up nodes in parallel ...
Lighting up nodes: [Node: pcvm1-1] [Node: pcvm1-3] [Node: pcvm1-2]
*** os_setup: Setting up virtual testbed nodes ...
TIMESTAMP: 07:58:13:681524 vnode_setup starting
pcvm1-1 is already setting up on local node pc1
pcvm1-3 is already setting up on local node pc1
pcvm1-2 is already setting up on local node pc1
vnode_setup running at parallelization: 10 wait_time: 3000
Vnode setup initiated on all nodes ...
TIMESTAMP: 07:58:14:988904 vnode_setup done
Waiting for nodes ...
*** os_setup: Still waiting for pcvm1-3 (RELOADING) - it's been 1 minute(s).
*** os_setup: Still waiting for pcvm1-3 (RELOADING) - it's been 2 minute(s).
*** os_setup: Still waiting for pcvm1-3 (RELOADING) - it's been 3 minute(s).
*** os_setup: Still waiting for pcvm1-3 (RELOADING) - it's been 4 minute(s).
*** os_setup: Still waiting for pcvm1-3 (RELOADING) - it's been 5 minute(s).
*** os_setup: Still waiting for pcvm1-3 (RELOADING) - it's been 6 minute(s).
pcvm1-1 has reported state ISUP
pcvm1-1 is alive and well
*** os_setup: Still waiting for pcvm1-3 (BOOTING) - it's been 7 minute(s).
*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 8 minute(s).
*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 9 minute(s).
*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 10 minute(s).
*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 11 minute(s).
*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 12 minute(s).
*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 13 minute(s).
*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 14 minute(s).
*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 15 minute(s).
*** Giving up on pcvm1-3 (TBSETUP) - it's been 15 minute(s). 
*** WARNING: os_setup: pcvm1-3 failed to boot
*** Giving up on pcvm1-2 (TBSETUP) - it's been 15 minute(s). 
*** WARNING: os_setup: pcvm1-2 failed to boot
Asking [pcvm] for volunteers
Asking [pc] for volunteers
*** ERROR: os_setup:
***   There were 2 failed nodes.
***   
***   1/1 pcvm's with a system osid of "CENTOS71-64-STD" failed to boot: (pcvm1-3)
***   1/1 pcvm's with a system osid of "FBSD102-64-STD" failed to boot: (pcvm1-2)
TIMESTAMP: 08:14:15:523928 os_setup finished
*** ERROR: tbswap: Failed to reset OS and reboot nodes.
Cleaning up after errors; will try again.
Stopping the event system
Checking for feature NewEventScheduler.
Closing TCP proxy ports...
TIMESTAMP: 08:14:17:065364 snmpit started
Removing VLANs.
snmpit: testbed/xen has no VLANs, skipping
TIMESTAMP: 08:14:18:198194 snmpit finished
Removing dynamic blobs.
Clearing shared port vlans.
Tearing down failed virtual nodes.
TIMESTAMP: 08:14:18:276280 vnode_setup -k started
pcvm1-2 failed to boot; skipping teardown.
pcvm1-3 failed to boot; skipping teardown.
vnode_setup running at parallelization: 10 wait_time: 3000
Vnode teardown finished.
TIMESTAMP: 08:14:19:570641 vnode_setup -k finished
Freeing failed nodes.
TIMESTAMP: 08:14:19:572343 nfree started
TIMESTAMP: 08:14:20:202508 nfree finished
Trying again...
Checking for feature ExternalNodeManagement.
Allocating public addresses...
Mapping to physical reality ...
TIMESTAMP: 08:14:20:692595 mapper wrapper started
Checking for feature OldMapper.
Checking for feature NewAssign.
Checking for feature ControlNetVlans.
Checking for feature LibVtopTest.
Starting the new and improved mapper wrapper.
GenFixNodes: No vnode for vhost-0 
Minimum nodes   = 0
Maximum nodes   = 0
Virtual nodes     = 3
Reserved vnodes   = 1
Reserved pnodes   = 1
Clearing physical state before updating.
Mapper loop 1
ptopargs: '-p testbed -e xen -v '
assign command: 'assign -uod -c .75 -H 3 -P -f text testbed-xen-3190.ptop testbed-xen-3190.vtop'
Reading assign results.
[Node: pc1] already reserved in holding reservation.
[Node: pcvm1-1] already reserved in holding reservation.
pc1
TIMESTAMP: 08:14:25:584052 mapper wrapper finished
Mapped to physical reality!
Fetching tarballs and RPMs (if any) ...
TIMESTAMP: 08:14:25:587581 tarfiles_setup started
TIMESTAMP: 08:14:26:040062 tarfiles_setup finished
Creating dynamic blobs.
Locking down arp entries
TIMESTAMP: 08:14:26:043943 arplockdown started
TIMESTAMP: 08:14:26:229505 arplockdown finished
Setting up mountpoints.
TIMESTAMP: 08:14:26:231241 mountpoints started
TIMESTAMP: 08:14:27:639764 mountpoints finished
TIMESTAMP: 08:14:27:640937 named started
Setting up named maps.
TIMESTAMP: 08:14:28:117419 named finished
Marking nodes for reboot.
A: pcvm1-1, RES_READY
A: pcvm1-2, RES_INIT_CLEAN
A: pcvm1-3, RES_INIT_CLEAN
A: pc1, RES_READY
TIMESTAMP: 08:14:28:123910 gentopofile started
Generating ltmap (again) ...
TIMESTAMP: 08:14:28:491811 gentopofile finished
snmpit: testbed/xen has no VLANs to create, skipping
Resetting OS and rebooting.
TIMESTAMP: 08:14:29:184672 launching os_setup
Setting up VLANs.
Checking for feature OldOsSetup.
TIMESTAMP: 08:14:29:188371 snmpit started
TIMESTAMP: 08:14:29:661714 os_setup started
Adding pcvm1-1 to type object pcvm
pcvm1-1 - [OS 10015:0 emulab-ops,UBUNTU14-64-STD]
Adding pcvm1-2 to type object pcvm
pcvm1-2 - [OS 10018:0 emulab-ops,FBSD102-64-STD]
osload: Using [Image 10018:0 emulab-ops,FBSD102-64-STD] for pcvm1-2
osload (pcvm1-2): Changing default OS to [OS 10018:0 emulab-ops,FBSD102-64-STD]
Setting up reload for pcvm1-2 (mode: Frisbee)
TIMESTAMP: 08:14:29:743537 osload: database setup done
osload: Done! There were 0 failures.
Adding pcvm1-3 to type object pcvm
pcvm1-3 - [OS 10016:0 emulab-ops,CENTOS71-64-STD]
osload: Using [Image 10016:0 emulab-ops,CENTOS71-64-STD] for pcvm1-3
osload (pcvm1-3): Changing default OS to [OS 10016:0 emulab-ops,CENTOS71-64-STD]
Setting up reload for pcvm1-3 (mode: Frisbee)
TIMESTAMP: 08:14:29:761805 osload: database setup done
osload: Done! There were 0 failures.
Adding pc1 to type object pc
pc1 - [OS 10021:0 emulab-ops,XEN44-64-STD]
Asking [pcvm] for volunteers
Skipping pcvm1-2 this time around
Skipping pcvm1-3 this time around
Skipping pcvm1-1 this time around
Asking [pc] for volunteers
[pc] returns volunteers: [Node: pc1]
Lighting up nodes in parallel ...
snmpit: testbed/xen has no VLANs to create, skipping
TIMESTAMP: 08:14:29:884517 snmpit finished
Clearing port counters.
TIMESTAMP: 08:14:29:888889 portstats started
TIMESTAMP: 08:14:30:477862 portstats finished
Lighting up nodes: [Node: pc1]
pc1 will be rebooted
reboot: no privs; invoking real nodereboot script!
reboot: Running in pipemode (23).
BATCH: rebooting pc1
reboot (pc1): Attempting to reboot ...
pc1: trying ssh reboot
pc1: ssh reboot returned 0
pc1: waiting 30s for reboot
pc1: ssh reboot failed, sending ipod
pc1: waiting 30s for ipod
pc1: rebooted
reboot (pc1): Successful!
reboot: Done. There were 0 failures.
reboot (pc1): child returned 0 status.
Waiting for nodes ...
*** os_setup: Still waiting for pc1 (BOOTING) - it's been 1 minute(s).
pc1 has reported state ISUP
pc1 is alive and well
Asking [pcvm] for volunteers
pcvm1-2 will boot up with pc1
pcvm1-3 will boot up with pc1
pcvm1-1 will boot up with pc1
[pcvm] returns volunteers: [Node: pcvm1-2] [Node: pcvm1-3] [Node: pcvm1-1]
Asking [pc] for volunteers
Lighting up nodes in parallel ...
Lighting up nodes: [Node: pcvm1-2] [Node: pcvm1-3] [Node: pcvm1-1]
*** os_setup: Setting up virtual testbed nodes ...
TIMESTAMP: 08:17:13:494438 vnode_setup starting
pcvm1-2 is already setting up on local node pc1
pcvm1-3 is already setting up on local node pc1
pcvm1-1 is already setting up on local node pc1
vnode_setup running at parallelization: 10 wait_time: 3000
Vnode setup initiated on all nodes ...
TIMESTAMP: 08:17:14:821429 vnode_setup done
Waiting for nodes ...
pcvm1-1 has reported state ISUP
pcvm1-1 is alive and well
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 1 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 2 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 3 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 4 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 5 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 6 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 7 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 8 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 9 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 10 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 11 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 12 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 13 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 14 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 15 minute(s).
*** Giving up on pcvm1-2 (TBSETUP) - it's been 15 minute(s). 
*** WARNING: os_setup: pcvm1-2 failed to boot
*** Giving up on pcvm1-3 (TBSETUP) - it's been 15 minute(s). 
*** WARNING: os_setup: pcvm1-3 failed to boot
Asking [pcvm] for volunteers
Asking [pc] for volunteers
*** ERROR: os_setup:
***   There were 2 failed nodes.
***   
***   1/1 pcvm's with a system osid of "CENTOS71-64-STD" failed to boot: (pcvm1-3)
***   1/1 pcvm's with a system osid of "FBSD102-64-STD" failed to boot: (pcvm1-2)
TIMESTAMP: 08:33:11:243502 os_setup finished
*** ERROR: tbswap: Failed to reset OS and reboot nodes.
Cleaning up after errors; will try again.
Stopping the event system
Checking for feature NewEventScheduler.
Closing TCP proxy ports...
TIMESTAMP: 08:33:12:484524 snmpit started
Removing VLANs.
snmpit: testbed/xen has no VLANs, skipping
TIMESTAMP: 08:33:13:174801 snmpit finished
Removing dynamic blobs.
Clearing shared port vlans.
Tearing down failed virtual nodes.
TIMESTAMP: 08:33:13:180791 vnode_setup -k started
pcvm1-2 failed to boot; skipping teardown.
pcvm1-3 failed to boot; skipping teardown.
vnode_setup running at parallelization: 10 wait_time: 3000
Vnode teardown finished.
TIMESTAMP: 08:33:14:467631 vnode_setup -k finished
Freeing failed nodes.
TIMESTAMP: 08:33:14:469313 nfree started
TIMESTAMP: 08:33:15:017888 nfree finished
Trying again...
Checking for feature ExternalNodeManagement.
Allocating public addresses...
Mapping to physical reality ...
TIMESTAMP: 08:33:15:392499 mapper wrapper started
Checking for feature OldMapper.
Checking for feature NewAssign.
Checking for feature ControlNetVlans.
Checking for feature LibVtopTest.
Starting the new and improved mapper wrapper.
GenFixNodes: No vnode for vhost-0 
Minimum nodes   = 0
Maximum nodes   = 0
Virtual nodes     = 3
Reserved vnodes   = 1
Reserved pnodes   = 1
Clearing physical state before updating.
Mapper loop 1
ptopargs: '-p testbed -e xen -v '
assign command: 'assign -uod -c .75 -H 3 -P -f text testbed-xen-3520.ptop testbed-xen-3520.vtop'
Reading assign results.
[Node: pcvm1-1] already reserved in holding reservation.
[Node: pc1] already reserved in holding reservation.
pc1
TIMESTAMP: 08:33:19:941841 mapper wrapper finished
Mapped to physical reality!
Fetching tarballs and RPMs (if any) ...
TIMESTAMP: 08:33:19:945413 tarfiles_setup started
TIMESTAMP: 08:33:20:327251 tarfiles_setup finished
Creating dynamic blobs.
Locking down arp entries
TIMESTAMP: 08:33:20:330817 arplockdown started
TIMESTAMP: 08:33:20:512573 arplockdown finished
Setting up mountpoints.
TIMESTAMP: 08:33:20:514164 mountpoints started
TIMESTAMP: 08:33:21:915819 mountpoints finished
TIMESTAMP: 08:33:21:916999 named started
Setting up named maps.
TIMESTAMP: 08:33:22:276238 named finished
Marking nodes for reboot.
A: pcvm1-1, RES_READY
A: pcvm1-2, RES_INIT_CLEAN
A: pcvm1-3, RES_INIT_CLEAN
A: pc1, RES_READY
TIMESTAMP: 08:33:22:283133 gentopofile started
Generating ltmap (again) ...
TIMESTAMP: 08:33:22:643382 gentopofile finished
snmpit: testbed/xen has no VLANs to create, skipping
Resetting OS and rebooting.
TIMESTAMP: 08:33:23:335037 launching os_setup
Setting up VLANs.
Checking for feature OldOsSetup.
TIMESTAMP: 08:33:23:338749 snmpit started
TIMESTAMP: 08:33:23:733530 os_setup started
Adding pcvm1-1 to type object pcvm
pcvm1-1 - [OS 10015:0 emulab-ops,UBUNTU14-64-STD]
Adding pcvm1-2 to type object pcvm
pcvm1-2 - [OS 10018:0 emulab-ops,FBSD102-64-STD]
osload: Using [Image 10018:0 emulab-ops,FBSD102-64-STD] for pcvm1-2
osload (pcvm1-2): Changing default OS to [OS 10018:0 emulab-ops,FBSD102-64-STD]
Setting up reload for pcvm1-2 (mode: Frisbee)
TIMESTAMP: 08:33:23:815986 osload: database setup done
osload: Done! There were 0 failures.
Adding pcvm1-3 to type object pcvm
pcvm1-3 - [OS 10016:0 emulab-ops,CENTOS71-64-STD]
osload: Using [Image 10016:0 emulab-ops,CENTOS71-64-STD] for pcvm1-3
osload (pcvm1-3): Changing default OS to [OS 10016:0 emulab-ops,CENTOS71-64-STD]
Setting up reload for pcvm1-3 (mode: Frisbee)
TIMESTAMP: 08:33:23:834152 osload: database setup done
osload: Done! There were 0 failures.
Adding pc1 to type object pc
pc1 - [OS 10021:0 emulab-ops,XEN44-64-STD]
Asking [pc] for volunteers
[pc] returns volunteers: [Node: pc1]
Asking [pcvm] for volunteers
Skipping pcvm1-1 this time around
Skipping pcvm1-3 this time around
Skipping pcvm1-2 this time around
Lighting up nodes in parallel ...
snmpit: testbed/xen has no VLANs to create, skipping
TIMESTAMP: 08:33:24:034685 snmpit finished
Clearing port counters.
TIMESTAMP: 08:33:24:038983 portstats started
TIMESTAMP: 08:33:24:617263 portstats finished
Lighting up nodes: [Node: pc1]
pc1 will be rebooted
reboot: no privs; invoking real nodereboot script!
reboot: Running in pipemode (23).
BATCH: rebooting pc1
reboot (pc1): Attempting to reboot ...
pc1: trying ssh reboot
pc1: ssh reboot returned 0
pc1: waiting 30s for reboot
pc1: ssh reboot failed, sending ipod
pc1: waiting 30s for ipod
pc1: rebooted
reboot (pc1): Successful!
reboot: Done. There were 0 failures.
reboot (pc1): child returned 0 status.
Waiting for nodes ...
*** os_setup: Still waiting for pc1 (BOOTING) - it's been 1 minute(s).
pc1 has reported state ISUP
pc1 is alive and well
Asking [pc] for volunteers
Asking [pcvm] for volunteers
pcvm1-1 will boot up with pc1
pcvm1-3 will boot up with pc1
pcvm1-2 will boot up with pc1
[pcvm] returns volunteers: [Node: pcvm1-1] [Node: pcvm1-3] [Node: pcvm1-2]
Lighting up nodes in parallel ...
Lighting up nodes: [Node: pcvm1-1] [Node: pcvm1-3] [Node: pcvm1-2]
*** os_setup: Setting up virtual testbed nodes ...
TIMESTAMP: 08:35:56:295760 vnode_setup starting
pcvm1-1 is already setting up on local node pc1
pcvm1-3 is already setting up on local node pc1
pcvm1-2 is already setting up on local node pc1
vnode_setup running at parallelization: 10 wait_time: 3000
Vnode setup initiated on all nodes ...
TIMESTAMP: 08:35:57:642580 vnode_setup done
Waiting for nodes ...
pcvm1-1 has reported state TBFAILED
*** WARNING: os_setup: pcvm1-1 reported a TBFAILED event
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 1 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 2 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 3 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 4 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 5 minute(s).
*** os_setup: Swap canceled; will terminate os_setup early!
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 6 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 7 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 8 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 9 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 10 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 11 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 12 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 13 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 14 minute(s).
*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 15 minute(s).
*** Giving up on pcvm1-2 (TBSETUP) - it's been 15 minute(s). 
*** WARNING: os_setup: pcvm1-2 failed to boot
*** Giving up on pcvm1-3 (TBSETUP) - it's been 15 minute(s). 
*** WARNING: os_setup: pcvm1-3 failed to boot
Asking [pc] for volunteers
Asking [pcvm] for volunteers
*** ERROR: os_setup:
***   There were 3 failed nodes.
***   
***   1/1 pcvm's with a system osid of "UBUNTU14-64-STD" failed to boot: (pcvm1-1)
***   1/1 pcvm's with a system osid of "CENTOS71-64-STD" failed to boot: (pcvm1-3)
***   1/1 pcvm's with a system osid of "FBSD102-64-STD" failed to boot: (pcvm1-2)
TIMESTAMP: 08:51:54:246438 os_setup finished
*** ERROR: tbswap: Failed to reset OS and reboot nodes.
Not retrying due to error type.
Cleaning up after errors.
Stopping the event system
Checking for feature NewEventScheduler.
Closing TCP proxy ports...
Releasing public address pools...
TIMESTAMP: 08:51:55:493509 snmpit started
Removing VLANs.
snmpit: testbed/xen has no VLANs, skipping
TIMESTAMP: 08:51:56:188569 snmpit finished
Removing dynamic blobs.
Clearing shared port vlans.
Tearing down virtual nodes.
TIMESTAMP: 08:51:56:194398 vnode_setup -k started
pcvm1-1 failed to boot; skipping teardown.
pcvm1-2 failed to boot; skipping teardown.
pcvm1-3 failed to boot; skipping teardown.
vnode_setup running at parallelization: 10 wait_time: 3000
Vnode teardown finished.
TIMESTAMP: 08:51:57:536644 vnode_setup finished
Removing logical wires.
Freeing nodes.
TIMESTAMP: 08:51:57:539518 nfree started
Releasing all nodes from experiment [Experiment: testbed/xen].
Moving [Node: pc1] to [Experiment: emulab-ops/reloadpending]
TIMESTAMP: 08:51:58:137337 nfree finished
Resetting DB.
*** ERROR: tbswap: Experiment swap-in canceled by user.
Failingly finished swap-in for testbed/xen. 08:51:58:152221
TIMESTAMP: 08:51:58:152764 tbswap in finished (failed)
*** ERROR: batchexp: tbswap in failed!
Cleaning up and exiting with status 1 ... 
Running 'tbend -f -e 153'
Beginning cleanup for testbed/xen. 08:51:58:549682
Clearing out virtual state.
Removing visualization data...
Cleanup finished! 08:51:58:836379
**** Experimental information, please ignore ****
Session ID = 52283
Likely Cause of the Problem:
  Experiment swap-in canceled by user.
Cause: canceled
Confidence: 0.7
Script: tbswap

Possible Error Before Cancellation:
  There were 2 failed nodes.
  
  1/1 pcvm's with a system osid of "CENTOS71-64-STD" failed to boot: (pcvm1-3)
  1/1 pcvm's with a system osid of "FBSD102-64-STD" failed to boot: (pcvm1-2)
Cause: unknown
Confidence: 0.7
Script: os_setup
**** End experimental information ****
SENDMAIL: Could not finish sendmail: 
Removing experiment directories ... 


pcvm.JPG



Leigh Stoller

unread,
Mar 7, 2019, 10:53:29 AM3/7/19
to emulab...@googlegroups.com
at 12:53 AM, eric <eric147...@gmail.com> wrote:

> I was successful in the experiment of creating two virtual nodes and one link, but when I created three virtual nodes and two links experiments, only the node connecting the two links could be successfully started. The other two nodes are always blocked in the state of TBSETUP. This is the NS script,log,and the screenshot of pcvm.
> I hope to get some useful suggestions, thank you!

Hi. Can you please run this on boss and send us the output:

boss> testbed-version

Leigh

eric

unread,
Mar 7, 2019, 8:33:25 PM3/7/19
to emulab-admins
This is the output of testbed-version.

testbed-version.jpg

eric

unread,
Mar 12, 2019, 5:09:13 AM3/12/19
to emulab-admins


The node is always in this state (tbsetup1) when I experiment. The boss server can ping three virtual nodes, but only node0 can ping the boss, and the three virtual nodes can communicate with each other. Boot_log is part of the log on node1 (/var/log/boot.log)

tbsetup1.JPG


boot_log.jpg

Reply all
Reply to author
Forward
0 new messages