Virtual node experiment that created multiple links failed

[eric]

I was successful in the experiment of creating two virtual nodes and one link, but when I created three virtual nodes and two links experiments, only the node connecting the two links could be successfully started. The other two nodes are always blocked in the state of TBSETUP. This is the NS script,log,and the screenshot of pcvm.

I hope to get some useful suggestions, thank you!

NS script:

set ns [new Simulator]

source tb_compat.tcl

# Nodes

set node0 [$ns node]

set node1 [$ns node]

set node2 [$ns node]

tb-set-vlink-emulation vlan

tb-set-node-os $node0 UBUNTU14-64-STD

tb-set-node-os $node1 FBSD102-64-STD

tb-set-node-os $node2 CENTOS71-64-STD

tb-set-hardware $node0 pcvm

tb-set-hardware $node1 pcvm

tb-set-hardware $node2 pcvm

tb-set-node-failure-action $node0 "nonfatal"

tb-set-node-failure-action $node1 "nonfatal"

tb-set-node-failure-action $node2 "nonfatal"

set link0 [$ns duplex-link $node0 $node1 100000.0kb 0.0ms DropTail]

set link1 [$ns duplex-link $node0 $node2 100000.0kb 0.0ms DropTail]

$ns rtproto Static

$ns run

log:

Running 'tbprerun  -e 153 xen.ns'

Beginning pre run for testbed/xen. 07:53:35:119614

Running parser ... 07:53:35:299260

Parser done! 07:53:38:106349

Precomputing visualization ...

Image rendering proceeding in background mode ...

Setting up static routes (if requested) ... 

Generating topomap ...

Doing a pre-assign: '/usr/testbed/bin/vtopgen -p testbed xen' ...

Checking for feature LibVtopTest.

Minimum nodes   = 0

Maximum nodes   = 0

Virtual nodes     = 3

Writing environment strings ...

Setting up additional program agent support ...

Setting up additional network agent support ...

Writing program agent info ...

Pre run finished. 07:53:39:997825

Running 'tbswap in  testbed xen'

Beginning swap-in for testbed/xen (153). 03/07/2019 07:53:40

TIMESTAMP: 07:53:40:452393 tbswap in started

Checking for feature SyncVlans.

Checking with Admission Control ...

Checking for feature ExternalNodeManagement.

Allocating public addresses...

Mapping to physical reality ...

TIMESTAMP: 07:53:40:834374 mapper wrapper started

Checking for feature OldMapper.

Checking for feature NewAssign.

Checking for feature ControlNetVlans.

Checking for feature LibVtopTest.

Starting the new and improved mapper wrapper.

Minimum nodes   = 0

Maximum nodes   = 0

Virtual nodes     = 3

Clearing physical state before updating.

Mapper loop 1

ptopargs: '-p testbed -e xen -v '

assign command: 'assign -uod -c .75 -H 3 -P -f text testbed-xen-2787.ptop testbed-xen-2787.vtop'

Reading assign results.

pc1

TIMESTAMP: 07:53:44:706591 mapper wrapper finished

Mapped to physical reality!

Fetching tarballs and RPMs (if any) ...

TIMESTAMP: 07:53:44:711472 tarfiles_setup started

TIMESTAMP: 07:53:45:094412 tarfiles_setup finished

Creating dynamic blobs.

Locking down arp entries

TIMESTAMP: 07:53:45:163461 arplockdown started

TIMESTAMP: 07:53:45:344630 arplockdown finished

Setting up mountpoints.

TIMESTAMP: 07:53:45:346258 mountpoints started

TIMESTAMP: 07:53:47:809865 mountpoints finished

TIMESTAMP: 07:53:47:811002 named started

Setting up named maps.

TIMESTAMP: 07:53:48:168703 named finished

TIMESTAMP: 07:53:48:170786 gentopofile started

Generating ltmap (again) ...

TIMESTAMP: 07:53:48:528793 gentopofile finished

snmpit: testbed/xen has no VLANs to create, skipping

Resetting OS and rebooting.

TIMESTAMP: 07:53:49:220143 launching os_setup

Checking for feature OldOsSetup.

Setting up VLANs.

TIMESTAMP: 07:53:49:223782 snmpit started

TIMESTAMP: 07:53:49:617712 os_setup started

Adding pcvm1-1 to type object pcvm

pcvm1-1 - [OS 10015:0 emulab-ops,UBUNTU14-64-STD]

osload: Using [Image 10015:0 emulab-ops,UBUNTU14-64-STD] for pcvm1-1

osload (pcvm1-1): Changing default OS to [OS 10015:0 emulab-ops,UBUNTU14-64-STD]

Setting up reload for pcvm1-1 (mode: Frisbee)

TIMESTAMP: 07:53:49:696156 osload: database setup done

osload: Done! There were 0 failures.

Adding pcvm1-2 to type object pcvm

pcvm1-2 - [OS 10018:0 emulab-ops,FBSD102-64-STD]

osload: Using [Image 10018:0 emulab-ops,FBSD102-64-STD] for pcvm1-2

osload (pcvm1-2): Changing default OS to [OS 10018:0 emulab-ops,FBSD102-64-STD]

Setting up reload for pcvm1-2 (mode: Frisbee)

TIMESTAMP: 07:53:49:714599 osload: database setup done

osload: Done! There were 0 failures.

Adding pcvm1-3 to type object pcvm

pcvm1-3 - [OS 10016:0 emulab-ops,CENTOS71-64-STD]

osload: Using [Image 10016:0 emulab-ops,CENTOS71-64-STD] for pcvm1-3

osload (pcvm1-3): Changing default OS to [OS 10016:0 emulab-ops,CENTOS71-64-STD]

Setting up reload for pcvm1-3 (mode: Frisbee)

TIMESTAMP: 07:53:49:732624 osload: database setup done

osload: Done! There were 0 failures.

Adding pc1 to type object pc

pc1 - [OS 10021:0 emulab-ops,XEN44-64-STD]

Asking [pcvm] for volunteers

Skipping pcvm1-1 this time around

Skipping pcvm1-3 this time around

Skipping pcvm1-2 this time around

Asking [pc] for volunteers

[pc] returns volunteers: [Node: pc1]

Lighting up nodes in parallel ...

snmpit: testbed/xen has no VLANs to create, skipping

TIMESTAMP: 07:53:49:920577 snmpit finished

Setting up email lists.

TIMESTAMP: 07:53:49:926349 genelists started

TIMESTAMP: 07:53:50:167573 genelists finished

Clearing port counters.

TIMESTAMP: 07:53:50:169285 portstats started

TIMESTAMP: 07:53:50:747006 portstats finished

Lighting up nodes: [Node: pc1]

pc1 will be reloaded with [Image 10021:0 emulab-ops,XEN44-64-STD]

osload: Using [Image 10021:0 emulab-ops,XEN44-64-STD] for pc1

osload (pc1): Changing default OS to [OS 10021:0 emulab-ops,XEN44-64-STD]

Setting up reload for pc1 (mode: Frisbee)

TIMESTAMP: 07:53:50:900324 osload: database setup done

osload: Issuing reboot for pc1 and then waiting ...

TIMESTAMP: 07:53:50:902664 osload: event handler enabled

reboot: no privs; invoking real nodereboot script!

reboot: Running in pipemode (27).

BATCH: rebooting pc1

reboot (pc1): Attempting to reboot ...

pc1: in PXEWAIT, sending PXEWAKEUP

reboot (pc1): Successful!

reboot: Done. There were 0 failures.

reboot (pc1): child returned 0 status.

Waiting for pc1 to finish reloading

Thu Mar  7 07:53:51 CST 2019

TIMESTAMP: 07:53:51:232382 osload: starting reload-done wait

osload: eventhandler: pc1 => RELOADSETUP @ 1551916456

got 1 events

osload: eventhandler: pc1 => RELOADING @ 1551916461

got 1 events

osload (pc1): still waiting; it has been 1 minute(s)

osload (pc1): still waiting; it has been 2 minute(s)

osload (pc1): left reloading mode at Thu Mar  7 07:56:12 CST 2019

osload: Done! There were 0 failures.

reload (pc1): child returned 0 status.

Waiting for nodes ...

*** os_setup: Still waiting for pc1 (BOOTING) - it's been 1 minute(s).

pc1 has reported state ISUP

pc1 is alive and well

Asking [pcvm] for volunteers

pcvm1-1 will boot up with pc1

pcvm1-3 will boot up with pc1

pcvm1-2 will boot up with pc1

[pcvm] returns volunteers: [Node: pcvm1-1] [Node: pcvm1-3] [Node: pcvm1-2]

Asking [pc] for volunteers

Lighting up nodes in parallel ...

Lighting up nodes: [Node: pcvm1-1] [Node: pcvm1-3] [Node: pcvm1-2]

*** os_setup: Setting up virtual testbed nodes ...

TIMESTAMP: 07:58:13:681524 vnode_setup starting

pcvm1-1 is already setting up on local node pc1

pcvm1-3 is already setting up on local node pc1

pcvm1-2 is already setting up on local node pc1

vnode_setup running at parallelization: 10 wait_time: 3000

Vnode setup initiated on all nodes ...

TIMESTAMP: 07:58:14:988904 vnode_setup done

Waiting for nodes ...

*** os_setup: Still waiting for pcvm1-3 (RELOADING) - it's been 1 minute(s).

*** os_setup: Still waiting for pcvm1-3 (RELOADING) - it's been 2 minute(s).

*** os_setup: Still waiting for pcvm1-3 (RELOADING) - it's been 3 minute(s).

*** os_setup: Still waiting for pcvm1-3 (RELOADING) - it's been 4 minute(s).

*** os_setup: Still waiting for pcvm1-3 (RELOADING) - it's been 5 minute(s).

*** os_setup: Still waiting for pcvm1-3 (RELOADING) - it's been 6 minute(s).

pcvm1-1 has reported state ISUP

pcvm1-1 is alive and well

*** os_setup: Still waiting for pcvm1-3 (BOOTING) - it's been 7 minute(s).

*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 8 minute(s).

*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 9 minute(s).

*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 10 minute(s).

*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 11 minute(s).

*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 12 minute(s).

*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 13 minute(s).

*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 14 minute(s).

*** os_setup: Still waiting for pcvm1-3 (TBSETUP) - it's been 15 minute(s).

*** Giving up on pcvm1-3 (TBSETUP) - it's been 15 minute(s). 

*** WARNING: os_setup: pcvm1-3 failed to boot

*** Giving up on pcvm1-2 (TBSETUP) - it's been 15 minute(s). 

*** WARNING: os_setup: pcvm1-2 failed to boot

Asking [pcvm] for volunteers

Asking [pc] for volunteers

*** ERROR: os_setup:

***   There were 2 failed nodes.

***   

***   1/1 pcvm's with a system osid of "CENTOS71-64-STD" failed to boot: (pcvm1-3)

***   1/1 pcvm's with a system osid of "FBSD102-64-STD" failed to boot: (pcvm1-2)

TIMESTAMP: 08:14:15:523928 os_setup finished

*** ERROR: tbswap: Failed to reset OS and reboot nodes.

Cleaning up after errors; will try again.

Stopping the event system

Checking for feature NewEventScheduler.

Closing TCP proxy ports...

TIMESTAMP: 08:14:17:065364 snmpit started

Removing VLANs.

snmpit: testbed/xen has no VLANs, skipping

TIMESTAMP: 08:14:18:198194 snmpit finished

Removing dynamic blobs.

Clearing shared port vlans.

Tearing down failed virtual nodes.

TIMESTAMP: 08:14:18:276280 vnode_setup -k started

pcvm1-2 failed to boot; skipping teardown.

pcvm1-3 failed to boot; skipping teardown.

vnode_setup running at parallelization: 10 wait_time: 3000

Vnode teardown finished.

TIMESTAMP: 08:14:19:570641 vnode_setup -k finished

Freeing failed nodes.

TIMESTAMP: 08:14:19:572343 nfree started

TIMESTAMP: 08:14:20:202508 nfree finished

Trying again...

Checking for feature ExternalNodeManagement.

Allocating public addresses...

Mapping to physical reality ...

TIMESTAMP: 08:14:20:692595 mapper wrapper started

Checking for feature OldMapper.

Checking for feature NewAssign.

Checking for feature ControlNetVlans.

Checking for feature LibVtopTest.

Starting the new and improved mapper wrapper.

GenFixNodes: No vnode for vhost-0 

Minimum nodes   = 0

Maximum nodes   = 0

Virtual nodes     = 3

Reserved vnodes   = 1

Reserved pnodes   = 1

Clearing physical state before updating.

Mapper loop 1

ptopargs: '-p testbed -e xen -v '

assign command: 'assign -uod -c .75 -H 3 -P -f text testbed-xen-3190.ptop testbed-xen-3190.vtop'

Reading assign results.

[Node: pc1] already reserved in holding reservation.

[Node: pcvm1-1] already reserved in holding reservation.

pc1

TIMESTAMP: 08:14:25:584052 mapper wrapper finished

Mapped to physical reality!

Fetching tarballs and RPMs (if any) ...

TIMESTAMP: 08:14:25:587581 tarfiles_setup started

TIMESTAMP: 08:14:26:040062 tarfiles_setup finished

Creating dynamic blobs.

Locking down arp entries

TIMESTAMP: 08:14:26:043943 arplockdown started

TIMESTAMP: 08:14:26:229505 arplockdown finished

Setting up mountpoints.

TIMESTAMP: 08:14:26:231241 mountpoints started

TIMESTAMP: 08:14:27:639764 mountpoints finished

TIMESTAMP: 08:14:27:640937 named started

Setting up named maps.

TIMESTAMP: 08:14:28:117419 named finished

Marking nodes for reboot.

A: pcvm1-1, RES_READY

A: pcvm1-2, RES_INIT_CLEAN

A: pcvm1-3, RES_INIT_CLEAN

A: pc1, RES_READY

TIMESTAMP: 08:14:28:123910 gentopofile started

Generating ltmap (again) ...

TIMESTAMP: 08:14:28:491811 gentopofile finished

snmpit: testbed/xen has no VLANs to create, skipping

Resetting OS and rebooting.

TIMESTAMP: 08:14:29:184672 launching os_setup

Setting up VLANs.

Checking for feature OldOsSetup.

TIMESTAMP: 08:14:29:188371 snmpit started

TIMESTAMP: 08:14:29:661714 os_setup started

Adding pcvm1-1 to type object pcvm

pcvm1-1 - [OS 10015:0 emulab-ops,UBUNTU14-64-STD]

Adding pcvm1-2 to type object pcvm

pcvm1-2 - [OS 10018:0 emulab-ops,FBSD102-64-STD]

osload: Using [Image 10018:0 emulab-ops,FBSD102-64-STD] for pcvm1-2

osload (pcvm1-2): Changing default OS to [OS 10018:0 emulab-ops,FBSD102-64-STD]

Setting up reload for pcvm1-2 (mode: Frisbee)

TIMESTAMP: 08:14:29:743537 osload: database setup done

osload: Done! There were 0 failures.

Adding pcvm1-3 to type object pcvm

pcvm1-3 - [OS 10016:0 emulab-ops,CENTOS71-64-STD]

osload: Using [Image 10016:0 emulab-ops,CENTOS71-64-STD] for pcvm1-3

osload (pcvm1-3): Changing default OS to [OS 10016:0 emulab-ops,CENTOS71-64-STD]

Setting up reload for pcvm1-3 (mode: Frisbee)

TIMESTAMP: 08:14:29:761805 osload: database setup done

osload: Done! There were 0 failures.

Adding pc1 to type object pc

pc1 - [OS 10021:0 emulab-ops,XEN44-64-STD]

Asking [pcvm] for volunteers

Skipping pcvm1-2 this time around

Skipping pcvm1-3 this time around

Skipping pcvm1-1 this time around

Asking [pc] for volunteers

[pc] returns volunteers: [Node: pc1]

Lighting up nodes in parallel ...

snmpit: testbed/xen has no VLANs to create, skipping

TIMESTAMP: 08:14:29:884517 snmpit finished

Clearing port counters.

TIMESTAMP: 08:14:29:888889 portstats started

TIMESTAMP: 08:14:30:477862 portstats finished

Lighting up nodes: [Node: pc1]

pc1 will be rebooted

reboot: no privs; invoking real nodereboot script!

reboot: Running in pipemode (23).

BATCH: rebooting pc1

reboot (pc1): Attempting to reboot ...

pc1: trying ssh reboot

pc1: ssh reboot returned 0

pc1: waiting 30s for reboot

pc1: ssh reboot failed, sending ipod

pc1: waiting 30s for ipod

pc1: rebooted

reboot (pc1): Successful!

reboot: Done. There were 0 failures.

reboot (pc1): child returned 0 status.

Waiting for nodes ...

*** os_setup: Still waiting for pc1 (BOOTING) - it's been 1 minute(s).

pc1 has reported state ISUP

pc1 is alive and well

Asking [pcvm] for volunteers

pcvm1-2 will boot up with pc1

pcvm1-3 will boot up with pc1

pcvm1-1 will boot up with pc1

[pcvm] returns volunteers: [Node: pcvm1-2] [Node: pcvm1-3] [Node: pcvm1-1]

Asking [pc] for volunteers

Lighting up nodes in parallel ...

Lighting up nodes: [Node: pcvm1-2] [Node: pcvm1-3] [Node: pcvm1-1]

*** os_setup: Setting up virtual testbed nodes ...

TIMESTAMP: 08:17:13:494438 vnode_setup starting

pcvm1-2 is already setting up on local node pc1

pcvm1-3 is already setting up on local node pc1

pcvm1-1 is already setting up on local node pc1

vnode_setup running at parallelization: 10 wait_time: 3000

Vnode setup initiated on all nodes ...

TIMESTAMP: 08:17:14:821429 vnode_setup done

Waiting for nodes ...

pcvm1-1 has reported state ISUP

pcvm1-1 is alive and well

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 1 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 2 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 3 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 4 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 5 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 6 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 7 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 8 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 9 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 10 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 11 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 12 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 13 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 14 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 15 minute(s).

*** Giving up on pcvm1-2 (TBSETUP) - it's been 15 minute(s). 

*** WARNING: os_setup: pcvm1-2 failed to boot

*** Giving up on pcvm1-3 (TBSETUP) - it's been 15 minute(s). 

*** WARNING: os_setup: pcvm1-3 failed to boot

Asking [pcvm] for volunteers

Asking [pc] for volunteers

*** ERROR: os_setup:

***   There were 2 failed nodes.

***   

***   1/1 pcvm's with a system osid of "CENTOS71-64-STD" failed to boot: (pcvm1-3)

***   1/1 pcvm's with a system osid of "FBSD102-64-STD" failed to boot: (pcvm1-2)

TIMESTAMP: 08:33:11:243502 os_setup finished

*** ERROR: tbswap: Failed to reset OS and reboot nodes.

Cleaning up after errors; will try again.

Stopping the event system

Checking for feature NewEventScheduler.

Closing TCP proxy ports...

TIMESTAMP: 08:33:12:484524 snmpit started

Removing VLANs.

snmpit: testbed/xen has no VLANs, skipping

TIMESTAMP: 08:33:13:174801 snmpit finished

Removing dynamic blobs.

Clearing shared port vlans.

Tearing down failed virtual nodes.

TIMESTAMP: 08:33:13:180791 vnode_setup -k started

pcvm1-2 failed to boot; skipping teardown.

pcvm1-3 failed to boot; skipping teardown.

vnode_setup running at parallelization: 10 wait_time: 3000

Vnode teardown finished.

TIMESTAMP: 08:33:14:467631 vnode_setup -k finished

Freeing failed nodes.

TIMESTAMP: 08:33:14:469313 nfree started

TIMESTAMP: 08:33:15:017888 nfree finished

Trying again...

Checking for feature ExternalNodeManagement.

Allocating public addresses...

Mapping to physical reality ...

TIMESTAMP: 08:33:15:392499 mapper wrapper started

Checking for feature OldMapper.

Checking for feature NewAssign.

Checking for feature ControlNetVlans.

Checking for feature LibVtopTest.

Starting the new and improved mapper wrapper.

GenFixNodes: No vnode for vhost-0 

Minimum nodes   = 0

Maximum nodes   = 0

Virtual nodes     = 3

Reserved vnodes   = 1

Reserved pnodes   = 1

Clearing physical state before updating.

Mapper loop 1

ptopargs: '-p testbed -e xen -v '

assign command: 'assign -uod -c .75 -H 3 -P -f text testbed-xen-3520.ptop testbed-xen-3520.vtop'

Reading assign results.

[Node: pcvm1-1] already reserved in holding reservation.

[Node: pc1] already reserved in holding reservation.

pc1

TIMESTAMP: 08:33:19:941841 mapper wrapper finished

Mapped to physical reality!

Fetching tarballs and RPMs (if any) ...

TIMESTAMP: 08:33:19:945413 tarfiles_setup started

TIMESTAMP: 08:33:20:327251 tarfiles_setup finished

Creating dynamic blobs.

Locking down arp entries

TIMESTAMP: 08:33:20:330817 arplockdown started

TIMESTAMP: 08:33:20:512573 arplockdown finished

Setting up mountpoints.

TIMESTAMP: 08:33:20:514164 mountpoints started

TIMESTAMP: 08:33:21:915819 mountpoints finished

TIMESTAMP: 08:33:21:916999 named started

Setting up named maps.

TIMESTAMP: 08:33:22:276238 named finished

Marking nodes for reboot.

A: pcvm1-1, RES_READY

A: pcvm1-2, RES_INIT_CLEAN

A: pcvm1-3, RES_INIT_CLEAN

A: pc1, RES_READY

TIMESTAMP: 08:33:22:283133 gentopofile started

Generating ltmap (again) ...

TIMESTAMP: 08:33:22:643382 gentopofile finished

snmpit: testbed/xen has no VLANs to create, skipping

Resetting OS and rebooting.

TIMESTAMP: 08:33:23:335037 launching os_setup

Setting up VLANs.

Checking for feature OldOsSetup.

TIMESTAMP: 08:33:23:338749 snmpit started

TIMESTAMP: 08:33:23:733530 os_setup started

Adding pcvm1-1 to type object pcvm

pcvm1-1 - [OS 10015:0 emulab-ops,UBUNTU14-64-STD]

Adding pcvm1-2 to type object pcvm

pcvm1-2 - [OS 10018:0 emulab-ops,FBSD102-64-STD]

osload: Using [Image 10018:0 emulab-ops,FBSD102-64-STD] for pcvm1-2

osload (pcvm1-2): Changing default OS to [OS 10018:0 emulab-ops,FBSD102-64-STD]

Setting up reload for pcvm1-2 (mode: Frisbee)

TIMESTAMP: 08:33:23:815986 osload: database setup done

osload: Done! There were 0 failures.

Adding pcvm1-3 to type object pcvm

pcvm1-3 - [OS 10016:0 emulab-ops,CENTOS71-64-STD]

osload: Using [Image 10016:0 emulab-ops,CENTOS71-64-STD] for pcvm1-3

osload (pcvm1-3): Changing default OS to [OS 10016:0 emulab-ops,CENTOS71-64-STD]

Setting up reload for pcvm1-3 (mode: Frisbee)

TIMESTAMP: 08:33:23:834152 osload: database setup done

osload: Done! There were 0 failures.

Adding pc1 to type object pc

pc1 - [OS 10021:0 emulab-ops,XEN44-64-STD]

Asking [pc] for volunteers

[pc] returns volunteers: [Node: pc1]

Asking [pcvm] for volunteers

Skipping pcvm1-1 this time around

Skipping pcvm1-3 this time around

Skipping pcvm1-2 this time around

Lighting up nodes in parallel ...

snmpit: testbed/xen has no VLANs to create, skipping

TIMESTAMP: 08:33:24:034685 snmpit finished

Clearing port counters.

TIMESTAMP: 08:33:24:038983 portstats started

TIMESTAMP: 08:33:24:617263 portstats finished

Lighting up nodes: [Node: pc1]

pc1 will be rebooted

reboot: no privs; invoking real nodereboot script!

reboot: Running in pipemode (23).

BATCH: rebooting pc1

reboot (pc1): Attempting to reboot ...

pc1: trying ssh reboot

pc1: ssh reboot returned 0

pc1: waiting 30s for reboot

pc1: ssh reboot failed, sending ipod

pc1: waiting 30s for ipod

pc1: rebooted

reboot (pc1): Successful!

reboot: Done. There were 0 failures.

reboot (pc1): child returned 0 status.

Waiting for nodes ...

*** os_setup: Still waiting for pc1 (BOOTING) - it's been 1 minute(s).

pc1 has reported state ISUP

pc1 is alive and well

Asking [pc] for volunteers

Asking [pcvm] for volunteers

pcvm1-1 will boot up with pc1

pcvm1-3 will boot up with pc1

pcvm1-2 will boot up with pc1

[pcvm] returns volunteers: [Node: pcvm1-1] [Node: pcvm1-3] [Node: pcvm1-2]

Lighting up nodes in parallel ...

Lighting up nodes: [Node: pcvm1-1] [Node: pcvm1-3] [Node: pcvm1-2]

*** os_setup: Setting up virtual testbed nodes ...

TIMESTAMP: 08:35:56:295760 vnode_setup starting

pcvm1-1 is already setting up on local node pc1

pcvm1-3 is already setting up on local node pc1

pcvm1-2 is already setting up on local node pc1

vnode_setup running at parallelization: 10 wait_time: 3000

Vnode setup initiated on all nodes ...

TIMESTAMP: 08:35:57:642580 vnode_setup done

Waiting for nodes ...

pcvm1-1 has reported state TBFAILED

*** WARNING: os_setup: pcvm1-1 reported a TBFAILED event

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 1 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 2 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 3 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 4 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 5 minute(s).

*** os_setup: Swap canceled; will terminate os_setup early!

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 6 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 7 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 8 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 9 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 10 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 11 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 12 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 13 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 14 minute(s).

*** os_setup: Still waiting for pcvm1-2 (TBSETUP) - it's been 15 minute(s).

*** Giving up on pcvm1-2 (TBSETUP) - it's been 15 minute(s). 

*** WARNING: os_setup: pcvm1-2 failed to boot

*** Giving up on pcvm1-3 (TBSETUP) - it's been 15 minute(s). 

*** WARNING: os_setup: pcvm1-3 failed to boot

Asking [pc] for volunteers

Asking [pcvm] for volunteers

*** ERROR: os_setup:

***   There were 3 failed nodes.

***   

***   1/1 pcvm's with a system osid of "UBUNTU14-64-STD" failed to boot: (pcvm1-1)

***   1/1 pcvm's with a system osid of "CENTOS71-64-STD" failed to boot: (pcvm1-3)

***   1/1 pcvm's with a system osid of "FBSD102-64-STD" failed to boot: (pcvm1-2)

TIMESTAMP: 08:51:54:246438 os_setup finished

*** ERROR: tbswap: Failed to reset OS and reboot nodes.

Not retrying due to error type.

Cleaning up after errors.

Stopping the event system

Checking for feature NewEventScheduler.

Closing TCP proxy ports...

Releasing public address pools...

TIMESTAMP: 08:51:55:493509 snmpit started

Removing VLANs.

snmpit: testbed/xen has no VLANs, skipping

TIMESTAMP: 08:51:56:188569 snmpit finished

Removing dynamic blobs.

Clearing shared port vlans.

Tearing down virtual nodes.

TIMESTAMP: 08:51:56:194398 vnode_setup -k started

pcvm1-1 failed to boot; skipping teardown.

pcvm1-2 failed to boot; skipping teardown.

pcvm1-3 failed to boot; skipping teardown.

vnode_setup running at parallelization: 10 wait_time: 3000

Vnode teardown finished.

TIMESTAMP: 08:51:57:536644 vnode_setup finished

Removing logical wires.

Freeing nodes.

TIMESTAMP: 08:51:57:539518 nfree started

Releasing all nodes from experiment [Experiment: testbed/xen].

Moving [Node: pc1] to [Experiment: emulab-ops/reloadpending]

TIMESTAMP: 08:51:58:137337 nfree finished

Resetting DB.

*** ERROR: tbswap: Experiment swap-in canceled by user.

Failingly finished swap-in for testbed/xen. 08:51:58:152221

TIMESTAMP: 08:51:58:152764 tbswap in finished (failed)

*** ERROR: batchexp: tbswap in failed!

Cleaning up and exiting with status 1 ... 

Running 'tbend -f -e 153'

Beginning cleanup for testbed/xen. 08:51:58:549682

Clearing out virtual state.

Removing visualization data...

Cleanup finished! 08:51:58:836379

**** Experimental information, please ignore ****

Session ID = 52283

Likely Cause of the Problem:

  Experiment swap-in canceled by user.

Cause: canceled

Confidence: 0.7

Script: tbswap

Possible Error Before Cancellation:

  There were 2 failed nodes.

  

  1/1 pcvm's with a system osid of "CENTOS71-64-STD" failed to boot: (pcvm1-3)

  1/1 pcvm's with a system osid of "FBSD102-64-STD" failed to boot: (pcvm1-2)

Cause: unknown

Confidence: 0.7

Script: os_setup

**** End experimental information ****

SENDMAIL: Could not finish sendmail: 

Removing experiment directories ... 

[Leigh Stoller]

at 12:53 AM, eric <eric147...@gmail.com> wrote:

> I was successful in the experiment of creating two virtual nodes and one link, but when I created three virtual nodes and two links experiments, only the node connecting the two links could be successfully started. The other two nodes are always blocked in the state of TBSETUP. This is the NS script,log,and the screenshot of pcvm.
> I hope to get some useful suggestions, thank you!

Hi. Can you please run this on boss and send us the output:

boss> testbed-version

Leigh

[eric]

This is the output of testbed-version.

[eric]

The node is always in this state (tbsetup1) when I experiment. The boss server can ping three virtual nodes, but only node0 can ping the boss, and the three virtual nodes can communicate with each other. Boot_log is part of the log on node1 (/var/log/boot.log)