Re: Multiple Issues with SD-Fabric Installation
Chan, Charles
Hi David,
Please see my comments below.
Thanks.
Charles Chan, PhD (he/him)
Cloud Software Development Engineer
Software Product Incubation | Network and Edge Group
Intel Corporation | intel.com
From:
David Lake <d.l...@surrey.ac.uk>
Date: Wednesday, August 17, 2022 at 04:37
To: Cascone, Carmelo <carmelo...@intel.com>, Chan, Charles <charle...@intel.com>
Cc: he...@opennetworking.org <he...@opennetworking.org>
Subject: Multiple Issues with SD-Fabric Installation
Carmelo/Charles
Please excuse the direct contact. I’ve tried to contact the SD-Fabric development and support team but I’ve had no replies to the issues I’ve raised.
I am attempting to build a very simple SD-Fabric installation but hitting multiple issues and am at the stage where I require direct assistance. There simply is not enough detail or clarity in the currently available documentation to troubleshoot.
1) Pfcp-agent is failing to start. I see this error in the log:
root@cp-node:/home/admin5g# kubectl logs -n sdfabric sdfabric-pfcp-agent-0
time="2022-08-17T11:15:53Z" level=info msg="{Mode: AccessIface:{IfName:} CoreIface:{IfName:} CPIface:{Peers:[] UseFQDN:false NodeID:pfcp-agent HTTPPort:8080 Dnn:internet EnableUeIPAlloc:false UEIPPool:10.250.0.0/24} P4rtcIface:{SliceID:0 AccessIP:172.17.0.1/32 P4rtcServer:sdfabric-onos-classic-hs P4rtcPort:51001 QFIToTC:map[] DefaultTC:3} EnableP4rt:true EnableFlowMeasure:false SimInfo:{MaxSessions:0 StartUEIP:<nil> StartENBIP:<nil> StartAUPFIP:<nil> N6AppIP:<nil> N9AppIP:<nil> StartN3TEID: StartN9TEID:} ConnTimeout:0 ReadTimeout:15 EnableNotifyBess:false EnableEndMarker:false NotifySockAddr: EndMarkerSockAddr: LogLevel:info QciQosConfig:[] SliceMeterConfig:{N6RateBps:0 N6BurstBytes:0 N3RateBps:0 N3BurstBytes:0} MaxReqRetries:5 RespTimeout:2s EnableHBTimer:false HeartBeatInterval:}" func=main.main file="/pfcpiface/cmd/pfcpiface/main.go:37"
time="2022-08-17T11:15:53Z" level=fatal msg="Unable to resolve hostname pfcp-agent lookup pfcp-agent on 10.96.0.10:53: no such host" func=github.com/omec-project/upf-epc/pfcpiface.NewUPF file="/pfcpiface/pfcpiface/upf.go:109"
(This error also occurs if I enable pfcp-agent in the dev-values.yaml file which you have provided in the 1.2.1-dev release).
[Charles] Did you have pfcp-agent started in the same K8s namespace as ONOS? Can you share the dump of kubectl pods and services?
2) I see an error in the sdfabric-onos-classic-0 log:
2022-08-16 09:44:45,431 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-int.bmv2 (fingerprint=5e:ab:5c:73:80:ab:1c:5d)
2022-08-16 09:44:45,432 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-int.mavericks_sde_9_7_0 (fingerprint=a5:57:71:11:11:5b:61:46)
2022-08-16 09:44:45,432 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-int.montara_sde_9_7_0 (fingerprint=a5:57:71:11:11:5b:61:46)
2022-08-16 09:44:45,433 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-upf-int.bmv2 (fingerprint=1f:73:8f:88:e2:e2:fc:1d)
2022-08-16 09:44:45,433 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-upf-int.mavericks_sde_9_7_0 (fingerprint=69:b1:92:e5:41:01:c3:
52)
2022-08-16 09:44:45,433 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-upf-int.montara_sde_9_7_0 (fingerprint=69:b1:92:e5:41:01:c3:52
)
2022-08-16 09:44:45,433 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-upf.bmv2 (fingerprint=31:54:23:60:b7:db:ae:c8)
2022-08-16 09:44:45,434 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-upf.mavericks_sde_9_7_0 (fingerprint=fa:34:cb:eb:cc:19:81:fe)
2022-08-16 09:44:45,434 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-upf.montara_sde_9_7_0 (fingerprint=fa:34:cb:eb:cc:19:81:fe)
2022-08-16 09:44:45,434 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric.bmv2 (fingerprint=48:79:3a:45:d3:fe:d7:6d)
2022-08-16 09:44:45,434 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric.mavericks_sde_9_7_0 (fingerprint=c5:34:c6:e8:b7:06:80:a5)
2022-08-16 09:44:45,435 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric.montara_sde_9_7_0 (fingerprint=c5:34:c6:e8:b7:06:80:a5)
2022-08-16 09:44:45,435 INFO [PipeconfLoader] Started
2022-08-16 09:44:45,445 INFO [IntManager] Started
2022-08-16 09:44:45,497 INFO [SlicingManager] Processing slice event MapEvent{name=fabric-tna-slice, type=INSERT, key=SliceStoreKey{sliceId=0, trafficClass=BEST_E
FFORT}, newValue=Versioned{value=TrafficClassDescription{tc=BEST_EFFORT, qid=0, maxRateBps=UNLIMITED, gminRateBps=0, isSystemTc=false}, version=1129, creationTime=
2022-08-16T09:44:45.494Z}, oldValue=null}
2022-08-16 09:44:45,498 INFO [SlicingManager] Processing Default TC event MapEvent{name=fabric-tna-default-tc, type=INSERT, key=0, newValue=Versioned{value=BEST_E
FFORT, version=1130, creationTime=2022-08-16T09:44:45.496Z}, oldValue=null}
2022-08-16 09:44:45,498 INFO [SlicingManager] Started
2022-08-16 09:44:45,501 INFO [NetcfgSlicingProvider] Started
2022-08-16 09:44:45,526 INFO [StatisticManager] Started
2022-08-16 09:44:45,551 INFO [HighlightManager] Started
2022-08-16 09:44:45,552 INFO [ApiDocManager] Fabric-tna REST API registered at /onos/fabrictna
2022-08-16 09:44:45,553 INFO [FeaturesServiceImpl] Done.
2022-08-16 09:44:45,554 INFO [ApplicationManager] Application org.stratumproject.fabric-tna has been activated
2022-08-16 09:44:52,580 ERROR [Tools] No such component
2022-08-16 09:44:57,600 ERROR [Tools] No such component
2022-08-16 09:45:02,610 ERROR [Tools] No such component
2022-08-16 09:45:07,620 ERROR [Tools] No such component
2022-08-16 09:45:12,632 ERROR [Tools] No such component
2022-08-16 09:45:17,641 ERROR [Tools] No such component
2022-08-16 09:45:22,651 ERROR [Tools] No such component
2022-08-16 09:45:27,660 ERROR [Tools] No such component
2022-08-16 09:45:32,670 ERROR [Tools] No such component
2022-08-16 09:45:37,679 ERROR [Tools] No such component
2022-08-16 09:45:42,689 ERROR [Tools] No such component
2022-08-16 09:45:47,699 ERROR [Tools] No such component
2022-08-16 09:45:52,708 ERROR [Tools] No such component
2022-08-16 09:45:57,718 ERROR [Tools] No such component
2022-08-16 09:46:02,727 ERROR [Tools] No such component
2022-08-16 09:46:07,739 ERROR [Tools] No such component
2022-08-16 09:46:12,749 ERROR [Tools] No such component
2022-08-16 09:46:17,758 ERROR [Tools] No such component
This message repeats every 5 seconds.
[Charles] I haven’t seen this before. Can you attach the full ONOS log from the very beginning?
3) There is no attempt to load the Stratum image on the switch containers. I’ve checked that the Kubernetes taints are correct and the hosts are reachable but the Helm deployment appears not to reach this point. I have no idea why.
[Charles] Do you see a Stratum daemonset being deployed on K8s? Can you share the output of kubectl describe of that daemonset? Typically it will tell you why a pod can’t be scheduled at the bottom.
I’d be grateful if you could assign a resource to work with me to resolve these issues so that I may achieve a working UP4 + PFCP system with the EdgeCore switches.
[Charles] Unfortunately we can’t offer dedicated engineering resource to support open-source project at this moment. We will try to answer questions from the community at our best.
Thank you in advance
David Lake
5G & 6G Innovation Centres
Institute for Communication Systems (ICS)
University of Surrey
Guildford
GU2 7XH
David Lake
Charles
Thanks for the response.
- [Charles] Did you have pfcp-agent started in the same K8s namespace as ONOS? Can you share the dump of kubectl pods and services?
root@cp-node:/home/admin5g# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default busybox 1/1 Running 65 2d17h
default dnsutils 1/1 Running 23 23h
kube-system coredns-66bff467f8-9lw58 1/1 Running 0 47d
kube-system coredns-66bff467f8-qv7vd 1/1 Running 1 47d
kube-system etcd-cp-node 1/1 Running 2 47d
kube-system kube-apiserver-cp-node 1/1 Running 2 47d
kube-system kube-controller-manager-cp-node 1/1 Running 1 47d
kube-system kube-flannel-ds-fkfcc 1/1 Running 0 47d
kube-system kube-flannel-ds-w8txw 1/1 Running 3 47d
kube-system kube-flannel-ds-z4qh6 1/1 Running 0 45d
kube-system kube-flannel-ds-zhwxz 1/1 Running 3 47d
kube-system kube-proxy-cp56k 1/1 Running 0 47d
kube-system kube-proxy-fhwg8 1/1 Running 3 47d
kube-system kube-proxy-hwnd5 1/1 Running 0 45d
kube-system kube-proxy-ts6fp 1/1 Running 1 47d
kube-system kube-scheduler-cp-node 1/1 Running 1 47d
sdfabric sdfabric-atomix-0 1/1 Running 0 2d
sdfabric sdfabric-onos-classic-0 1/1 Running 0 2d
sdfabric sdfabric-onos-classic-onos-config-loader-59d64bd8dc-wq48d 1/1 Running 0 2d
sdfabric sdfabric-pfcp-agent-0 0/1 CrashLoopBackOff 567 2d
root@cp-node:/home/admin5g#
root@cp-node:/home/admin5g# kubectl -n sdfabric get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
sdfabric-atomix-api ClusterIP 10.104.61.48 <none> 5678/TCP 2d
sdfabric-atomix-hs ClusterIP None <none> 5679/TCP 2d
sdfabric-onos-classic-hs ClusterIP None <none> 51001/TCP,9876/TCP,8101/TCP,8181/TCP 2d
sdfabric-pfcp-agent ClusterIP 10.100.0.173 <none> 8805/UDP 2d
sdfabric-pfcp-agent-rest ClusterIP 10.98.103.31 <none> 8080/TCP 2d
root@cp-node:/home/admin5g#
- I haven’t seen this before. Can you attach the full ONOS log from the very beginning?
This error seems to coincide with the attempt to load the necfg:
root@cp-node:/home/admin5g# kubectl logs sdfabric-onos-classic-onos-config-loader-59d64bd8dc-wq48d -n sdfabric
-----------------------------------------
Timestamp is
Tue Aug 16 09:43:50 UTC 2022
-----------------------------------------
Waiting for ONOS Cluster to be ready
-----------------------------------------
Timestamp is
Tue Aug 16 09:44:01 UTC 2022
-----------------------------------------
Waiting for ONOS Cluster to be ready
-----------------------------------------
Timestamp is
Tue Aug 16 09:44:11 UTC 2022
-----------------------------------------
Waiting for ONOS Cluster to be ready
-----------------------------------------
Timestamp is
Tue Aug 16 09:44:21 UTC 2022
-----------------------------------------
Waiting for ONOS Cluster to be ready
-----------------------------------------
Timestamp is
Tue Aug 16 09:44:31 UTC 2022
-----------------------------------------
Waiting for ONOS Cluster to be ready
-----------------------------------------
Timestamp is
Tue Aug 16 09:44:42 UTC 2022
-----------------------------------------
Waiting for ONOS Cluster to be ready
-----------------------------------------
Timestamp is
Tue Aug 16 09:44:52 UTC 2022
-----------------------------------------
Waiting for ONOS Cluster to be ready
-----------------------------------------
Loading netcfg into ONOS
-----------------------------------------
cat: can't open '/opt/configs/netcfg.json': No such file or directory
sed: can't read /opt/configs/netcfg.json: No such file or directory
Update is unnecessary since the netcfg on the ONOS cluster is up to date
-----------------------------------------
Loading component configs into ONOS
-----------------------------------------
-----------------------------------------
Check that component netcfg is active
-----------------------------------------
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
(REPEATS EVERY 5 SECONDS)
ONOS Log:
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ /_/ / / /_/ /\ \
\____/_/|_/\____/___/
Documentation: wiki.onosproject.org
Tutorials: tutorials.onosproject.org
Mailing lists: lists.onosproject.org
Come help out! Find out how at: contribute.onosproject.org
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'logout' to exit ONOS session.
karaf@root > log 10:07:53
karaf@root log> display 10:07:59
10:06:14.502 INFO [ApplicationManager] Application org.onosproject.protocols.gnmi has been activated
10:06:14.505 INFO [FeaturesServiceImpl] Adding features: onos-providers-general/[2.5.8.SNAPSHOT,2.5.8.SNAPSHOT]
10:06:14.521 INFO [BgpSessionManager] BgpSessionManager started
10:06:14.596 INFO [FpmManager] clearRoutes is true
10:06:14.596 INFO [FpmManager] PD pushing is disabled.
10:06:14.626 INFO [FpmManager] Started
10:06:14.634 INFO [ObjectiveTracker] Started
10:06:14.637 INFO [PathManager] Started
10:06:14.644 INFO [NetworkConfigHostProvider] Started
10:06:14.649 INFO [NeighbourResolutionManager] IPv6 neighbor discovery is disabled
10:06:14.649 INFO [NeighbourResolutionManager] Address resolution protocol is enabled
10:06:14.650 INFO [NeighbourResolutionManager] Configured. Request intercepts is enabled
10:06:14.732 INFO [LldpLinkProvider] Settings: enabled=true, useBDDP=true, probeRate=3000, staleLinkAge=10000, maxLLDPage=1000, useStaleLinkAge=true
10:06:14.732 INFO [LldpLinkProvider] Started
10:06:14.744 INFO [Dhcp4HandlerImpl] Learning routes from DHCP leasequery is disabled
10:06:14.751 INFO [Dhcp6HandlerImpl] Learning routes from DHCP leasequery is disabled
10:06:14.763 INFO [DhcpRelayManager] Address resolution protocol is enabled
10:06:14.765 INFO [DhcpRelayManager] DhcpRelay poll interval new 86400 old 86400
10:06:14.765 INFO [DhcpRelayManager] DhcpRelay FPM is disabled
10:06:14.769 INFO [DhcpRelayManager] DHCP-RELAY Started
10:06:14.780 INFO [IntentManager] Started
10:06:14.789 INFO [IntentCleanup] Started
10:06:14.800 INFO [IntentConfigurableRegistrator] Started
10:06:14.812 INFO [ProtectedTransportIntentCompiler] started
10:06:14.838 INFO [UiSharedTopologyModel] Started
10:06:14.839 INFO [LionUtils] System Default Locale: [en_US]
10:06:14.840 INFO [LionUtils] Language: [en], Country: [US]
10:06:14.840 INFO [LionUtils] Setting runtime locale to: [en_US]
10:06:14.846 INFO [BundleStitcher] Generated LION bundle: LionBundle{id=core.fw.Mast, #items=7}
10:06:14.850 INFO [BundleStitcher] Generated LION bundle: LionBundle{id=core.fw.Nav, #items=15}
10:06:14.851 INFO [BundleStitcher] Generated LION bundle: LionBundle{id=core.fw.QuickHelp, #items=5}
10:06:14.852 INFO [BundleStitcher] Generated LION bundle: LionBundle{id=core.view.App, #items=32}
10:06:14.853 INFO [BundleStitcher] Generated LION bundle: LionBundle{id=core.view.Cluster, #items=23}
10:06:14.855 INFO [BundleStitcher] Generated LION bundle: LionBundle{id=core.view.Topo, #items=160}
10:06:14.856 INFO [BundleStitcher] Generated LION bundle: LionBundle{id=core.view.Flow, #items=25}
10:06:14.887 INFO [UiExtensionManager] Started
10:06:14.893 INFO [GroupManager] Configured. PurgeOnDisconnection is disabled
10:06:14.894 INFO [HostManager] Configured. monitorHosts disabled
10:06:14.894 INFO [HostManager] Configured. probeRate 30000
10:06:14.895 INFO [HostManager] Removal of duplicate ip address is disabled
10:06:14.895 INFO [HostManager] Configured. greedyLearningIpv6 disabled
10:06:14.895 INFO [HostManager] Configured. hostMoveTrackerEnabled disabled
10:06:14.898 INFO [DefaultTopology] Setting new default maximum paths count to -1
10:06:14.898 INFO [DistributedTopologyStore] Settings: linkWeightFunction=hopCount
10:06:14.901 INFO [DistributedStatisticStore] Setting: messageHandlerThreadPoolSize=4
10:06:14.902 INFO [CoreManager] Settings: sharedThreadPoolSize=30, maxEventTimeLimit=2000, sharedThreadPerformanceCheck=false
10:06:14.937 INFO [DistributedMeterStore] Started
10:06:14.943 INFO [MeterManager] Configured. PurgeOnDisconnection is disabled
10:06:14.944 INFO [MeterManager] UserDefinedIndex is disabled
10:06:14.946 INFO [MeterManager] Started
10:06:15.091 INFO [FeaturesServiceImpl] Changes to perform:
10:06:15.092 INFO [FeaturesServiceImpl] Region: root
10:06:15.092 INFO [FeaturesServiceImpl] Bundles to install:
10:06:15.092 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-providers-general-device/2.5.8-SNAPSHOT
10:06:15.093 INFO [FeaturesServiceImpl] Installing bundles:
10:06:15.093 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-providers-general-device/2.5.8-SNAPSHOT
10:06:15.112 INFO [FeaturesServiceImpl] Starting bundles:
10:06:15.112 INFO [FeaturesServiceImpl] org.onosproject.onos-providers-general-device/2.5.8.SNAPSHOT
10:06:15.129 INFO [StatsPoller] Started
10:06:15.129 INFO [GeneralDeviceProvider] Configured. checkupInterval is configured to 10 seconds
10:06:15.129 INFO [GeneralDeviceProvider] Configured. statsPollInterval is configured to 10 seconds
10:06:15.129 INFO [GeneralDeviceProvider] Started
10:06:15.129 INFO [FeaturesServiceImpl] Done.
10:06:15.130 INFO [ApplicationManager] Application org.onosproject.generaldeviceprovider has been activated
10:06:15.134 INFO [FeaturesServiceImpl] Adding features: onos-protocols-gnoi/[2.5.8.SNAPSHOT,2.5.8.SNAPSHOT]
10:06:15.378 INFO [Dhcp4HandlerImpl] Learning routes from DHCP leasequery is disabled
10:06:15.380 INFO [IntentCleanup] Settings: enabled=true, period=5, retryThreshold=5
10:06:15.381 INFO [DhcpRelayManager] Address resolution protocol is enabled
10:06:15.383 INFO [DhcpRelayManager] DhcpRelay poll interval new 86400 old 86400
10:06:15.384 INFO [DhcpRelayManager] DhcpRelay FPM is disabled
10:06:15.388 INFO [GeneralDeviceProvider] Configured. checkupInterval is configured to 10 seconds
10:06:15.388 INFO [GeneralDeviceProvider] Configured. statsPollInterval is configured to 10 seconds
10:06:15.390 INFO [NeighbourResolutionManager] IPv6 neighbor discovery is disabled
10:06:15.390 INFO [NeighbourResolutionManager] Address resolution protocol is enabled
10:06:15.390 INFO [NeighbourResolutionManager] Configured. Request intercepts is enabled
10:06:15.400 INFO [Dhcp6HandlerImpl] Learning routes from DHCP leasequery is disabled
10:06:15.402 INFO [UiExtensionManager] Settings: trafficRefresh=5000
10:06:15.404 INFO [FpmManager] clearRoutes is true
10:06:15.405 INFO [FpmManager] PD pushing is disabled.
10:06:15.406 INFO [LldpLinkProvider] Settings: enabled=true, useBDDP=true, probeRate=3000, staleLinkAge=10000, maxLLDPage=1000, useStaleLinkAge=true
10:06:15.407 INFO [MeterManager] Configured. PurgeOnDisconnection is disabled
10:06:15.409 INFO [MeterManager] UserDefinedIndex is disabled
10:06:15.412 INFO [GnmiControllerImpl] Configured. readPortId is configured to false
10:06:15.451 INFO [AtomixClusterStore] Updated node onos-0 state to READY
10:06:15.454 INFO [FpmManager] Receives ClusterEvent INSTANCE_READY for onos-0
10:06:15.462 INFO [FpmManager] 0 FPM routes have been updated to route store
10:06:15.621 INFO [FeaturesServiceImpl] Changes to perform:
10:06:15.621 INFO [FeaturesServiceImpl] Region: root
10:06:15.621 INFO [FeaturesServiceImpl] Bundles to install:
10:06:15.622 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-gnoi-api/2.5.8-SNAPSHOT
10:06:15.622 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-gnoi-ctl/2.5.8-SNAPSHOT
10:06:15.622 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-gnoi-stub/2.5.8-SNAPSHOT
10:06:15.623 INFO [FeaturesServiceImpl] Installing bundles:
10:06:15.623 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-gnoi-api/2.5.8-SNAPSHOT
10:06:15.625 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-gnoi-ctl/2.5.8-SNAPSHOT
10:06:15.626 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-gnoi-stub/2.5.8-SNAPSHOT
10:06:15.636 INFO [FeaturesServiceImpl] Starting bundles:
10:06:15.637 INFO [FeaturesServiceImpl] org.onosproject.onos-protocols-gnoi-stub/2.5.8.SNAPSHOT
10:06:15.637 INFO [FeaturesServiceImpl] org.onosproject.onos-protocols-gnoi-api/2.5.8.SNAPSHOT
10:06:15.638 INFO [FeaturesServiceImpl] org.onosproject.onos-protocols-gnoi-ctl/2.5.8.SNAPSHOT
10:06:15.642 INFO [GnoiControllerImpl] Started
10:06:15.642 INFO [FeaturesServiceImpl] Done.
10:06:15.643 INFO [ApplicationManager] Application org.onosproject.protocols.gnoi has been activated
10:06:15.645 INFO [FeaturesServiceImpl] Adding features: onos-drivers-gnoi/[2.5.8.SNAPSHOT,2.5.8.SNAPSHOT]
10:06:16.202 INFO [FeaturesServiceImpl] Changes to perform:
10:06:16.202 INFO [FeaturesServiceImpl] Region: root
10:06:16.202 INFO [FeaturesServiceImpl] Bundles to install:
10:06:16.202 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-drivers-gnoi/2.5.8-SNAPSHOT
10:06:16.203 INFO [FeaturesServiceImpl] Installing bundles:
10:06:16.203 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-drivers-gnoi/2.5.8-SNAPSHOT
10:06:16.210 INFO [FeaturesServiceImpl] Starting bundles:
10:06:16.210 INFO [FeaturesServiceImpl] org.onosproject.onos-drivers-gnoi/2.5.8.SNAPSHOT
10:06:16.215 INFO [GnoiDriversLoader] Started
10:06:16.216 INFO [FeaturesServiceImpl] Done.
10:06:16.216 INFO [ApplicationManager] Application org.onosproject.drivers.gnoi has been activated
10:06:16.223 INFO [FeaturesServiceImpl] Adding features: onos-protocols-p4runtime/[2.5.8.SNAPSHOT,2.5.8.SNAPSHOT]
10:06:16.737 INFO [FeaturesServiceImpl] Changes to perform:
10:06:16.738 INFO [FeaturesServiceImpl] Region: root
10:06:16.738 INFO [FeaturesServiceImpl] Bundles to install:
10:06:16.738 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-p4runtime-api/2.5.8-SNAPSHOT
10:06:16.738 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-p4runtime-ctl/2.5.8-SNAPSHOT
10:06:16.738 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-p4runtime-model/2.5.8-SNAPSHOT
10:06:16.738 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-p4runtime-proto/2.5.8-SNAPSHOT
10:06:16.738 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-p4runtime-utils/2.5.8-SNAPSHOT
10:06:16.739 INFO [FeaturesServiceImpl] Installing bundles:
10:06:16.739 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-p4runtime-api/2.5.8-SNAPSHOT
10:06:16.741 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-p4runtime-ctl/2.5.8-SNAPSHOT
10:06:16.742 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-p4runtime-model/2.5.8-SNAPSHOT
10:06:16.743 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-p4runtime-proto/2.5.8-SNAPSHOT
10:06:16.747 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-protocols-p4runtime-utils/2.5.8-SNAPSHOT
10:06:16.757 INFO [FeaturesServiceImpl] Starting bundles:
10:06:16.757 INFO [FeaturesServiceImpl] org.onosproject.onos-protocols-p4runtime-proto/2.5.8.SNAPSHOT
10:06:16.758 INFO [FeaturesServiceImpl] org.onosproject.onos-protocols-p4runtime-utils/2.5.8.SNAPSHOT
10:06:16.759 INFO [FeaturesServiceImpl] org.onosproject.onos-protocols-p4runtime-model/2.5.8.SNAPSHOT
10:06:16.760 INFO [FeaturesServiceImpl] org.onosproject.onos-protocols-p4runtime-api/2.5.8.SNAPSHOT
10:06:16.760 INFO [FeaturesServiceImpl] org.onosproject.onos-protocols-p4runtime-ctl/2.5.8.SNAPSHOT
10:06:16.771 INFO [DistributedMasterElectionIdStore] Started
10:06:16.774 INFO [P4RuntimeControllerImpl] Started
10:06:16.774 INFO [FeaturesServiceImpl] Done.
10:06:16.774 INFO [ApplicationManager] Application org.onosproject.protocols.p4runtime has been activated
10:06:16.778 INFO [FeaturesServiceImpl] Adding features: onos-providers-p4runtime/[2.5.8.SNAPSHOT,2.5.8.SNAPSHOT]
10:06:17.316 INFO [FeaturesServiceImpl] Changes to perform:
10:06:17.316 INFO [FeaturesServiceImpl] Region: root
10:06:17.316 INFO [FeaturesServiceImpl] Bundles to install:
10:06:17.316 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-providers-p4runtime-packet/2.5.8-SNAPSHOT
10:06:17.317 INFO [FeaturesServiceImpl] Installing bundles:
10:06:17.318 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-providers-p4runtime-packet/2.5.8-SNAPSHOT
10:06:17.325 INFO [FeaturesServiceImpl] Starting bundles:
10:06:17.325 INFO [FeaturesServiceImpl] org.onosproject.onos-providers-p4runtime-packet/2.5.8.SNAPSHOT
10:06:17.333 INFO [P4RuntimePacketProvider] Settings: p4RuntimePacketProviderWorkers=4
10:06:17.333 INFO [P4RuntimePacketProvider] Started
10:06:17.333 INFO [FeaturesServiceImpl] Done.
10:06:17.333 INFO [ApplicationManager] Application org.onosproject.p4runtime has been activated
10:06:17.336 INFO [FeaturesServiceImpl] Adding features: onos-drivers-p4runtime/[2.5.8.SNAPSHOT,2.5.8.SNAPSHOT]
10:06:17.874 INFO [FeaturesServiceImpl] Changes to perform:
10:06:17.874 INFO [FeaturesServiceImpl] Region: root
10:06:17.874 INFO [FeaturesServiceImpl] Bundles to install:
10:06:17.874 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-drivers-p4runtime/2.5.8-SNAPSHOT
10:06:17.875 INFO [FeaturesServiceImpl] Installing bundles:
10:06:17.875 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-drivers-p4runtime/2.5.8-SNAPSHOT
10:06:17.885 INFO [FeaturesServiceImpl] Starting bundles:
10:06:17.886 INFO [FeaturesServiceImpl] org.onosproject.onos-drivers-p4runtime/2.5.8.SNAPSHOT
10:06:17.895 INFO [P4RuntimeDriversLoader] Started
10:06:17.907 INFO [DistributedP4RuntimeActionProfileGroupMirror] Started
10:06:17.915 INFO [DistributedP4RuntimeActionProfileMemberMirror] Started
10:06:17.923 INFO [DistributedP4RuntimeDefaultEntryMirror] Started
10:06:17.933 INFO [DistributedP4RuntimePreEntryMirror] Started
10:06:17.941 INFO [DistributedP4RuntimeTableMirror] Started
10:06:17.942 INFO [FeaturesServiceImpl] Done.
10:06:17.942 INFO [ApplicationManager] Application org.onosproject.drivers.p4runtime has been activated
10:06:17.944 INFO [FeaturesServiceImpl] Adding features: onos-pipelines-basic/[2.5.8.SNAPSHOT,2.5.8.SNAPSHOT]
10:06:18.503 INFO [FeaturesServiceImpl] Changes to perform:
10:06:18.504 INFO [FeaturesServiceImpl] Region: root
10:06:18.504 INFO [FeaturesServiceImpl] Bundles to install:
10:06:18.504 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-pipelines-basic/2.5.8-SNAPSHOT
10:06:18.505 INFO [FeaturesServiceImpl] Installing bundles:
10:06:18.505 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-pipelines-basic/2.5.8-SNAPSHOT
10:06:18.513 INFO [FeaturesServiceImpl] Starting bundles:
10:06:18.514 INFO [FeaturesServiceImpl] org.onosproject.onos-pipelines-basic/2.5.8.SNAPSHOT
10:06:18.844 INFO [PiPipeconfManager] New pipeconf registered: org.onosproject.pipelines.basic (fingerprint=c3:4a:1b:ef:32:19:e4:f4)
10:06:18.845 INFO [PiPipeconfManager] New pipeconf registered: org.onosproject.pipelines.int (fingerprint=5b:7d:f9:2d:dc:5f:97:4c)
10:06:18.845 INFO [FeaturesServiceImpl] Done.
10:06:18.845 INFO [ApplicationManager] Application org.onosproject.pipelines.basic has been activated
10:06:18.853 INFO [FeaturesServiceImpl] Adding features: onos-drivers-gnmi/[2.5.8.SNAPSHOT,2.5.8.SNAPSHOT]
10:06:19.384 INFO [FeaturesServiceImpl] Changes to perform:
10:06:19.384 INFO [FeaturesServiceImpl] Region: root
10:06:19.384 INFO [FeaturesServiceImpl] Bundles to install:
10:06:19.384 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-drivers-gnmi/2.5.8-SNAPSHOT
10:06:19.386 INFO [FeaturesServiceImpl] Installing bundles:
10:06:19.386 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-drivers-gnmi/2.5.8-SNAPSHOT
10:06:19.396 INFO [FeaturesServiceImpl] Starting bundles:
10:06:19.397 INFO [FeaturesServiceImpl] org.onosproject.onos-drivers-gnmi/2.5.8.SNAPSHOT
10:06:19.403 INFO [GnmiDriversLoader] Started
10:06:19.403 INFO [FeaturesServiceImpl] Done.
10:06:19.404 INFO [ApplicationManager] Application org.onosproject.drivers.gnmi has been activated
10:06:19.426 INFO [FeaturesServiceImpl] Adding features: onos-drivers-stratum/[2.5.8.SNAPSHOT,2.5.8.SNAPSHOT]
10:06:20.057 INFO [FeaturesServiceImpl] Changes to perform:
10:06:20.058 INFO [FeaturesServiceImpl] Region: root
10:06:20.058 INFO [FeaturesServiceImpl] Bundles to install:
10:06:20.058 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-drivers-stratum/2.5.8-SNAPSHOT
10:06:20.059 INFO [FeaturesServiceImpl] Installing bundles:
10:06:20.059 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-drivers-stratum/2.5.8-SNAPSHOT
10:06:20.067 INFO [FeaturesServiceImpl] Starting bundles:
10:06:20.067 INFO [FeaturesServiceImpl] org.onosproject.onos-drivers-stratum/2.5.8.SNAPSHOT
10:06:20.072 INFO [StratumDriversLoader] Started
10:06:20.073 INFO [FeaturesServiceImpl] Done.
10:06:20.073 INFO [ApplicationManager] Application org.onosproject.drivers.stratum has been activated
10:06:20.079 INFO [FeaturesServiceImpl] Adding features: onos-apps-portloadbalancer/[2.5.8.SNAPSHOT,2.5.8.SNAPSHOT]
10:06:20.560 INFO [FeaturesServiceImpl] Changes to perform:
10:06:20.561 INFO [FeaturesServiceImpl] Region: root
10:06:20.561 INFO [FeaturesServiceImpl] Bundles to install:
10:06:20.561 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-apps-portloadbalancer/2.5.8-SNAPSHOT
10:06:20.562 INFO [FeaturesServiceImpl] Installing bundles:
10:06:20.562 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-apps-portloadbalancer/2.5.8-SNAPSHOT
10:06:20.570 INFO [FeaturesServiceImpl] Starting bundles:
10:06:20.571 INFO [FeaturesServiceImpl] org.onosproject.onos-apps-portloadbalancer/2.5.8.SNAPSHOT
10:06:20.573 INFO [CommandExtension] Registering commands for bundle org.onosproject.onos-apps-portloadbalancer/2.5.8.SNAPSHOT
10:06:20.617 INFO [PortLoadBalancerManager] Started
10:06:20.617 INFO [FeaturesServiceImpl] Done.
10:06:20.618 INFO [ApplicationManager] Application org.onosproject.portloadbalancer has been activated
10:06:20.652 INFO [FeaturesServiceImpl] Adding features: onos-drivers-barefoot/[2.5.8.SNAPSHOT,2.5.8.SNAPSHOT]
10:06:21.287 INFO [FeaturesServiceImpl] Changes to perform:
10:06:21.287 INFO [FeaturesServiceImpl] Region: root
10:06:21.287 INFO [FeaturesServiceImpl] Bundles to install:
10:06:21.287 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-drivers-barefoot/2.5.8-SNAPSHOT
10:06:21.289 INFO [FeaturesServiceImpl] Installing bundles:
10:06:21.289 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-drivers-barefoot/2.5.8-SNAPSHOT
10:06:21.297 INFO [FeaturesServiceImpl] Starting bundles:
10:06:21.298 INFO [FeaturesServiceImpl] org.onosproject.onos-drivers-barefoot/2.5.8.SNAPSHOT
10:06:21.302 INFO [BarefootDriversLoader] Started
10:06:21.303 INFO [FeaturesServiceImpl] Done.
10:06:21.303 INFO [ApplicationManager] Application org.onosproject.drivers.barefoot has been activated
10:06:21.310 INFO [FeaturesServiceImpl] Adding features: onos-providers-host/[2.5.8.SNAPSHOT,2.5.8.SNAPSHOT]
10:06:21.712 INFO [FeaturesServiceImpl] Changes to perform:
10:06:21.712 INFO [FeaturesServiceImpl] Region: root
10:06:21.712 INFO [FeaturesServiceImpl] Bundles to install:
10:06:21.712 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-providers-host/2.5.8-SNAPSHOT
10:06:21.713 INFO [FeaturesServiceImpl] Installing bundles:
10:06:21.713 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-providers-host/2.5.8-SNAPSHOT
10:06:21.727 INFO [FeaturesServiceImpl] Starting bundles:
10:06:21.728 INFO [FeaturesServiceImpl] org.onosproject.onos-providers-host/2.5.8.SNAPSHOT
10:06:21.754 INFO [HostLocationProvider] Host removal on port/device down events is not configured, using current value of true
10:06:21.754 INFO [HostLocationProvider] Configured. Using ARP is enabled
10:06:21.754 INFO [HostLocationProvider] Configured. Using IPv6 NDP Neighbor Solicitation and Advertisement is disabled
10:06:21.754 INFO [HostLocationProvider] Configured. Using IPv6 NDP Router Solicitation and Advertisement is disabled
10:06:21.755 INFO [HostLocationProvider] Configured. Using DHCP is disabled
10:06:21.755 INFO [HostLocationProvider] Configured. Using DHCP6 is disabled
10:06:21.755 INFO [HostLocationProvider] Configured. Request intercepts is enabled
10:06:21.755 INFO [HostLocationProvider] Configured. Multihoming is disabled
10:06:21.764 INFO [HostLocationProvider] Started with Application ID 41
10:06:21.764 INFO [FeaturesServiceImpl] Done.
10:06:21.765 INFO [ApplicationManager] Application org.onosproject.hostprovider has been activated
10:06:21.782 INFO [FeaturesServiceImpl] Adding features: onos-apps-mcast/[2.5.8.SNAPSHOT,2.5.8.SNAPSHOT]
10:06:21.999 INFO [HostLocationProvider] Host removal on port/device down events is not configured, using current value of true
10:06:21.999 INFO [HostLocationProvider] Configured. Using ARP is enabled
10:06:22.000 INFO [HostLocationProvider] Configured. Using IPv6 NDP Neighbor Solicitation and Advertisement is disabled
10:06:22.000 INFO [HostLocationProvider] Configured. Using IPv6 NDP Router Solicitation and Advertisement is disabled
10:06:22.000 INFO [HostLocationProvider] Configured. Using DHCP is disabled
10:06:22.000 INFO [HostLocationProvider] Configured. Using DHCP6 is disabled
10:06:22.000 INFO [HostLocationProvider] Configured. Request intercepts is enabled
10:06:22.000 INFO [HostLocationProvider] Configured. Multihoming is disabled
10:06:22.391 INFO [FeaturesServiceImpl] Changes to perform:
10:06:22.391 INFO [FeaturesServiceImpl] Region: root
10:06:22.391 INFO [FeaturesServiceImpl] Bundles to install:
10:06:22.391 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-apps-mcast-api/2.5.8-SNAPSHOT
10:06:22.391 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-apps-mcast-cli/2.5.8-SNAPSHOT
10:06:22.391 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-apps-mcast-impl/2.5.8-SNAPSHOT
10:06:22.391 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-apps-mcast-web/2.5.8-SNAPSHOT
10:06:22.394 INFO [FeaturesServiceImpl] Installing bundles:
10:06:22.394 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-apps-mcast-api/2.5.8-SNAPSHOT
10:06:22.396 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-apps-mcast-cli/2.5.8-SNAPSHOT
10:06:22.397 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-apps-mcast-impl/2.5.8-SNAPSHOT
10:06:22.398 INFO [FeaturesServiceImpl] mvn:org.onosproject/onos-apps-mcast-web/2.5.8-SNAPSHOT
10:06:22.409 INFO [FeaturesServiceImpl] Starting bundles:
10:06:22.410 INFO [FeaturesServiceImpl] org.onosproject.onos-apps-mcast-api/2.5.8.SNAPSHOT
10:06:22.411 INFO [FeaturesServiceImpl] org.onosproject.onos-apps-mcast-cli/2.5.8.SNAPSHOT
10:06:22.414 INFO [CommandExtension] Registering commands for bundle org.onosproject.onos-apps-mcast-cli/2.5.8.SNAPSHOT
10:06:22.415 INFO [FeaturesServiceImpl] org.onosproject.onos-apps-mcast-web/2.5.8.SNAPSHOT
10:06:22.454 INFO [HttpServiceFactoryImpl] Binding bundle: [org.onosproject.onos-apps-mcast-web [263]] to http service
10:06:22.458 INFO [ApiDocManager] Multicast API registered at /onos/mcast
10:06:22.460 INFO [McastServiceCodecRegistrator] Started
10:06:22.460 INFO [FeaturesServiceImpl] org.onosproject.onos-apps-mcast-impl/2.5.8.SNAPSHOT
10:06:22.461 INFO [ServletContainerInitializerScanner] will add org.eclipse.jetty.websocket.server.NativeWebSocketServletContainerInitializer to ServletContainerInitializers
10:06:22.461 INFO [ServletContainerInitializerScanner] added ServletContainerInitializer: org.eclipse.jetty.websocket.server.NativeWebSocketServletContainerInitializer
10:06:22.462 INFO [ServletContainerInitializerScanner] will add org.apache.jasper.servlet.JasperInitializer to ServletContainerInitializers
10:06:22.462 INFO [ServletContainerInitializerScanner] Skipt org.apache.jasper.servlet.JasperInitializer, because specialized handler will be present
10:06:22.462 INFO [ServletContainerInitializerScanner] will add org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer to ServletContainerInitializers
10:06:22.482 INFO [DistributedMcastRoutesStore] Started
10:06:22.487 INFO [MulticastRouteManager] Started
10:06:22.488 INFO [FeaturesServiceImpl] Done.
10:06:22.488 INFO [ApplicationManager] Application org.onosproject.mcast has been activated
10:06:22.498 INFO [FeaturesServiceImpl] Adding features: up4-app/[1.2.0.SNAPSHOT,1.2.0.SNAPSHOT]
10:06:22.580 INFO [ServletContainerInitializerScanner] added ServletContainerInitializer: org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer
10:06:22.581 INFO [HttpServiceContext] registering context WebAppHttpContext{org.onosproject.onos-apps-mcast-web - 263}, with context-name: onos/mcast
10:06:22.584 INFO [HttpServiceContext] registering JasperInitializer
10:06:22.586 INFO [WebSocketServerFactory] No DecoratedObjectFactory provided, using new org.eclipse.jetty.util.DecoratedObjectFactory[decorators=1]
10:06:22.590 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/c-1_0-rt.tld
10:06:22.592 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/c-1_0.tld
10:06:22.594 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/c-1_1.tld
10:06:22.596 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/c.tld
10:06:22.598 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/fmt-1_0-rt.tld
10:06:22.600 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/fmt-1_0.tld
10:06:22.602 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/fmt.tld
10:06:22.603 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/fn.tld
10:06:22.604 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/permittedTaglibs.tld
10:06:22.604 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/scriptfree.tld
10:06:22.605 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/sql-1_0-rt.tld
10:06:22.606 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/sql-1_0.tld
10:06:22.608 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/sql.tld
10:06:22.608 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/x-1_0-rt.tld
10:06:22.610 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/x-1_0.tld
10:06:22.612 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/x.tld
10:06:22.642 INFO [ContextHandler] Started HttpServiceContext{httpContext=WebAppHttpContext{org.onosproject.onos-apps-mcast-web - 263}}
10:06:22.969 INFO [FeaturesServiceImpl] Changes to perform:
10:06:22.970 INFO [FeaturesServiceImpl] Region: root
10:06:22.970 INFO [FeaturesServiceImpl] Bundles to install:
10:06:22.970 INFO [FeaturesServiceImpl] mvn:org.omecproject/up4-api/1.2.0-SNAPSHOT
10:06:22.970 INFO [FeaturesServiceImpl] mvn:org.omecproject/up4-app/1.2.0-SNAPSHOT
10:06:22.971 INFO [FeaturesServiceImpl] Installing bundles:
10:06:22.971 INFO [FeaturesServiceImpl] mvn:org.omecproject/up4-api/1.2.0-SNAPSHOT
10:06:22.972 INFO [FeaturesServiceImpl] mvn:org.omecproject/up4-app/1.2.0-SNAPSHOT
10:06:22.982 INFO [FeaturesServiceImpl] Starting bundles:
10:06:22.982 INFO [FeaturesServiceImpl] org.omecproject.up4-api/1.2.0.SNAPSHOT
10:06:22.983 INFO [FeaturesServiceImpl] org.omecproject.up4-app/1.2.0.SNAPSHOT
10:06:22.991 INFO [CommandExtension] Registering commands for bundle org.omecproject.up4-app/1.2.0.SNAPSHOT
10:06:23.000 INFO [DistributedUp4Store] Started
10:06:23.008 INFO [Up4DeviceManager] Starting...
10:06:23.019 INFO [Up4DeviceManager] Started.
10:06:23.028 INFO [Up4NorthComponent] Starting...
10:06:23.147 INFO [Up4NorthComponent] UP4 gRPC server started on port 51001
10:06:23.147 INFO [Up4NorthComponent] Started.
10:06:23.147 INFO [FeaturesServiceImpl] Done.
10:06:23.148 INFO [ApplicationManager] Application org.omecproject.up4 has been activated
10:06:23.155 INFO [FeaturesServiceImpl] Adding features: segmentrouting-app/[3.3.0.SNAPSHOT,3.3.0.SNAPSHOT]
10:06:23.838 INFO [FeaturesServiceImpl] Changes to perform:
10:06:23.839 INFO [FeaturesServiceImpl] Region: root
10:06:23.839 INFO [FeaturesServiceImpl] Bundles to install:
10:06:23.839 INFO [FeaturesServiceImpl] mvn:org.onosproject/segmentrouting-api/3.3.0-SNAPSHOT
10:06:23.839 INFO [FeaturesServiceImpl] mvn:org.onosproject/segmentrouting-impl/3.3.0-SNAPSHOT
10:06:23.839 INFO [FeaturesServiceImpl] mvn:org.onosproject/segmentrouting-web/3.3.0-SNAPSHOT
10:06:23.840 INFO [FeaturesServiceImpl] Installing bundles:
10:06:23.840 INFO [FeaturesServiceImpl] mvn:org.onosproject/segmentrouting-api/3.3.0-SNAPSHOT
10:06:23.842 INFO [FeaturesServiceImpl] mvn:org.onosproject/segmentrouting-impl/3.3.0-SNAPSHOT
10:06:23.845 INFO [FeaturesServiceImpl] mvn:org.onosproject/segmentrouting-web/3.3.0-SNAPSHOT
10:06:23.859 INFO [FeaturesServiceImpl] Starting bundles:
10:06:23.860 INFO [FeaturesServiceImpl] org.onosproject.segmentrouting-api/3.3.0.SNAPSHOT
10:06:23.860 INFO [FeaturesServiceImpl] org.onosproject.segmentrouting-impl/3.3.0.SNAPSHOT
10:06:23.874 INFO [CommandExtension] Registering commands for bundle org.onosproject.segmentrouting-impl/3.3.0.SNAPSHOT
10:06:24.165 INFO [SegmentRoutingManager] Configuring network before adding listeners
10:06:24.165 INFO [SegmentRoutingManager] Configuring network ...
10:06:24.166 INFO [SegmentRoutingManager] Buffering config calls for 2 secs
10:06:24.166 INFO [SegmentRoutingManager] Creating new DeviceConfiguration
10:06:24.174 INFO [SegmentRoutingManager] Creating new DefaultRoutingHandler
10:06:24.208 INFO [LinkHandler] Loading stored links
10:06:24.209 INFO [SegmentRoutingManager] Started
10:06:24.246 INFO [PhasedRecoveryManager] Started
10:06:24.307 INFO [PolicyManager] Started
10:06:24.382 INFO [XconnectManager] Started
10:06:24.383 INFO [FeaturesServiceImpl] org.onosproject.segmentrouting-web/3.3.0.SNAPSHOT
10:06:24.425 INFO [HttpServiceFactoryImpl] Binding bundle: [org.onosproject.segmentrouting-web [268]] to http service
10:06:24.429 INFO [ApiDocManager] Segment Routing REST API registered at /onos/segmentrouting
10:06:24.429 INFO [FeaturesServiceImpl] Done.
10:06:24.429 INFO [ApplicationManager] Application org.onosproject.segmentrouting has been activated
10:06:24.434 INFO [ServletContainerInitializerScanner] will add org.eclipse.jetty.websocket.server.NativeWebSocketServletContainerInitializer to ServletContainerInitializers
10:06:24.434 INFO [ServletContainerInitializerScanner] added ServletContainerInitializer: org.eclipse.jetty.websocket.server.NativeWebSocketServletContainerInitializer
10:06:24.435 INFO [ServletContainerInitializerScanner] will add org.apache.jasper.servlet.JasperInitializer to ServletContainerInitializers
10:06:24.435 INFO [ServletContainerInitializerScanner] Skipt org.apache.jasper.servlet.JasperInitializer, because specialized handler will be present
10:06:24.435 INFO [ServletContainerInitializerScanner] will add org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer to ServletContainerInitializers
10:06:24.436 INFO [FeaturesServiceImpl] Adding features: fabric-tna/[1.2.0.SNAPSHOT,1.2.0.SNAPSHOT]
10:06:24.497 INFO [NeighbourResolutionManager] IPv6 neighbor discovery is disabled
10:06:24.498 INFO [NeighbourResolutionManager] Address resolution protocol is enabled
10:06:24.498 INFO [NeighbourResolutionManager] Configured. Request intercepts is disabled
10:06:24.510 INFO [HostManager] Configured. monitorHosts disabled
10:06:24.510 INFO [HostManager] Configured. probeRate 30000
10:06:24.510 INFO [HostManager] Removal of duplicate ip address is enabled
10:06:24.510 INFO [HostManager] Configured. greedyLearningIpv6 enabled
10:06:24.511 INFO [HostManager] Configured. hostMoveTrackerEnabled disabled
10:06:24.512 INFO [DhcpRelayManager] Address resolution protocol is disabled
10:06:24.515 INFO [DhcpRelayManager] DhcpRelay poll interval new 86400 old 86400
10:06:24.516 INFO [DhcpRelayManager] DhcpRelay FPM is disabled
10:06:24.519 INFO [LldpLinkProvider] Settings: enabled=true, useBDDP=true, probeRate=3000, staleLinkAge=15000, maxLLDPage=1000, useStaleLinkAge=true
10:06:24.520 INFO [FlowRuleManager] Configured. AllowExtraneousRules is disabled
10:06:24.520 INFO [FlowRuleManager] Configured. importExtraneousRules is disabled
10:06:24.520 INFO [FlowRuleManager] Configured. PurgeOnDisconnection is disabled
10:06:24.520 INFO [FlowRuleManager] Configured. FallbackFlowPollFrequency is 4 seconds
10:06:24.523 INFO [HostLocationProvider] Host removal on port/device down events is not configured, using current value of true
10:06:24.523 INFO [HostLocationProvider] Configured. Using ARP is enabled
10:06:24.523 INFO [HostLocationProvider] Configured. Using IPv6 NDP Neighbor Solicitation and Advertisement is disabled
10:06:24.523 INFO [HostLocationProvider] Configured. Using IPv6 NDP Router Solicitation and Advertisement is disabled
10:06:24.523 INFO [HostLocationProvider] Configured. Using DHCP is disabled
10:06:24.524 INFO [HostLocationProvider] Configured. Using DHCP6 is disabled
10:06:24.524 INFO [HostLocationProvider] Configured. Request intercepts is disabled
10:06:24.524 INFO [HostLocationProvider] Configured. Multihoming is enabled
10:06:24.534 INFO [MeterManager] Configured. PurgeOnDisconnection is disabled
10:06:24.535 INFO [MeterManager] UserDefinedIndex is disabled
10:06:24.537 INFO [GroupManager] Configured. PurgeOnDisconnection is disabled
10:06:24.580 INFO [ServletContainerInitializerScanner] added ServletContainerInitializer: org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer
10:06:24.580 INFO [HttpServiceContext] registering context WebAppHttpContext{org.onosproject.segmentrouting-web - 268}, with context-name: onos/segmentrouting
10:06:24.584 INFO [HttpServiceContext] registering JasperInitializer
10:06:24.585 INFO [WebSocketServerFactory] No DecoratedObjectFactory provided, using new org.eclipse.jetty.util.DecoratedObjectFactory[decorators=1]
10:06:24.590 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/c-1_0-rt.tld
10:06:24.593 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/c-1_0.tld
10:06:24.595 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/c-1_1.tld
10:06:24.596 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/c.tld
10:06:24.598 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/fmt-1_0-rt.tld
10:06:24.600 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/fmt-1_0.tld
10:06:24.602 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/fmt.tld
10:06:24.604 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/fn.tld
10:06:24.604 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/permittedTaglibs.tld
10:06:24.605 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/scriptfree.tld
10:06:24.606 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/sql-1_0-rt.tld
10:06:24.607 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/sql-1_0.tld
10:06:24.608 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/sql.tld
10:06:24.609 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/x-1_0-rt.tld
10:06:24.611 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/x-1_0.tld
10:06:24.612 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/x.tld
10:06:24.649 INFO [ContextHandler] Started HttpServiceContext{httpContext=WebAppHttpContext{org.onosproject.segmentrouting-web - 268}}
10:06:25.089 INFO [FeaturesServiceImpl] Changes to perform:
10:06:25.090 INFO [FeaturesServiceImpl] Region: root
10:06:25.090 INFO [FeaturesServiceImpl] Bundles to install:
10:06:25.090 INFO [FeaturesServiceImpl] mvn:org.stratumproject/fabric-tna/1.2.0-SNAPSHOT
10:06:25.091 INFO [FeaturesServiceImpl] Installing bundles:
10:06:25.091 INFO [FeaturesServiceImpl] mvn:org.stratumproject/fabric-tna/1.2.0-SNAPSHOT
10:06:25.108 INFO [FeaturesServiceImpl] Starting bundles:
10:06:25.109 INFO [FeaturesServiceImpl] org.stratumproject.fabric-tna/1.2.0.SNAPSHOT
10:06:25.115 INFO [CommandExtension] Registering commands for bundle org.stratumproject.fabric-tna/1.2.0.SNAPSHOT
10:06:25.154 INFO [HttpServiceFactoryImpl] Binding bundle: [org.stratumproject.fabric-tna [269]] to http service
10:06:25.165 INFO [ServletContainerInitializerScanner] will add org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer to ServletContainerInitializers
10:06:25.338 INFO [ServletContainerInitializerScanner] added ServletContainerInitializer: org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer
10:06:25.339 INFO [ServletContainerInitializerScanner] will add org.eclipse.jetty.websocket.server.NativeWebSocketServletContainerInitializer to ServletContainerInitializers
10:06:25.339 INFO [ServletContainerInitializerScanner] added ServletContainerInitializer: org.eclipse.jetty.websocket.server.NativeWebSocketServletContainerInitializer
10:06:25.339 INFO [ServletContainerInitializerScanner] will add org.apache.jasper.servlet.JasperInitializer to ServletContainerInitializers
10:06:25.339 INFO [ServletContainerInitializerScanner] Skipt org.apache.jasper.servlet.JasperInitializer, because specialized handler will be present
10:06:25.340 INFO [HttpServiceContext] registering context WebAppHttpContext{org.stratumproject.fabric-tna - 269}, with context-name: onos/fabrictna
10:06:25.343 INFO [HttpServiceContext] registering JasperInitializer
10:06:25.344 INFO [WebSocketServerFactory] No DecoratedObjectFactory provided, using new org.eclipse.jetty.util.DecoratedObjectFactory[decorators=1]
10:06:25.348 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/c-1_0-rt.tld
10:06:25.350 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/c-1_0.tld
10:06:25.352 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/c-1_1.tld
10:06:25.353 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/c.tld
10:06:25.354 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/fmt-1_0-rt.tld
10:06:25.356 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/fmt-1_0.tld
10:06:25.358 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/fmt.tld
10:06:25.359 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/fn.tld
10:06:25.359 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/permittedTaglibs.tld
10:06:25.360 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/scriptfree.tld
10:06:25.360 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/sql-1_0-rt.tld
10:06:25.361 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/sql-1_0.tld
10:06:25.363 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/sql.tld
10:06:25.363 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/x-1_0-rt.tld
10:06:25.365 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/x-1_0.tld
10:06:25.366 INFO [TldScanner] found TLD bundle://115.0:1/META-INF/x.tld
10:06:25.391 INFO [ContextHandler] Started HttpServiceContext{httpContext=WebAppHttpContext{org.stratumproject.fabric-tna - 269}}
10:06:25.424 INFO [MeterManager] Configured. PurgeOnDisconnection is disabled
10:06:25.427 INFO [MeterManager] UserDefinedIndex is enabled
10:06:25.429 INFO [GnmiControllerImpl] Configured. readPortId is configured to true
10:06:26.166 INFO [SegmentRoutingManager] Reacting to config changes after buffer delay
10:06:26.167 INFO [DefaultRoutingHandler] Starting to populate all routing rules
10:06:26.172 WARN [DefaultRoutingHandler] Failed to repopulate all routing rules.
10:06:42.917 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-int.bmv2 (fingerprint=5e:ab:5c:73:80:ab:1c:5d)
10:06:42.917 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-int.mavericks_sde_9_7_0 (fingerprint=a5:57:71:11:11:5b:61:46)
10:06:42.917 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-int.montara_sde_9_7_0 (fingerprint=a5:57:71:11:11:5b:61:46)
10:06:42.917 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-upf-int.bmv2 (fingerprint=1f:73:8f:88:e2:e2:fc:1d)
10:06:42.917 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-upf-int.mavericks_sde_9_7_0 (fingerprint=69:b1:92:e5:41:01:c3:52)
10:06:42.918 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-upf-int.montara_sde_9_7_0 (fingerprint=69:b1:92:e5:41:01:c3:52)
10:06:42.918 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-upf.bmv2 (fingerprint=31:54:23:60:b7:db:ae:c8)
10:06:42.918 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-upf.mavericks_sde_9_7_0 (fingerprint=fa:34:cb:eb:cc:19:81:fe)
10:06:42.918 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric-upf.montara_sde_9_7_0 (fingerprint=fa:34:cb:eb:cc:19:81:fe)
10:06:42.918 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric.bmv2 (fingerprint=48:79:3a:45:d3:fe:d7:6d)
10:06:42.918 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric.mavericks_sde_9_7_0 (fingerprint=c5:34:c6:e8:b7:06:80:a5)
10:06:42.918 INFO [PiPipeconfManager] New pipeconf registered: org.stratumproject.fabric.montara_sde_9_7_0 (fingerprint=c5:34:c6:e8:b7:06:80:a5)
10:06:42.918 INFO [PipeconfLoader] Started
10:06:42.928 INFO [IntManager] Started
10:06:42.989 INFO [SlicingManager] Started
10:06:42.990 INFO [SlicingManager] Processing Default TC event MapEvent{name=fabric-tna-default-tc, type=INSERT, key=0, newValue=Versioned{value=BEST_EFFORT, version=1130, creationTime=2022-08-18T10:06:42.987Z}, oldValue=null}
10:06:42.991 INFO [SlicingManager] Processing slice event MapEvent{name=fabric-tna-slice, type=INSERT, key=SliceStoreKey{sliceId=0, trafficClass=BEST_EFFORT}, newValue=Versioned{value=TrafficClassDescription{tc=BEST_EFFORT, qid=0, maxRateBps=UNLIMITED, gminRateBps=0, isSystemTc=false}, version=1129, creationTime=2022-08-18T10:06:42.986Z}, oldValue=null}
10:06:42.992 INFO [NetcfgSlicingProvider] Started
10:06:43.028 INFO [StatisticManager] Started
10:06:43.048 INFO [HighlightManager] Started
10:06:43.050 INFO [ApiDocManager] Fabric-tna REST API registered at /onos/fabrictna
10:06:43.050 INFO [FeaturesServiceImpl] Done.
10:06:43.051 INFO [ApplicationManager] Application org.stratumproject.fabric-tna has been activated
10:06:50.300 ERROR [Tools] No such component
10:06:55.318 ERROR [Tools] No such component
10:07:00.329 ERROR [Tools] No such component
10:07:05.338 ERROR [Tools] No such component
10:07:10.348 ERROR [Tools] No such component
10:07:15.357 ERROR [Tools] No such component
10:07:20.367 ERROR [Tools] No such component
10:07:25.376 ERROR [Tools] No such component
10:07:30.386 ERROR [Tools] No such component
10:07:31.625 INFO [OpenSSHKeyPairProvider] Creating ssh server private key at /root/onos/apache-karaf-4.2.14/etc/host.key
10:07:31.628 INFO [OpenSSHKeyPairGenerator] generateKeyPair(RSA) generating host key - size=2048
10:07:35.395 ERROR [Tools] No such component
10:07:40.405 ERROR [Tools] No such component
10:07:45.413 ERROR [Tools] No such component
10:07:50.424 ERROR [Tools] No such component
10:07:53.204 INFO [ServerSessionImpl] Session karaf@/127.0.0.1:54664 authenticated
10:07:55.434 ERROR [Tools] No such component
10:08:00.443 ERROR [Tools] No such component
karaf@root log>
- [Charles] Do you see a Stratum daemonset being deployed on K8s? Can you share the output of kubectl describe of that daemonset? Typically it will tell you why a pod can’t be scheduled at the bottom.
No. There is no attempt to build anything on the switches. Describe of the DS:
root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl describe ds -n sdfabric
Name: stratum
Selector: name=stratum
Node-Selector: node-role.aetherproject.org=switch,node-role.kubernetes.io=switch
Labels: app.kubernetes.io/managed-by=Helm
Annotations: deprecated.daemonset.template.generation: 1
meta.helm.sh/release-name: sdfabric
meta.helm.sh/release-namespace: sdfabric
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Scheduled with Up-to-date Pods: 0
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: name=stratum
Annotations: fluentbit.io/parser: stratum
Init Containers:
init-huge:
Image: alpine
Port: <none>
Host Port: <none>
Command:
sh
-c
sysctl vm.nr_hugepages && sysctl vm.nr_hugepages=128
Environment: <none>
Mounts: <none>
config-getter:
Image: alpine/git
Port: <none>
Host Port: <none>
Command:
/opt/configs/stratum-config
Environment:
NODE_NAME: (v1:spec.nodeName)
CFG_MOUNT: /config
Mounts:
/config from config-dir (rw)
/opt/chassis_config/ from chassis-config (rw)
/opt/configs from configmap-config (rw)
Containers:
stratum-bfrt:
Image: docker.io/stratumproject/stratum-bfrt:22.08.12-9.7.0
Port: <none>
Host Port: <none>
Command:
/usr/bin/stratum-startup
Environment:
CFG_MOUNT: /config
Mounts:
/config from config-dir (rw)
/dev from dev (rw)
/etc/onl from etc-onl (rw)
/lib/modules/4.14.49-OpenNetworkLinux from lib-modules (rw)
/sys from sys (rw)
/usr/bin/stratum-startup from configmap-startup (rw,path="stratum-startup")
/var/log/stratum from log (rw)
Volumes:
dev:
Type: HostPath (bare host directory volume)
Path: /dev
HostPathType:
sys:
Type: HostPath (bare host directory volume)
Path: /sys
HostPathType:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules/4.14.49-OpenNetworkLinux
HostPathType:
log:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
etc-onl:
Type: HostPath (bare host directory volume)
Path: /etc/onl
HostPathType:
config-dir:
Type: HostPath (bare host directory volume)
Path: /config
HostPathType:
configmap-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: stratum-config-scripts
Optional: false
chassis-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: stratum-chassis-configs
Optional: false
configmap-startup:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: stratum-startup-scripts
Optional: false
Events: <none>
root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric#
Let me know if there is anything else that would help pinpoint these issues.
Thanks
David
sdfabr...@opennetworking.org <sdfabr...@opennetworking.org> on behalf of Chan, Charles <charle...@intel.com>
Date: Thursday, 18 August 2022 at 10:40
To: Lake, David (PG/R - Comp Sci & Elec Eng) <d.l...@surrey.ac.uk>
Cc: Cascone, Carmelo <carmelo...@intel.com>, sdfabr...@opennetworking.org <sdfabr...@opennetworking.org>
Subject: [sdfabric-dev] Re: Multiple Issues with SD-Fabric Installation
--
You received this message because you are subscribed to the Google Groups "SDFABRIC-Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
sdfabric-dev...@opennetworking.org.
To view this discussion on the web visit
https://groups.google.com/a/opennetworking.org/d/msgid/sdfabric-dev/BN6PR1101MB2177FC5733C9874C794B2BA7876D9%40BN6PR1101MB2177.namprd11.prod.outlook.com.
David Lake
Charles
Do you have any update as to what these faults are please and how I can overcome them?
Thank you
David
David Lake
Charles
I’ve had no update from the installation issues that I have raised and no communication from anyone on the mailing list or in the development team.
Can I have some assistance please.
David
.
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/sdfabric-dev/DB7PR06MB47922E678C0BE68C633165C6B5709%40DB7PR06MB4792.eurprd06.prod.outlook.com.
Chan, Charles
Hi David,
Our team has been focusing on an internal deliverable these days so please understand that there could be delays in our responses.
Please find the rest of my comments below.
Thanks.
Charles Chan, PhD (he/him)
Cloud Software Development Engineer
Software Product Incubation | Network and Edge Group
Intel Corporation | intel.com
From: 'David Lake' via SDFABRIC-Dev <sdfabr...@opennetworking.org>
Date: Thursday, August 18, 2022 at 03:41
To: Chan, Charles <charle...@intel.com>
Cc: Cascone, Carmelo <carmelo...@intel.com>, sdfabr...@opennetworking.org <sdfabr...@opennetworking.org>
Subject: [sdfabric-dev] Re: Multiple Issues with SD-Fabric Installation
Charles
Thanks for the response.
1) [Charles] Did you have pfcp-agent started in the same K8s namespace as ONOS? Can you share the dump of kubectl pods and services?
[Charles] Looks like PFCP agent isn’t started correctly. Please try kubectl describe / kubectl logs on the sdfabric-pfcp-agent-0 pod and see if you can get more information.
2) I haven’t seen this before. Can you attach the full ONOS log from the very beginning?
[Charles] This is expected. It is not able to reach ONOS since ONOS is not fully ready. It should be gone once the ONOS issue is resolved
[Charles] Hmm… this is indeed weird. Can you show the output of kubectl describe of one of the ONOS pod? I’d like to confirm which sdfabric-onos image you are using.
karaf@root log>
3) [Charles] Do you see a Stratum daemonset being deployed on K8s? Can you share the output of kubectl describe of that daemonset? Typically it will tell you why a pod can’t be scheduled at the bottom.
[Charles] This doesn’t seem right. The “Desired Number of Nodes Scheduled” should equal to the number of switch you have.
Can you show the output of kubectl get nodes?
.
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/sdfabric-dev/DB7PR06MB479216DDB9A735AD12691B58B56D9%40DB7PR06MB4792.eurprd06.prod.outlook.com.
David Lake
- Kubectl describe sdfabric-pfcp-agent-0 -n sdfabric gives no result.
- Kubectl logs sdfabric-pfcp-agent-0 -n sdfabric shows the error I detailed before:
- Kubectl get nodes shows:
Sent: Wednesday, August 24, 2022 7:36:58 PM
David Lake
Charles and all
I have been looking at this issue now for many weeks and provided the logs you asked for.
- The error with the pfcp-agent is that it is asking for a DNS resolution for ‘pfcp-agent.’ There is a comment in the code that this needs to be ‘fixed with CI’ which seems to me to mean that the team knows there is a problem here but has not documented it. If I manually add an entry for ‘pfcp-agent’ in CoreDNS then pfcp-agent progresses beyond this point but in the absence of documentation, I do not know if this is correct thing to be doing.
- Netcfg is still failing with the error 404 when the ONOS Configuration Loader tries to load the config into ONOS. Again, in the absence of documentation, it is impossible for me to fault-find.
I would appreciate some help from the wider ONF team.
Thank you
David
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/sdfabric-dev/DB7PR06MB47927249EC102FCA3768B616B5739%40DB7PR06MB4792.eurprd06.prod.outlook.com.
Chan, Charles
Hi David,
(1) I am adding Hungwei to the list. I recall he had seen a similar issue in the past so hopefully he can provide some guidance.
(2) Can you provide the digest of the sdfabric-onos image you deployed? The
latest release uses 1.1.1 image, which should have
sha256:523b981405857c008c1dd6ca348b4239c38fe15ceb0f376f6d00c1a9990b4a3e
as the image digest.
Thanks,
Charles Chan, PhD (he/him)
Cloud Software Development Engineer
Software Product Incubation | Network and Edge Group
Intel Corporation | intel.com
From: David Lake <d.l...@surrey.ac.uk>
Date: Friday, September 2, 2022 at 05:26
To: Chan, Charles <charle...@intel.com>
Cc: Cascone, Carmelo <carmelo...@intel.com>, sdfabr...@opennetworking.org <sdfabr...@opennetworking.org>
Subject: Re: Multiple Issues with SD-Fabric Installation
Charles and all
I have been looking at this issue now for many weeks and provided the logs you asked for.
1) The error with the pfcp-agent is that it is asking for a DNS resolution for ‘pfcp-agent.’ There is a comment in the code that this needs to be ‘fixed with CI’ which seems to me to mean that the team knows there is a problem here but has not documented it. If I manually add an entry for ‘pfcp-agent’ in CoreDNS then pfcp-agent progresses beyond this point but in the absence of documentation, I do not know if this is correct thing to be doing.
2) Netcfg is still failing with the error 404 when the ONOS Configuration Loader tries to load the config into ONOS. Again, in the absence of documentation, it is impossible for me to fault-find.
David Lake
Hi Charles
- Sdfabric-onos is set in my yaml file:
onos-classic:
image:
repository: opennetworking/sdfabric-onos
tag: 1.1.1
- I’ve ‘exec-ed’ to the container running the config-loader program and I can see that the 404 error is because config-loader is checking for http://sdfabric-onos-classic-hs:8181/onos/v1/network/configuration/netcfg
This does not exist in ONOS. I’ve run up a standalone version of ONOS and verified that a standard netcfg ‘POST’ of a json config works but that this netcfg is not reported.
This is the response to a curl from the config-loader container for general configuration and for configuration/netcfg:
bash-5.0# curl -sSL -u karaf:karaf -X GET http://sdfabric-onos-classic-hs:8181/onos/v1/configuration
{"org.onosproject.net.host.impl.HostManager":{"hostMoveTrackerEnabled":"false","offendingHostExpiryInMins":"1","probeRate":"30000","hostMoveThresholdInMillis":"200000","monitorHosts":"false","allowDuplicateIps":"false","greedyLearningIpv6":"true","offendingHostClearThreadPool":"10","hostMoveCounter":"3"},"org.onosproject.dhcprelay.Dhcp4HandlerImpl":{"learnRouteFromLeasequery":"false"},"org.onosproject.store.flow.impl.ECFlowRuleStore":{"antiEntropyPeriod":"5000","persistenceEnabled":"false","msgHandlerPoolSize":"8","backupPeriod":"2000","backupCount":"2"},"org.onosproject.grpc.ctl.GrpcChannelControllerImpl":{"enableMessageLog":"false"},"org.onosproject.cluster.impl.MastershipManager":{"useRegionForBalanceRoles":"false","rebalanceRolesOnUpgrade":"true"},"org.onosproject.net.flowobjective.impl.FlowObjectiveManager":{"accumulatorMaxObjectives":"1000","accumulatorMaxBatchMillis":"500","numThreads":"4","accumulatorMaxIdleMillis":"10"},"org.onosproject.net.intent.impl.IntentCleanup":{"period":"5","enabled":"true","retryThreshold":"5"},"org.onosproject.dhcprelay.DhcpRelayManager":{"dhcpPollInterval":"86400","dhcpFpmEnabled":"false","dhcpHostRelearnProbeCount":"3","dhcpHostRelearnProbeInterval":"500","arpEnabled":"false"},"org.onosproject.net.flow.impl.FlowRuleManager":{"importExtraneousRules":"false","fallbackFlowPollFrequency":"4","purgeOnDisconnection":"false","allowExtraneousRules":"false"},"org.onosproject.store.topology.impl.DistributedTopologyStore":{"linkWeightFunction":"hopCount","maxPaths":"-1"},"org.onosproject.routeservice.store.RouteStoreImpl":{"distributed":"true"},"org.onosproject.net.pi.impl.PiPipeconfWatchdogManager":{"probeInterval":"15"},"org.omecproject.up4.impl.Up4DeviceManager":{"upfReconcileInterval":"30"},"org.onosproject.net.intent.impl.compiler.IntentConfigurableRegistrator":{"labelSelection":"RANDOM","useFlowObjectives":"false","useCopyTtl":"false","optLabelSelection":"NONE","optimizeInstructions":"false"},"org.onosproject.provider.general.device.impl.GeneralDeviceProvider":{"checkupInterval":"10","statsPollInterval":"10"},"org.onosproject.store.cluster.impl.DistributedLeadershipStore":{"electionTimeoutMillis":"2500"},"org.onosproject.net.neighbour.impl.NeighbourResolutionManager":{"ndpEnabled":"false","requestInterceptsEnabled":"false","arpEnabled":"true"},"org.onosproject.segmentrouting.SegmentRoutingManager":{"routeSimplification":"false","routeDoubleTaggedHosts":"false","respondToUnknownHosts":"true","defaultInternalVlan":"4094","pwTransportVlan":"4090","activeProbing":"true","singleHomedDown":"false","symmetricProbing":"false"},"org.onosproject.store.packet.impl.DistributedPacketStore":{"messageHandlerThreadPoolSize":"4"},"org.onosproject.dhcprelay.Dhcp6HandlerImpl":{"learnRouteFromLeasequery":"false"},"org.onosproject.store.intent.impl.GossipIntentStore":{"persistenceEnabled":"false"},"org.onosproject.ui.impl.UiExtensionManager":{"trafficRefreshMs":"5000"},"org.onosproject.net.topology.impl.DefaultTopologyProvider":{"maxBatchMs":"50","maxEvents":"1000","maxIdleMs":"10"},"org.onosproject.net.intent.impl.IntentManager":{"numThreads":"12","skipReleaseResourcesOnWithdrawal":"false"},"org.onosproject.routing.fpm.FpmManager":{"pdPushNextHopIPv4":"","clearRoutes":"true","pdPushEnabled":"false","pdPushNextHopIPv6":""},"org.onosproject.store.statistic.impl.DistributedFlowStatisticStore":{"messageHandlerThreadPoolSize":"4"},"org.onosproject.provider.p4runtime.packet.impl.P4RuntimePacketProvider":{"workers":"4"},"org.onosproject.segmentrouting.phasedrecovery.impl.PhasedRecoveryManager":{"phasedRecovery":"false"},"org.onosproject.provider.lldp.impl.LldpLinkProvider":{"probeRate":"3000","maxDiscoveryDelayMs":"1000","enabled":"true","useBddp":"true","useStaleLinkAge":"true","staleLinkAge":"15000"},"org.onosproject.provider.host.impl.HostLocationProvider":{"useDhcp6":"false","requestArp":"true","hostRemovalEnabled":"true:Boolean=true","requestIpv6ND":"false","requestIpv6NdpRsRa":"false","requestInterceptsEnabled":"false","multihomingEnabled":"true","useDhcp":"false"},"org.onosproject.net.driver.impl.DriverRegistryManager":{"requiredDrivers":"default"},"org.onosproject.net.meter.impl.MeterManager":{"fallbackMeterPollFrequency":"3","numThreads":"12","userDefinedIndex":"true","purgeOnDisconnection":"false"},"org.onosproject.net.flowobjective.impl.InOrderFlowObjectiveManager":{"objectiveTimeoutMs":"15000","accumulatorMaxObjectives":"1000","accumulatorMaxBatchMillis":"500","numThreads":"4","accumulatorMaxIdleMillis":"10"},"org.onosproject.audit.impl.AuditManager":{"auditEnabled":"false","auditLogger":"securityAudit"},"org.onosproject.store.statistic.impl.DistributedStatisticStore":{"messageHandlerThreadPoolSize":"4"},"org.onosproject.store.group.impl.DistributedGroupStore":{"allowExtraneousGroups":"false","garbageCollect":"false","gcThresh":"6"},"org.onosproject.core.impl.CoreManager":{"maxEventTimeLimit":"2000","sharedThreadPerformanceCheck":"false","sharedThreadPoolSize":"30"},"org.onosproject.gnmi.ctl.GnmiControllerImpl":{"readPortId":"true"},"org.onosproject.net.intent.impl.installer.FlowRuleIntentInstaller":{"nonDisruptiveInstallationWaitingTime":"1"},"org.onosproject.net.group.impl.GroupManager":{"fallbackGroupPollFrequency":"3","purgeOnDisconnection":"false"}}
bash-5.0# curl -sSL -u karaf:karaf -X GET http://sdfabric-onos-classic-hs:8181/onos/v1/configuration/netcfg
{"code":404,"message":"No such component"}bash-5.0#
David
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/sdfabric-dev/BN6PR1101MB217770D161444A0F94F76EDE877D9%40BN6PR1101MB2177.namprd11.prod.outlook.com.
Chan, Charles
The URL in (2) seems incorrect. The network config endpoint should look like:
http://sdfabric-onos-classic-hs:8181/onos/v1/network/configuration/
(without the “netcfg” suffix)
That netcfg endpoint is specified here as part of the helm chart:
onos-classic/templates/onos-config-loader-script.yaml Line 36.
But it can be altered by overriding the NETCFG_URL in helm value. Can you check if that’s what happened?
Thanks,
Charles Chan, PhD (he/him)
Cloud Software Development Engineer
Software Product Incubation | Network and Edge Group
Intel Corporation | intel.com
From: David Lake <d.l...@surrey.ac.uk>
Date: Saturday, September 3, 2022 at 00:48
To: Chan, Charles <charle...@intel.com>
Cc: Cascone, Carmelo <carmelo...@intel.com>, sdfabr...@opennetworking.org <sdfabr...@opennetworking.org>, Chiu, HungWei <hungwe...@intel.com>
Subject: Re: Multiple Issues with SD-Fabric Installation
Hi Charles
1) Sdfabric-onos is set in my yaml file:
onos-classic:
image:
repository: opennetworking/sdfabric-onos
tag: 1.1.1
2) I’ve ‘exec-ed’ to the container running the config-loader program and I can see that the 404 error is because config-loader is checking for http://sdfabric-onos-classic-hs:8181/onos/v1/network/configuration/netcfg
David Lake
Charles
I don’t have a file onos-config-loader-script.yaml anywhere on my system.
The end of the URL is read in loader.sh running in the onos-loader container:
CFGS=$(ls /opt/configs | grep -v netcfg.json);
for CFG in ${CFGS};
do
CFG_FILE="/opt/configs/$CFG"
echo "-----------------------------------------"
echo "Check that component $CFG is active";
echo "-----------------------------------------"
until curl --fail -sSL -u $USER:$PASSWORD -X GET "${COMPONENT_CFG_URL}/$CFG";
do
echo "Waiting for $CFG to be active";
sleep 5;
done
The URL used for testing to see if ONOS is ready to accept netconf is COMPONENT_CFG_URL which is derived from the yaml file. The netcfg is tagged onto the end by parsing the files in /opt/configs. The file ‘netcfg’ is generated from parsing the yaml file. If I look at /opt/configs/netcfg it is the exact json entries in my yaml file.
I’ve tried to modify the loader.sh file directly in the container but the filesystem is read only…
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/sdfabric-dev/BN6PR1101MB2177DA8CA7F6A6944716B35A877D9%40BN6PR1101MB2177.namprd11.prod.outlook.com.
David Lake
There is something VERY wrong with the onos-classic-loader logic.
Loader.sh is pulling ‘netcfg’ as $CFG from the parsing by Helm of the yaml file.
It then looks for http://host:8181/onos/v1/network/configuration/netcfg to prove that the netcfg service is active but this is incorrect.
The script is also using the same URL to download the currently running config which is also incorrect.
So the reconciliation loop is not going to work given this logic.
I think that the currently-running network configuration can be obtained by http://host:8181/onos/v1/network/configuration
The only way to run the loader is to disable the reconciliation mode because that is using the correct URL for netcfg and Helm parses the config to /opt/configs/netcfg.json on the onos-classic-loader container.
I’ll spend some time trying to work out how to fix this once I understand the logic more.
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/sdfabric-dev/DB7PR06MB4792EDD7C5432FB5B4F18A20B57C9%40DB7PR06MB4792.eurprd06.prod.outlook.com.
David Lake
Charles
I’m giving up trying to debug the loader.sh process for now – the URLs and logic in the process are very wrong so I’m not sure how anyone has ever managed to get this to work.
Next problem:
- I’ve sent the Labels and Taints exactly as per the website instructions, verified this with kubectl and made sure that the NodeSelector and NoExecute are correct but there is still no attempt to load the stratum pod onto the switches
- Later today I’m going to break this out to a standalone yaml file and try to load the pod manually using kubectl
My question:
- Is the stratum pod initiated as part of the Helm deployment – in other words should I see this active on the switches if I run ‘kubectl get pods -n sdfabric’ ? I don’t see this but I do see the two switches as worker nodes with correct Taints and Labels:
Name: 5g1
Roles: worker
Labels: beta.kubernetes.io/arch=amd64
node-role.kubernetes.io=switch
node-role.kubernetes.io/worker=worker
Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"be:61:7e:8f:41:30"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 10.5.23.21
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Fri, 01 Jul 2022 10:07:47 +0000
Taints: node-role.kubernetes.io=switch:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: 5g1
AcquireTime: <unset>
RenewTime: Wed, 07 Sep 2022 07:07:14 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Fri, 01 Jul 2022 11:08:56 +0000 Fri, 01 Jul 2022 11:08:56 +0000 FlannelIsUp Flannel is running on this node
MemoryPressure False Wed, 07 Sep 2022 07:06:17 +0000 Fri, 01 Jul 2022 10:07:47 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 07 Sep 2022 07:06:17 +0000 Fri, 01 Jul 2022 10:07:47 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 07 Sep 2022 07:06:17 +0000 Fri, 01 Jul 2022 10:07:47 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 07 Sep 2022 07:06:17 +0000 Fri, 01 Jul 2022 10:53:25 +0000 KubeletReady kubelet is posting ready status. WARNING: CPU hardcapping unsupported
Addresses:
InternalIP: 10.5.23.21
Hostname: 5g1
Capacity:
cpu: 8
ephemeral-storage: 88235280Ki
hugepages-1Gi: 0
hugepages-2Mi: 256Mi
memory: 8046400Ki
pods: 110
Allocatable:
cpu: 8
ephemeral-storage: 81317633914
hugepages-1Gi: 0
hugepages-2Mi: 256Mi
memory: 7681856Ki
pods: 110
System Info:
Machine ID: 6ed8d19bd40f4b5fbbfa982611c6e247
System UUID: 03000200-0400-0500-0006-000700080009
Boot ID: e48121ef-8886-46d4-b7fd-0ee026f49aa7
Kernel Version: 4.14.49-OpenNetworkLinux
OS Image: Debian GNU/Linux 9 (stretch)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.15
Kubelet Version: v1.18.20
Kube-Proxy Version: v1.18.20
PodCIDR: 10.244.1.0/24
PodCIDRs: 10.244.1.0/24
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system kube-flannel-ds-fkfcc 100m (1%) 100m (1%) 50Mi (0%) 50Mi (0%) 67d
kube-system kube-proxy-cp56k 0 (0%) 0 (0%) 0 (0%) 0 (0%) 67d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 100m (1%) 100m (1%)
memory 50Mi (0%) 50Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events: <none>
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/sdfabric-dev/DB7PR06MB479245440D10BB973F25DF7FB57F9%40DB7PR06MB4792.eurprd06.prod.outlook.com.
Chan, Charles
The default toleration should be node-role.aetherproject.org=switch, not node-role.kubernetes.io=switch
See this part of the helm value: sdfabric/values.yaml - sdfabric-helm-charts - Gitiles (opencord.org)
It seems the documentation is wrong. Try this and let me know if it works.
Charles Chan, PhD (he/him)
Cloud Software Development Engineer
Software Product Incubation | Network and Edge Group
Intel Corporation | intel.com
From: David Lake <d.l...@surrey.ac.uk>
Date: Wednesday, September 7, 2022 at 00:10
To: Chan, Charles <charle...@intel.com>
Cc: Cascone, Carmelo <carmelo...@intel.com>, sdfabr...@opennetworking.org <sdfabr...@opennetworking.org>, Chiu, HungWei <hungwe...@intel.com>
Subject: Re: Multiple Issues with SD-Fabric Installation
node-role.kubernetes.io=switch
David Lake
Charles
Thank you!
That has fixed the Stratum issue and I now see the pod deployed correctly. I’m going to take a look at the config loader logic now.
David
From: Chan, Charles <charle...@intel.com>
Date: Wednesday, 7 September 2022 at 08:32
To: Lake, David (PG/R - Comp Sci & Elec Eng) <d.l...@surrey.ac.uk>
Cc: Cascone, Carmelo <carmelo...@intel.com>, sdfabr...@opennetworking.org <sdfabr...@opennetworking.org>, Chiu, HungWei <hungwe...@intel.com>
Subject: Re: Multiple Issues with SD-Fabric Installation