I’m trying to run SD-Fabric as a UP4 with two EdgeCore switches and a single worker running ONOS.
My k8s nodes and taints are:
root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl get node -o custom-columns=NAME:.metadata.name,TAINT:.spec.taints
NAME TAINT
5g1 [map[effect:NoSchedule key:node-role.kubernetes.io value:switch]]
5g2 [map[effect:NoSchedule key:node-role.kubernetes.io value:switch]]
cp-node [map[effect:NoSchedule key:node-role.kubernetes.io/master]]
onos <none>
root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl get nodes
NAME STATUS ROLES AGE VERSION
5g1 Ready worker 45d v1.18.20
5g2 Ready worker 42d v1.18.20
cp-node Ready master 45d v1.18.20
onos Ready worker 44d v1.18.20
root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric#
My yaml configuration file is attached.
When I try to load the system, I get a ‘CrashLoopBackoff’ from the PFCP agent and because I never see ‘readiness’ from it, none of the other configuration is implemented:
root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl get pods -n sdfabric
NAME READY STATUS RESTARTS AGE
sdfabric-atomix-0 1/1 Running 0 4m7s
sdfabric-onos-classic-0 1/1 Running 0 4m7s
sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f 1/1 Running 0 4m7s
sdfabric-pfcp-agent-0 0/1 CrashLoopBackOff 5 4m7s
This is what I see in the log:
m26s Normal SuccessfulCreate statefulset/sdfabric-atomix create Pod sdfabric-atomix-0 in StatefulSet sdfabric-atomix successful
2m26s Normal Scheduled pod/sdfabric-pfcp-agent-0 Successfully assigned sdfabric/sdfabric-pfcp-agent-0 to onos
2m26s Normal SuccessfulCreate statefulset/sdfabric-onos-classic create Pod sdfabric-onos-classic-0 in StatefulSet sdfabric-onos-classic successful
2m26s Normal ScalingReplicaSet deployment/sdfabric-onos-classic-onos-config-loader Scaled up replica set sdfabric-onos-classic-onos-config-loader-59d64bd8dc to 1
2m26s Normal SuccessfulCreate replicaset/sdfabric-onos-classic-onos-config-loader-59d64bd8dc Created pod: sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f
2m26s Normal Scheduled pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f Successfully assigned sdfabric/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f to onos
2m26s Normal Scheduled pod/sdfabric-atomix-0 Successfully assigned sdfabric/sdfabric-atomix-0 to onos
2m26s Normal Scheduled pod/sdfabric-onos-classic-0 Successfully assigned sdfabric/sdfabric-onos-classic-0 to onos
2m26s Normal SuccessfulCreate statefulset/sdfabric-pfcp-agent create Pod sdfabric-pfcp-agent-0 in StatefulSet sdfabric-pfcp-agent successful
2m25s Normal Started pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f Started container onos-config-loader
2m25s Warning FailedMount pod/sdfabric-onos-classic-0 MountVolume.SetUp failed for volume "probe-scripts" : failed to sync configmap cache: timed out waiting for the condition
2m25s Warning FailedMount pod/sdfabric-onos-classic-0 MountVolume.SetUp failed for volume "init-scripts" : failed to sync configmap cache: timed out waiting for the condition
2m25s Warning FailedMount pod/sdfabric-onos-classic-0 MountVolume.SetUp failed for volume "logging-configmap" : failed to sync configmap cache: timed out waiting for the condition
2m25s Normal Created pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f Created container onos-config-loader
2m25s Normal Pulled pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f Container image "opencord/onos-classic-helm-utils:0.1.0" already present on machine
2m24s Normal Pulled pod/sdfabric-atomix-0 Container image "atomix/atomix:3.1.12" already present on machine
2m24s Normal Pulled pod/sdfabric-atomix-0 Container image "ubuntu:16.04" already present on machine
2m24s Normal Created pod/sdfabric-atomix-0 Created container configure
2m24s Normal Created pod/sdfabric-atomix-0 Created container atomix
2m24s Normal Started pod/sdfabric-atomix-0 Started container atomix
2m24s Normal Started pod/sdfabric-atomix-0 Started container configure
2m23s Normal Started pod/sdfabric-onos-classic-0 Started container onos-classic-init
2m23s Normal Created pod/sdfabric-onos-classic-0 Created container onos-classic-init
2m23s Normal Pulled pod/sdfabric-onos-classic-0 Container image "tutum/dnsutils:latest" already present on machine
2m22s Normal Started pod/sdfabric-onos-classic-0 Started container onos-classic-logging
2m22s Normal Created pod/sdfabric-onos-classic-0 Created container onos-classic-logging
2m22s Normal Pulled pod/sdfabric-onos-classic-0 Container image "alpine" already present on machine
2m21s Normal Started pod/sdfabric-onos-classic-0 Started container onos-classic
2m21s Normal Pulled pod/sdfabric-onos-classic-0 Container image "opennetworking/sdfabric-onos:1.1.1" already present on machine
2m21s Normal Created pod/sdfabric-onos-classic-0 Created container onos-classic
109s Warning Unhealthy pod/sdfabric-onos-classic-0 Readiness probe failed: {"id":"onos-0","ip":"10.244.2.34","tcpPort":9876,"status":"READY","lastUpdate":"1660562485597","humanReadableLastUpdate":"6s ago"}
{"message":"ready"}
{"message":"ready"}
{"message":"ready"}
{"message":"ready"}
{"message":"ready"}
{"message":"pending"}
org.onosproject.segmentrouting is not yet ready
94s Warning Unhealthy pod/sdfabric-onos-classic-0 Readiness probe failed: {"id":"onos-0","ip":"10.244.2.34","tcpPort":9876,"status":"READY","lastUpdate":"1660562485597","humanReadableLastUpdate":"20s ago"}
{"message":"ready"}
{"message":"ready"}
{"message":"ready"}
{"message":"ready"}
{"message":"ready"}
{"message":"ready"}
{"message":"ready"}
{"message":"pending"}
org.stratumproject.fabric-tna is not yet ready
90s Warning Unhealthy pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f Readiness probe failed:
54s Normal Pulled pod/sdfabric-pfcp-agent-0 Container image "omecproject/upf-epc-pfcpiface:master-2f04e3a" already present on machine
54s Normal Created pod/sdfabric-pfcp-agent-0 Created container sdfabric-pfcp-agent
54s Normal Started pod/sdfabric-pfcp-agent-0 Started container sdfabric-pfcp-agent
41s Warning BackOff pod/sdfabric-pfcp-agent-0 Back-off restarting failed container
21s Normal NoPods poddisruptionbudget/sdfabric-atomix-pdb No matching pods found
Not sure what I’m doing wrong and would appreciate some help?
Thank you
David
5G & 6G Innovation Centres
Institute for Communication Systems (ICS)
University of Surrey
Guildford
GU2 7XH
I think I’ve found the cause of the pfcp-agent crash but I’ve no idea how to fix it.
root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl logs sdfabric-pfcp-agent-0 -p -n sdfabric
unable to retrieve container logs for docker://86f5fc74c403b16970b639e56b4be03d8c308d133ab2cc8b081cb29314895bfbroot@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl logs sdfabric-pfcp-agent-0 -p -n sdfabric
time="2022-08-15T14:00:01Z" level=info msg="{Mode: AccessIface:{IfName:} CoreIface:{IfName:} CPIface:{Peers:[] UseFQDN:false NodeID:pfcp-agent HTTPPort:8080 Dnn:internet EnableUeIPAlloc:false UEIPPool:10.250.0.0/24} P4rtcIface:{SliceID:0 AccessIP:172.17.0.1/32 P4rtcServer:sdfabric-onos-classic-hs P4rtcPort:51001 QFIToTC:map[] DefaultTC:3} EnableP4rt:true EnableFlowMeasure:false SimInfo:{MaxSessions:0 StartUEIP:<nil> StartENBIP:<nil> StartAUPFIP:<nil> N6AppIP:<nil> N9AppIP:<nil> StartN3TEID: StartN9TEID:} ConnTimeout:0 ReadTimeout:15 EnableNotifyBess:false EnableEndMarker:false NotifySockAddr: EndMarkerSockAddr: LogLevel:info QciQosConfig:[] SliceMeterConfig:{N6RateBps:0 N6BurstBytes:0 N3RateBps:0 N3BurstBytes:0} MaxReqRetries:5 RespTimeout:2s EnableHBTimer:false HeartBeatInterval:}" func=main.main file="/pfcpiface/cmd/pfcpiface/main.go:37"
time="2022-08-15T14:00:01Z" level=fatal msg="Unable to resolve hostname pfcp-agent lookup pfcp-agent on 10.96.0.10:53: no such host" func=github.com/omec-project/upf-epc/pfcpiface.NewUPF file="/pfcpiface/pfcpiface/upf.go:109"
Can someone advise on this please?
Thanks
David
And another problem.
ONOS is running – I can ssh into it.
David
kubectl logs --namespace=sdfabric sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f
Waiting for ONOS Cluster to be ready
-----------------------------------------
Loading netcfg into ONOS
-----------------------------------------
cat: can't open '/opt/configs/netcfg.json': No such file or directory
sed: can't read /opt/configs/netcfg.json: No such file or directory
Update is unnecessary since the netcfg on the ONOS cluster is up to date
-----------------------------------------
Loading component configs into ONOS
-----------------------------------------
-----------------------------------------
Check that component netcfg is active
-----------------------------------------
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
curl: (22) The requested URL returned error: 404
Waiting for netcfg to be active
This just repeats
--
You received this message because you are subscribed to the Google Groups "SDFABRIC-Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
sdfabric-dev...@opennetworking.org.
To view this discussion on the web visit
https://groups.google.com/a/opennetworking.org/d/msgid/sdfabric-dev/DB7PR06MB4792F9DAB4B8FC5D9174D9B5B5689%40DB7PR06MB4792.eurprd06.prod.outlook.com.