Issues running SD-Fabric

David Lake

unread,

Aug 15, 2022, 7:27:56 AM8/15/22

to sdfabr...@opennetworking.org

I’m trying to run SD-Fabric as a UP4 with two EdgeCore switches and a single worker running ONOS.

My k8s nodes and taints are:

root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl get node -o custom-columns=NAME:.metadata.name,TAINT:.spec.taints

NAME TAINT

5g1 [map[effect:NoSchedule key:node-role.kubernetes.io value:switch]]

5g2 [map[effect:NoSchedule key:node-role.kubernetes.io value:switch]]

cp-node [map[effect:NoSchedule key:node-role.kubernetes.io/master]]

onos <none>

root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl get nodes

NAME STATUS ROLES AGE VERSION

5g1 Ready worker 45d v1.18.20

5g2 Ready worker 42d v1.18.20

cp-node Ready master 45d v1.18.20

onos Ready worker 44d v1.18.20

root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric#

My yaml configuration file is attached.

When I try to load the system, I get a ‘CrashLoopBackoff’ from the PFCP agent and because I never see ‘readiness’ from it, none of the other configuration is implemented:

root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl get pods -n sdfabric

NAME READY STATUS RESTARTS AGE

sdfabric-atomix-0 1/1 Running 0 4m7s

sdfabric-onos-classic-0 1/1 Running 0 4m7s

sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f 1/1 Running 0 4m7s

sdfabric-pfcp-agent-0 0/1 CrashLoopBackOff 5 4m7s

This is what I see in the log:

m26s Normal SuccessfulCreate statefulset/sdfabric-atomix create Pod sdfabric-atomix-0 in StatefulSet sdfabric-atomix successful

2m26s Normal Scheduled pod/sdfabric-pfcp-agent-0 Successfully assigned sdfabric/sdfabric-pfcp-agent-0 to onos

2m26s Normal SuccessfulCreate statefulset/sdfabric-onos-classic create Pod sdfabric-onos-classic-0 in StatefulSet sdfabric-onos-classic successful

2m26s Normal ScalingReplicaSet deployment/sdfabric-onos-classic-onos-config-loader Scaled up replica set sdfabric-onos-classic-onos-config-loader-59d64bd8dc to 1

2m26s Normal SuccessfulCreate replicaset/sdfabric-onos-classic-onos-config-loader-59d64bd8dc Created pod: sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f

2m26s Normal Scheduled pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f Successfully assigned sdfabric/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f to onos

2m26s Normal Scheduled pod/sdfabric-atomix-0 Successfully assigned sdfabric/sdfabric-atomix-0 to onos

2m26s Normal Scheduled pod/sdfabric-onos-classic-0 Successfully assigned sdfabric/sdfabric-onos-classic-0 to onos

2m26s Normal SuccessfulCreate statefulset/sdfabric-pfcp-agent create Pod sdfabric-pfcp-agent-0 in StatefulSet sdfabric-pfcp-agent successful

2m25s Normal Started pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f Started container onos-config-loader

2m25s Warning FailedMount pod/sdfabric-onos-classic-0 MountVolume.SetUp failed for volume "probe-scripts" : failed to sync configmap cache: timed out waiting for the condition

2m25s Warning FailedMount pod/sdfabric-onos-classic-0 MountVolume.SetUp failed for volume "init-scripts" : failed to sync configmap cache: timed out waiting for the condition

2m25s Warning FailedMount pod/sdfabric-onos-classic-0 MountVolume.SetUp failed for volume "logging-configmap" : failed to sync configmap cache: timed out waiting for the condition

2m25s Normal Created pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f Created container onos-config-loader

2m25s Normal Pulled pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f Container image "opencord/onos-classic-helm-utils:0.1.0" already present on machine

2m24s Normal Pulled pod/sdfabric-atomix-0 Container image "atomix/atomix:3.1.12" already present on machine

2m24s Normal Pulled pod/sdfabric-atomix-0 Container image "ubuntu:16.04" already present on machine

2m24s Normal Created pod/sdfabric-atomix-0 Created container configure

2m24s Normal Created pod/sdfabric-atomix-0 Created container atomix

2m24s Normal Started pod/sdfabric-atomix-0 Started container atomix

2m24s Normal Started pod/sdfabric-atomix-0 Started container configure

2m23s Normal Started pod/sdfabric-onos-classic-0 Started container onos-classic-init

2m23s Normal Created pod/sdfabric-onos-classic-0 Created container onos-classic-init

2m23s Normal Pulled pod/sdfabric-onos-classic-0 Container image "tutum/dnsutils:latest" already present on machine

2m22s Normal Started pod/sdfabric-onos-classic-0 Started container onos-classic-logging

2m22s Normal Created pod/sdfabric-onos-classic-0 Created container onos-classic-logging

2m22s Normal Pulled pod/sdfabric-onos-classic-0 Container image "alpine" already present on machine

2m21s Normal Started pod/sdfabric-onos-classic-0 Started container onos-classic

2m21s Normal Pulled pod/sdfabric-onos-classic-0 Container image "opennetworking/sdfabric-onos:1.1.1" already present on machine

2m21s Normal Created pod/sdfabric-onos-classic-0 Created container onos-classic

109s Warning Unhealthy pod/sdfabric-onos-classic-0 Readiness probe failed: {"id":"onos-0","ip":"10.244.2.34","tcpPort":9876,"status":"READY","lastUpdate":"1660562485597","humanReadableLastUpdate":"6s ago"}

{"message":"ready"}

{"message":"pending"}

org.onosproject.segmentrouting is not yet ready

94s Warning Unhealthy pod/sdfabric-onos-classic-0 Readiness probe failed: {"id":"onos-0","ip":"10.244.2.34","tcpPort":9876,"status":"READY","lastUpdate":"1660562485597","humanReadableLastUpdate":"20s ago"}

{"message":"ready"}

{"message":"pending"}

org.stratumproject.fabric-tna is not yet ready

90s Warning Unhealthy pod/sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f Readiness probe failed:

54s Normal Pulled pod/sdfabric-pfcp-agent-0 Container image "omecproject/upf-epc-pfcpiface:master-2f04e3a" already present on machine

54s Normal Created pod/sdfabric-pfcp-agent-0 Created container sdfabric-pfcp-agent

54s Normal Started pod/sdfabric-pfcp-agent-0 Started container sdfabric-pfcp-agent

41s Warning BackOff pod/sdfabric-pfcp-agent-0 Back-off restarting failed container

21s Normal NoPods poddisruptionbudget/sdfabric-atomix-pdb No matching pods found

Not sure what I’m doing wrong and would appreciate some help?

Thank you

David

signature_3900680105

5G & 6G Innovation Centres

Institute for Communication Systems (ICS)
University of Surrey
Guildford
GU2 7XH

myvalues.yaml.rtf

David Lake

unread,

Aug 15, 2022, 10:56:26 AM8/15/22

to sdfabr...@opennetworking.org

I think I’ve found the cause of the pfcp-agent crash but I’ve no idea how to fix it.

root@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl logs sdfabric-pfcp-agent-0 -p -n sdfabric

unable to retrieve container logs for docker://86f5fc74c403b16970b639e56b4be03d8c308d133ab2cc8b081cb29314895bfbroot@cp-node:/home/admin5g/sdfabric-helm-charts/sdfabric# kubectl logs sdfabric-pfcp-agent-0 -p -n sdfabric

time="2022-08-15T14:00:01Z" level=info msg="{Mode: AccessIface:{IfName:} CoreIface:{IfName:} CPIface:{Peers:[] UseFQDN:false NodeID:pfcp-agent HTTPPort:8080 Dnn:internet EnableUeIPAlloc:false UEIPPool:10.250.0.0/24} P4rtcIface:{SliceID:0 AccessIP:172.17.0.1/32 P4rtcServer:sdfabric-onos-classic-hs P4rtcPort:51001 QFIToTC:map[] DefaultTC:3} EnableP4rt:true EnableFlowMeasure:false SimInfo:{MaxSessions:0 StartUEIP:<nil> StartENBIP:<nil> StartAUPFIP:<nil> N6AppIP:<nil> N9AppIP:<nil> StartN3TEID: StartN9TEID:} ConnTimeout:0 ReadTimeout:15 EnableNotifyBess:false EnableEndMarker:false NotifySockAddr: EndMarkerSockAddr: LogLevel:info QciQosConfig:[] SliceMeterConfig:{N6RateBps:0 N6BurstBytes:0 N3RateBps:0 N3BurstBytes:0} MaxReqRetries:5 RespTimeout:2s EnableHBTimer:false HeartBeatInterval:}" func=main.main file="/pfcpiface/cmd/pfcpiface/main.go:37"

time="2022-08-15T14:00:01Z" level=fatal msg="Unable to resolve hostname pfcp-agent lookup pfcp-agent on 10.96.0.10:53: no such host" func=github.com/omec-project/upf-epc/pfcpiface.NewUPF file="/pfcpiface/pfcpiface/upf.go:109"

Can someone advise on this please?

Thanks

David

David Lake

unread,

Aug 15, 2022, 1:04:58 PM8/15/22

to sdfabr...@opennetworking.org

And another problem.

ONOS is running – I can ssh into it.

David

kubectl logs --namespace=sdfabric sdfabric-onos-classic-onos-config-loader-59d64bd8dc-2zr9f

Waiting for ONOS Cluster to be ready

-----------------------------------------

Loading netcfg into ONOS

-----------------------------------------

cat: can't open '/opt/configs/netcfg.json': No such file or directory

sed: can't read /opt/configs/netcfg.json: No such file or directory

Update is unnecessary since the netcfg on the ONOS cluster is up to date

-----------------------------------------

Loading component configs into ONOS

-----------------------------------------

Check that component netcfg is active

-----------------------------------------

curl: (22) The requested URL returned error: 404

Waiting for netcfg to be active

curl: (22) The requested URL returned error: 404

Waiting for netcfg to be active

curl: (22) The requested URL returned error: 404

Waiting for netcfg to be active

This just repeats

--
You received this message because you are subscribed to the Google Groups "SDFABRIC-Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sdfabric-dev...@opennetworking.org.
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/sdfabric-dev/DB7PR06MB4792F9DAB4B8FC5D9174D9B5B5689%40DB7PR06MB4792.eurprd06.prod.outlook.com.

Reply all

Reply to author

Forward