KubeVirt on rPi4 demo issues

Mazzystr

unread,

Oct 12, 2021, 3:05:36 PM10/12/21

to kubevirt-dev

Hi everyone,

We are in emergency mode for the demo for Kubevirt on rPi. Our demo is in shambles with KubeVirt components crashlooping.

We see things crashlooping

kubectl get all -A | egrep "NAMESPACE|kube"

NAMESPACE NAME READY STATUS RESTARTS AGE
kubevirt pod/virt-api-db7f949b4-596m9 0/1 CrashLoopBackOff 99 (2m59s ago) 9h
kubevirt pod/virt-controller-7b7d99ff5-44tc4 0/1 CrashLoopBackOff 137 (4m39s ago) 9h
kubevirt pod/virt-handler-44zq6 0/1 CrashLoopBackOff 111 (3m27s ago) 9h
kubevirt pod/virt-handler-wqlsd 0/1 CrashLoopBackOff 111 (115s ago) 9h
kubevirt pod/virt-operator-55bbc8984f-9h4xq 0/1 CrashLoopBackOff 99 (3m50s ago) 9h
kubevirt pod/virt-operator-55bbc8984f-md8jx 0/1 CrashLoopBackOff 99 (4m1s ago) 9h
kubevirt pod/virt-operator-55bbc8984f-vfvwv 0/1 Terminating 0 20h

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kubevirt deployment.apps/virt-api 0/2 2 0 23h
kubevirt deployment.apps/virt-controller 0/2 2 0 23h
kubevirt deployment.apps/virt-operator 0/2 2 0 23h

NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/coredns-58c49ff546 2 2 2 9h
kube-system replicaset.apps/coredns-744bb8d495 0 0 0 9h
kube-system replicaset.apps/coredns-78fcd69978 0 0 0 6d23h
kubevirt replicaset.apps/virt-api-db7f949b4 2 2 0 23h
kubevirt replicaset.apps/virt-controller-7b7d99ff5 2 2 0 23h
kubevirt replicaset.apps/virt-operator-55bbc8984f 2 2 0 23h

NAMESPACE NAME AGE PHASE
kubevirt kubevirt.kubevirt.io/kubevirt 23h Deploying

We need some help debugging what's going on.

I'm happy to open up the Zoom call or we can tack on a debugging session after tomorrow's community call.

Thanks!

/Chris Callegari

Mazzystr

unread,

Oct 13, 2021, 12:58:59 PM10/13/21

to kubevirt-dev

I'm making some forward steps. I pulled one rpi out of the cluster and installed minikube k8s with KubeVirt. KubeVirt fully deploys. I am now able to launch a virtual machine.

# kubectl get all -A

NAMESPACE NAME READY STATUS RESTARTS AGE

default pod/virt-launcher-testvm-9zh5v 2/2 Running 0 18m
kube-system pod/coredns-78fcd69978-5gbjp 1/1 Running 0 65m
kube-system pod/etcd-gold1 1/1 Running 0 66m
kube-system pod/kube-apiserver-gold1 1/1 Running 0 66m
kube-system pod/kube-controller-manager-gold1 1/1 Running 0 66m
kube-system pod/kube-proxy-t9gns 1/1 Running 0 65m
kube-system pod/kube-scheduler-gold1 1/1 Running 0 66m
kube-system pod/storage-provisioner 1/1 Running 1 (45m ago) 65m
kubevirt pod/virt-api-db7f949b4-5vx9w 1/1 Running 0 53m
kubevirt pod/virt-api-db7f949b4-l72kw 1/1 Running 0 53m
kubevirt pod/virt-controller-7b7d99ff5-9m7mt 1/1 Running 0 52m
kubevirt pod/virt-controller-7b7d99ff5-hpxcd 1/1 Running 1 (45m ago) 52m
kubevirt pod/virt-handler-x6sl4 1/1 Running 0 52m
kubevirt pod/virt-operator-55bbc8984f-5dssj 1/1 Running 0 58m
kubevirt pod/virt-operator-55bbc8984f-zqkhq 1/1 Running 1 (45m ago) 58m

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 66m
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 66m
kubevirt service/kubevirt-operator-webhook ClusterIP 10.109.138.249 <none> 443/TCP 53m
kubevirt service/kubevirt-prometheus-metrics ClusterIP 10.97.246.93 <none> 443/TCP 53m
kubevirt service/virt-api ClusterIP 10.109.169.118 <none> 443/TCP 53m

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE

kube-system daemonset.apps/kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 66m
kubevirt daemonset.apps/virt-handler 1 1 1 1 1 kubernetes.io/os=linux 52m

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 1/1 1 1 66m
kubevirt deployment.apps/virt-api 2/2 2 2 53m
kubevirt deployment.apps/virt-controller 2/2 2 2 52m
kubevirt deployment.apps/virt-operator 2/2 2 2 58m

NAMESPACE NAME DESIRED CURRENT READY AGE

kube-system replicaset.apps/coredns-78fcd69978 1 1 1 65m
kubevirt replicaset.apps/virt-api-db7f949b4 2 2 2 53m
kubevirt replicaset.apps/virt-controller-7b7d99ff5 2 2 2 52m
kubevirt replicaset.apps/virt-operator-55bbc8984f 2 2 2 58m

NAMESPACE NAME AGE STATUS READY
default virtualmachine.kubevirt.io/testvm 18m Running True

NAMESPACE NAME AGE PHASE IP NODENAME READY
default virtualmachineinstance.kubevirt.io/testvm 18m Running 172.17.0.10 gold1 True

NAMESPACE NAME AGE PHASE
kubevirt kubevirt.kubevirt.io/kubevirt 53m Deployed

Next problems

* I am unable to ping the vm IP yet I can ping the docker hub ip of 172.17.0.1

* I need to compile virtctl for arm. See https://github.com/kubevirt/kubevirt/issues/3558

Mazzystr

unread,

Oct 14, 2021, 12:47:09 PM10/14/21

to kubevirt-dev

virtctl successfully compiles for ARM. The build guide is well written and works as advertised. This was the first time I built KubeVirt from scratch.

Howard with ARM pinged me in slack with troubleshooting tips. I am now able to run KubeVirt on ARM and run ARM virtual machines! UEFI boot is a requirement for ARM virtual machines. UEFI boot requires 256 mb of mem or else the vm doesn't start. The strange thing is virt-laucher reported successful launch of the vm. qemu-kvm process is running on the node but the vm remains inaccessible on the network and there are no messages on the console.

Work needs to be done here to let the user know there is a vm post issue.

Ref links

[1] https://kubevirt.io/user-guide/operations/installation/#experimental-arm64-developer-builds

[2] https://github.com/kubevirt/kubevirt/issues/3558

[3] https://kubevirt.io/user-guide/operations/installation/#experimental-arm64-developer-builds

[4] https://github.com/kubevirt/kubevirt/blob/main/docs/getting-started.md

Reply all

Reply to author

Forward