Why does the kubectl command display an exception after upgrading kubevirt?

121 views
Skip to first unread message

meteor

unread,
Dec 26, 2023, 9:04:07 PM12/26/23
to kubevirt-dev
I used the following command to upgrade kubevirt from v1.1.0 to v1.1.1, and found that the kubectl command output was abnormal after all pod updates were completed.
export RELEASE=v1.1.1
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml


root@sse-jq-111-19:~# kubectl get nodes
E1227 09:30:06.987268 1417204 memcache.go:255] couldn't get resource list for subresources.kubevirt.io/v1alpha3: the server is currently unable to handle the request
E1227 09:30:06.991328 1417204 memcache.go:255] couldn't get resource list for subresources.kubevirt.io/v1: the server is currently unable to handle the request
E1227 09:30:06.992233 1417204 memcache.go:106] couldn't get resource list for subresources.kubevirt.io/v1: the server is currently unable to handle the request
E1227 09:30:06.994871 1417204 memcache.go:106] couldn't get resource list for subresources.kubevirt.io/v1alpha3: the server is currently unable to handle the request
E1227 09:30:06.998699 1417204 memcache.go:106] couldn't get resource list for subresources.kubevirt.io/v1alpha3: the server is currently unable to handle the request
E1227 09:30:06.999630 1417204 memcache.go:106] couldn't get resource list for subresources.kubevirt.io/v1: the server is currently unable to handle the request
E1227 09:30:07.002278 1417204 memcache.go:106] couldn't get resource list for subresources.kubevirt.io/v1: the server is currently unable to handle the request
E1227 09:30:07.003110 1417204 memcache.go:106] couldn't get resource list for subresources.kubevirt.io/v1alpha3: the server is currently unable to handle the request
NAME            STATUS   ROLES           AGE   VERSION
sse-jq-111-19   Ready    control-plane   29d   v1.26.2
sse-jq-111-43   Ready    <none>          29d   v1.26.2
root@sse-jq-111-19:~#

I tried to reboot the operating system of the k8s master and the problem persisted. An attempt to upgrade kubectl's virt plugin and virtctl command did not resolve the issue.
How to solve the above problems ?

meteor

unread,
Dec 28, 2023, 4:27:03 AM12/28/23
to kubevirt-dev
I found two exceptions in apiservice, as follows:
v1.subresources.kubevirt.io            kubevirt/virt-api               False (FailedDiscoveryCheck)   37m
v1alpha3.subresources.kubevirt.io      kubevirt/virt-api               False (FailedDiscoveryCheck)   37m

Use the describe apiservice command to check the following information:
...
 Message:               failing or missing response from https://10.107.158.221:443/apis/subresources.kubevirt.io/v1: Get "https://10.107.158.221:443/apis/subresources.kubevirt.io/v1": dial tcp 10.107.158.221:443: connect: connection refused
...


I don't know if it has anything to do with my renewal of the k8s certificates.

meteor

unread,
Dec 28, 2023, 10:47:55 PM12/28/23
to kubevirt-dev
Trying various methods failed to solve the above problems. Finally, the entire kubevirt, including CDI, is cleaned, kubevirt v1.1.0 and CDI are redeployed, and when all components are Ready, it is upgraded to v1.1.1 again. The above abnormal phenomenon has not been repeated. 
Then run the kubeadm certs renew all command to renew the certificate and restart the OS. The above abnormal phenomenon has not been repeated. Although the problem was solved, the cause was still unknown.

Or Mergi

unread,
Mar 18, 2024, 6:50:46 AM3/18/24
to meteor, kubevirt-dev
Hi, just stumbled in this email

FWIW, last time I had similar issue with Kubevirt API services, I had to configure Kubevirt nodes as follows:
sysctl fs.inotify.max_user_instances=512
sysctl fs.inotify.max_user_watches=1048576

In case you run KinD cluster:
podman network inspect kind | grep network_interface
"network_interface": "<net name>",        
sysctl net.ipv4.conf.<net name>.rp_filter=2
systemctl daemon-reload


--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/92c6c19e-8d5a-4139-8e8e-7900b44a1b1dn%40googlegroups.com.


--
Or Mergi
Reply all
Reply to author
Forward
0 new messages