GKE: K8S Control Plane failt to reach kubectl service. Error from server: error dialing backend: dial tcp 10.15.10.25:10250: i/o timeout

131 views
Skip to first unread message

Joey Wang

unread,
Nov 4, 2021, 1:48:42 PM11/4/21
to gce-discussion
Error from server: error dialing backend: dial tcp 10.150.1.45:10250: i/o timeout

kubectl exec/logs kubctl top node

They all failed because of this. 

I've confirmed 10250 are open on all node in the cluster. 
But kubectl --1> API Server -2-> Node kubelet

2 part always IO timeout.

kubectl get po, manage the pods are all OK.

I checked the firewall rule even add 0.0.0.0/0 to node:10250 still no luck.

Please help me. I have run out of ideas. Everything is working just 10 days ago then suddenly kubectl exec is not working.

Thanks

Joey.

Joey Wang

unread,
Nov 4, 2021, 6:16:04 PM11/4/21
to gce-discussion
Ssh tunnels are not supported from gke 1.22 onward, and konnectivity proxy is not enabled please reach out to gke support to enable konnectivity proxy on your cluster.

Screenshot 2021-11-04 at 22.06.51.png

Maybe this the reason:

1. konnectivity is removed in June.
2. ssh tunnel is used.
3. ssh tunnel is not working and 10250 not working at 1.22.1300
4. when upgrading to 1.22.1901. konnectivity has to be enabled.

Any idea how to reach out to the gke support? 

Thanks.

Adebisi Ibirogba

unread,
Nov 5, 2021, 12:14:48 PM11/5/21
to gce-discussion
Q: Any idea how to reach out to the gke support?
Ans: Yes, you can reach out via Public issue tracker. You can use "Create new Google Kubernetes Engine issue"

Joey Wang

unread,
Nov 14, 2021, 5:36:03 PM11/14/21
to gce-discussion
updates:
The probelm is fixed after google Engineer enable konnectivity proxy again on the control plane/master node.
Reply all
Reply to author
Forward
0 new messages