GKE: K8S Control Plane failt to reach kubectl service. Error from server: error dialing backend: dial tcp 10.15.10.25:10250: i/o timeout

[Joey Wang]

Error from server: error dialing backend: dial tcp 10.150.1.45:10250: i/o timeout

kubectl exec/logs kubctl top node

They all failed because of this. 

I've confirmed 10250 are open on all node in the cluster. 

But kubectl --1> API Server -2-> Node kubelet

2 part always IO timeout.

kubectl get po, manage the pods are all OK.

I checked the firewall rule even add 0.0.0.0/0 to node:10250 still no luck.

Please help me. I have run out of ideas. Everything is working just 10 days ago then suddenly kubectl exec is not working.

Thanks

Joey.

[Joey Wang]

Ssh tunnels are not supported from gke 1.22 onward, and konnectivity proxy is not enabled please reach out to gke support to enable konnectivity proxy on your cluster.

Maybe this the reason:

1. konnectivity is removed in June.

2. ssh tunnel is used.

3. ssh tunnel is not working and 10250 not working at 1.22.1300

4. when upgrading to 1.22.1901. konnectivity has to be enabled.

Any idea how to reach out to the gke support? 

Thanks.

[Adebisi Ibirogba]

Q: Any idea how to reach out to the gke support?

Ans: Yes, you can reach out via Public issue tracker. You can use "Create new Google Kubernetes Engine issue"

[Joey Wang]

updates:

The probelm is fixed after google Engineer enable konnectivity proxy again on the control plane/master node.