kube-dns not able to connect to kubernetes service

[Manoj Khotele]

Hello,

I have a kubernetes v 1.3.4 cluster created on aws. It was running for more than 90 days.

And now suddenly it has started complaining. It could be that somebody enforced restarting by deleting the pod.

E1109 05:54:44.906919       1 reflector.go:216] pkg/dns/dns.go:154: Failed to list *api.Endpoints: Get https://10.0.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.0.0.1:443: i/o timeout

E1109 05:54:44.907207       1 reflector.go:216] pkg/dns/dns.go:155: Failed to list *api.Service: Get https://10.0.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.0.0.1:443: i/o timeout

Though, I can reach to kubernetes service successfully from outside the k8s-cluster using basic authentication.

What is going wrong? How can i recover from this problem?

Best regards,

Manoj

[Brandon Philips]

Did you try deleting the pod? Can you try launching another pod from the same box and see if you can reach the API server at that IP? I forget if kube-dns has bash but something like this: http://kubernetes.io/docs/user-guide/getting-into-containers/

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Manoj Khotele]

I had restarted the DNS pod, but of no use.

Finally I ended up deleting the cluster and recreating another one.

Fortunately this was dev cluster. Now I am evaluating and preparing a plan to migrate the deployments to new cluster without any downtime and without loosing data of statefull application. 

Any tips for this goal are most welcome.

BR, Manoj

You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/1XabiMWRvyQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.

[Brandon Philips]

OK, well please file an issue and cc this thread if it happens again. There wasn't enough details in your original report to really have a hunch about the issue. Next time please include as many of these details as you can: https://github.com/kubernetes/kubernetes/blob/master/.github/ISSUE_TEMPLATE.md 

[Manoj Khotele]

A note for those who may face such issue again.

I am not sure, but, it could be that, different team members were sshing into the minions and performing some tests, which includes installing some software.

I am wondering if this could have corrupted the setup. ideally it should not because of the way k8s uses manifests.

But to avoid such things in future, we have asked team members to avoid sshing into minions and do anything which can change the setup there.

BR,

Manoj