kubectl get nodes - NotReady

5,918 views
Skip to first unread message

brant4test

unread,
May 18, 2017, 4:17:22 PM5/18/17
to CoreOS User

Hi , Team,
What would be the reason for this error? Any suggestions? Thanks!

core@ip-10-3-0-11 ~ $ ./kubectl get nodes
NAME        STATUS                        AGE       VERSION
10.3.0.11   NotReady,SchedulingDisabled   58m       v1.6.3+coreos.0
core@ip
-10-3-0-11 ~ $ ./kubectl get pods --all-namespaces
NAMESPACE     NAME                                READY     STATUS    RESTARTS   AGE
kube
-system   kube-apiserver-10.3.0.11            1/1       Running   0          1h
kube
-system   kube-controller-manager-10.3.0.11   1/1       Running   0          1h
kube
-system   kube-proxy-10.3.0.11                1/1       Running   0          1h
kube
-system   kube-scheduler-10.3.0.11            1/1       Running   0          1h
core@ip
-10-3-0-11 ~ $ ./kubectl get svc --all-namespaces
NAMESPACE   NAME         CLUSTER
-IP   EXTERNAL-IP   PORT(S)   AGE
default     kubernetes   10.3.0.1     <none>        443/TCP   1h
core@ip
-10-3-0-11 ~ $ ./kubectl describe nodes
Name:            10.3.0.11
Role:            
Labels:            beta.kubernetes.io/arch=amd64
            beta
.kubernetes.io/os=linux
            kubernetes
.io/hostname=10.3.0.11
Annotations:        node.alpha.kubernetes.io/ttl=0
            volumes
.kubernetes.io/controller-managed-attach-detach=true
Taints:            <none>
CreationTimestamp:    Thu, 18 May 2017 19:08:49 +0000
Phase:            
Conditions:
 
Type            Status    LastHeartbeatTime            LastTransitionTime            Reason                Message
 
----            ------    -----------------            ------------------            ------                -------
 
OutOfDisk         False     Thu, 18 May 2017 20:07:56 +0000     Thu, 18 May 2017 19:08:49 +0000     KubeletHasSufficientDisk     kubelet has sufficient disk space available
 
MemoryPressure     False     Thu, 18 May 2017 20:07:56 +0000     Thu, 18 May 2017 19:08:49 +0000     KubeletHasSufficientMemory     kubelet has sufficient memory available
 
DiskPressure         False     Thu, 18 May 2017 20:07:56 +0000     Thu, 18 May 2017 19:08:49 +0000     KubeletHasNoDiskPressure     kubelet has no disk pressure
 
Ready         False     Thu, 18 May 2017 20:07:56 +0000     Thu, 18 May 2017 19:08:49 +0000     KubeletNotReady         runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:        10.3.0.11,10.3.0.11,10.3.0.11
Capacity:
 cpu
:        1
 memory
:    1018868Ki
 pods
:        110
Allocatable:
 cpu
:        1
 memory
:    916468Ki
 pods
:        110
System Info:
 
Machine ID:            d6477cfb290f4f7cb97b9f9bbee06927
 
System UUID:            EC25B3A6-51A5-299F-F2BF-8F43889AE6F7
 
Boot ID:            2ae27138-2047-4a67-b767-8e41a56fdbb8
 
Kernel Version:        4.9.24-coreos
 OS
Image:            Container Linux by CoreOS 1353.7.0 (Ladybug)
 
Operating System:        linux
 
Architecture:            amd64
 
Container Runtime Version:    docker://1.12.6
 
Kubelet Version:        v1.6.3+coreos.0
 
Kube-Proxy Version:        v1.6.3+coreos.0
ExternalID:            10.3.0.11
Non-terminated Pods:        (4 in total)
 
Namespace            Name                        CPU Requests    CPU Limits    Memory Requests    Memory Limits
 
---------            ----                        ------------    ----------    ---------------    -------------
  kube
-system            kube-apiserver-10.3.0.11            0 (0%)        0 (0%)        0 (0%)        0 (0%)
  kube
-system            kube-controller-manager-10.3.0.11        200m (20%)    0 (0%)        0 (0%)        0 (0%)
  kube
-system            kube-proxy-10.3.0.11                0 (0%)        0 (0%)        0 (0%)        0 (0%)
  kube
-system            kube-scheduler-10.3.0.11            100m (10%)    0 (0%)        0 (0%)        0 (0%)
Allocated resources:
 
(Total limits may be over 100 percent, i.e., overcommitted.)
  CPU
Requests    CPU Limits    Memory Requests    Memory Limits
 
------------    ----------    ---------------    -------------
 
300m (30%)    0 (0%)        0 (0%)        0 (0%)
Events:
 
FirstSeen    LastSeen    Count    From            SubObjectPath    Type        Reason            Message
 
---------    --------    -----    ----            -------------    --------    ------            -------
 
1h        1h        1    kubelet, 10.3.0.11            Warning        ImageGCFailed        unable to find data for container /
 
1h        1h        1    kubelet, 10.3.0.11            Normal        Starting        Starting kubelet.
 
1h        1h        1    kubelet, 10.3.0.11            Normal        NodeNotSchedulable    Node 10.3.0.11 status is now: NodeNotSchedulable
 
1h        59m        42    kubelet, 10.3.0.11            Normal        NodeHasSufficientDisk    Node 10.3.0.11 status is now: NodeHasSufficientDisk
 
1h        59m        42    kubelet, 10.3.0.11            Normal        NodeHasNoDiskPressure    Node 10.3.0.11 status is now: NodeHasNoDiskPressure
 
1h        59m        42    kubelet, 10.3.0.11            Normal        NodeHasSufficientMemory    Node 10.3.0.11 status is now: NodeHasSufficientMemory
 
44m        44m        1    kubelet, 10.3.0.11            Normal        Starting        Starting kubelet.
 
44m        44m        1    kubelet, 10.3.0.11            Warning        ImageGCFailed        unable to find data for container /
 
44m        44m        1    kubelet, 10.3.0.11            Normal        NodeNotSchedulable    Node 10.3.0.11 status is now: NodeNotSchedulable
 
44m        44m        1    kubelet, 10.3.0.11            Normal        NodeHasSufficientDisk    Node 10.3.0.11 status is now: NodeHasSufficientDisk
 
44m        44m        1    kubelet, 10.3.0.11            Normal        NodeHasNoDiskPressure    Node 10.3.0.11 status is now: NodeHasNoDiskPressure
 
44m        44m        1    kubelet, 10.3.0.11            Normal        NodeHasSufficientMemory    Node 10.3.0.11 status is now: NodeHasSufficientMemory
 
32m        32m        1    kubelet, 10.3.0.11            Normal        Starting        Starting kubelet.
 
32m        32m        1    kubelet, 10.3.0.11            Warning        ImageGCFailed        unable to find data for container /
 
32m        32m        1    kubelet, 10.3.0.11            Normal        NodeHasSufficientDisk    Node 10.3.0.11 status is now: NodeHasSufficientDisk
 
32m        32m        1    kubelet, 10.3.0.11            Normal        NodeHasSufficientMemory    Node 10.3.0.11 status is now: NodeHasSufficientMemory
 
32m        32m        1    kubelet, 10.3.0.11            Normal        NodeHasNoDiskPressure    Node 10.3.0.11 status is now: NodeHasNoDiskPressure
 
32m        32m        1    kubelet, 10.3.0.11            Normal        NodeNotSchedulable    Node 10.3.0.11 status is now: NodeNotSchedulable
 
28m        28m        1    kubelet, 10.3.0.11            Normal        NodeNotSchedulable    Node 10.3.0.11 status is now: NodeNotSchedulable
 
28m        28m        1    kubelet, 10.3.0.11            Warning        ImageGCFailed        unable to find data for container /
 
28m        28m        1    kubelet, 10.3.0.11            Normal        NodeHasSufficientDisk    Node 10.3.0.11 status is now: NodeHasSufficientDisk
 
28m        28m        1    kubelet, 10.3.0.11            Normal        NodeHasSufficientMemory    Node 10.3.0.11 status is now: NodeHasSufficientMemory
 
28m        28m        1    kubelet, 10.3.0.11            Normal        NodeHasNoDiskPressure    Node 10.3.0.11 status is now: NodeHasNoDiskPressure
 
28m        28m        1    kubelet, 10.3.0.11            Normal        Starting        Starting kubelet.
 
13m        13m        1    kubelet, 10.3.0.11            Normal        Starting        Starting kubelet.
 
13m        13m        1    kubelet, 10.3.0.11            Warning        ImageGCFailed        unable to find data for container /
 
13m        13m        1    kubelet, 10.3.0.11            Normal        NodeHasSufficientDisk    Node 10.3.0.11 status is now: NodeHasSufficientDisk
 
13m        13m        1    kubelet, 10.3.0.11            Normal        NodeHasSufficientMemory    Node 10.3.0.11 status is now: NodeHasSufficientMemory
 
13m        13m        1    kubelet, 10.3.0.11            Normal        NodeHasNoDiskPressure    Node 10.3.0.11 status is now: NodeHasNoDiskPressure
 
13m        13m        1    kubelet, 10.3.0.11            Normal        NodeNotSchedulable    Node 10.3.0.11 status is now: NodeNotSchedulable
 
51s        51s        1    kubelet, 10.3.0.11            Normal        Starting        Starting kubelet.
 
51s        51s        1    kubelet, 10.3.0.11            Warning        ImageGCFailed        unable to find data for container /
 
51s        51s        1    kubelet, 10.3.0.11            Normal        NodeHasSufficientDisk    Node 10.3.0.11 status is now: NodeHasSufficientDisk
 
51s        51s        1    kubelet, 10.3.0.11            Normal        NodeHasSufficientMemory    Node 10.3.0.11 status is now: NodeHasSufficientMemory
 
51s        51s        1    kubelet, 10.3.0.11            Normal        NodeHasNoDiskPressure    Node 10.3.0.11 status is now: NodeHasNoDiskPressure
 
51s        51s        1    kubelet, 10.3.0.11            Normal        NodeNotSchedulable    Node 10.3.0.11 status is now: NodeNotSchedulable


Kyle Brown

unread,
May 18, 2017, 4:28:37 PM5/18/17
to brant4test, CoreOS User
Seems like the error is pretty straight forward. The network is not properly configured.
```
NetworkReady
=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
```

Is this a Tectonic cluster or are you using another tool to deploy? Perhaps following other instructions? 

Cheers,
Kyle Brown

Kyle Brown

unread,
May 18, 2017, 4:55:11 PM5/18/17
to brant carrey, brant4test, CoreOS User
Did you deploy by following these instructions? 


If so, you should probably start with:

kubectl get daemonset cilium-net-controller
kubectl describe ds cilium-net-controller
kubectl get events
kubectl logs pod/<cilium-pod>

Cheers,
Kyle Brown


On Thu, May 18, 2017 at 1:43 PM brant carrey <brant...@hotmail.com> wrote:
Hi, kyle,
I'm trying to build K8s+CoreOS+Cilium stack step by step manually.

Following the Coreos + Calico (substituted with Cilium) guide step by step on coreos official website. https://coreos.com/kubernetes/docs/latest/deploy-master.html

Any clues to troubleshoot in details?

Thanks!

Best Wishes,
Brant
Reply all
Reply to author
Forward
0 new messages