Kubernetes / CentOS7 / VirtualBox - CrashLoopBackOff during start the kubernetes-panel pod

[michels...@gmail.com]

I need a help! :-|


I have created 2 virtual machines in VirtualBox, the master node and the minion node. The kubernete-panel has been installed and during pod startup, the CrashLoopBackOff error is displayed. I do not understand why this problem occurs and I would like help. Below is the information for this issue.

Let me know if you need more information.



tks!



#######################################################################################



kubectl describe pod kubernetes-dashboard-1872455951-wprxz --namespace=kube-system

#######################################################################################




Name: kubernetes-dashboard-1872455951-wprxz

Namespace: kube-system

Node: 192.168.10.212/192.168.10.212

Start Time: Thu, 09 Feb 2017 23:38:53 -0200

Labels: app=kubernetes-dashboard

pod-template-hash=1872455951

Status: Running

IP: 172.17.15.2

Controllers: ReplicaSet/kubernetes-dashboard-1872455951

Containers:

kubernetes-dashboard:

Container ID: docker://0dc7baaf145abb504a2dc02e1a81fe641ce9c42f6f0f10336da23c64a0087b47

Image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1

Image ID: docker-pullable://gcr.io/google_containers/kubernetes-dashboard-amd64@sha256:46a09eb9c611e625e7de3fcf325cf78e629d002e57dc80348e9b0638338206b5

Port: 9090/TCP

State: Waiting

Reason: CrashLoopBackOff

Last State: Terminated

Reason: Error

Exit Code: 1

Started: Mon, 20 Feb 2017 20:54:18 -0300

Finished: Mon, 20 Feb 2017 20:54:19 -0300

Ready: False

Restart Count: 100

Liveness: http-get http://:9090/ delay=30s timeout=30s period=10s #success=1 #failure=3

Volume Mounts: <none>

Environment Variables: <none>

Conditions:

Type Status

Initialized True

Ready False

PodScheduled True

No volumes.

QoS Class: BestEffort

Tolerations: <none>

Events:

FirstSeen LastSeen Count From SubobjectPath Type Reason Message

--------- -------- ----- ---- ------------- -------- ------ -------

1m 1m 1 {kubelet 192.168.10.212} spec.containers{kubernetes-dashboard} Normal Created Created container with docker id 60960478f88d; Security:[seccomp=unconfined]

1m 1m 1 {kubelet 192.168.10.212} spec.containers{kubernetes-dashboard} Normal Started Started container with docker id 60960478f88d

1m 1m 1 {kubelet 192.168.10.212} spec.containers{kubernetes-dashboard} Normal Started Started container with docker id d7de24bdfcff

1m 1m 1 {kubelet 192.168.10.212} spec.containers{kubernetes-dashboard} Normal Created Created container with docker id d7de24bdfcff; Security:[seccomp=unconfined]

1m 1m 3 {kubelet 192.168.10.212} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "kubernetes-dashboard" with CrashLoopBackOff: "Back-off 10s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-1872455951-wprxz_kube-system(ae11254d-ef31-11e6-81a6-0800273291c3)"




1m 1m 1 {kubelet 192.168.10.212} spec.containers{kubernetes-dashboard} Normal Created Created container with docker id 8565c3e46650; Security:[seccomp=unconfined]

1m 1m 1 {kubelet 192.168.10.212} spec.containers{kubernetes-dashboard} Normal Started Started container with docker id 8565c3e46650

1m 57s 2 {kubelet 192.168.10.212} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "kubernetes-dashboard" with CrashLoopBackOff: "Back-off 20s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-1872455951-wprxz_kube-system(ae11254d-ef31-11e6-81a6-0800273291c3)"




1m 44s 4 {kubelet 192.168.10.212} spec.containers{kubernetes-dashboard} Normal Pulling pulling image "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1"

1m 40s 5 {kubelet 192.168.10.212} Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.

1m 40s 4 {kubelet 192.168.10.212} spec.containers{kubernetes-dashboard} Normal Pulled Successfully pulled image "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1"

38s 38s 1 {kubelet 192.168.10.212} spec.containers{kubernetes-dashboard} Normal Started Started container with docker id 0dc7baaf145a

38s 38s 1 {kubelet 192.168.10.212} spec.containers{kubernetes-dashboard} Normal Created Created container with docker id 0dc7baaf145a; Security:[seccomp=unconfined]

1m 9s 9 {kubelet 192.168.10.212} spec.containers{kubernetes-dashboard} Warning BackOff Back-off restarting failed docker container

37s 9s 4 {kubelet 192.168.10.212} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "kubernetes-dashboard" with CrashLoopBackOff: "Back-off 40s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-1872455951-wprxz_kube-system(ae11254d-ef31-11e6-81a6-0800273291c3)"




###################################################################################################################################################################################################################################################################################################################################################################################################################################################







#######################################################################################

/etc/hosts

#######################################################################################

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6




192.168.10.201 master.host

192.168.10.211 master1.host

192.168.10.202 minion1.host

192.168.10.203 minion2.host

192.168.10.212 minionA.host




#######################################################################################







#######################################################################################

/etc/kubernetes/config (master1.host)

#######################################################################################

###

# kubernetes system config

#

# The following values are used to configure various aspects of all

# kubernetes services, including

#

# kube-apiserver.service

# kube-controller-manager.service

# kube-scheduler.service

# kubelet.service

# kube-proxy.service

# logging to stderr means we get it in the systemd journal

KUBE_LOGTOSTDERR="--logtostderr=true"




# journal message level, 0 is debug

KUBE_LOG_LEVEL="--v=0"




# Should this cluster be allowed to run privileged docker containers

KUBE_ALLOW_PRIV="--allow-privileged=false"




# How the controller-manager, scheduler, and proxy find the apiserver

KUBE_MASTER="--master=http://127.0.0.1:8080"

###################################################################################################################################################################################################################################################################################################################################################################################################################################################










#######################################################################################

/etc/kubernetes/apiserver (master1.host)

#######################################################################################

###

# kubernetes system config

#

# The following values are used to configure the kube-apiserver

#







KUBE_API_ADDRESS="--address=0.0.0.0"

KUBE_API_PORT="--port=8080"

KUBELET_PORT="--kubelet_port=10250"

KUBE_ETCD_SERVERS="--etcd_servers=http://127.0.0.1:2379"

KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

KUBE_ADMISSION_CONTROL="--admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"

KUBE_API_ARGS=""










# The address on the local server to listen to.

#KUBE_API_ADDRESS="--insecure-bind-address=127.0.0.1"




# The port on the local server to listen on.

# KUBE_API_PORT="--port=8080"




# Port minions listen on

# KUBELET_PORT="--kubelet-port=10250"




# Comma separated list of nodes in the etcd cluster

#KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"




# Address range to use for services

#KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"




# default admission control policies

#KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"




# Add your own!

#KUBE_API_ARGS=""

###################################################################################################################################################################################################################################################################################################################################################################################################################################################







#######################################################################################

[root@master1-host etc]# hostnamectl

#######################################################################################

Static hostname: master1-host

Icon name: computer-vm

Chassis: vm

Machine ID: 2f97f9f83b814338963a7c00ce6870c6

Boot ID: 144946d199c8433c8c7aefa385f40cdc

Virtualization: kvm

Operating System: CentOS Linux 7 (Core)

CPE OS Name: cpe:/o:centos:centos:7

Kernel: Linux 3.10.0-514.6.1.el7.x86_64

Architecture: x86-64

###################################################################################################################################################################################################################################################################################################################################################################################################################################################







#######################################################################################

/etc/hosts (minionA.host)

#######################################################################################







127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6







192.168.10.201 master.host

192.168.10.211 master1.host

192.168.10.202 minion1.host

192.168.10.203 minion2.host

192.168.10.212 minionA.host

###################################################################################################################################################################################################################################################################################################################################################################################################################################################







#######################################################################################

[root@minionA-host ~]# hostnamectl

#######################################################################################

Static hostname: minionA-host

Icon name: computer-vm

Chassis: vm

Machine ID: 2f97f9f83b814338963a7c00ce6870c6

Boot ID: 0a1ffe7f3c5d48faadeb797346ca1c0f

Virtualization: kvm

Operating System: CentOS Linux 7 (Core)

CPE OS Name: cpe:/o:centos:centos:7

Kernel: Linux 3.10.0-514.6.1.el7.x86_64

Architecture: x86-64




###################################################################################################################################################################################################################################################################################################################################################################################################################################################







#######################################################################################

[root@minionA-host ~]# kubectl logs kubernetes-dashboard-1872455951-wprxz --namespace=kube-system

#######################################################################################

Using HTTP port: 9090

Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: invalid configuration: no configuration has been provided

Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md

[root@minionA-host ~]#

###################################################################################################################################################################################################################################################################################################################################################################################################################################################







#######################################################################################

/etc/kubernetes/config (minionA.host)

#######################################################################################

###

# kubernetes system config

#

# The following values are used to configure various aspects of all

# kubernetes services, including

#

# kube-apiserver.service

# kube-controller-manager.service

# kube-scheduler.service

# kubelet.service

# kube-proxy.service

# logging to stderr means we get it in the systemd journal

KUBE_LOGTOSTDERR="--logtostderr=true"




# journal message level, 0 is debug

KUBE_LOG_LEVEL="--v=0"




# Should this cluster be allowed to run privileged docker containers

KUBE_ALLOW_PRIV="--allow-privileged=false"




# How the controller-manager, scheduler, and proxy find the apiserver

#KUBE_MASTER="--master=http://127.0.0.1:8080"







KUBE_MASTER="--master=http://192.168.10.211:8080"

###################################################################################################################################################################################################################################################################################################################################################################################################################################################







#######################################################################################

/etc/kubernetes/apiserver (minionA.host)

#######################################################################################

###

# kubernetes system config

#

# The following values are used to configure the kube-apiserver

#




# The address on the local server to listen to.

KUBE_API_ADDRESS="--insecure-bind-address=master1.host"




# The port on the local server to listen on.

KUBE_API_PORT="--port=8080"




# Port minions listen on

# KUBELET_PORT="--kubelet-port=10250"




# Comma separated list of nodes in the etcd cluster

KUBE_ETCD_SERVERS="--etcd-servers=http://master1.host:2379"




# Address range to use for services

KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"




# default admission control policies

KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"




# Add your own!

KUBE_API_ARGS=""

###################################################################################################################################################################################################################################################################################################################################################################################################################################################







#######################################################################################

/etc/kubernetes/kubelet (minionA.host)

#######################################################################################

###

# kubernetes kubelet (minion) config




# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)

#KUBELET_ADDRESS="--address=127.0.0.1"




# The port for the info server to serve on

# KUBELET_PORT="--port=10250"




# You may leave this blank to use the actual hostname

#KUBELET_HOSTNAME="--hostname-override=127.0.0.1"




# location of the api-server

#KUBELET_API_SERVER="--api-servers=http://127.0.0.1:8080"




# pod infrastructure container

#KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"




# Add your own!

#KUBELET_ARGS=""













KUBELET_ADDRESS="--address=0.0.0.0"

KUBELET_PORT="--port=10250"

# change the hostname to this host’s IP address

KUBELET_HOSTNAME="--hostname_override=192.168.10.212"

KUBELET_API_SERVER="--api_servers=http://192.168.10.211:8080"

KUBELET_ARGS=""




###################################################################################################################################################################################################################################################################################################################################################################################################################################################

[root@minionA-host ~]# kubectl logs kubernetes-dashboard-1872455951-wprxz --namespace=kube-system
#######################################################################################
Using HTTP port: 9090
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: invalid configuration: no configuration has been provided
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
[root@minionA-host ~]#

[Guangya Liu]

Your container was created on "192.168.10.212", for such case, it is encouraged to ssh to `192.168.10.212` and then `docker ps -a` to get the exited container, and then `docker logs ID` to see what is wrong with the container.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Guangya Liu]

Or else directly `kubectl logs kubernetes-dashboard-1872455951-wprxz --namespace=kube-system`

[vasista...@gmail.com]

Hi ,

I'm also getting the same problem.
I have exactly the same setup & configuration as michels...@gmail.com.

I have tried and finally checked various places for solutions but no luck.

This is currently a blocker for me.

Any help to resolve this is highly appreciated.

And idea what am I missing ?

- Vasista

[Andrew Smith]

When you say "no luck" do you mean that you don't see anything in the container logs when using the commands provided?

That should give you the clue as to way the container is exiting.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--

Andy Smith

http://andrewmichaelsmith.com | @bingleybeep

[vasista...@gmail.com]

Hi Andrew,

When I do

kubectl describe po <POD_NAME>

I can see following error:
------------------------------------------------------------------
15s 3s 2 {kubelet 192.168.170.32} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "ipf-reference" with CrashLoopBackOff: "Back-off 20s restarting failed container=ipf-reference pod=ipf-deployment-2355814806-404xk_default(3deecee9-2691-11e7-acf1-000c290d68bc)"
------------------------------------------------------------------

When I do kubectl logs <POD_NAME>

I don't get anything back

So I logged in to Minion 1:
I did docker logs on the exited container and I don't get anything back.

docker logs <CONATINER_ID>

Am I doing anything wrong ?

- Vasista

On Friday, 21 April 2017 12:32:20 UTC+1, Andrew Smith wrote:
> When you say "no luck" do you mean that you don't see anything in the container logs when using the commands provided?
>
>
> That should give you the clue as to way the container is exiting.
>
>
> On 21 April 2017 at 11:11, <vasista...@gmail.com> wrote:
> Hi ,
>
>
>
> I'm also getting the same problem.
>
> I have exactly the same setup & configuration as michels...@gmail.com.
>
>
>
> I have tried and finally checked various places for solutions but no luck.
>
>
>
> This is currently a blocker for me.
>
>
>
>
>
> Any help to resolve this is highly appreciated.
>
>
>
> And idea what am I missing ?
>
>
>
> - Vasista
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
>
> To post to this group, send email to kubernet...@googlegroups.com.

[vasista...@gmail.com]

Hi Andrew, Any help in this regard?

On Friday, 21 April 2017 12:32:20 UTC+1, Andrew Smith wrote:

> When you say "no luck" do you mean that you don't see anything in the container logs when using the commands provided?
>
>
> That should give you the clue as to way the container is exiting.
>
>
> On 21 April 2017 at 11:11, <vasista...@gmail.com> wrote:
> Hi ,
>
>
>
> I'm also getting the same problem.
>
> I have exactly the same setup & configuration as michels...@gmail.com.
>
>
>
> I have tried and finally checked various places for solutions but no luck.
>
>
>
> This is currently a blocker for me.
>
>
>
>
>
> Any help to resolve this is highly appreciated.
>
>
>
> And idea what am I missing ?
>
>
>
> - Vasista
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
>
> To post to this group, send email to kubernet...@googlegroups.com.

[roland...@gmail.com]

Hi Vasista,
I had the same issue.
No logs and same error.
What worked for me is removing the ports and restartpolicy section from my yaml file that creates the deployment.
Probably it was trying to expose the container on port 80.
I am still learning kubernetes as well so not sure whats exactly going on... but hope it points you to the right direction.

----------------------
GOODconfig.yaml
----------------------
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: net
spec:
replicas: 1
template:
metadata:
labels:
app: net
spec:
containers:
- name: net
image: ubuntu
imagePullPolicy: IfNotPresent

----------------------
BADconfig.yaml
----------------------
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: net
spec:
replicas: 1
template:
metadata:
labels:
app: net
spec:
containers:
- name: net
image: ubuntu
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
restartPolicy: Always