Heapster metrics now being shown on the kubernetes dashboard.

492 views

Skip to first unread message

Ashish r

unread,

Jul 19, 2018, 11:38:21 AM7/19/18

to Kubernetes user discussion and Q&A

Hi all,

I have trouble about dashboard UI. I cannot see Heapster information such as CPU usage and memory usage in dashboard.
I can only see the normal information such as pods status and deployments status.

I built a kubernetes environment by kubeadm.
Here is my dashboard yaml file.
It is almost same as default file, but I added NodePort setting in 'kind: Service'.

apiVersion: v1

kind: Secret

metadata:

labels:

k8s-app: kubernetes-dashboard

namespace: kube-system

type: Opaque

---

# ------------------- Dashboard Service Account ------------------- #

apiVersion: v1

kind: ServiceAccount

metadata:

labels:

k8s-app: kubernetes-dashboard

namespace: kube-system

---

# ------------------- Dashboard Role & Role Binding ------------------- #

kind: Role

apiVersion: rbac.authorization.k8s.io/v1

metadata:

namespace: kube-system

rules:

# Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.

- apiGroups: [""]

resources: ["secrets"]

verbs: ["create"]

# Allow Dashboard to create 'kubernetes-dashboard-settings' config map.

- apiGroups: [""]

resources: ["configmaps"]

verbs: ["create"]

# Allow Dashboard to get, update and delete Dashboard exclusive secrets.

- apiGroups: [""]

resources: ["secrets"]

resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"]

verbs: ["get", "update", "delete"]

# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.

- apiGroups: [""]

resources: ["configmaps"]

resourceNames: ["kubernetes-dashboard-settings"]

verbs: ["get", "update"]

# Allow Dashboard to get metrics from heapster.

- apiGroups: [""]

resources: ["services"]

resourceNames: ["heapster"]

verbs: ["proxy"]

- apiGroups: [""]

resources: ["services/proxy"]

resourceNames: ["heapster", "http:heapster:", "https:heapster:"]

verbs: ["get"]

---

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

namespace: kube-system

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: Role

subjects:

- kind: ServiceAccount

namespace: kube-system

---

# ------------------- Dashboard Deployment ------------------- #

kind: Deployment

apiVersion: apps/v1beta2

metadata:

labels:

k8s-app: kubernetes-dashboard

namespace: kube-system

spec:

replicas: 1

revisionHistoryLimit: 10

selector:

matchLabels:

k8s-app: kubernetes-dashboard

template:

metadata:

labels:

k8s-app: kubernetes-dashboard

spec:

containers:

- name: kubernetes-dashboard

image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3

ports:

- containerPort: 8443

protocol: TCP

args:

- --auto-generate-certificates

# Uncomment the following line to manually specify Kubernetes API server Host

# If not specified, Dashboard will attempt to auto discover the API server and connect

# to it. Uncomment only if the default does not work.

# - --apiserver-host=http://my-address:port

volumeMounts:

- name: kubernetes-dashboard-certs

mountPath: /certs

# Create on-disk volume to store exec logs

- mountPath: /tmp

livenessProbe:

httpGet:

scheme: HTTPS

path: /

port: 8443

initialDelaySeconds: 30

timeoutSeconds: 30

volumes:

- name: kubernetes-dashboard-certs

secret:

secretName: kubernetes-dashboard-certs

- name: tmp-volume

emptyDir: {}

serviceAccountName: kubernetes-dashboard

# Comment the following tolerations if Dashboard must not be deployed on master

tolerations:

- key: node-role.kubernetes.io/master

effect: NoSchedule

---

# ------------------- Dashboard Service ------------------- #

kind: Service

apiVersion: v1

metadata:

labels:

k8s-app: kubernetes-dashboard

namespace: kube-system

spec:

ports:

- port: 443

targetPort: 8443

selector:

k8s-app: kubernetes-dashboard

Heapster Configuration:

I have used the YAML file for Heapster from here: https://gist.github.com/chukaofili/b5e5380bdac703abd1e8a7f0b9f0d6df

My environment information is follows.

# cat /etc/redhat-release

Red Hat Enterprise Linux Server release 7.5 (Maipo)

# kubeadm version

kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

# kubectl get all -n kube-system

NAME READY STATUS RESTARTS AGE

pod/coredns-78fcdf6894-s9d9w 1/1 Running 0 8d

pod/coredns-78fcdf6894-sljjw 1/1 Running 0 8d

pod/etcd-ip-172-30-0-131.us-west-1.compute.internal 1/1 Running 2 8d

pod/etcd-ip-172-30-0-192.us-west-1.compute.internal 1/1 Running 1 8d

pod/etcd-ip-172-30-1-174.us-west-1.compute.internal 1/1 Running 1 8d

pod/heapster-6cc56b8ff9-tbt9g 1/1 Running 0 4m

pod/kube-apiserver-ip-172-30-0-131.us-west-1.compute.internal 1/1 Running 2 8d

pod/kube-apiserver-ip-172-30-0-192.us-west-1.compute.internal 1/1 Running 1 8d

pod/kube-apiserver-ip-172-30-1-174.us-west-1.compute.internal 1/1 Running 1 8d

pod/kube-controller-manager-ip-172-30-0-131.us-west-1.compute.internal 1/1 Running 2 8d

pod/kube-controller-manager-ip-172-30-0-192.us-west-1.compute.internal 1/1 Running 1 8d

pod/kube-controller-manager-ip-172-30-1-174.us-west-1.compute.internal 1/1 Running 1 8d

pod/kube-proxy-4m4kl 1/1 Running 0 9d

pod/kube-proxy-7crkc 1/1 Running 2 9d

pod/kube-proxy-cqsd6 1/1 Running 0 9d

pod/kube-proxy-d998b 1/1 Running 1 9d

pod/kube-proxy-dkpw2 1/1 Running 2 9d

pod/kube-scheduler-ip-172-30-0-131.us-west-1.compute.internal 1/1 Running 2 8d

pod/kube-scheduler-ip-172-30-0-192.us-west-1.compute.internal 1/1 Running 1 8d

pod/kube-scheduler-ip-172-30-1-174.us-west-1.compute.internal 1/1 Running 1 8d

pod/kubernetes-dashboard-64576f8f86-p5rb9 1/1 Running 0 5d

pod/monitoring-grafana-764779695-xvb5h 1/1 Running 0 4m

pod/monitoring-influxdb-6d7bc5546f-lp2h4 1/1 Running 0 4m

pod/tiller-deploy-759cb9df9-97tpc 1/1 Running 0 7d

pod/weave-net-dv7hk 2/2 Running 1 9d

pod/weave-net-k8mm5 2/2 Running 4 9d

pod/weave-net-kqdc7 2/2 Running 2 9d

pod/weave-net-pdrtf 2/2 Running 0 9d

pod/weave-net-x65z5 2/2 Running 2 9d

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

service/heapster ClusterIP 10.104.126.61 <none> 80/TCP 4m

service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 9d

service/kubernetes-dashboard NodePort 10.106.207.208 <none> 443:31726/TCP 6d

service/monitoring-grafana ClusterIP 10.111.137.140 <none> 80/TCP 4m

service/monitoring-influxdb ClusterIP 10.98.147.113 <none> 8086/TCP 4m

service/tiller-deploy ClusterIP 10.96.15.172 <none> 44134/TCP 8d

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE

daemonset.apps/kube-proxy 5 5 5 5 5 beta.kubernetes.io/arch=amd64 9d

daemonset.apps/weave-net 5 5 5 5 5 <none> 9d

NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE

deployment.apps/coredns 2 2 2 2 9d

deployment.apps/heapster 1 1 1 1 4m

deployment.apps/kubernetes-dashboard 1 1 1 1 6d

deployment.apps/monitoring-grafana 1 1 1 1 4m

deployment.apps/monitoring-influxdb 1 1 1 1 4m

deployment.apps/tiller-deploy 1 1 1 1 8d

NAME DESIRED CURRENT READY AGE

replicaset.apps/coredns-78fcdf6894 2 2 2 9d

replicaset.apps/heapster-6cc56b8ff9 1 1 1 4m

replicaset.apps/kubernetes-dashboard-64576f8f86 1 1 1 6d

replicaset.apps/monitoring-grafana-764779695 1 1 1 4m

replicaset.apps/monitoring-influxdb-6d7bc5546f 1 1 1 4m

replicaset.apps/tiller-deploy-597c48f967 0 0 0 8d

replicaset.apps/tiller-deploy-759cb9df9 1 1 1 7d

The dashboard logs show the following errors:

# kubectl logs -n kube-system kubernetes-dashboard-64576f8f86-p5rb9 | grep error

2018/07/16 15:01:17 http: TLS handshake error from 10.38.0.0:53199: tls: first record does not look like a TLS handshake

2018/07/16 15:01:28 http: TLS handshake error from 10.38.0.0:53198: EOF

2018/07/16 15:01:29 http: TLS handshake error from 10.38.0.0:53201: tls: first record does not look like a TLS handshake

2018/07/16 15:01:40 http: TLS handshake error from 10.38.0.0:53200: EOF

2018/07/16 15:05:20 http: TLS handshake error from 10.32.0.1:53227: tls: first record does not look like a TLS handshake

2018/07/16 15:05:20 http: TLS handshake error from 10.32.0.1:53226: tls: first record does not look like a TLS handshake

2018/07/16 15:06:06 http: TLS handshake error from 10.32.0.1:53228: EOF

2018/07/16 15:06:06 http: TLS handshake error from 10.32.0.1:53229: tls: first record does not look like a TLS handshake

2018/07/16 15:06:11 http: TLS handshake error from 10.32.0.1:53230: tls: first record does not look like a TLS handshake

2018/07/16 15:06:41 http: TLS handshake error from 10.32.0.1:53231: EOF

2018/07/16 17:02:16 http2: server: error reading preface from client 10.32.0.1:54503: read tcp 10.38.0.2:8443->10.32.0.1:54503: read: connection reset by peer

2018/07/17 14:59:00 http: TLS handshake error from 10.38.0.0:57174: tls: first record does not look like a TLS handshake

2018/07/17 14:59:14 http: TLS handshake error from 10.38.0.0:57175: EOF

2018/07/19 15:17:04 http: TLS handshake error from 10.38.0.0:63307: EOF

2018/07/19 15:17:32 Metric client health check failed: an error on the server ("[-]healthz failed: could not get the latest data batch\nhealthz check failed") has prevented the request from succeeding. Retrying in 30 seconds.

2018/07/19 15:18:02 Metric client health check failed: an error on the server ("[-]healthz failed: could not get the latest data batch\nhealthz check failed") has prevented the request from succeeding. Retrying in 30 seconds.

Below is the Heapster logs:

kubectl logs -n kube-system heapster-6cc56b8ff9-tbt9g | grep error

E0719 15:18:05.007331 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.131:10255/stats/container/": Post http://172.30.0.131:10255/stats/container/: dial tcp 172.30.0.131:10255: getsockopt: connection refused

E0719 15:18:05.011941 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.1.120:10255/stats/container/": Post http://172.30.1.120:10255/stats/container/: dial tcp 172.30.1.120:10255: getsockopt: connection refused

E0719 15:18:05.027411 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.1.174:10255/stats/container/": Post http://172.30.1.174:10255/stats/container/: dial tcp 172.30.1.174:10255: getsockopt: connection refused

E0719 15:18:05.036160 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.41:10255/stats/container/": Post http://172.30.0.41:10255/stats/container/: dial tcp 172.30.0.41:10255: getsockopt: connection refused

E0719 15:18:05.039040 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.192:10255/stats/container/": Post http://172.30.0.192:10255/stats/container/: dial tcp 172.30.0.192:10255: getsockopt: connection refused

E0719 15:19:05.019275 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.192:10255/stats/container/": Post http://172.30.0.192:10255/stats/container/: dial tcp 172.30.0.192:10255: getsockopt: connection refused

E0719 15:19:05.028251 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.1.174:10255/stats/container/": Post http://172.30.1.174:10255/stats/container/: dial tcp 172.30.1.174:10255: getsockopt: connection refused

E0719 15:19:05.029726 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.1.120:10255/stats/container/": Post http://172.30.1.120:10255/stats/container/: dial tcp 172.30.1.120:10255: getsockopt: connection refused

E0719 15:19:05.032867 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.41:10255/stats/container/": Post http://172.30.0.41:10255/stats/container/: dial tcp 172.30.0.41:10255: getsockopt: connection refused

E0719 15:19:05.037851 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.131:10255/stats/container/": Post http://172.30.0.131:10255/stats/container/: dial tcp 172.30.0.131:10255: getsockopt: connection refused

E0719 15:20:05.005946 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.1.120:10255/stats/container/": Post http://172.30.1.120:10255/stats/container/: dial tcp 172.30.1.120:10255: getsockopt: connection refused

E0719 15:20:05.011213 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.1.174:10255/stats/container/": Post http://172.30.1.174:10255/stats/container/: dial tcp 172.30.1.174:10255: getsockopt: connection refused

E0719 15:20:05.014956 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.192:10255/stats/container/": Post http://172.30.0.192:10255/stats/container/: dial tcp 172.30.0.192:10255: getsockopt: connection refused

E0719 15:20:05.028179 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.131:10255/stats/container/": Post http://172.30.0.131:10255/stats/container/: dial tcp 172.30.0.131:10255: getsockopt: connection refused

E0719 15:20:05.032871 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.41:10255/stats/container/": Post http://172.30.0.41:10255/stats/container/: dial tcp 172.30.0.41:10255: getsockopt: connection refused

E0719 15:21:05.004697 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.1.120:10255/stats/container/": Post http://172.30.1.120:10255/stats/container/: dial tcp 172.30.1.120:10255: getsockopt: connection refused

E0719 15:21:05.005081 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.131:10255/stats/container/": Post http://172.30.0.131:10255/stats/container/: dial tcp 172.30.0.131:10255: getsockopt: connection refused

E0719 15:21:05.028308 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.1.174:10255/stats/container/": Post http://172.30.1.174:10255/stats/container/: dial tcp 172.30.1.174:10255: getsockopt: connection refused

E0719 15:21:05.034900 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.192:10255/stats/container/": Post http://172.30.0.192:10255/stats/container/: dial tcp 172.30.0.192:10255: getsockopt: connection refused

E0719 15:21:05.039941 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.41:10255/stats/container/": Post http://172.30.0.41:10255/stats/container/: dial tcp 172.30.0.41:10255: getsockopt: connection refused

E0719 15:22:05.003089 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.131:10255/stats/container/": Post http://172.30.0.131:10255/stats/container/: dial tcp 172.30.0.131:10255: getsockopt: connection refused

E0719 15:22:05.031173 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.192:10255/stats/container/": Post http://172.30.0.192:10255/stats/container/: dial tcp 172.30.0.192:10255: getsockopt: connection refused

E0719 15:22:05.033722 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.1.120:10255/stats/container/": Post http://172.30.1.120:10255/stats/container/: dial tcp 172.30.1.120:10255: getsockopt: connection refused

E0719 15:22:05.038925 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.41:10255/stats/container/": Post http://172.30.0.41:10255/stats/container/: dial tcp 172.30.0.41:10255: getsockopt: connection refused

E0719 15:22:05.039964 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.1.174:10255/stats/container/": Post http://172.30.1.174:10255/stats/container/: dial tcp 172.30.1.174:10255: getsockopt: connection refused

E0719 15:23:05.000785 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.1.120:10255/stats/container/": Post http://172.30.1.120:10255/stats/container/: dial tcp 172.30.1.120:10255: getsockopt: connection refused

E0719 15:23:05.010076 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.131:10255/stats/container/": Post http://172.30.0.131:10255/stats/container/: dial tcp 172.30.0.131:10255: getsockopt: connection refused

E0719 15:23:05.019256 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.1.174:10255/stats/container/": Post http://172.30.1.174:10255/stats/container/: dial tcp 172.30.1.174:10255: getsockopt: connection refused

E0719 15:23:05.027067 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.192:10255/stats/container/": Post http://172.30.0.192:10255/stats/container/: dial tcp 172.30.0.192:10255: getsockopt: connection refused

E0719 15:23:05.041090 1 kubelet.go:231] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.30.0.41:10255/stats/container/": Post http://172.30.0.41:10255/stats/container/: dial tcp 172.30.0.41:10255: getsockopt: connection refused