kube-apiserver high CPU and requests

[carlo...@gmail.com]

We have a Kubernetes 1.7.8 clusters deployed with Kops 1.7 in HA with three masters. The cluster has 10 nodes and around 400 pods.

The cluster has heapster, prometheus, and ELK (collecting logs for some pods).

We are seeing a very high activity in the masters, over 90% of CPU used by the api-server.

Checking prometheus numbers we can see that near 5000 requests to the kube-apiserver are WATCH verbs, the rest are less than 50 request (GET, LIST, PATCH, PUT).

Almost all requests are reported with client "Go-Http-client/2.0" (the default User Agent for the Go HTTP library).

We increased the masters size (doubling their CPU) and the CPU stilled to the same high CPU (now the double).

Is this a normal situation?

How can we debug which are the pods sending these requests? (How can we add the source IP to the kube-apiserver logs?)

[kube-apiserver.manifest][1]

Thanks, Charles

The apiserver.manifest can be seen in https://pastebin.com/nGxSXuZb

[Daniel Smith]

I thought source IP was already in the kube-apiserver logs?

You may also be able to take a clue from what objects are being watched.

re: the apiserver manifest: You should upgrade to etcd3 & proto.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.