kube api-server can't authenticate to kubelet when using kubectl logs?

[Scalefastr]

I'm setting up a kubernetes clusters via "the hard way" but I"m stuck.

Right now I'm using flannel (tried canal too) and the apiserver runs with a 'kubernetes' cert.

I get this when I run kubectl:

root@host-9c16fd7a ~ # kubectl logs busybox-855686df5d-ln6ww
Error from server (Forbidden): Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy) ( pods/log busybox-855686df5d-ln6ww)

Then on the kubelet node I get the following error (see below)

I think what's happening is that the 'kubernetes' user doesn't have the proper permissions but I can't figure out actually how to configure it as the documentation seems sparse/complicated on this issue.

I've definitely RTFMd but can't figure this out.

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: I0208 19:24:34.381382   10257 server.go:248] Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: I0208 19:24:34.381576   10257 server.go:796] GET /containerLogs/default/busybox-855686df5d-ln6ww/busybox: (5.610932ms) 403

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: goroutine 963 [running]:

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc42025ea10, 0x193)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc42025ea10, 0x193)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteHeader(0xc421c3f0e0, 0x193)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:201 +0x41

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteErrorString(0xc421c3f0e0, 0x193, 0xc420f879a0, 0x48, 0x4, 0xc420f879a0)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:181 +0x46

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: k8s.io/kubernetes/pkg/kubelet/server.(*Server).InstallAuthFilter.func1(0xc420febbc0, 0xc421c3f0e0, 0xc420febcb0)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:249 +0x4b5

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*FilterChain).ProcessFilter(0xc420febcb0, 0xc420febbc0, 0xc421c3f0e0)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/filter.go:19 +0x68

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc4208c23f0, 0x5769940, 0xc42025ea10, 0xc421036500)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:274 +0x8ff

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).(k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch)-fm(0x5769940, 0xc42025ea10, 0xc421036500)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x48

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: net/http.HandlerFunc.ServeHTTP(0xc4201a2240, 0x5769940, 0xc42025ea10, 0xc421036500)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /usr/local/go/src/net/http/server.go:1918 +0x44

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: net/http.(*ServeMux).ServeHTTP(0xc42075c9c0, 0x5769940, 0xc42025ea10, 0xc421036500)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /usr/local/go/src/net/http/server.go:2254 +0x130

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).ServeHTTP(0xc4208c23f0, 0x5769940, 0xc42025ea10, 0xc421036500)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:292 +0x4d

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: k8s.io/kubernetes/pkg/kubelet/server.(*Server).ServeHTTP(0xc420880690, 0x5769940, 0xc42025ea10, 0xc421036500)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:795 +0x106

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: net/http.serverHandler.ServeHTTP(0xc420a092b0, 0x576a580, 0xc421797c00, 0xc421036500)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /usr/local/go/src/net/http/server.go:2619 +0xb4

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: net/http.(*conn).serve(0xc4203e03c0, 0x576ca00, 0xc42183ff00)

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /usr/local/go/src/net/http/server.go:1801 +0x71d

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: created by net/http.(*Server).Serve

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:         /usr/local/go/src/net/http/server.go:2720 +0x288

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]: logging error output: "Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy)"

Feb 08 19:24:34 host-d9c9d5e1.instances.us-west-1.scalefastr.cloud kubelet[10257]:  [[Go-http-client/1.1] 195.201.30.240:58019]

[lig...@gmail.com]

The required permissions for the various kubelet endpoints are referenced here:
https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#kubelet-authorization

https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/cluster-roles.yaml#L883-L916 is an example clusterrole that grants those permissions you can bind to your apiserver user with a clusterrolebinding