Error from server (ServiceUnavailable): the server is currently unable to handle the request

3,269 views
Skip to first unread message

ailso...@gmail.com

unread,
Mar 20, 2020, 10:48:20 AM3/20/20
to kubernetes-sig-instrumentation
So,

I've tryied to install the metrics-server, at first following your guide on the github...didn't work.

After try all possible known fixes at your github docs I install it with helm - wasn't the best option for me cause I'm trying to learn how to deploy a k8s cluster the hard way - and I'm still not abble to acess metrics.

The cluster has 3 nodes (2 masters and 1 worker)

Configuration on each node:
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
$ helm version
version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:        18.04
Codename:       bionic 

How do I get it to work? 
 

ailso...@gmail.com

unread,
Mar 20, 2020, 2:16:08 PM3/20/20
to kubernetes-sig-instrumentation
$ kubectl get apiservice | grep metrics
v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        5h12m
$ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
Error from server (ServiceUnavailable): the server is currently unable to handle the request
$ kubectl get --raw "/apis/" | grep metrics
... 
{"name":"metrics.k8s.io","versions":[{"groupVersion":"metrics.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"metrics.k8s.io/v1beta1","version":"v1beta1"}}]}
$ kubectl get --raw "/apis/metrics.k8s.io/v1beta1"
Error from server (ServiceUnavailable): the server is currently unable to handle the request

Marek Siarkowicz

unread,
Mar 20, 2020, 2:23:53 PM3/20/20
to ailso...@gmail.com, kubernetes-sig-instrumentation
Have you checked logs?

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-instrumentation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-instru...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-instrumentation/09c6fc02-afc5-4124-8f9c-175ad27a6e91%40googlegroups.com.

ailso...@gmail.com

unread,
Mar 20, 2020, 2:38:13 PM3/20/20
to kubernetes-sig-instrumentation
The pod logs? Yes, I did. That is the only lines on it. 

$ kubectl logs metrics-server-758857cbfd-vtvlx -n kube-system
I0320 12:56:46.304314       1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0320 12:56:47.237240       1 secure_serving.go:116] Serving securely on [::]:8443
Have you checked logs?

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-instrumentation+unsubscribe@googlegroups.com.

Marek Siarkowicz

unread,
Mar 20, 2020, 3:04:50 PM3/20/20
to ailso...@gmail.com, kubernetes-sig-instrumentation
Missing logs in MS means that your request doesn't reach it.
This could mean that your kube-apiserver cannot access aggregated apiserver within the cluster. Check logs of your kube-apiserver

You can read more about it in MS README, requirements section:

Have you checked logs?

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-instru...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-instrumentation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-instru...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-instrumentation/382cc220-eaf4-45b6-964c-efc3ec2a97c2%40googlegroups.com.

ailso...@gmail.com

unread,
Mar 20, 2020, 4:05:24 PM3/20/20
to kubernetes-sig-instrumentation
That is the log from apiserver that clearer show that the master isn't reaching the metrics-server
E0320 19:43:05.560702       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: Error trying to reach service: 'dial tcp 10.111.255.156:443: i/o timeout', Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0320 19:43:05.560744       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0320 19:43:09.601158       1 available_controller.go:419] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.111.255.156:443/apis/metrics.k8s.io/v1beta1: Get https://10.111.255.156:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
 
That is the apiserver description where shows the options activated...I'm new to this and I've deployed the cluster with kubeadm...It created the apiserver with all these options.
... 
    Command:
      kube-apiserver
      --advertise-address=...
      --allow-privileged=true
      --authorization-mode=Node,RBAC
      --client-ca-file=/etc/kubernetes/pki/ca.crt
      --enable-admission-plugins=NodeRestriction
      --enable-bootstrap-token-auth=true
      --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
      --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
      --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
      --etcd-servers=https://127.0.0.1:2379
      --insecure-port=0
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
      --requestheader-allowed-names=front-proxy-client
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
      --secure-port=6443
      --service-account-key-file=/etc/kubernetes/pki/sa.pub
      --service-cluster-ip-range=10.96.0.0/12
      --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
      --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
...
Clearly the master can't connect to the pods...so, how to do that? These articles that you posted doesn't tell much...at least to me :)
Have you checked logs?

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-instrumentation+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-instrumentation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-instrumentation+unsubscribe@googlegroups.com.

Marek Siarkowicz

unread,
Mar 20, 2020, 4:09:37 PM3/20/20
to ailso...@gmail.com, kubernetes-sig-instrumentation
I don't know much about configuring kubeadm, maybe someone else can help?

Best would be to ask on kubeadm slack/mailing list. ;)

Have you checked logs?

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-instru...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-instrumentation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-instru...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-instrumentation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-instru...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-instrumentation/63d350f2-8485-4c73-a287-d80a134bfd8c%40googlegroups.com.

ailso...@gmail.com

unread,
Mar 20, 2020, 4:29:47 PM3/20/20
to kubernetes-sig-instrumentation
Ok...thank you. I'll wait someone else tell me something or keep searching 'till my brain melt :)
Have you checked logs?

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-instrumentation+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-instrumentation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-instrumentation+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-instrumentation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-instrumentation+unsubscribe@googlegroups.com.

Mike Spreitzer

unread,
Mar 20, 2020, 4:58:45 PM3/20/20
to ailso...@gmail.com, kubernetes-sig-instrumentation
Your immediate problem is a networking failure, so you can debug that.

First and foremost, get a pod listing and see if there really is a server
at 10.111.255.156 . Check that it is the server you intend. Check that
it really is listening at TCP port 443. On the node running the
kube-apiserver, try `curl -k https://10.111.255.156/`, see what happens.
If you quickly get any response, then your question is why the
kube-apiserver does not. I myself would dive straight down to the wire,
use `tcpdump -i $the_relevant_interface -w $some_file host 10.111.255.156`
to capture packets while you try again. Another thing you can try is to
use `nsenter -n` on that node to try the curl from the kube-apiserver's
network namespace.

Regards,
Mike

kubernetes-sig-...@googlegroups.com wrote on 03/20/2020
04:05:24 PM:
kubernetes-sig-instru...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/
> d/msgid/kubernetes-sig-instrumentation/09c6fc02-
> afc5-4124-8f9c-175ad27a6e91%40googlegroups.com.
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-instrumentation" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to
kubernetes-sig-instru...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/
> d/msgid/kubernetes-sig-instrumentation/382cc220-eaf4-45b6-964c-
> efc3ec2a97c2%40googlegroups.com.
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-instrumentation" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to
kubernetes-sig-instru...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/
> d/msgid/kubernetes-sig-instrumentation/63d350f2-8485-4c73-a287-
> d80a134bfd8c%40googlegroups.com.

ailso...@gmail.com

unread,
Mar 20, 2020, 5:28:58 PM3/20/20
to kubernetes-sig-instrumentation
$ kubectl get pods -l app=metrics-server -n kube-system -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP               NODE     NOMINATED NODE   READINESS GATES
metrics-server-758857cbfd-vtvlx   1/1     Running   1          8h    192.168.179.13   k8s-02   <none>           <none>

Above the pod IP, below the service running as ClusterIP on the port 443

$ kubectl get svc -l app=metrics-server -n kube-system
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
metrics-server   ClusterIP   10.111.255.156   <none>        443/TCP   8h

The curl on the node running the pod (k8s-02) and one of the master nodes (k8s-03)

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403

The curl on the main master node (k8s-01)

curl: (7) Failed to connect to 10.111.255.156 port 443: Connection timed out

The node status

$ kubectl get nodes -o wide
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-01   Ready    master   28d   v1.17.2   10.1.1.101    <none>        Ubuntu 18.04.2 LTS   4.15.0-54-generic   docker://19.3.4
k8s-02   Ready    <none>   14d   v1.17.3   10.1.1.240    <none>        Ubuntu 18.04.2 LTS   4.15.0-54-generic   docker://19.3.7
k8s-03   Ready    master   14d   v1.17.3   10.1.1.97     <none>        Ubuntu 18.04.2 LTS   4.15.0-54-generic   docker://19.3.7

Maybe the version? It's the only difference between them.

But I beleive that this shouldn't be the problem.


On Friday, March 20, 2020 at 5:58:45 PM UTC-3, Mike Spreitzer wrote:
Your immediate problem is a networking failure, so you can debug that.

First and foremost, get a pod listing and see if there really is a server
at 10.111.255.156 .  Check that it is the server you intend.  Check that
it really is listening at TCP port 443.  On the node running the
kube-apiserver, try `curl -k https://10.111.255.156/`, see what happens.
If you quickly get any response, then your question is why the
kube-apiserver does not.  I myself would dive straight down to the wire,
use `tcpdump -i $the_relevant_interface -w $some_file host 10.111.255.156`
to capture packets while you try again.  Another thing you can try is to
use `nsenter -n` on that node to try the curl from the kube-apiserver's
network namespace.

Regards,
Mike

> To view this discussion on the web visit https://groups.google.com/
> d/msgid/kubernetes-sig-instrumentation/09c6fc02-
> afc5-4124-8f9c-175ad27a6e91%40googlegroups.com.
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-instrumentation" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to
> To view this discussion on the web visit https://groups.google.com/
> d/msgid/kubernetes-sig-instrumentation/382cc220-eaf4-45b6-964c-
> efc3ec2a97c2%40googlegroups.com.
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-instrumentation" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to

Mike Spreitzer

unread,
Mar 20, 2020, 5:46:38 PM3/20/20
to ailso...@gmail.com, kubernetes-sig-instrumentation
You could test whether your problem is correlated with the version
difference by trying the `curl` from k8s-03.

You could test whether your problem is in the kube Service plumbing by
trying `curl https://192.168.179.13:$the_relevant_port`. Look into the
details of the metrics-server Service to find the right port number.

If the problem is in the Service plumbing then you could look into that
plumbing to see how it is going wrong, but you will be more concerned with
why. The how might provide clues to the why. I would also look to see if
other Services have broken plumbing.

If the problem is _not_ in the Service plumbing then it is elsewhere in
the networking. I would try `tcpdump` on both your chosen client node and
on k8s-02, while you try the `curl` to the pod's IP; see where stuff is
failing to happen.

Regards,
Mike

kubernetes-sig-...@googlegroups.com wrote on 03/20/2020
05:28:58 PM:
> IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
> k8s-01 Ready master 28d v1.17.2 10.1.1.101 <none>
> Ubuntu 18.04.2 LTS 4.15.0-54-generic docker://19.3.4
> k8s-02 Ready <none> 14d v1.17.3 10.1.1.240 <none>
> Ubuntu 18.04.2 LTS 4.15.0-54-generic docker://19.3.7
> k8s-03 Ready master 14d v1.17.3 10.1.1.97 <none>
> Ubuntu 18.04.2 LTS 4.15.0-54-generic docker://19.3.7
>
> Maybe the version? It's the only difference between them.
>
> But I beleive that this shouldn't be the problem.
>
>
> On Friday, March 20, 2020 at 5:58:45 PM UTC-3, Mike Spreitzer wrote:
> Your immediate problem is a networking failure, so you can debug that.
>
> First and foremost, get a pod listing and see if there really is a
server
> at 10.111.255.156 . Check that it is the server you intend. Check that

> it really is listening at TCP port 443. On the node running the
> kube-apiserver, try `curl -k https://10.111.255.156/`, see what happens.

> If you quickly get any response, then your question is why the
> kube-apiserver does not. I myself would dive straight down to the wire,

> use `tcpdump -i $the_relevant_interface -w $some_file host
10.111.255.156`
> to capture packets while you try again. Another thing you can try is to

> use `nsenter -n` on that node to try the curl from the kube-apiserver's
> network namespace.
>
> Regards,
> Mike
>
> kubernetes-sig-...@googlegroups.com wrote on 03/20/2020
> kubernetes-sig-instru...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/
> > d/msgid/kubernetes-sig-instrumentation/09c6fc02-
> > afc5-4124-8f9c-175ad27a6e91%40googlegroups.com.
> > --
> > You received this message because you are subscribed to the Google
> > Groups "kubernetes-sig-instrumentation" group.
> > To unsubscribe from this group and stop receiving emails from it,
> > send an email to
> kubernetes-sig-instru...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/
> > d/msgid/kubernetes-sig-instrumentation/382cc220-eaf4-45b6-964c-
> > efc3ec2a97c2%40googlegroups.com.
> > --
> > You received this message because you are subscribed to the Google
> > Groups "kubernetes-sig-instrumentation" group.
> > To unsubscribe from this group and stop receiving emails from it,
> > send an email to
> kubernetes-sig-instru...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/
> > d/msgid/kubernetes-sig-instrumentation/63d350f2-8485-4c73-a287-
> > d80a134bfd8c%40googlegroups.com.

> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-instrumentation" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to
kubernetes-sig-instru...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/
> d/msgid/kubernetes-sig-instrumentation/46469e08-4104-4bb1-bfd2-
> b743e8a94d16%40googlegroups.com.

ailso...@gmail.com

unread,
Mar 20, 2020, 6:12:10 PM3/20/20
to kubernetes-sig-instrumentation
From the k8s-03 -- The pod ip have changed so don't mind about that

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}

From k8s-01

curl: (7) Failed to connect to 192.168.165.249 port 8443: Connection timed out

Same old, same old. That leaves me the network problem...I'll check that.

Thanks.

On Friday, March 20, 2020 at 6:46:38 PM UTC-3, Mike Spreitzer wrote:
You could test whether your problem is correlated with the version
difference by trying the `curl` from k8s-03.

You could test whether your problem is in the kube Service plumbing by
trying `curl https://192.168.179.13:$the_relevant_port`.  Look into the
details of the metrics-server Service to find the right port number.

If the problem is in the Service plumbing then you could look into that
plumbing to see how it is going wrong, but you will be more concerned with
why.  The how might provide clues to the why.  I would also look to see if
other Services have broken plumbing.

If the problem is _not_ in the Service plumbing then it is elsewhere in
the networking.  I would try `tcpdump` on both your chosen client node and
on k8s-02, while you try the `curl` to the pod's IP; see where stuff is
failing to happen.

Regards,
Mike

> > To view this discussion on the web visit https://groups.google.com/
> > d/msgid/kubernetes-sig-instrumentation/09c6fc02-
> > afc5-4124-8f9c-175ad27a6e91%40googlegroups.com.
> > --
> > You received this message because you are subscribed to the Google
> > Groups "kubernetes-sig-instrumentation" group.
> > To unsubscribe from this group and stop receiving emails from it,
> > send an email to
> > To view this discussion on the web visit https://groups.google.com/
> > d/msgid/kubernetes-sig-instrumentation/382cc220-eaf4-45b6-964c-
> > efc3ec2a97c2%40googlegroups.com.
> > --
> > You received this message because you are subscribed to the Google
> > Groups "kubernetes-sig-instrumentation" group.
> > To unsubscribe from this group and stop receiving emails from it,
> > send an email to
> > To view this discussion on the web visit https://groups.google.com/
> > d/msgid/kubernetes-sig-instrumentation/63d350f2-8485-4c73-a287-
> > d80a134bfd8c%40googlegroups.com.

> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-instrumentation" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to
Reply all
Reply to author
Forward
0 new messages