Custometrics 503 error

[kumar k]

Hi,

I am getting below error for custom metrics on prometheus adaptor.

Can someone please help?

kubectl get apiservice

v1beta1.custom.metrics.k8s.io          dock/prometheus-adapter   False (FailedDiscoveryCheck)   2d4h

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 -v9

I1122 22:12:38.509257   17365 loader.go:375] 

I1122 22:12:38.511514   17365 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e" 'https://XXXXXX.us-east-1.eks.amazonaws.com/apis/custom.metrics.k8s.io/v1beta1'

I1122 22:12:39.614349   17365 round_trippers.go:443] GET https://XXXXXX.us-east-1.eks.amazonaws.com/apis/custom.metrics.k8s.io/v1beta1 503 Service Unavailable in 1102 milliseconds

I1122 22:12:39.614434   17365 round_trippers.go:449] Response Headers:

I1122 22:12:39.614457   17365 round_trippers.go:452]     Content-Length: 20

I1122 22:12:39.614474   17365 round_trippers.go:452]     Date: Mon, 23 Nov 2020 03:12:39 GMT

I1122 22:12:39.614492   17365 round_trippers.go:452]     Audit-Id: f167be23-5f10-451c-b423-9ed6362cb672

I1122 22:12:39.614515   17365 round_trippers.go:452]     Cache-Control: no-cache, private

I1122 22:12:39.614537   17365 round_trippers.go:452]     Content-Type: text/plain; charset=utf-8

I1122 22:12:39.614566   17365 round_trippers.go:452]     X-Content-Type-Options: nosniff

I1122 22:12:39.615355   17365 request.go:1068] Response Body: service unavailable

I1122 22:12:39.616357   17365 helpers.go:216] server response object: [{

  "metadata": {},

  "status": "Failure",

  "message": "the server is currently unable to handle the request",

  "reason": "ServiceUnavailable",

  "details": {

    "causes": [

      {

        "reason": "UnexpectedServerResponse",

        "message": "service unavailable"

      }

    ]

  },

  "code": 503

}]

F1122 22:12:39.616722   17365 helpers.go:115] Error from server (ServiceUnavailable): the server is currently unable to handle the request

[kumar k]

When i do describe API,I am seeing 404 error.

Can someone please help

kubectl describe apiservice v1beta1.custom.metrics.k8s.io
Name:         v1beta1.custom.metrics.k8s.io
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2020-11-20T22:20:54Z
  Resource Version:    126789181
  Self Link:           /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.custom.metrics.k8s.io
  UID:                 06b5e355-8a67-4f17-b1ec-7c655d33e83e
Spec:
  Group:                     custom.metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            monitoring-base-prometheus-adapter
    Namespace:       dock
    Port:            443
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2020-11-20T22:20:54Z
    Message:               failing or missing response from https://:6443/apis/custom.metrics.k8s.io/v1beta1: bad status from https://:6443/apis/custom.metrics.k8s.io/v1beta1: 404
    Reason:                FailedDiscoveryCheck
    Status:                False
    Type:                  Available
Events:                    <none>

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ce7ff7eb-cf6d-47a2-88cf-7323e805c732n%40googlegroups.com.

[Rohit Mittapalli]

Hey Kumar,

Were you able to get this resolved? Seeing a similar issue on our end.