knative-serving v1beta1.custom.metrics.k8s.io api issue

[Pradeep Aradhya]

Hello All,

I am facing issues in using knative-serving after installation. I'm not able to deploy "HelloWorld-go" example from Knative website. 

My configurations are: 

K8 on AWS via KOPS. 

K8 version:

$ kubectl version --short

Client Version: v1.15.5

Server Version: v1.15.5

I have installed knative vai glooctl, all the knative deployments/services and gloo deployments/services are running. 

$ ~/Downloads/glooctl-darwin-amd64 version

Client: {"version":"1.1.0"}

Server: {"type":"Knative","kubernetes":{"containers":[{"Tag":"1.1.0","Name":"discovery","Registry":"quay.io/solo-io"},{"Tag":"1.1.0","Name":"gloo","Registry":"quay.io/solo-io"},{"Tag":"1.1.0","Name":"ingress","Registry":"quay.io/solo-io"},{"Tag":"1.1.0","Name":"gloo-envoy-wrapper","Registry":"quay.io/solo-io"},{"Tag":"1.1.0","Name":"gloo-envoy-wrapper","Registry":"quay.io/solo-io"}],"namespace":"gloo-system"}}

$ kubectl get pods -n knative-serving

NAME                             READY   STATUS    RESTARTS   AGE

activator-5cf5f5df77-25z24       1/1     Running   0          6m20s

autoscaler-6b5799b5d-rw5mf       1/1     Running   0          6m19s

autoscaler-hpa-f97ff947f-922dk   1/1     Running   0          6m19s

controller-7db8ff7b65-xzbr6      1/1     Running   0          6m19s

webhook-6c59b8f657-5zvjx         1/1     Running   0          6m18s

$ kubectl get pods -n gloo-system

NAME                                      READY   STATUS    RESTARTS   AGE

discovery-58fc7fd5df-4k9rx                1/1     Running   0          6m23s

gloo-8bbffd497-c6tq2                      1/1     Running   0          6m23s

ingress-5d98446899-4p5d5                  1/1     Running   0          6m23s

knative-external-proxy-5487c76bbc-j4rtx   1/1     Running   0          6m23s

knative-internal-proxy-b85646577-hlg99    1/1     Running   0          6m23s

When I try to install as in knative website, I get the following error:

$ kubectl apply -f serving.yaml -v 6

I1218 01:17:07.252211   98240 loader.go:359] Config loaded from file:  /Users/pradeara/.kube/config

I1218 01:17:07.379126   98240 round_trippers.go:438] GET https://api.kops01.scholastic-labs.io/openapi/v2?timeout=32s 200 OK in 122 milliseconds

I1218 01:17:07.506907   98240 round_trippers.go:438] GET https://api.kops01.scholastic-labs.io/apis/custom.metrics.k8s.io/v1beta1?timeout=32s 503 Service Unavailable in 26 milliseconds

I1218 01:17:07.524950   98240 request.go:1150] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }

I1218 01:17:07.524991   98240 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request

I1218 01:17:07.546449   98240 round_trippers.go:438] GET https://api.kops01.scholastic-labs.io/apis/serving.knative.dev/v1/namespaces/default/services/helloworld-go 404 Not Found in 19 milliseconds

I1218 01:17:37.566771   98240 round_trippers.go:438] POST https://api.kops01.scholastic-labs.io/apis/serving.knative.dev/v1/namespaces/default/services 500 Internal Server Error in 30019 milliseconds

I1218 01:17:37.573359   98240 helpers.go:199] server response object: [{

  "kind": "Status",

  "apiVersion": "v1",

  "metadata": {},

  "status": "Failure",

  "message": "error when creating \"serving.yaml\": Internal error occurred: failed calling webhook \"webhook.serving.knative.dev\": Post https://webhook.knative-serving.svc:443/?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)",

  "reason": "InternalError",

  "details": {

    "causes": [

      {

        "message": "failed calling webhook \"webhook.serving.knative.dev\": Post https://webhook.knative-serving.svc:443/?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

      }

    ]

  },

  "code": 500

}]

F1218 01:17:37.574038   98240 helpers.go:114] Error from server (InternalError): error when creating "serving.yaml": Internal error occurred: failed calling webhook "webhook.serving.knative.dev": Post https://webhook.knative-serving.svc:443/?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

As, you can see in the highlighted the error says, it was not able to reach custom.metrics.k8s.io. When I search for that API, I get it is not "true" in AVAILABLE cloumn

$ kubectl get apiservices | grep metrics

v1beta1.custom.metrics.k8s.io               knative-serving/autoscaler   False (FailedDiscoveryCheck)   13m

I have tried deleting that API and reinstalling from https://github.com/knative/serving/blob/master/config/custom-metrics-apiservice.yaml

But, it still goes back to same status. I am not able to find anything on the internet as well. 

Please let me know where I am doing wrong. I have tried with different versions of Knative+Gloo. 

Note: The knative installation should be done via gloo. 

Thank You.