Request blocked due to unsupported error code: 122

809 views
Skip to first unread message

andreas....@castlebravo.se

unread,
May 19, 2018, 10:58:59 PM5/19/18
to Google Cloud Endpoints
Hi!

I have deployed a REST api to GKE with tls and endpoints through Open API. I have configured x-google-allow: "all" and for non-specified paths everything works as normal. For configured paths endpoints correctly requires a key, but when requested with a valid key I get the following error:
{
 "code": 13,
 "message": "Request blocked due to unsupported error code: 122",
 "details": [
  {
   "stackEntries": [],
   "detail": "service_control"
  }
 ]
}


This happens on ALL configured paths, and the requests never reach the backend, which would show at least trace logging..

I've been through all the troubleshooting docs I could find, checked the /endpoint_status etc. But everything looks legit, the correct service config seems to be fetched.. I'll include what I think are the most relevant pieces of setup, but feel free to ask for more!

Open API spec:
# [START swagger]
swagger: "2.0"
info:
  title: "Telescope Data Service"
  version: "1.0.1"
x-google-endpoints:
  allowCors: "true"
  target: "130.211.38.252"
x-google-allow: "all"
basePath: "/"
consumes:
- "application/json"
produces:
- "application/json"
schemes:
- "https"
- "http"
paths:
  "/service/events":
    get:
      description: "Get event stream"
      operationId: "events"
      produces:
      - "application/x-ndjson"
      responses:
        200:
          description: "Successful operation"
      parameters:
      - name: "ver"
        in: "query"
        type: "string"
# A few similar paths removed for brevity...
# This section requires all requests to any path to require an API key.
security:
- api_key: []
securityDefinitions:
  # This section configures basic authentication with an API key.
  api_key:
    type: "apiKey"
    name: "key"
    in: "query"


Here are the relevant parts of k8s manifests:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: telescope-backend
  namespace: telescope
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: telescope-backend
    spec:
      volumes:
      - name: google-cloud-key
        secret:
          secretName: adc-key
      containers:
      - name: esp
        args: [
          "--http_port=8080",
          "--backend=127.0.0.1:8081",
          "--rollout_strategy=managed",
          "--healthz=healthz",
        ]
        ports:
          - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 20
          timeoutSeconds: 1
      - name: telescope
        volumeMounts:
        - name: google-cloud-key
          mountPath: /var/secrets/google
        env:
        - name: GCLOUD_PROJECT
          value: testserver-xxx
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/secrets/google/key.json
        ports:
        - containerPort: 8081



/Andreas K

andreas....@castlebravo.se

unread,
May 20, 2018, 6:22:16 PM5/20/18
to Google Cloud Endpoints
I've spent the better part of the weekend on this now.  I've tried without the x-google-endpoints config by port-forwarding 8080 and 8081, but I still get the same error on 8080 while my api works on 8081.. Through googling, the only other mention I've found of the error is https://groups.google.com/forum/#!topic/google-cloud-endpoints/BtqrMnVJu_Q and in that case the error started when changing from cloud.goog to a custom domain. I don't see how that relates to my case, and that problem was never resolved anyways..

This is my full esp log after a fresh container start and one failed request:
INFO:Fetching an access token from the metadata service
INFO:Fetching the service config ID from the rollouts service
INFO:Fetching the service configuration from the service management service
nginx: [warn] Using trusted CA certificates file: /etc/nginx/trusted-ca-certificates.crt
127.0.0.1 - - [20/May/2018:21:58:05 +0000] "GET /service/events?key=AIzaSyxxxx&ver=0 HTTP/1.1" 500 229 "-" "curl/7.54.0"


And as I've mentioned the 500 isn't sent from my backend.

This is how the errors are logged in stackdriver:
{
 insertId:  "c2ef9152-af2a-4061-9b3e-d0fecc51aba9@a1"  
 jsonPayload: {
  api_key:  "AIzaSyDaQgxxxx"   
  api_method:  "1.telescope_endpoints_testserver_xxxx_cloud_goog.Events"   
  api_name:  "1.telescope_endpoints_testserver_xxxx_cloud_goog"   
  api_version:  "1.0.1"   
  error_cause:  "service_control"   
  http_method:  "GET"   
  http_response_code:  500   
  location:  "europe-west1-b"   
  log_message:  "Method: 1.telescope_endpoints_testserver_xxxx_cloud_goog.Events"   
  producer_project_id:  "testserver-xxxx"   
  request_latency_in_ms:  406   
  request_size_in_bytes:  142   
  response_size_in_bytes:  403   
  timestamp:  1526853485.7306347   
  url:  "/service/events?key=AIzaSyDaxxxx&ver=0"   
 }
 logName:  "projects/testserver-xxxx/logs/telescope.endpoints.testserver-xxxx.cloud.goog%2Fendpoints_log"  
 receiveTimestamp:  "2018-05-20T21:58:12.959123784Z"  
 resource: {
  labels: {
   location:  "europe-west1-b"    
   method:  "1.telescope_endpoints_testserver_xxxx_cloud_goog.Events"    
   project_id:  "testserver-xxxx"    
   version:  "1.0.1"    
  }
  type:  "api"   
 }
 severity:  "ERROR"  
 timestamp:  "2018-05-20T21:58:05.730634594Z"  
}


The fact that non-configured paths works when I use x-google-allow: "all", isn't that somehow telling to where the problem could be?

Would really appreciate some help. Thanks in advance.

/Andreas

D. T.

unread,
May 21, 2018, 2:12:34 PM5/21/18
to andreas....@castlebravo.se, Google Cloud Endpoints
Has the project owning the API key activated the service?

Daniel Tang | Software Engineer | ta...@google.com | Google LLC



--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/140dd80e-b05b-4b62-9870-9c840491d6da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Christophe Taton

unread,
May 21, 2018, 5:22:36 PM5/21/18
to Google Cloud Endpoints
Error code 122 has been added recently (see https://github.com/googleapis/googleapis/pull/499) and ESP might not have been updated accordingly yet.
It apparently indicates an issue with the API key used (eg. if the API key has been restricted).

andreas....@castlebravo.se

unread,
May 21, 2018, 7:51:19 PM5/21/18
to Google Cloud Endpoints
The problem was the key being restricted to the wrong project. A less cryptic error message or docs could have saved me a good few hours... but.. onwards and upwards..

Thanks both of you who directed me to the problem! Good catch.
Reply all
Reply to author
Forward
0 new messages