Request blocked due to unsupported error code: 122

[andreas....@castlebravo.se]

Hi!

I have deployed a REST api to GKE with tls and endpoints through Open API. I have configured x-google-allow: "all" and for non-specified paths everything works as normal. For configured paths endpoints correctly requires a key, but when requested with a valid key I get the following error:

curl https://telescope.endpoints.testserver-xxx.cloud.goog/service/events\?key\=AIzaSyDaQg9EBzg1REiJDxxxx\&ver\=0

{

 "code": 13,

 "message": "Request blocked due to unsupported error code: 122",

 "details": [

  {

   "@type": "type.googleapis.com/google.rpc.DebugInfo",

   "stackEntries": [],

   "detail": "service_control"

  }

 ]

}

This happens on ALL configured paths, and the requests never reach the backend, which would show at least trace logging..

I've been through all the troubleshooting docs I could find, checked the /endpoint_status etc. But everything looks legit, the correct service config seems to be fetched.. I'll include what I think are the most relevant pieces of setup, but feel free to ask for more!

Open API spec:

# [START swagger]

swagger: "2.0"

info:

  title: "Telescope Data Service"

  version: "1.0.1"

host: "telescope.endpoints.testserver-xxx.cloud.goog"

x-google-endpoints:

- name: "telescope.endpoints.testserver-xxx.cloud.goog"

  allowCors: "true"

  target: "130.211.38.252"

x-google-allow: "all"

basePath: "/"

consumes:

- "application/json"

produces:

- "application/json"

schemes:

- "https"

- "http"

paths:

  "/service/events":

    get:

      description: "Get event stream"

      operationId: "events"

      produces:

      - "application/x-ndjson"

      responses:

        200:

          description: "Successful operation"

      parameters:

      - name: "ver"

        in: "query"

        type: "string"
# A few similar paths removed for brevity...

# This section requires all requests to any path to require an API key.

security:

- api_key: []

securityDefinitions:

  # This section configures basic authentication with an API key.

  api_key:

    type: "apiKey"

    name: "key"

    in: "query"

Here are the relevant parts of k8s manifests:

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

  name: telescope-backend

  namespace: telescope

spec:

  replicas: 2

  template:

    metadata:

      labels:

        app: telescope-backend

    spec:

      volumes:

      - name: google-cloud-key

        secret:

          secretName: adc-key

      containers:

      - name: esp

        image: gcr.io/endpoints-release/endpoints-runtime:1.19

        args: [

          "--http_port=8080",

          "--backend=127.0.0.1:8081",

          "--service=telescope.endpoints.testserver-xxx.cloud.goog",

          "--rollout_strategy=managed",

          "--healthz=healthz",

        ]

        ports:

          - containerPort: 8080

        readinessProbe:

          httpGet:

            path: /healthz

            port: 8080

          initialDelaySeconds: 20

          timeoutSeconds: 1

      - name: telescope

        image: gcr.io/testserver-xxx/telescope:0.2.2

        volumeMounts:

        - name: google-cloud-key

          mountPath: /var/secrets/google

        env:

        - name: GCLOUD_PROJECT

          value: testserver-xxx

        - name: GOOGLE_APPLICATION_CREDENTIALS

          value: /var/secrets/google/key.json

        ports:

        - containerPort: 8081

/Andreas K

[andreas....@castlebravo.se]

I've spent the better part of the weekend on this now.  I've tried without the x-google-endpoints config by port-forwarding 8080 and 8081, but I still get the same error on 8080 while my api works on 8081.. Through googling, the only other mention I've found of the error is https://groups.google.com/forum/#!topic/google-cloud-endpoints/BtqrMnVJu_Q and in that case the error started when changing from cloud.goog to a custom domain. I don't see how that relates to my case, and that problem was never resolved anyways..

This is my full esp log after a fresh container start and one failed request:

INFO:Fetching an access token from the metadata service

INFO:Fetching the service config ID from the rollouts service

INFO:Fetching the service configuration from the service management service

nginx: [warn] Using trusted CA certificates file: /etc/nginx/trusted-ca-certificates.crt

127.0.0.1 - - [20/May/2018:21:58:05 +0000] "GET /service/events?key=AIzaSyxxxx&ver=0 HTTP/1.1" 500 229 "-" "curl/7.54.0"

And as I've mentioned the 500 isn't sent from my backend.

This is how the errors are logged in stackdriver:

{

 insertId:  "c2ef9152-af2a-4061-9b3e-d0fecc51aba9@a1"  

 jsonPayload: {

  api_key:  "AIzaSyDaQgxxxx"   

  api_method:  "1.telescope_endpoints_testserver_xxxx_cloud_goog.Events"   

  api_name:  "1.telescope_endpoints_testserver_xxxx_cloud_goog"   

  api_version:  "1.0.1"   

  error_cause:  "service_control"   

  http_method:  "GET"   

  http_response_code:  500   

  location:  "europe-west1-b"   

  log_message:  "Method: 1.telescope_endpoints_testserver_xxxx_cloud_goog.Events"   

  producer_project_id:  "testserver-xxxx"   

  request_latency_in_ms:  406   

  request_size_in_bytes:  142   

  response_size_in_bytes:  403   

  timestamp:  1526853485.7306347   

  url:  "/service/events?key=AIzaSyDaxxxx&ver=0"   

 }

 logName:  "projects/testserver-xxxx/logs/telescope.endpoints.testserver-xxxx.cloud.goog%2Fendpoints_log"  

 receiveTimestamp:  "2018-05-20T21:58:12.959123784Z"  

 resource: {

  labels: {

   location:  "europe-west1-b"    

   method:  "1.telescope_endpoints_testserver_xxxx_cloud_goog.Events"    

   project_id:  "testserver-xxxx"    

   service:  "telescope.endpoints.testserver-xxxx.cloud.goog"    

   version:  "1.0.1"    

  }

  type:  "api"   

 }

 severity:  "ERROR"  

 timestamp:  "2018-05-20T21:58:05.730634594Z"  

}

The fact that non-configured paths works when I use x-google-allow: "all", isn't that somehow telling to where the problem could be?

Would really appreciate some help. Thanks in advance.

/Andreas

[D. T.]

Has the project owning the API key activated the service?

Daniel Tang |

Software Engineer |

ta...@google.com |

Google LLC

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/140dd80e-b05b-4b62-9870-9c840491d6da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christophe Taton]

Error code 122 has been added recently (see https://github.com/googleapis/googleapis/pull/499) and ESP might not have been updated accordingly yet.

It apparently indicates an issue with the API key used (eg. if the API key has been restricted).

[andreas....@castlebravo.se]

The problem was the key being restricted to the wrong project. A less cryptic error message or docs could have saved me a good few hours... but.. onwards and upwards..

Thanks both of you who directed me to the problem! Good catch.