Cloud Endpoints with kubernetes and GraphQL

[Seun Ore]

Hello guys,

Nice to be here. I have a problem getting graphql to work just behind Cloud endpoint.

I have several microservices (about 14 ) running on GKE. There is Graphql just in front these microservices. This is the design of the developers. I am a DevOps engineer and part of my job is to deploy those microservices for frontend engineers to consume the APIs. I followed https://cloud.google.com/endpoints/docs/openapi/get-started-kubernetes-engine#configuring-endpoints-dns and this NEXT 2017 https://www.youtube.com/watch?v=bR9hEyZ9774 to quickly get onboard with with Cloud Endpoints. Never worked with it before. Neither have I worked with GraphQL before as well.

Here is my openapi.yaml file:

# [START swagger]

swagger: "2.0"

info:

description: "API Gateway Deployment."

title: "Roava API Gateway"

version: "1.0.0"

host: "staging.example.io"

x-google-endpoints:

- name: "staging.example.io"

# [END swagger]

consumes:

- "application/json"

produces:

- "application/json"

schemes:

# Uncomment the next line if you configure SSL for this API.

- "https"

paths:

"/graphql":

get:

description: "Echo back a given message."

operationId: "ms-api"

produces:

- "application/json"

responses:

200:

description: "ms-api"

schema:

$ref: "#/definitions/echoMessage"

parameters:

- description: "Message to echo"

in: body

name: message

required: true

schema:

$ref: "#/definitions/echoMessage"

security:

- api_key: []

"/auth/info/googlejwt":

get:

description: "Returns the requests' authentication information."

operationId: "auth_info_google_jwt"

produces:

- "application/json"

responses:

200:

description: "Authentication info."

schema:

$ref: "#/definitions/authInfoResponse"

security:

- google_jwt: []

"/auth/info/googleidtoken":

get:

description: "Returns the requests' authentication information."

operationId: "authInfoGoogleIdToken"

produces:

- "application/json"

responses:

200:

description: "Authentication info."

schema:

$ref: "#/definitions/authInfoResponse"

security:

- google_id_token: []

"/auth/info/firebase":

get:

description: "Returns the requests' authentication information."

operationId: "authInfoFirebase"

produces:

- "application/json"

responses:

200:

description: "Authentication info."

schema:

$ref: "#/definitions/authInfoResponse"

security:

- firebase: []

"/auth/info/auth0":

get:

description: "Returns the requests' authentication information."

operationId: "auth_info_auth0_jwk"

produces:

- "application/json"

responses:

200:

description: "Authentication info."

schema:

$ref: "#/definitions/authInfoResponse"

security:

- auth0_jwk: []

definitions:

echoMessage:

type: "object"

properties:

message:

type: "string"

authInfoResponse:

properties:

id:

type: "string"

email:

type: "string"

# This section requires all requests to any path to require an API key.

securityDefinitions:

# This section configures basic authentication with an API key.

api_key:

type: "apiKey"

name: "AIza"

in: "query"

# This section configures authentication using Google API Service Accounts

# to sign a json web token. This is mostly used for server-to-server

# communication.

google_jwt:

authorizationUrl: ""

flow: "implicit"

type: "oauth2"

# This must match the 'iss' field in the JWT.

x-google-issuer: "jwt-client.endpoints.sample.google.com"

# Update this with your service account's email address.

x-google-jwks_uri: "https://www.googleapis.com/service_accounts/v1/jwk/YOUR-SERVICE-ACCOUNT-EMAIL"

# This must match the "aud" field in the JWT. You can add multiple audiences to accept JWTs from multiple clients.

x-google-audiences: "echo.endpoints.sample.google.com"

# This section configures authentication using Google OAuth2 ID Tokens.

# ID Tokens can be obtained using OAuth2 clients, and can be used to access

# your API on behalf of a particular user.

google_id_token:

authorizationUrl: ""

flow: "implicit"

type: "oauth2"

x-google-issuer: "https://accounts.google.com"

x-google-jwks_uri: "https://www.googleapis.com/oauth2/v3/certs"

x-google-audiences: "YOUR-CLIENT-ID"

# This section configures authentication using Firebase Auth.

firebase:

authorizationUrl: ""

flow: "implicit"

type: "oauth2"

x-google-issuer: "https://securetoken.google.com/YOUR-PROJECT-ID"

x-google-audiences: "YOUR-PROJECT-ID"

# This section configures authentication using Auth0 (https://auth0.com).

auth0_jwk:

# Update YOUR-ACCOUNT-NAME with your Auth0 account name.

authorizationUrl: "https://YOUR-ACCOUNT-NAME.auth0.com/authorize"

flow: "implicit"

type: "oauth2"

x-google-issuer: "https://YOUR-ACCOUNT-NAME.auth0.com/"

# Update this with your service account's email address.

x-google-jwks_uri: "https://YOUR-ACCOUNT-NAME.auth0.com/.well-known/jwks.json"

# Replace with your client ID, found in the Auth0 console.

x-google-audiences: "YOUR-CLIENT-ID"

 Here is a snippet of k8s manifest containing google endpoints runtime docker image

spec:

      containers:

      # [START esp]

      - name: esp

        image: eu.gcr.io/***/platform/endpoints-runtime:1

        args: [

          "--http2_port=8081",

          "--backend=ms-api:9000",

          "--service=staging.example.io",

          "--rollout_strategy=managed",

          "--version=2020-11-30r1",

        ]

      # [END esp]

        ports:

          - containerPort: 8081

      - image: eu.gcr.io/***/services/ms.api:0.2.1

        imagePullPolicy: Always

        tag: 

        name: ms-api

        ports:

        - name: ms-api

          containerPort: 8000

After configuring API key and deploying openapi.yaml, everything seems to work well. However, the authentication part failed. Basically there is no difference between passing API key to curl request and not doing so at all. Everyone can still access those APIs on staging.example.roava. I concluded that this is not working because I still haven't found a way to put graphQL behind Cloud Endpoints.

How does Cloud Endpoints integrate with GraphQL for microservices running on GKE?

[Seun Ore]

I forgot to say that these APIs are written in Go language. 

[Xuyang(Jason) Tao]

This is a well-known api key limitation that Cloud Endpoints only accepts three kinds of api key locations. Other than that, "Endpoints might accept them and output a warning. However, the API key securty definitions are ignored in incoming requests"

So, in your openAPI yaml, it should be

api_key:

  type: "apiKey"

  name: "key"

  in: "query"

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/4f35d90b-4c14-49c3-bbae-500ca267248dn%40googlegroups.com.

--

Xuyang(Jason) Tao tao...@google.com Service Infrastructure Software Engineer

[Seun Ore]

This is exactly what I did in that section. And of course its properly indebted. 

api_key:

  type: "apiKey"

  name: "AIzaSyCdcV8x6WCZaffmUnEVoMLOaKA0GjdkjsajdfkafsdufkvuqFYs"

  in: "query"

This never worked. Not sure if there is something I am missing here.

[Wayne Zhang]

You should not specify your api-key in the openapi spec.   You do it in our client actual request

when you make a Http call  

curl http://your-host/path?key=API-KEY

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/0ab68039-89d3-44af-a449-794fc76a9d61n%40googlegroups.com.

[Seun Ore]

What then indicate that API key is enabled within openapi spec? 

You mean that that section would remain like below:

api_key:

  type: "apiKey"

  name: "key"

  in: "query"

[Xuyang(Jason) Tao]

For the apiKey configured to be set in the query parameter, the value "key" under "name" is the variable name for you to pass the query parameter.

api_key:

  type: "apiKey"

  name: "key"

  in: "query"

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/CAGJCt2PKx8f4FycOusS1rLHqBUktY686NDyx8QZvYh8SEFXg4A%40mail.gmail.com.

[Seun Ore]

I should add that curl http://your-host/path?key=API-KEY and curl http://your-host/path actually produced the same result. My expectation is that without specifying the key in curl request, the request should fail. But as it stands now, both curl requests actually produced positive response

On Wednesday, 2 December 2020 at 20:39:19 UTC+1 qiwz...@google.com wrote:

[Teju Nareddy]

The `api_key.name` field in the OpenAPI spec tells Cloud Endpoints where to find the API Key in a request. You want Cloud Endpoints to check the `key` query param for the api key (http://example.com/path?key=API_KEY). So you need to leave the value hardcoded as `key`. Just having the `api_key` section in your OpenAPI spec tells Cloud Endpoints to check the key.

api_key:

  type: "apiKey"

  name: "key"

  in: "query"

Please remember to delete and re-create a new API key since you posted the original one publicly.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/CAGJCt2PKx8f4FycOusS1rLHqBUktY686NDyx8QZvYh8SEFXg4A%40mail.gmail.com.

--

Teju Nareddy nare...@google.com Software Engineer

[Seun Ore]

Thanks @Teju Nareddy.
That was just a dummy key.  I think I understand your point. I will edit now and redeploy endpoints

[Seun Ore]

I just redeployed endpoints and nothing has changed. I still can send curl request without key and getting positive response. That request should be failing basically. So the api section exactly looks like this now:

api_key:

  type: "apiKey"

  name: "key"

  in: "query"

On Wednesday, 2 December 2020 at 20:49:19 UTC+1 Teju Nareddy wrote:

[Mike D]

We had the same problem when setting up GCE.  We're building our endpoints in GAE + Endpoints Framework (with java).   I think the fix for us was to use the SPI servlet.  The default servlet class (and I don't recall what it was) had to be replaced.  Not sure if this is the problem you are having.

<servlet>

<servlet-name>EndpointsServlet</servlet-name>

<servlet-class>com.google.api.server.spi.EndpointsServlet</servlet-class>

<init-param>

<param-name>services</param-name>

<param-value>com.company.MyApi</param-value>

</init-param>

</servlet>

[Wayne Zhang]

Could you send us the result of "gcloud endpoints configs describe CONFIG_ID --service=SERVICE_NAME"

SERVICE_NAME is staging.example.io

You can get CONFIG_ID from "gcloud endpoints configs list --service=SERVICE_NAME".  The first one.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/f81528ce-2ced-4f0f-bf73-8f9998bb58b0n%40googlegroups.com.

[Mike D]

This is what I was referring to in terms of the API key being ignored.  It resolved our problem.

https://stackoverflow.com/questions/51217920/the-apikeyrequired-is-not-working-in-google-cloud-endpoints-project

[Seun Ore]

Glad to know I am not the only person facing this. Thanks for sharing. My environment is GKE + graphQL. The microservices are all running behind GraphQL

[Wayne Zhang]

Mike, your issue could be different.   I looked at your openapi config,  it seems that your api-key definition is fine.   But you are using Endpoint Framework, it is a totally different implementation than ESP.  If your open api spec is correct,  then the only thing I  can think of is that the consumer project(the project you create your api-key) did not enable your API.

For ESP,  if it is not enabled,   the request should fail.  I don't know the behaviour of Endpoint Framework. 

For Seun issue,  I need to check "gcloud endpoints configs" result to see if api-key is configured correctly.  If you like, you can send me your too,  I can check it for you.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/afc69d8b-d2b9-4b4b-b51a-c4d1c4a6b7c4n%40googlegroups.com.

[Seun Ore]

I posted result of gcloud endpoints configs describe CONFIG-ID --service=staging.example.io
Its attached to this too

[Wayne Zhang]

From your config, this path "/msapi"  requires api-key,   all other paths prefixed "/auth/info"  don't need api-key.   If /msapi still works without api-key,   then it could be a bug

BTW,  since your ESP docker image is eu.gcr.io/***/platform/endpoints-runtime:1  which is not our registry,  

it may not be the latest version.   Could you update it to the latest version.  

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/000675a0-3d6f-4574-9ee9-77266e46a3e1n%40googlegroups.com.

[Seun Ore]

Thanks. I will update that and revert. For the endpoints-runtime image, we normally retag all open source images and push to our private GCR. I will also check for latest updates too. I will revert shortly.

[Seun Ore]

paths:

"/graphql":

get:

description: "Echo back a given message."

operationId: "ms-api"

produces:

- "application/json"

responses:

200:

description: "ms-api"

schema:

$ref: "#/definitions/echoMessage"

parameters:

- description: "Message to echo"

in: body

name: message

required: true

schema:

$ref: "#/definitions/echoMessage"

security:

- api_key: []                         ## Is this where I have to manually specify api key like - api-key: [ jksajsnjaKSNLKJSJD ]?

BTW, there is indeed a new version of endpoints-image version 1.54.0 as against 1 that I was using. I have redeployed with the latest version 1.54.0

On Wednesday, 2 December 2020 at 22:01:49 UTC+1 qiwz...@google.com wrote:

[Seun Ore]

In fact, after updating the endpoints-runtime image to 1.54.0 and manually inputting my api key in  api_key section just under security, nothing actually changed. The behavior has remained the same. Meaning that the addition or omission of API key has no effect whatsoever. My APIs remain opened to the world.

[Wayne Zhang]

BTW,  could you post an intended xml openapi config?   it is very hard to read your openapi config without indentation. 

I think your securityDefinition is wrong in your last openapi config.  It should be:

securityDefinitions:

   # This section configures basic authentication with an API key.

   api_key:

       type: "apiKey"

       name: "key"   <===== this has to be "key"

       in: "query"

Your actual api-key should be from your request,  not in the openapi config.    It should be in the call:   

http://staging.example.io/graphql?key=YOUR-API-KEY

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/cabe947d-9a86-4e60-a2ef-d78fe916ef14n%40googlegroups.com.

[Seun Ore]

That is exactly how it is in the openapi spec file:

 api_key:

       type: "apiKey"

       name: "key"   

       in: "query"

Actually, when you open staging.example.io, it open up a graphql playground (I hope I am right ) with URL that point to staging.example.io/graphql 

I think my challenge basically is getting graphql to sit just behind cloud endpoint just the same microservices sit behind graphql. I will try and attach my openapi spec so you can see everything with indentation. 

[Seun Ore]

[Seun Ore]

I remember that when I followed youtube tutorials and google documentation, things worked pretty fine as there was no graphql in the picture. I have attached my openapi spec properly yaml formatted.

On Wednesday, 2 December 2020 at 23:37:53 UTC+1 qiwz...@google.com wrote:

[Wayne Zhang]

Thanks,  this indented openapi.yaml is much better to read.  

I found some discrepancy between "gcloud endpoints configs describe" result and your openapi.yaml.   In your openapi.yanl, you have a path "/ms-api"  with "get" 

But in the "gcloud" result,  it is using "/graphql" with "get".

was the openapi.yaml  used to generate "gcloud" results?   

After you change openapi.yaml, you need to run "gcloud endpoints service deploy" 

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/24d88107-c5ba-4f85-92fb-ec10b0da3915n%40googlegroups.com.

[Seun Ore]

Yes. In an attempt to get solution, I had different implementations. I guess I forgot to revert to previous state before sending config info.  Even with that, the result has remained the same. 

The ms-api is one of the microservices built by developers but one that acts more like gateway to the remaining microservices. So in my Ingress object, this is the backend service through which all other services can be reached.

Now I have changed path to /ms-api in openapi spec and then redeployed endpoints. I have also updated my manifest to reflect the latest CONFIG-ID and have redeployed. 
Attached now are the latest valued of gcloud endpoints configs describe CONFIG-ID --service=staging.example.io and a portion of k8s manifest containing ESP and ms-api

[Seun Ore]

Can one simply assume that this is a bug? I am sorry how can one report this as bug on github?

[Wayne Zhang]

This is a very basic feature of ESP, requiring api-key and checking it.   There are so many users using it. I don't believe it did not work.   

I noticed some problem in your container configuration

"--http2_port=8081",
          "--backend=ms-api:9000",
          "--service=staging.example.io

",
          "--rollout_strategy=managed",
          "--version=2020-12-03r0",

Is this typo:  your backend is using port 8000,  but here the flag --backend is using 9000

BTW,  if you use "managed" rollout_strategy,  you don't need to specify --version.  It means,  if there is a newer config version, it will automatically applied,  you don't even need to start the ESP container. 

How do you make the call in the client?   I believe your call may go to some other places,  it was NOT go to ESP.

if you are using "staging.example.io" domain name,   Did you associate that domain name with the correct IP?

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/2060b0e0-b349-4a0e-9c13-17c1fc342b02n%40googlegroups.com.

[Seun Ore]

Thank you.  I tried to make few editing before posting certain config on open internet. The original backend port is 9000. I changed to 8000. Apologies for the confusion. Same logic goes for URL staging.example.io and project name.

I take note of your explanation around   "--rollout_strategy=managed". Thanks

To make curl request, I simply do : 
curl --location --request POST 'https://staging.example.io/graphql' --header 'Content-Type: application/json' --data-raw '{"query":"mutation createPhone ($input: CreatePhoneInput!) {\n    createPhone (input: $input) {\n        success\n        message\n        token\n    }\n}","variables":{"input":{"phone":"09082911633","device":{"os":"Android 10","brand":"Samsung","deviceId":"Device ID","deviceToken":"Device TOken"}}}}'

Without specifying API KEY, I expect this curl request to be denied. But here is the result:
{"data":{"createPhone":{"success":true,"message":"phone number created","token":"01ern3mzayj2vwmp36mnv142tn"}}}

With API KEY: 
curl --location --request POST 'https://staging.example.io/graphql?key=$ENDPOINTS_KEY' --header 'Content-Type: application/json' --data-raw '{"query":"mutation createPhone ($input: CreatePhoneInput!) {\n    createPhone (input: $input) {\n        success\n        message\n        token\n    }\n}","variables":{"input":{"phone":"09082911633","device":{"os":"Android 10","brand":"Samsung","deviceId":"Device ID","deviceToken":"Device TOken"}}}}'

Result is the same as above which is : {"data":{"createPhone":{"success":true,"message":"phone number created","token":"01ern3mzayj2vwmp36mnv142tn"}}}

Without the API KEY, request should be denied. Same thing applied when I tried to access this over browser.

The staging.example.com point to a static global LoadBalancer on GCP. I associated this to an ingress object and then configure DNS entries for routing traffics to relevant backend services. This way, we minimize public IPs usages for many of our resources.

[Wayne Zhang]

I strongly suspect that the traffic did not go through ESP.  Could you check your DNS record to see if it maps to ESP IP.  Another way to confirm it is to look at ESP container log after adding --enable_debug flag in ESP container args.  

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/f69de7d3-27cd-4627-b2f6-d876b2f2c93cn%40googlegroups.com.

[Seun Ore]

I included --enable_debug in esp args and removed CONFIG_ID and then redeployed.

Here is log from esp: kubectl logs pod-name -c esp

INFO:Fetching an access token from the metadata service

INFO:Fetching the service configuration from the service management service

INFO:Attribute zone: europe-west2-c

INFO:Attribute project_id: my-project-id

INFO:Attribute kube_env: KUBE_ENV

nginx: [warn] Using trusted CA certificates file: /etc/nginx/trusted-ca-certificates.crt

Here is logs from  ms-api : kubectl logs pod-name -c ms-api

{"level":"info","msg":"Connect to http://0.0.0.0:9000/ for GraphQL playground","time":"2020-12-03T09:16:39+01:00"}

[cors] 2020/12/03 09:18:57 Handler: Actual request

[cors] 2020/12/03 09:18:57   Actual request no headers added: missing origin

2020/12/03 09:18:58 [ms-api-68b8dcb79c-glthz/yXQkth1F0o-000001] "POST http://staging.example.io/graphql HTTP/1.1" from 10.0.32.100:60630 - 200 111B in 984.606415ms

[cors] 2020/12/03 09:19:14 Handler: Actual request

[cors] 2020/12/03 09:19:14   Actual request no headers added: missing origin

2020/12/03 09:19:14 [ms-api-68b8dcb79c-glthz/yXQkth1F0o-000002] "POST http://staging.example.io/graphql HTTP/1.1" from 10.0.32.100:60630 - 200 111B in 98.522741ms

[cors] 2020/12/03 09:19:29 Handler: Actual request

[cors] 2020/12/03 21:10:44 Handler: Actual request

[cors] 2020/12/03 21:10:44   Actual request no headers added: missing origin

2020/12/03 21:10:45 [ms-api-68b8dcb79c-glthz/yXQkth1F0o-000005] "POST http://staging.example.io/graphql HTTP/1.1" from 10.0.32.100:44616 - 200 111B in 996.870325ms

[cors] 2020/12/03 21:16:16 Handler: Actual request

[cors] 2020/12/03 21:16:16   Actual request no headers added: missing origin

2020/12/03 21:16:16 [ms-api-68b8dcb79c-glthz/yXQkth1F0o-000006] "POST http://staging.example.io/graphql?key=$ENDPOINTS_KEY HTTP/1.1" from 10.0.32.100:45810 - 200 111B in 235.102564ms

[Seun Ore]

Could you check your DNS record to see if it maps to ESP IP.

How do I figure out esp IP?

[Wayne Zhang]

From ESP container log, the traffic did not go to ESP.    Here is how to find out ESP IP

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/23178057-0b68-428a-ae05-89b399e7f2d5n%40googlegroups.com.

[Seun Ore]

Yes of course. I redeployed endpoints with those additions:

host: "staging.example.io"

x-google-endpoints:

- name: "staging.example.io"

  target: "34.89.30.xxx"

The result hasn't changed. ESP logs remain the same as above. I am seriously missing something. 

[Seun Ore]

Below are the list of running services: gcloud services list

NAME                                 TITLE

apigateway.googleapis.com            API Gateway API

bigquery.googleapis.com              BigQuery API

bigquerystorage.googleapis.com       BigQuery Storage API

cloudapis.googleapis.com             Google Cloud APIs

clouddebugger.googleapis.com         Cloud Debugger API

cloudresourcemanager.googleapis.com  Cloud Resource Manager API

cloudtrace.googleapis.com            Cloud Trace API

compute.googleapis.com               Compute Engine API

container.googleapis.com             Kubernetes Engine API

containerregistry.googleapis.com     Container Registry API

datastore.googleapis.com             Cloud Datastore API

deploymentmanager.googleapis.com     Cloud Deployment Manager V2 API

dns.googleapis.com                   Cloud DNS API

endpoints.googleapis.com             Google Cloud Endpoints

endpointsportal.googleapis.com       Cloud Endpoints Portal

iam.googleapis.com                   Identity and Access Management (IAM) API

iamcredentials.googleapis.com        IAM Service Account Credentials API

logging.googleapis.com               Cloud Logging API

monitoring.googleapis.com            Cloud Monitoring API

oslogin.googleapis.com               Cloud OS Login API

pubsub.googleapis.com                Cloud Pub/Sub API

redis.googleapis.com                 Google Cloud Memorystore for Redis API

servicecontrol.googleapis.com        Service Control API

servicemanagement.googleapis.com     Service Management API

serviceusage.googleapis.com          Service Usage API

sql-component.googleapis.com         Cloud SQL

sqladmin.googleapis.com              Cloud SQL Admin API

staging.example.io                   roava API Gateway

storage-api.googleapis.com           Google Cloud Storage JSON API

storage-component.googleapis.com     Cloud Storage

storage.googleapis.com               Cloud Storage API

[Wayne Zhang]

Could you use ESP IP directly in the client?  Check the ESP log to see the traffic go there.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/2d3e7938-3e77-4feb-9e7a-3436b7f072c0n%40googlegroups.com.

[Seun Ore]

I did exactly that like below:

curl --location --request POST 'http://34.89.83.221:9000/graphql' --header 'Content-Type: application/json' --data-raw '{"query":"mutation createPhone ($input: CreatePhoneInput!) {\n    createPhone (input: $input) {\n        success\n        message\n        token\n    }\n}","variables":{"input":{"phone":"09082911633","device":{"os":"Android 10","brand":"Samsung","deviceId":"Device ID","deviceToken":"Device TOken"}}}}'lt 

The result is the same as before:

{"data":{"createPhone":{"success":true,"message":"phone number created","token":"01ernbm9bae7fv4wpw56xatt2w"}}}

The esp log hasn't changed too

INFO:Fetching an access token from the metadata service

INFO:Fetching the service config ID from the rollouts service

INFO:Fetching the service configuration from the service management service

INFO:Attribute zone: europe-west2-c

INFO:Attribute project_id: my-project-id

INFO:Attribute kube_env: KUBE_ENV

nginx: [warn] Using trusted CA certificates file: /etc/nginx/trusted-ca-certificates.crt

[Wayne Zhang]

is port 9000 your backend, not ESP,  ESP port is 8081?  

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/84636b1b-9bef-4455-b909-61233f9449e6n%40googlegroups.com.

[Seun Ore]

My backend runs on port 9000 yes. ESP port is 8081

spec:

containers:

# [START esp]

- name: esp

image: eu.gcr.io/***/platform/endpoints-runtime:1.54.0

args: [

"--http2_port=8081",

"--backend=ms-api:9000",

"--service=staging.example.io",

"--rollout_strategy=managed",

"--enable_debug"

]

ports:

- containerPort: 8081

- image: eu.gcr.io/***/services/ms.api:0.2.1

imagePullPolicy: Always

tag:

name: ms-api

ports:

- name: ms-api

containerPort: 9000

[Wayne Zhang]

So,  when you use port 9000, you are talking to your backend directly.  your request bypassed ESP.   You should use port 8081.  But I don't know how your service config,   Your GKE "service" may have port config too,  

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/6bb896ee-306c-49b8-95f8-5861217bafcen%40googlegroups.com.

[Seun Ore]

I can no longer post messages. They get deleted immediately. Have I broken a rule?

[Seun Ore]

These are my ms-api deployment manifest and the corresponding service manifest file.

[Wayne Zhang]

What is your current issue now?   

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/87426c90-be0a-4b70-882e-fd461aa83a7an%40googlegroups.com.

[Seun Ore]

The situation has remained the same. The issue you raised, which is that, traffic isn't going through ESP, is the main problem here. I am looking at this tutorial on medium, which built on top of google documentation  to further troubleshoot. I am really frustrated and pressure is mounting on me to secure our APIs. More services are being added weekly. 

[Wayne Zhang]

How can we help you to debug the "traffic is not going through ESP" problem?  What IP:port were you used when making the calls?  Does that IP:port point to the ESP container? 

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/c8978e0d-585a-47c8-828c-36c000f3e2cfn%40googlegroups.com.

[Seun Ore]

The esp container listens on port 8081 and my service container runs on port 9000 as shown below:

spec:

  containers:

# [START esp]

  - name: esp

  image: eu.gcr.io/***/platform/endpoints-runtime:1.54.0

  args: [

  "--http_port=8081",

  "--backend=127.0.0.1:9000",

  "--service=staging.example.io",

  "--rollout_strategy=managed",

  "--enable_debug"

]

# [END esp]

  ports:

  - containerPort: 8081

  - image: eu.gcr.io/***/services/ms.api:0.2.1

    imagePullPolicy: Always

   tag:

   name: ms-api

   ports:

    - name: ms-api

    containerPort: 9000

k get service -n staging -owide

NAME               TYPE               CLUSTER-IP        EXTERNAL-IP        PORT(S)                         AGE                  SELECTOR

ms-api          LoadBalancer        10.6.216.226        34.89.83.221             9000:30190/TCP              4d12h                  app=ms-api

Then curl request with API KEY

curl --location --request POST 'http://34.89.83.221:9000/graphql' --header 'Content-Type: application/json' --data-raw '{"query":"mutation createPhone ($input: CreatePhoneInput!) {\n    createPhone (input: $input) {\n        success\n        message\n        token\n    }\n}","variables":{"input":{"phone":"09082911633","device":{"os":"Android 10","brand":"Samsung","deviceId":"Device ID","deviceToken":"Device TOken"}}}}'

Response
{"data":{"createPhone":{"success":true,"message":"phone number created","token":"01erqy6sm64he4zf5n3tkqztd4"}}}

The response should have produced authentication error since it is without API key

ESP LOGS

INFO:Fetching an access token from the metadata service

INFO:Fetching the service config ID from the rollouts service

INFO:Fetching the service configuration from the service management service

INFO:Attribute zone: europe-west2-c

INFO:Attribute project_id: roava-io

INFO:Attribute kube_env: KUBE_ENV

nginx: [warn] Using trusted CA certificates file: /etc/nginx/trusted-ca-certificates.crt

MY SERVICE LOGS

[cors] 2020/12/04 23:33:20 Handler: Actual request

[cors] 2020/12/04 23:33:20   Actual request no headers added: missing origin

2020/12/04 23:33:20 [ms-api-5474bd8f69-5sfzg/piSvvIaZCD-000001] "POST http://34.89.83.221:9000/graphql HTTP/1.1" from 192.168.0.27:22948 - 200 111B in 235.541238ms

The traffic obviously didn't go through esp but instead being received by my service ms-api

[Wayne Zhang]

Sorry,   I did not mean to ask you to change the GKE container port.  I wanted you to change in your client request when you make a http call.  Sorry for the confusion.

curl http://ESP_POD_IP:8081  // The traffic should go to ESP

If you call

curl http://ESP_POD_IP:9000  // The traffic should go to your backend directly, bypass ESP

On Thu, Dec 3, 2020 at 3:47 PM Seun Ore <se...@roava.app> wrote:

I changed my service port ms-api:9000 to ms-api:8081. of course this required that I effect the same change across all instances where it appeared including ms-api service manifest. Here is the result:

INFO:Fetching an access token from the metadata service

INFO:Fetching the service config ID from the rollouts service

INFO:Fetching the service configuration from the service management service

INFO:Attribute zone: europe-west2-c

INFO:Attribute project_id: roava-io

INFO:Attribute kube_env: KUBE_ENV

nginx: [warn] Using trusted CA certificates file: /etc/nginx/trusted-ca-certificates.crt

2020/12/03 23:30:59 [emerg] 1#1: bind() to 0.0.0.0:8081 failed (98: Address already in use)

2020/12/03 23:30:59 [emerg] 1#1: bind() to 0.0.0.0:8081 failed (98: Address already in use)

2020/12/03 23:30:59 [emerg] 1#1: bind() to 0.0.0.0:8081 failed (98: Address already in use)

2020/12/03 23:30:59 [emerg] 1#1: bind() to 0.0.0.0:8081 failed (98: Address already in use)

2020/12/03 23:30:59 [emerg] 1#1: bind() to 0.0.0.0:8081 failed (98: Address already in use)

2020/12/03 23:30:59 [emerg] 1#1: still could not bind()

Was concerned about this error because these are two different containers. This shouldn't happen

I have concern around the --backend flag. In this google tutorial,  the --backend flag points to 127.0.0.1:8080 and not the sample API echo. Truth is that it worked as I followed through. 

Let me share my complete deployment and service files. I am very desperate to get this sorted out.

On Friday, 4 December 2020 at 00:10:42 UTC+1 qiwz...@google.com wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/fef4e509-1dcc-4da2-b9bc-8c94ac9394b4n%40googlegroups.com.

[Seun Ore]

apiVersion: v1

kind: Service

metadata:

labels:

name: ms-api

name: ms-api

spec:

type: LoadBalancer

ports:

- protocol: "TCP"

port: 9000

targetPort: 9000

selector:

app: ms-api

On Sat, 5 Dec 2020 at 00:28, Wayne Zhang <qiwz...@google.com> wrote:

May I see your GKE "service" configuration?  The one with LoadBalancer,  how do you map 9000 to target port.  

It is so confusing that your service port is 9000,  the same as your backend port.  

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/ea4dcd18-278f-47ee-9804-f1856bdd13fan%40googlegroups.com.

[Wayne Zhang]

In GKE,  all containers in a pod share the same port space.   Your backend is using 8081,  so ESP container should not use it.  Why do you want ESP to use the same port as your backend?

On Thu, Dec 10, 2020 at 2:31 AM Seun Ore <se...@roava.app> wrote:

I changed my service port ms-api:9000 to ms-api:8081. of course this required that I effect the same change across all instances where it appeared including ms-api service manifest. Here is the result:

INFO:Fetching an access token from the metadata service

INFO:Fetching the service config ID from the rollouts service

INFO:Fetching the service configuration from the service management service

INFO:Attribute zone: europe-west2-c

INFO:Attribute project_id: my-project-id

INFO:Attribute kube_env: KUBE_ENV

nginx: [warn] Using trusted CA certificates file: /etc/nginx/trusted-ca-certificates.crt

2020/12/03 23:30:59 [emerg] 1#1: bind() to 0.0.0.0:8081 failed (98: Address already in use)

2020/12/03 23:30:59 [emerg] 1#1: bind() to 0.0.0.0:8081 failed (98: Address already in use)

2020/12/03 23:30:59 [emerg] 1#1: bind() to 0.0.0.0:8081 failed (98: Address already in use)

2020/12/03 23:30:59 [emerg] 1#1: bind() to 0.0.0.0:8081 failed (98: Address already in use)

2020/12/03 23:30:59 [emerg] 1#1: bind() to 0.0.0.0:8081 failed (98: Address already in use)

2020/12/03 23:30:59 [emerg] 1#1: still could not bind()

Was concerned about this error because these are two different containers. This shouldn't happen

I have concern around the --backend flag. In this google tutorial,  the --backend flag points to 127.0.0.1:8080 and not the sample API echo. Truth is that it worked as I followed through. 

Let me share my complete deployment and service files. I am very desperate to get this sorted out.

On Friday, 4 December 2020 at 00:10:42 UTC+1 qiwz...@google.com wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/e1519385-b357-4d86-bf34-23942e65f408n%40googlegroups.com.