Fairly basic config doesnt seem to be working

[dan.b...@theladbiblegroup.com]

Hi all, i'm having a bit of trouble configuring Endpoints and was hoping someone could help.

Our swagger file seems to work and i can see all the expected endpoints listed in the endpoints UI

I'm using ESP in kubernetes with Endpoints and im getting some odd error messages - i'm using a project key which has been added to the kube config with -k and the key is visible in the list of authorised keys in the API credentials page. I haven't enabled any  other security in the api spec but our api does internally check for a JWT bearer token.

2017-07-20T16:24:11.475914471Z 10.40.0.1 - - [20/Jul/2017:16:24:11 +0000] "GET /v1.0/videos?filters=type.equals(%27standard) HTTP/1.1" 502 186 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

2017-07-20T16:24:11.476006798Z 2017/07/20 16:24:11 [error] 10#10: *21 no live upstreams while connecting to upstream, client: 10.40.0.1, server: , request: "GET /v1.0/videos?filters=type.equals(%27standard) HTTP/1.1", upstream: "http://app_server/v1.0/videos?filters=type.equals(%27standard)", host: "staging-api.projectoneshot.com"

2017-07-20T16:24:12.067839329Z 2017/07/20 16:24:12 [error] 10#10: *21 connect() failed (111: Connection refused) while connecting to upstream, client: 10.40.0.1, server: , request: "GET /v1.0/videos?filters=type.equals(%27standard) HTTP/1.1", upstream: "http://[::1]:3000/v1.0/videos?filters=type.equals(%27standard)", host: "staging-api.projectoneshot.com"

2017-07-20T16:24:12.067895487Z 2017/07/20 16:24:12 [warn] 10#10: *21 upstream server temporarily disabled while connecting to upstream, client: 10.40.0.1, server: , request: "GET /v1.0/videos?filters=type.equals(%27standard) HTTP/1.1", upstream: "http://[::1]:3000/v1.0/videos?filters=type.equals(%27standard)", host: "staging-api.projectoneshot.com"

2017-07-20T16:24:12.069024156Z 2017/07/20 16:24:12 [error] 10#10: *21 upstream prematurely closed connection while reading response header from upstream, client: 10.40.0.1, server: , request: "GET /v1.0/videos?filters=type.equals(%27standard) HTTP/1.1", upstream: "http://127.0.0.1:3000/v1.0/videos?filters=type.equals(%27standard)", host: "staging-api.projectoneshot.com"

2017-07-20T16:24:12.069054906Z 2017/07/20 16:24:12 [warn] 10#10: *21 upstream server temporarily disabled while reading response header from upstream, client: 10.40.0.1, server: , request: "GET /v1.0/videos?filters=type.equals(%27standard) HTTP/1.1", upstream: "http://127.0.0.1:3000/v1.0/videos?filters=type.equals(%27standard)", host: "staging-api.projectoneshot.com"

2017-07-20T16:24:12.069091189Z 10.40.0.1 - - [20/Jul/2017:16:24:12 +0000] "GET /v1.0/videos?filters=type.equals(%27standard) HTTP/1.1" 502 186 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

2017-07-20T16:24:12.690780462Z 2017/07/20 16:24:12[error]10#10: Failed to call https://servicecontrol.googleapis.com/v1/services/staging-api.endpoints.euw-gcp-oneshot-001.cloud.goog:report, Error: FORBIDDEN: server response status code: 403, Response body: /Request had insufficient authentication scopes.

2017-07-20T16:24:12.690830183Z [libprotobuf ERROR external/servicecontrol_client_git/src/service_control_client_impl.cc:182] Failed in Report call: Service control request failed with HTTP response code 403

I'm only passing traffic from external port 80 to 8082 on the esp container then on to 3000 in our API container so i don't think there's any certificates in the mix.

Anyone got any ideas what might be causing this?

Many thanks

Dan

[Dan Ciruli]

Dan -

Was your API key issued in the same project you are using to manage the API? If not (if the client project is a different project), did you "enable" the API in the API section?

- The Other Dan

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/b6624fd0-98c7-4ff3-ac08-dd679c9a0798%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

DC

[dan.b...@theladbiblegroup.com]

Hi Dan,

Yep the keys are all from the same project and the API is enabled, I disabled and re-enabled it just to make sure.

Is there anywhere else that the permissions might need to be set?

[dan.b...@theladbiblegroup.com]

Turns out the Kubernetes cluster I was running my container on didn't have access to the API - just added allow access to all API setting to the nodes 

[ak...@refuel4.com]

https://cloud.google.com/service-control/reference/rest/v1/services/report

Authorization

Requires one of the following OAuth scopes:

• https://www.googleapis.com/auth/servicecontrol
• https://www.googleapis.com/auth/cloud-platform

[anggah...@gmail.com]

Hi Dan, I'm new in endpoints and was trying to make one on my own following exactly the same like tutorial on this link: https://cloud.google.com/endpoints/docs/grpc/get-started-grpc-kubernetes-engine. But when I logs to the esp's pod (kubectl logs POD ESP_CONTAINER), I got this message:

INFO:Fetching an access token from the metadata service

INFO:Fetching the service config ID from the rollouts service

INFO:Fetching the service configuration from the service management service

INFO:Attribute zone: asia-southeast1-a

INFO:Attribute project_id: XXXX-XXX

INFO:Attribute kube_env: KUBE_ENV

nginx: [warn] Using trusted CA certificates file: /etc/nginx/trusted-ca-certificates.crt

2018/11/03 23:16:17 [warn] 11#11: upstream server temporarily disabled

2018/11/03 23:16:17 [error] 11#11: upstream timed out (110: Connection timed out)

2018/11/03 23:17:17 [warn] 11#11: upstream server temporarily disabled

2018/11/03 23:17:17 [error] 11#11: upstream timed out (110: Connection timed out)

2018/11/03 23:18:17 [warn] 11#11: upstream server temporarily disabled

2018/11/03 23:18:17 [error] 11#11: upstream timed out (110: Connection timed out)

Is there something I missed?
Thanks.

[Andrew Gunsch]

+Wayne, do you know what those error logs might be referring to? Seeing as those are once/minute, that looks like it might be trying to do the service config fetch?

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/b6624fd0-98c7-4ff3-ac08-dd679c9a0798%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

DC

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/6d653c33-6981-4667-bffb-f3b642a2af6c%40googlegroups.com.

[Wayne Zhang]

Hi Andrew

I spotted two problems:

1) failed to call Report:  error is  

Failed to call https://servicecontrol.googleapis.com/v1/services/staging-api.endpoints.euw-gcp-oneshot-001.cloud.goog:report, Error: FORBIDDEN: server response status code: 403, Response body:    /Request had insufficient authentication scopes.

The producer project scope could not call ServiceControl.  So it is possible that it could not call ServiceManagement either.  which the log is showing time-out in every minute.

If users are using metadata server with default service_account,  need to ask them to check its scope. or the project scope. 

2) upstream connection closed:

upstream: "http://127.0.0.1:3000/v1.0/videos?filters=type.equals(%27standard)

around 2017-07-20T16:24:12.   I think this is due to backend application not responding.  It is calling port 3000 which is backend port.

Need to ask users to check their backend log for that time stamp, to see what the backend is doing.

Thanks

-Wayne