Logs are not generated at Cloud Endpoint Console ( Istio + ESP + GKE)

[Rishabh Jain]

Hi All,

I need your help here, I spent 10 days on it but didn't resolve yet.

My API endpoints are written in gRPC protocol, deployed on GKE (K8s) along with Istio service-mesh. For example:
Product Catalog service is exposed to <Istio-Ingress IP>:<Port>,  I followed these docs https://cloud.google.com/endpoints/docs/grpc/deploy-api-backend and https://cloud.google.com/endpoints/docs/grpc/transcoding to leverages the Cloud Endpoint.

I also did troubleshoot using this doc: https://cloud.google.com/endpoints/docs/grpc/troubleshoot-gke-deployment and it seems to ok, esp container is configured properly (I'm able to see HTTP rules and service name).

c

The problem is the logs are not generated, not shown in Cloud Endpoints console and in response at header I received this from Postman:  

 

Thanks in advance 

[Teju Nareddy]

Can you check the GKE container logs for the ESP container? There is probably a permission issue calling Google Service Control from ESP. The application logs should show any permissions issues.

This may be helpful to locate the container logs: https://cloud.google.com/blog/products/management-tools/finding-your-gke-logs

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/3e8dc4f7-8d0e-4f63-a2b9-466784ad589dn%40googlegroups.com.

--

Teju Nareddy nare...@google.com Software Engineer

[Wayne Zhang]

There are a couple known issues with Istio for Cloud Endpoints ESPv1: 

* ESP has to use Http/1.0 to talk to Google Metadata server for access token.  By default Http/1.0 is disabled in Envoy.

* Envoy ways of handling Http1.0 is not compatible with ESPv1. 

The work around is to config Istio Egress to bypass GKE metadata server IP at 169.254.169.254. To make sure this Egress call bypasses all Envoy proxy.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/CAMV0QbuAW7vCMHRpft78WJWZYvskr0N%3DHqxR2DGRRn3maRrs2Q%40mail.gmail.com.

[Rishabh Jain]

Thank you for the response,

To access the  GKE metadata server IP, I created a ServiceEntry in Istio along with I also used a service account. PFA of some snips which would be helpful:-

Below are the ESP container logs and I think It is able to get access token.

Below is my deployment file of 1 microservice- product catalog which I used in GKE.

When I run below command using grpcurl, I'm able to get a response (but still I didn't see logs in Cloud Endpoint dashboard)

I have created 2 different Cloud Endpoint service:

1. Based on Transcoding HTTP/JSON to gRPC using https://cloud.google.com/endpoints/docs/grpc/transcoding

2. Based on gRPC

Tested with both, but unable to get logs.

[Wayne Zhang]

If you are using a service account key file for ESP,  then ESP doesn't need to talk to GCP metadata server.  The problem may be the service account doesn't have permission to access the Endpoint Service.  

In order to confirm this: could you check the log in the ESP container.  It should have error logs about calling :report failed with some 4xx status code.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/ac9f60c9-257d-4de6-8e43-190aa016e21dn%40googlegroups.com.

[Rishabh Jain]

The service account has permissions which it requires acc. to doc:  https://cloud.google.com/endpoints/docs/grpc/running-esp-localdev#create_service_account  

In esp logs, I didn't see any such type of error and also in above-attached Cloud logging snip

[Wayne Zhang]

Hmm, are you sure your traffic routed to ESP?   Could you "exec" into ESP container, check its /etc/log/nginx/ folder for access.log file to see if your requests have been logged?

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/97886e27-c600-4468-bc62-f8b1277303a9n%40googlegroups.com.

[Rishabh Jain]

I unable to check it, in my case, the traffic should be routed: ( -> Istio-proxy -> ESP -> Service) these 3 containers are running in a POD. After exec, I didn't find the mentioned dir:

Kindly see the directory snips of esp container, under etc then Nginx and then endpoints

but I found the log file under var/log/nginx:

In an access log file, there is no entry but in error.log it has, PFA esp_error.log file

[Rishabh Jain]

PFA of response while hit the IP from Postman.

<Istio-Ingress-Gateway IP>:<PORT>/v1/products: this prefix is defined in the cloud endpoint which is mapped to product-catalog gRPC service method (ListProducts) 

 

product.endpoints.gcp-cloud-native-app-dev.cloud.goog: this service is deployed in cloud-endpoint.

While using GET, it response 405

If I don't pass the content-type, it shows me below res.

If I pass "application/json" in Content-Type, then got below response:

[Wayne Zhang]

From esp error log.  It seems that ESP was able to call Google service control report successfully.  If so,  Endpoints log should show up. 

This is usually how I query the Endpoints log in Cloud Console:  Endpoints top page -> Service list  (click at a service) -> graphs for that service,  -> at the right bottom, there is a link "view all logs"

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/6175c01f-513a-4e11-af47-ff85e24adcf1n%40googlegroups.com.

[Rishabh Jain]

Actually, this is problem the logs are not generated, I double-checked all APIs are enable that required for this task. I also checked as per your assist.

Kindly help on this, I have spent many days on it.

[Wayne Zhang]

In the Cloud Console, in the Endpoints page, did you see any traffic in the graph.  qps,  errors graphs for your service

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/91e26297-e44d-4839-8551-7c673eb4fb67n%40googlegroups.com.

[qiwz...@google.com]

I created a case with support.  I assume your

* service name is: product.endpoints.gcp-cloud-native-app-dev.cloud.goog

&  project is gcp-cloud-native-app-dev

We could not find any log entries for this project.   Could you double check it?

On Thursday, July 16, 2020 at 1:18:10 PM UTC-7 jrish...@gmail.com wrote:

No, The traffic of any type is not shown. I have attached the deployment file of K8s along with the file used in Cloud Endpoint.

I also attached a file that is used to create a gateway for this product service in Istio-Ingress Gateway.

Please check these files and let me know if there is any mistake.

* I had converted these YAML files to .txt because of error while posting.

[Eaton Zveare]

I am also having trouble seeing logs. I do see the problem, but am unsure how to resolve it.

I ran the add-iam-policy-binding command as displayed in the "Grant ESPv2 Beta permission" step here, but it doesn't seem to be enough? https://cloud.google.com/endpoints/docs/openapi/get-started-cloud-run#configure_esp

[Wayne Zhang]

Hi Eaton,  the "add-iam-policy-binding" command at that step is for allowing ESPv2 to call your backend if your backend is not public, or without the flag "--allow-unauthenticate".

Your logging program is different,  it is the account deploying ESPv2 is not allowed to use the Endpoints service config.   Could you make sure you are using the same project when "gcloud endpoints service deploy" to deploy your endpoints service config, and "gcloud run deploy" to deploy ESPv2 Cloud run service.

If they are in different projects, you have to set the permission properly for the service account deploying ESPv2. 

--
You received this message because you are subscribed to a topic in the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-cloud-endpoints/WjfzBjkjUQk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/95590cc4-df51-43e0-a7ea-a47aec838a33n%40googlegroups.com.

[Rishabh Jain]

Hi,

 Thanks to all for giving your precious time on my issue especially Wayne.

 My issue is resolved, now I'm getting logs at Cloud Endpoint console as well as on Cloud Logging. I'm happy because I learned a lot while troubleshooting this issue and solved it on my own.
 Wayne, you can now close this case with the support team.

[Wayne Zhang]

I am glad to hear that you resolved the issue.   I will close the case.  -Wayne

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/d4c16ab8-652a-410f-896d-472f88657d81n%40googlegroups.com.

[Eaton Zveare]

This is all in the same project.

When deploying ESPv2 and my backend Cloud Run services, I assign the same service account to both.

That service account has the "Service Controller" role. So, that should ensure ESPv2 can invoke the API, but it's still returning 403. Is there another role I am missing here? Roles assigned so far:

-Cloud Trace Agent

-Cloud Run Invoker

-Service Controller

[Wayne Zhang]

Has your project owning the service account enabled the "Google service control" API?

You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/d903081c-a3a0-4274-8ec0-e14486308f57n%40googlegroups.com.

[Eaton Zveare]

Ah, it was not! Strange because it was a new project and your tutorial says "In most cases, the gcloud endpoints services deploy command enables these required services", and I didn't explicitly disable it or use a third-party app. Maybe a bug?

Everything is working now though - thank you.