Getting Started with Endpoints for Cloud Run - Request Timeout or Request Too Large Errors

[John Ragone]

Preface

I've just finished the Getting Started with Endpoints for Cloud Run tutorial for the second time. When I first tried running this tutorial with my own Cloud Run app, I got either a 413 Request too large or 504 Upstream request timeout error. Thinking that I may have taken too large a leap trying to deploy Endpoints on my Cloud Run service, I started the tutorial over using only the tutorial code, i.e., creating a Cloud Run service with the Cloud Run quickstart and using the given YAML in the Endpoints tutorial. Unfortunately, again, after running through all the commands, enabling services, and deploying Endpoints, I get either 413 Request too large or 504 Upstream request timeout.

Issue

Following the Getting Started with Endpoints for Cloud Run, I get either a 413 Request too large or 504 Upstream request timeout error. I feel like I must be missing something major, as I can't find any other comments or documentation on this issue. Has anyone run into this issue before or does anyone have advice as to how I can debug the issue?

What I've Tried So Far

Beyond restarting the tutorial with the Cloud Run tutorial image, I've looked at the logs and seen that one request creates many error logs within milliseconds of each other. To be clear, the Cloud Run service is functioning, as I've tested both my own and Google's service without the Endpoints layer. I didn't have any issues accessing the services then.

Let me know what else I can provide, and thanks in advance for any help.

[Wayne Zhang]

Could you enable ESPv2 debug and look at log of ESPv2 service?   It may provide some clues.   You can enable it in this step

"gcloud run deploy CLOUD_RUN_SERVICE_NAME"

by adding this flag:

" --set-env-vars=ESPv2_ARGS=--enable_debug \

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/cf80f34d-0c17-4d36-86be-854ecab14bc4n%40googlegroups.com.

[John Ragone]

When I make one GET request to /hello, the logs have one 514 error and sixteen 413 warnings that span about 15 seconds since the call. I'm not sure why there's so many errors for just one request, but it seems after 15 seconds, it hits the default timeout restriction. The httpRequest requestSize property in the 413 errors ranges from 1500 to 15000. I don't see how this could trigger a 413, as even 15000 bytes is only 15kB. Could this be a misconfigured limit?

[John Ragone]

Sorry I should also mention this is after setting the environment variable.

[Wayne Zhang]

What is your request size in /hello?   What are all the other 6 requests?  are they all /hello?   If you don't mind, could you post the log here?

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/34ce7cdd-6ed2-4a56-936d-d44ab041a480n%40googlegroups.com.

[John Ragone]

I've attached an example of a 504 error and one of the number of 413 errors that occur from making a GET request to /hello. These requests were triggered just by pasting the URL in the browser.

[Teju Nareddy]

Thanks, you pasted the Endpoints access logs. That shows the 504 and 413 you mentioned.

Can you also post the application logs for the Cloud Run service `hello-test`? You can read about how to view the application logs here: https://cloud.google.com/run/docs/logging#viewing_logs

There may be a lot since you turned on `--enable_debug`. I believe there should be a way to download them and attach them here.

[John Ragone]

I'm having a little trouble getting the logs. It's true that there's a lot, and from the log viewer I can only download 300, which doesn't span very long in time. I created a sink for the logs, but there's nothing in it yet. I'll attach what I downloaded, though.

[Wayne Zhang]

My tip with viewing debug log is:  choose to "INFO" level first to find the request log line with that error,  then "pin" to it, and switch the level to "Debug" to get the log lines before that one.

BTW,  could you try to use "cURL" to send a single request, instead of using browser 

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/1ef99427-1faa-4168-9bcb-b14057bfbdd1n%40googlegroups.com.

[Teju Nareddy]

Those logs help a lot, I found some interesting things in the config that is being generated. I included a portion at the end of this message.

The problem is that ESPv2 is sending the request to itself instead of the backend. That would explain why you see more than 1 access log for every request you make. You essentially configured ESPv2 to proxy requests to itself, which turns into an infinite loop until Google / Cloud Run stops it.

In the config at the end of the message, you can see the `hostRewriteLiteral` has the ESPv2 address. It should have the backend address instead. 

My guess on why this is happening is that you misconfigured the openapi-run.yaml file. Specifically for the `x-google-backend`, you need to include the address of the URL of the backend Cloud Run service. Not the ESPv2 service. After all, it's called `x-google-backend` :)

I can update our documentation to clarify.

```

{
   ...
   "routeConfig":{
      "name":"local_route",
      "virtualHosts":[
         {
            "name":"backend",
            "domains":[
               "*"
            ],
            "routes":[
               {
                  ....
                  "route":{
                     "cluster":"hello-test-5flbx7dxjq-uc.a.run.app:443",
                     "hostRewriteLiteral":"hello-test-5flbx7dxjq-uc.a.run.app",
                     "timeout":"15s"
                  }
               }
            ]
         }
      ]
   },
   ...
}

```

You received this message because you are subscribed to a topic in the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-cloud-endpoints/xsrSkpCRmII/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/1ef99427-1faa-4168-9bcb-b14057bfbdd1n%40googlegroups.com.

--

Teju Nareddy nare...@google.com Software Engineer

[John Ragone]

Ohhhhh!!! Wow I really screwed that one up 😄. I don't know why I saw CLOUD_RUN_SERVICE_NAME, and didn't think beyond putting my backend service name there. I think I doomed myself once I made that mistake at the start, along with not considering the role ESPv2 was playing. Thanks a ton! That just cleared up a number of conceptual questions I had haha.

[Wayne Zhang]

Teju,  good catch.   Thanks.  -Wayne

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/CAMV0QbtwTF5VCOZnUCKa71maKrXXMrgyhLhgHHXGH54C08ph5g%40mail.gmail.com.

[Steve Geggie]

Hi Teju,

I'm also receiving 504 errors.     Using the "hello" demo, I can deploy the image and create the endpoints.   When I attempt to do the following per the ESPv2 document, things stop working:

1. chmod +x gcloud_build_image ./gcloud_build_image -s hello-ef4dv4yxsq-uc.a.run.app \ -c 2021-06-02r1 -p document-tools 
2. gcloud run deploy hello \ --image="gcr.io/document-tools/endpoints-runtime-serverless:2.26.1-hello-ef4dv4yxsq-uc.a.run.app-2021-06-02r1" \ --allow-unauthenticated \ --platform managed \ --project=document-tools
   

Thanks for any guidance you may provide.

Steve

[Vivek Jha]

I was stuck at this for past 2 days, exact same issue.
Was using host url in BACKEND_SERVICE_NAME -_-

Thanks Teju :)