Unstable latency inside of the Google infrasturcture during execution requests to Google AppEngine Standard

[Andrey Hudyakov]

Hello,

I faced strange latencies during execution HTTP requests to a Java (Spring Web) application which is deployed on Google AppEngine Standard.

For testing purposes, I've used

- VM in GCP and located in us-central1 region to make client requests

- Java Web App in Google AppEngine Standard and deployed in us-east1 region

Made exactly the same GET request to my application multiple times from VM in GCP. Below you can see a couple of request details

Attempt #1

curl output

time_namelookup: 0.004209

time_connect: 0.005623

time_appconnect: 0.01674

time_pretransfer: 0.016828

time_starttransfer: 0.246101

size_request: 280

--------

time_total: 0.24617

Google AppEngine Wallclock time

Total clock time in milliseconds spent by App Engine on the request. This time duration does not include time spent between the client and the server running the instance of your application (According to https://cloud.google.com/appengine/docs/standard/java/logs)

37ms

So I expect that latency on Google network can be calculated as

time_starttransfer - time_appconnect - warclock_time

Try to explan this formula: we subtract time to connect to Google Frontend Server and time of processing requst by GAE infrastructure from time to receive the first byte of a response

And we can see that we lost 192.361ms in Google Network

Attempt #2

curl output

time_namelookup: 0.012408

time_connect: 0.014181

time_appconnect: 0.024485

time_pretransfer: 0.024586

time_starttransfer: 0.107834

size_request: 280

--------

time_total: 0.107901

Google AppEngine Wallclock time

Total clock time in milliseconds spent by App Engine on the request. This time duration does not include time spent between the client and the server running the instance of your application (According to https://cloud.google.com/appengine/docs/standard/java/logs)

46ms

Now we lost only 37.349ms in Google Network

The difference between ~192ms and ~37ms makes me confused a bit.

Could you please help me to understand this issue and may be suggest what I can do with it to make this latency low?

[David (Cloud Platform Support)]

Hello,

Just by looking at this, it may be hard to tell why there’s a discrepancy between the first and second attempt. It could be related to cold starts 1,2. I would recommend you to contact GCP support as they are able to inspect your App Engine service and provide you with more information regarding this discrepancy on latency you are seeing.