AppEngine-Go Flex big difference in dashboard and logged request latency

[Andrii Sydorchuk]

Hi,

I am running go app in flexible environment writing request body to Google cloud Bigtable.

There seems to be a huge difference in latency as reported by AppEngine dashboard and actual request logs (screenshots attached).

P95 on the dashboard is around 300-400ms, while service logs suggest it should be around 20 ms.

Has anyone encountered this before? does it mean the requests spend additional 280-380 ms in some sort of AppEngine queue?

[Katayoon (Cloud Platform Support)]

The 95th percentile and 99th percentile values in the dashboard tell you the point at which 95% and 99% of your traffic is experiencing latency while in the logs you see the latency for each request.

[Andrii Sydorchuk]

Hi Katayoon,

Thank you for your message. I understand the difference between 95% and log latency.

The thing is that majority of logs have latency 20 ms, and there is only 0.005% of logs with latency higher than 200ms.

[Andrii Sydorchuk]

To verify my previous statement, I also created log sink to BigQuery.

With 12 000 rows in the table, please find the output of percentile queries below:

SELECT NTH(99, QUANTILES(httpRequest.latency, 101)) FROM [XXX:req_latency.appengine_googleapis_com_nginx_request_20180321]

0.99 latency: 0.023 seconds

SELECT NTH(999, QUANTILES(httpRequest.latency, 1001)) FROM [XXX:req_latency.appengine_googleapis_com_nginx_request_20180321]

0.999 latency: 0.141 seconds

SELECT NTH(9999, QUANTILES(httpRequest.latency, 10001)) FROM [XXX:req_latency.appengine_googleapis_com_nginx_request_20180321]

0.9999 latency: 0.715

There is definitely some big discrepancy here between what's reported by the AppEngine dashboard and actual logs latency.

[Katayoon (Cloud Platform Support)]

Hi Andril, I should note that Public Issue Tracker is for reporting Google Cloud Platform-end issues and feature requests. So, I recommend that you create a private Issue Tracker case and provide us your project number or create a case with the Cloud Support Team through your project console so that we would be able to dig into the issue.

[Andrii Sydorchuk]

Update:

My current understanding is that AppEngine dashboard shows latency from AppEngine load balancer.

So if a user and load balancer are in Australia while the actual service instance is in Europe, the latency displayed on the dashboard is going to be much higher than logs latency.

It would be really nice if Google took more care in explaining/documenting metrics they have in dashboards.

[Katayoon (Cloud Platform Support)]

Thanks for the update. I should note that the logging is being originally calculated based on the Epoch time and will be shown in your logging based on what Time Zone you have set in your "Logs Viewer". That’s why I recommended to create a private Issue Tracker case and provide us your project number for further investigation.