latency started spiking up dramatically today
49 views
Skip to first unread message
Wenzhe Gao
Sep 29, 2016, 8:27:28 PM9/29/16
to Google App Engine
Hi,
Our app has been running fine with low latency for months now. Latency started spiking up dramatically at around 6:50am this morning. No real changes on our side. I see quite a few datastore RPC deadline exceeded errors in the logs. Ex:
...
rpc = datastore_rpc.MultiRpc.wait_any(self.rpcs)
DeadlineExceededError: The overall deadline for responding to the HTTP request was exceeded.
See latency graph attached for reference. Any ideas what's going on?
Nick (Cloud Platform Support)
Oct 3, 2016, 10:37:33 AM10/3/16
to Google App Engine
Hey Wenzhe,
We have no incidents reported on the Cloud Status Dashboard for Datastore as of this moment. Sometimes performance may degrade for a short period of time dependent on Data-Center conditions. You should check out SLA on Datastore and consider implementing some exponential back-off logic in your code to handle the possibility of deadline exceeded errors. We have a document on this topic titled "Dealing with DeadlineExceededErrors".
I'll check to see if I can find any other explanation and will report back to this thread.
Cheers,
Nick
We have no incidents reported on the Cloud Status Dashboard for Datastore as of this moment. Sometimes performance may degrade for a short period of time dependent on Data-Center conditions. You should check out SLA on Datastore and consider implementing some exponential back-off logic in your code to handle the possibility of deadline exceeded errors. We have a document on this topic titled "Dealing with DeadlineExceededErrors".
I'll check to see if I can find any other explanation and will report back to this thread.
Cheers,
Nick
Cloud Platform Community Support
Nick (Cloud Platform Support)
Oct 3, 2016, 10:38:47 AM10/3/16
to Google App Engine
Hey Wenzhe,
Apologies for not including this in my last message, as I had believed your original message included the necessary information. Can you email me your project ID directly at pay...@google.com? I'll be glad to take a look for you after that. Is this still occurring, by the way?
Regards,
Apologies for not including this in my last message, as I had believed your original message included the necessary information. Can you email me your project ID directly at pay...@google.com? I'll be glad to take a look for you after that. Is this still occurring, by the way?
Regards,
Nick
Cloud Platform Community Support
Reply all
Reply to author
Forward
0 new messages