Incoming request dalyed by 4+ seconds

[Tomas]

Hello there,

I've noticed that last month was pretty bad for my java apps running on app engine. Almost everything was unbelievably slow:

1) saving simple java bean into cache consistently takes half second (sometime more)
2) saving same bean (4 String fields) into datastore can take 10+ seconds (Using Objectify)
3) also my incoming requests are being hold for 5 seconds before they reach my servlet
4) looks like external http fetches are also taking more than usual (used to do them in 30-50ms now they take 500ms)

I've been charged with $60 for last month (App engine front-end instances) while usually charge is around $2-3 per app... And it's very hard to trace the reason for a such high charge, for example my trace console wasn't able to load at all etc)...

Does anyone have the similar issue? Also how could I trace the reason, why is the request held 'somewhere' before actually being send to my servlet?

Thanks...

[Nicholas (Google Cloud Support)]

While the time spans you describe do sound fairly long considering the related actions, there isn't really enough information to know exactly what the root cause might be.

I would normally recommend consulting Stackdriver Trace to get more reliable statistics on the where the bulk of your application time is being spent.  This will likely reveal exact times for calls to memcache, Datastore and the URL Fetch service.  As for the overall time, you may look at the protoPayload.latency in the Stackdriver Logging log entry for your service to get the time during which the request was being handled by your application.

To compare the above measurements with the end-user experience, you may want to get a few HAR captures of requests to your application from a given client.  Studying this as well will allow you to compare the total time for the request/response exchange to the time spent by your application receiving/responding to the client request.

• Do you have any more specific time frames for when you noticed such latency?
  
• Are you using the standard or flexible environment?
  
• Where is your application serving and where are client requests coming from?
  
• How are you measuring the time taken for the various operations you listed?
  
• Are the more significant latencies specific to a particular service, endpoint, URL, etc.?
  

Regarding charges with App Engine front-end instance hours, there was a known issue with charges not being applied correctly.  I would recommend following that public issue closely for updates.  You may also receive communications if it was determined that your project(s) was/were affected.