Rapid increase in app latency

[Chase Slater]

Starting yesterday evening through today (6/24 - 6/25) I have experienced an increase in latency with my app.  My traffic, utilization, and memory usage have not increased.  I have also not made any changes to the app.  

Studying the logs, it appears that any request that reads from Datastore is having latency issues while requests that use the cache are fine.  The size of the Datastore queries have not increased materially.  

I've tried changing my instance class from F2 to F4 but the latency didn't improve.  

Any thoughts?

[Olu]

Hello, Chase

I understand you observed increased Latency in your App Engine application, which you found out was relating to the latency is from reads to your Datastore. There are a number of reasons that may be responsible for Datastore latency including hotspotting and ramping up traffic to a kind too rapidly, not following 500/50/5[1].  To determine specifically the root cause of the latency issue, I recommend that you contact the GCP Support Engineers[2][3]. This way a GCP Support engineer would be able to work closely with you, reviewing your application configuration and determine exactly what may be the exact cause of the long latency on reads to your Datastore.

Checking our internal issue tracker, I can confirm to you that there seems to be no incident relating to latency on Datastore. So, this issue might be specific to your setup, which would be better supported by a GCP Support Engineer collaborating with you to find out the cause. 

[1]https://cloud.google.com/datastore/docs/best-practices#ramping_up_traffic

[2]https://cloud.google.com/support/docs#contacting_technical_support

[3]https://cloud.google.com/support-hub

[Chase Slater]

Thanks for the reply!

I did some additional profiling and it turns out Datastore is not the issue but setting cache entries to Redis.  I have a RedisLabs Cloud deployment and my redis_client.get commands are taking <1s but my redis_client.set commands are taking 40 seconds, these used to be on the order of 1s.    

I've looked at the Redis SlowLog but my worst latencies are 10ms.  The objects I'm caching are 3MB or less and I'm well below my size and connection limits.  The .set() command is called once every few minutes.  

My app is located in us-central and my Redis Labs instance is deployed to us-central1-1.gce.cloud.redislabs.com. 

Note that I am not using Memorystore for Redis.  I am connecting to a RedisLabs cloud instance and billed directly by Redis Labs.  Any advice on troubleshooting?  It feels like a networking problem. 

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/jy3tAGsERJY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/d9b5e9ce-448c-4ab7-ac60-9b8507563650o%40googlegroups.com.

[Chase Slater]

I tried using Memorystore for Redis with the VPC connector and I'm still seeing the same issue.  If I try running the same commands from the cloud console I have no issue.  Something is going wrong inside the App Engine environment for me.  

Here is the psuedo code:

# see if data is cached
cached = redis_client.get('my_key')

if cached is not None:

    return cached

# data not cached, query from datastore

q = datastore_client.query(kind='MyKind')

result = list(q.fetch())

# cache data for next query

t = time.time()

success = redis_client.set('my_key', result, ex=7200)

logging.info(f'redis.set took {time.time() - t} seconds')  # this takes 35 seconds with memorystore for redis, <1 s from cloud console

[George (Cloud Platform Support)]

Hello Chase, 

As this issue is specific to your app, you should either contact a GCP Support Engineer, or, if you think this is GCP-related, you should open an issue in the Public Issue Tracker, marking it as private. Publicly visible postings as in this Group are not the proper way to report an issue, as often private information items such as project numbers are requested.