MemCache and NDB

[Ivo Bellin Salarin]

Hi all,

I have two questions concerning memcache and its performances.

I was reading the page https://cloud.google.com/appengine/articles/best-practices-for-app-engine-memcache#distribute_load_across_the_keyspace, where I found

*The Memcache viewer in the administration console displays a list of top keys, which can be used to identify bottlenecks*

In my administration console, this section is empty. (i.e. 'No hot keys')
But yes, I am sure that there are hot keys, since among the stackdriver insights I can see the following messages

Slow memcache calls.

Your app made 1 remote procedure calls to memcache that took more than 50 ms and the slowest call took 66 ms.

Consider reducing the value of max_concurrent_requests for your app.

1) Thus, where can I find the hottest keys of my dedicated memcache?

In the meanwhile, according to the messages I can find in StackDriver, the longest (thus, probably, hottest) memcache.get(s) are made by ndb.get(). In the code of ndb.get I can see that the cache keys are **not** distributed across the keyspace (as recommended by the page above).

I think that not using memcache (use_memcache=False) would be a worse solution.

2) Is there a way to force a better key distribution for NDB, apart from monkey patching /gcloud/platform/google_appengine/google/appengine/ext/ndb/context.py::Context._memcache_prefix with a time dependent % operation? :-)

[Nick (Cloud Platform Support)]

Hey Ivo,

It seems odd that you'd see nothing for hot keys. I've emailed you to ask for your project ID, so feel free to reply to that email when you get a chance. 

In the meantime, I'm also curious about your observation of the key distribution. Could you possibly show some sample keys, sanitized/obfuscated for security, but which will still give an indication of what you mean by the key space not being properly distributed-into by NDB? If you like, you can also include the keys, without alteration, in your private email reply.

Of course, feel free to add any other useful information or any additional questions.

Cheers,

Nick
Cloud Platform Community Support

[Nick (Cloud Platform Support)]

Hey Ivo,

The qualification for a hot key is a key that receives over 100 QPS in the memcache, as our docs say [1]. You have had a few hot keys over the course of the past few days, from our investigation. These ought to have been displayed. We've taken note of this and created an internal issue to track work on it. You can follow this public issue [2] for updates.

Cheers,

Nick

Cloud Platform Community Support

[1] https://cloud.google.com/appengine/docs/standard/java/memcache/using#monitoring_memcache
[2] https://issuetracker.google.com/36103562

On Friday, March 3, 2017 at 6:00:20 AM UTC-5, Ivo Bellin Salarin wrote:

[Ivo Bellin Salarin]

Sorry, Nick, but I still have some questions.
Which are those keys that you've discovered to bypass that threshold? Are they effectively the ones related to NDB?
If my hypothesis is right, how to distribute the charge in a better way?

Many thanks,
Ivo

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/555801c2-75f1-47c9-8470-a9df33520f2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.