Debugging Datastore Read usage

[Mark Cummins]

Hi,

One day last week our Datastore Read Operations were 5x their normal level, and we're trying to identify why.

The problem is almost certainly caused by some corner case in our own code. My question is, how can we go about identifying what caused this issue?

As far as I can see, our only way to keep track of Datastore Reads is via the billing page ("View Usage History"), which shows the total number of reads on a given day (and there is about a 2 day delay before the report is ready). Is there any way we can get more precise information about what caused our spike in Datastore Reads? If we could narrow it down to a particular handler, or even a particular time of day, we could sort it out pretty quickly. The main dashboard at least gives you a graph of memcache usage by time of day, but nothing for Datastore Reads that I can see?

Thanks,

Mark

[Evan Jones]

Datastore gets also cost us a lot. The thing we have discussed doing, but have not done yet, is put some instrumentation into the pre-call hook [1] to count the entities involved in query and gets. This would allow us to "slice" the data by entity, app engine service, and if you added stack trace sampling, by caller. If you try something like this, I would be very interested in hearing how it works! Good luck,

Evan

[1] http://blog.notdot.net/2009/11/API-call-hooks-for-fun-and-profit

[Alex (Cloud Platform Support)]

Hi Mark,

As another option, Google Stackdriver Monitoring allows you to retrieve relevant metrics about your Datastore operations. Among the available metrics listed here, you may be interested by the “api/request_count” metric that allows you to retrieve the number of datastore requests within a timeframe as narrow as an hour. You can test this feature on the Metrics Explorer directly after enabling it. Still, be aware that the number of requests to the datastore may not be representative of the actual number of read calls involved in these requests.

Let me know if that helps you in identifying the periods of your spikes.

Cheers,

Alex

[Mark Cummins]

Aha! Very useful, thanks!