How to check which urls are doing a lot of Datastore reads?

[John Del Rosario]

Starting a few days ago (June 17 to be precise), my Datastore Reads quota has more than quadrupled, with no changes in the codebase.
I'm thinking maybe some url is being abused or something, or maybe a broken task. 

But in the dashboard graphs, I don't see any spikes or strange behavior.

And my taskqueues look normal.

Where could I start looking for what's wrong?

[Nick (Cloud Platform Support)]

The AppStats tool contains much of the functionality you'd want if you were looking to profile your RPC calls (such as Datastore query), although knowing that a Datastore.query occurred won't tell you how many entities were fetched.

In order to get a better picture of the source of your Datastore reads, you should do an analysis of your codebase to determine where Datastore reads can occur, and also determine whether there are any limit() or offset() operations being applied. In the absence of these, . Then, associate these locations in code with the requests that will activate them. Finally, perform an analysis of your logs to determine how often each route is being called, thus allowing you to determine how often each potential Datastore read location is being called. In order to analyze your logs, you should use the Logs API.

If you have a handler on the route /account/details that will perform a Datastore read of the past N records for your user's account when /account/details?depth=N is requested, you'll be able to determine this information while iterating through your logs for the day, and tally up the number of reads that are likely to have occurred. 

Finally, you can read a collection of great tips on managing Datastore resource usage in this docs article: Managing App Resources.

[Chad Vincent]

Also, if you're using Memcache at all (manually or through something like Objectify), check the cache hit rate and see if there's been a change there.

[Nick (Cloud Platform Support)]

Good point, Chad, that would also be a possible cause of increase in Datastore reads. 

I realize that I forgot to fully finish editing my reply before. I meant to explain that if you make a Datastore query without limit(), you could fetch the entire number of entities in that Kind. If you use offset(), be aware that you are retrieving and throwing away (unused), any entities which are before the offset amount.