Question about Data Store Read operations

[Frank Mantek]

I am trying to figure out why my app is getting over quota. Under
storage is see this (limited to relevant data, stuff not shown is 0):

Datastore Write Operations 21% 0.01 of 0.05 Million Ops Okay
Datastore Read Operations 96% 0.05 of 0.05 Million Ops Limited
Datastore API Calls 2,381 Okay
Datastore Queries 213 Okay
Data Received from Datastore API 0.01 GBytes Okay
Datastore Entity Fetch Ops 47,914 Okay
Datastore Entity Put Ops 2,110 Okay
Datastore Index Write Ops 8,500 Okay
Datastore Query Ops 214 Okay

Now the only request i am currently running to test what is causing
the read operations to spike, shows this in appstats (per request):

memcache.Delete 6
memcache.Get 6
memcache.Set 5
urlfetch.Fetch 5
taskqueue.BulkAdd 1

None of those operations strike me as a datastore read.

Can someone shed light on this?

[Andrew Osipenko]

Hi Frank,

Have you solved it?

I have nearly the same problem.

I think that appstats do not count some datastore read operations. I thought my application accessed datastore during intialization before appstat filter is being invoked. But I can't find anything bad during my app initialization. Do you have any ideas?

[Robert Kluin]

Are you using sessions?

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/Tr6-P0a_XWAJ.
>
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Andrew Osipenko]

Hi Robert,

no,
session disabling is the first thing I did to reduce datastore access count.
See http://code.google.com/p/rent-map/source/browse/trunk/trunk/src/main/webapp/WEB-INF/appengine-web.xml

Second thing I did was memcache introducing. So I used all standard datastore friendly technics.

[Robert Kluin]

Hey Andrew,
What symptoms are you seeing exactly?

Robert

[Andrew Osipenko]

Hi Robert,

Thanks for you being interested to help me.

Let me tell my long sad story:
I have Java appengine app.
The app has only one page. This page performs search with various parameters, fetches 11 entities and displays search result.
My app has 3400 page views per day and each day it exceeds it's datastore read quota.

Lets count: (11 entities + 1 query) * 3400 = 41800
41800 is nearly 50000 (I have not counted other pages)
This looks ok.

After this I added memcache with refresh interval = 1 hour. But I am still getting datastore read quota exceeding.

Appstats reports the following data:

user.CreateLoginURL	674
datastore_v3.RunQuery	311
memcache.Get	309
memcache.Set	150
datastore_v3.Put	34
datastore_v3.Get	20
blobstore.CreateUploadURL	14
datastore_v3.Delete	8
mail.Send	3

Appstats data is not clear to me. According to docs it describes recent 1000 requests or so.
user.CreateLoginURL is called on each page, so appstats describe 674 page requests
datastore_v3.RunQuery=311
But why memcache.Set = 150? It should be = 311= datastore_v3.RunQuery = cache miss count

Perhaps appstats is not the app which can provide exact numbers.

My question is: do you have an idea why my app still get's datastore read quota exceeding even after memcache introducing?

-thanks,
Andrew

[Robert Kluin]

Hey Andrew,

  So you need to remember that running a query will result in many reads (the number of results, plus a scan iirc) -- not just one.  That could well be the source of your problem.

  As for why memcache isn't being effective, that's harder to say; that could be a coding issue or a memcache issue.

Robert

[Andrew Osipenko]

Hi Robert,

I got it!
I have not considered that query offset also causes small operation.

thank you very much Robert for your input.