Running Out of Database Read Operations

[David]

My application (studyvolved) was sending 3 post requests a second to
constantly update a user's page with the appropriate information. I
managed to succeed my 50k read operations in 13 successfully served
requests by 1 instance. This took between 30 seconds and 2 minutes. I
am reworking the app to only change on user refresh but I would
eventually like to return to the constantly updating model. Does
anyone have suggestions for how to do live updates without using too
many resources? What constitutes a read? Is a .gql() call more than
one read? Also, would my app's inefficiencies have shown up when I was
hosting it on my computer? It seemed fine then.

[alex]

David,

you might want to rethink the design of your app and use push updates instead of pulling:

http://code.google.com/appengine/docs/python/channel/overview.html

http://code.google.com/appengine/docs/java/channel/overview.html

Also, caching queries results will help you a lot.

[David Durst]

Thanks for the advice. But, won't channels limit me to 100 users per day? And does memcache pickup changes to the database quickly?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/9WVje6plTXkJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Robert Kluin]

Hey David,
A query counts as one read plus one read per entity returned. The
costs of various operation are detailed on the billings page:
http://code.google.com/appengine/docs/billing.html

You probably need to come up with a way to filter your queries so
they only return what is needed. Or, come up with some other way to
make the reads more efficient. There are a number of ways you could
achieve this, but without knowing more about what you're doing it is
hard to suggest something.

Most likely, you'd need (or want) to sync memcache with the
datastore yourself. To quickly test how much this would help you
could cache the query results, then clear the cache each time
something is added. It is simplistic, but it might give you an idea.

Alex's suggestion about the channel API is also good. The free
quotas are sufficient for small apps, but for anything actually using
resources they won't stretch very far.

Robert

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[Leandro Rezende]

Hi David, u must "Denormalize" as much as u can, and use Memcache.

I was with a problem like yours, i had to review all my code, to "Denormalize", but it didnt save enought "Data Read" so, i had to implement memcache.

i posted a simple code that u can replace your "GetObjectByID" for a function that uses memcache. (when u are querying only, when updating, u keep the GetObjectByID)

and, when u do a query, each query is +1 read, and each record in the query is another +1 read.

so, "Denormalizing" u will query less, and memcaching u will save on getobjectbyid.

i hope it helps

Leandro

2012/2/6 Robert Kluin <robert...@gmail.com>