Datastore Small Operations - datastore.get(key)

[cloudpre]

Hi

We have a pretty big app which has not been optimized till date. We
spend over $40/day just on the reads.

This is our current stats

Datastore Read Operations 46.74 Million Ops 46.69 $0.70/ Million
Ops $32.69
Datastore Small Operations 0.00 Million Ops 0.00 $0.10/ Million
Ops $0.00


I have been reading that small operations are much cheaper and they
are done based on key.

The following function is used at many places - but I do not see
counter moving from zero for small operations.

Is datastore.get(key) a small operation? If not, what should I change
the function to make it count towards small operations.

Thanks.

// Get Entity
public static Entity getEntityFromId(String entityName, String id)
{
// Get Datastore
DatastoreService datastore =
DatastoreServiceFactory.getDatastoreService();

// Get Entity
try
{
Key key = KeyFactory.stringToKey(id);
return datastore.get(key);
} catch (Exception e)
{
return null;
}
}

[Jeff Schnitzer]

Get-by-key is a read operation, not a small operation. Small
operations are things like count() index walks and keys-only queries
(although there is still 1 read operation per query).

Most read-heavy apps can benefit significantly from memcache. If
you're using the low-level api, try this:

http://code.google.com/p/objectify-appengine/wiki/MemcacheStandalone

Jeff

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>

[cloudpre]

Jeff - thanks for the note. I have been trying to add few items in
memcache manually.

Can put operations also be saved? Let's say I am updating the same
query again after few seconds.

Does it work flawlessly in the production? The last thing I want to
see is our thousands of customers coming back and complaining.

Cheers.

[Robert Kluin]

Hi,
Memcache is a cache, so you should not depend on it being there. If
you need something persisted, you will need to write it to the
datastore. You can stick it in memcache at write time, which will
help your reads though.

Robert

[Jeff Schnitzer]

On Wed, Apr 18, 2012 at 3:36 AM, cloudpre <pbx....@gmail.com> wrote:
> Jeff - thanks for the note. I have been trying to add few items in
> memcache manually.
>
> Can put operations also be saved? Let's say I am updating the same
> query again after few seconds.

I'm not sure what you mean here. The cache covers get(), put(), and
delete() operations. Queries do not affect the cache at all.
However, you can convert queries into keys-only queries followed by
batch get()s; this will cost small datastore operations for cache hits
rather than full read operations. Objectify4 will actually do this
for you.

> Does it work flawlessly in the production? The last thing I want to
> see is our thousands of customers coming back and complaining.

There have not been any complaints for the current cache code (3.1+),
and it is fairly widely used. Older versions of the cache code (3.0
and prior) had synchronization issues, but for 3.1 I rewrote it with
some help from Ari and Alfred (Google). It should be transactionally
safe - even under heavy contention - as long as you don't hit
DeadlineExceededExceptions.

Jeff

[cloudpre]

Thanks Robert.

Jeff - thanks for chipping in.

I wanted to check if puts are also cached - eg: entity with same
values being written over and over again - will they be actually be
written to datastore.

I will use 3.1 - I liked the Global cache very much.

Twig kept the syntax simpler but I do not like the lack of caching
which will affect the high-volume apps.




On Apr 18, 9:49 am, Jeff Schnitzer <j...@infohazard.org> wrote:

[Jeff Schnitzer]

On Wed, Apr 18, 2012 at 2:45 PM, cloudpre <pbx....@gmail.com> wrote:
> Thanks Robert.
>
> Jeff - thanks for chipping in.
>
> I wanted to check if puts are also cached - eg: entity with same
> values being written over and over again - will they be actually be
> written to datastore.

The cache is write-through; all put()s go straight to the datastore
and then expire the cache content. The cache is refreshed on a
subsequent get().

> I will use 3.1 - I liked the Global cache very much.
>
> Twig kept the syntax simpler but I do not like the lack of caching
> which will affect the high-volume apps.

If you're just using the low-level api, the CachingDatstoreService in
the Objectify 3.1 jar is the same as what's in trunk (Objectify4). If
you're actually thinking of using Objectify, grab trunk - Objectify4
has a more Twig-ish syntax and has folded in many of Twig's features
(with John Patterson's design help).

Jeff

[cloudpre]

Thanks. I will use CachingDatastoreService - As a suggestion, you
should put a list of customers using it so that they know that it's
proven.

On Apr 18, 12:22 pm, Jeff Schnitzer <j...@infohazard.org> wrote:

[Jeff Schnitzer]

Thanks... but I don't have "customers". Customers pay. This is free
software - I don't really know who is using it, I just take wild
guesses based on the # of people who have subscribed to the mailing
list and the # of questions on stackoverflow.

Jeff

[Peter Han]

"However, you can convert queries into keys-only queries followed by
batch get()s; this will cost small datastore operations for cache
hits
rather than full read operations"

keys-only query is a small datastore operation thats is clear, but
batch get of entities by key also? Thank you.





On Apr 18, 6:49 pm, Jeff Schnitzer <j...@infohazard.org> wrote:

[Jeff Schnitzer]

No, get-by-key that reaches the datastore is a Read Operation.
However, memcache hits are free.

Jeff

[cloudpre]

I settled with Generic DAO classes for Objectify wrapped by Jersey.

I am using Backbone for the frontend MVC - so far, I have to admit
that it is shaping up well.

Jerff - thanks for the library.