Datastore Small Operations optimization suggestions

[Martin Dvorak]

Hi all!

I'm looking for a lesson from 'pricing oriented programming'. My service has a REST-style endpoint that I use to replicate/export/import data. Surprisingly export consumes ~10x more Datastore Small Operations than Datastore Read Operations. Thus I'm almost running out of free quota. There are ~2500 entities, the export does 10x count()s and reads all the data as follows:

public List getPage(Class clazz, int from, int to, int pageSize) {

PersistenceManager pm = getPm();

try {

Query queryStatement = pm.newQuery(clazz);

if(from==0) {

queryStatement.setRange(from, to);

} else {

if(from>0) {

Cursor cursor = Cursor.fromWebSafeString(getTablePageCursors.get(clazz));

               Map<String, Object> extensionMap = new HashMap<String, Object>();

               extensionMap.put(JDOCursorHelper.CURSOR_EXTENSION, cursor);

               queryStatement.setExtensions(extensionMap);

               queryStatement.setRange(0, pageSize);

}

}

List<?> queryResultList = (List<?>)queryStatement.execute();

Cursor cursor = JDOCursorHelper.getCursor(queryResultList);

getTablePageCursors.put(clazz, cursor.toWebSafeString());

        

                    ...

} finally {

pm.close();

}

return result;

}

My question is what can I do to optimize the code. I don't think that count() is a problem. Can it be structure of entities exported? It seems that certain types consumer more small operations than others. Can it be caused by cursors? 

Any suggestion or idea will be highly appreciated!  

[sb]

memcache

http://code.google.com/appengine/docs/python/memcache/usingmemcache.html

On Nov 26, 11:54 am, Martin Dvorak <martin.dvo...@mindforger.com>
wrote:

> *Any suggestion or idea will be highly appreciated!  *

[Simon Knott]

Hi,

Your problem actually may be the counts - read this post by Alfred, who is part of the datastore team I believe.

https://groups.google.com/d/msg/google-appengine/LsVSKbFCLwI/FpYEd1fPFLYJ

Cheers,

Simon

[Martin Dvorak]

Obviously I use memcache through the application. Unfortunately this is not solution to my problem as the export I described above is typically used for backup before upgrades and majority of entities is read occasionally. It may be that I'm wrong. Is it a best practice to have a custom write through (mem)cache on top of repository and cache everything?

[Martin Dvorak]

Thanks, I have read this post before I sent my question to the group. However it 1 count() operation is much more "cheaper" that loading one entity (that in the collection I checked takes ~7 Datastore Key Fetch Ops on average).   

Actually I need to reformulate the question I originally asked:

  Is there anything I can do in order to decrease number of Datastore Small Operations beside attempting to memcache counts and the entire repository to memcache?