DatastoreTimeoutException when trying to iterate over 6 million rows

[scott]

Jeff,

We are getting a DatastoreTimeoutException when we try to iterate over
some 6 million rows of a certain kind. This is executing in a B4
backend. One interesting note is that the exception is thrown exactly
1 minute into the query. The ofy deadline has not been changed from
the default (tried setting to null as well). Also of note, we are
able to iterate over the the same set using the datastore api
directly. What is be correct way to iterate over a large set of
entities using ofy? What would cause the timeout at exactly 1 minute?

Thanks, scott

The code that generates the exception is:

for (Group group : ofy.query("Group.class").chunkSize(500))
oldGroups.put(group.groupId,group.id);

The exception is:

com.tempusnova.lib.servlet.BaseServlet service: The datastore
operation timed out, or the data was temporarily unavailable.
com.google.appengine.api.datastore.DatastoreTimeoutException: The
datastore operation timed out, or the data was temporarily
unavailable.
at
com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:
46)
at com.google.appengine.api.datastore.DatastoreApiHelper
$1.convertException(DatastoreApiHelper.java:76)
at
com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:
106)
at
com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:
72)
at
com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:
33)
at
com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:
86)
at
com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:
73)
at
com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:
156)
at
com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:
65)
at
com.googlecode.objectify.util.TranslatingIterator.hasNext(TranslatingIterator.java:
29)
at
com.googlecode.objectify.util.QueryResultIteratorWrapper.hasNext(QueryResultIteratorWrapper.java:
31)

[Jeff Schnitzer]

Weird coincidence, I just hit this GAE limitation myself. Although I
get a slightly different error message - probably because I'm running
it in a frontend task rather than a backend.

Even though you can run tasks for 10 minutes and backend requests for
aeons, any single individual query can only run for a much smaller
length of time. So you end up needing to grab a cursor and restart
the query every 30s or so. Maybe there's a good technical reason for
it but this is still incredibly frustrating.

See Alfred's comments at the end of this issue:

http://code.google.com/p/googleappengine/issues/detail?id=4432

Jeff

[scott]

Thanks Jeff. There is a lot of new thinking required when working
with all this distributed stuff. Glad you are here to help. - scott