[google-appengine] Counting query results > 1000

[Patrick Twohig]

I'm trying to figure out a way to count entities matching a certain criteria.  I was thinking of doing a key-only query and using PreparedQuery.countEntities() (Java).  However, when I do that, it always returns a value no higher than 1000.  Is there a way to get an accurate count of all entities matching a query without having to do something like....

int count = datastore.prepare( query.setKeysOnly() ).asList(FetchOptions.Builder.limit(Integer.MAX_VALUE)).size() ?

Thanks,
Patrick.


--
Patrick H. Twohig.

Namazu Studios
P.O. Box 34161
San Diego, CA 92163-4161

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Tim Hoffman]

Hi

You will need to use a cursor combined with a keys_only query (for
efficiency) and keep
resubmitting the cursor until you get all the results.

If you plan to do this in appengine process rather than via the
remote_api you will probably need to use a chain of tasks
to achieve the result if you have a lot of entities.

In python it looks something like the following (I don't do any work
with java on app engine)

def count(query):
i = 0
while True:
result = query.fetch(1000)
i = i + len(result)
if len(result) < 1000:
break
cursor = query.cursor()
query.with_cursor(cursor)
return i

myquery = mymodels.MyModel.all(keys_only=True)

x = count(myquery)

Hope this helps

Rgds

T

[Patrick Twohig]

I can't use a cursor, it has to happen in one request.  I tried using a sharded counter to provide an estimation of ranking, but that has proven unsuccessful.

[风笑雪]

def count(model, limit):
result = model.count(limit)
if limit > 1000 and result == 1000:
result = len(model.all(keys_only=True).fetch(limit))
return result

----------
keakon

2010/4/10 Patrick Twohig <pat...@namazustudios.com>:

[Patrick Twohig]

For whatever it's worth, just getting all the keys with a fetch limit of the largest possible value for an integer seems to work.  I have approximately 10,000 entities and counting the keys seems to work and not hit a deadline exception.

Cheers!

[Jeff Schnitzer]

"yet" ;-)

Jeff