Counts on big domains is very slow, here's potential solution

4 views

Skip to first unread message

Travis Reeder

unread,

Aug 16, 2010, 5:17:43 PM8/16/10

to simple-record

If you query a domain with a large number of objects, then you call results.size, SimpleRecord will run a COUNT query against SimpleDB. Currently, there is a bug that it may not count the entire domain since it should loop until no next_token is returned, but after fixing that, it can take a long time to get the final count value. It literally took 185 queries (got back next_token 185 times) on a domain with 26 million records.

So, I'm thinking it might make sense to add an optional setting that will store that count in a "counts" domain for a period of time before re-running the full count again. Then subsequent counts will do a direct select by item_name to get the count quickly.