Datastore async ops, query vs get, and batches

[Tim]

I'm trying to get rid of queries as much as I can (anticipating the move to HR datastore), and using pre-computed keys and lists of keys much more so I can simply db.get() items I need without needing to worry about eventual consistency of queries (or put my entire datastore into one huge entity group).

But one nice thing about using a db.Query as an iterable (this is python terminology) is that the query pre-fetches a few results and then lets you process them while more results are fetched, which sounds nice and efficient, whereas db.get() blocks until it returns a complete list and db.get_async() returns a future of a similarly completely fulfilled list.

Short of breaking up my list of keys and doing my own chunking (feels ugly - and how do I know if the chunksize I use is counter-productive), the other option would be to do a query with "KEY IN" but I note that use of "IN" is subject to a maximum of 30 items.

So what I'm really after is something like db.get_iterable() (returns immediately the iterator blocks until each value is returned, I'd suggest this isn't obliged to return results in the same order as the request list) or an async API that lets me get started with partial results without having to wait for everything ("fetch()" ?).

Is there a better way for me to do this now?

Am I plain wrong to even worry about this?

Or is this maybe a "yeah, we know... coming soon" type question?

Cheers

--

Tim