How to do a batch query of the datastore?

[John Beckett]

Is there a way of batch querying the datastore in a single API request similar to the way there can be a batch get (via GetMulti) once one already has the keys?

[John Beckett]

I realised that this may be better asked in the general Google App Engine group, and so I have asked it there as well at this URL: https://groups.google.com/forum/#!topic/google-appengine/23x6CBBCp8Y

[Dave Day]

There's no way in the underlying API to perform multiple queries as a single call, so if you do need to use multiple queries then you will have to make multiple calls. It is possible to perform each inside its own goroutine to reduce latency, but that won't reduce the number of calls.

Is it possible to restructure your data to avoid needing to do a query? Reading the linked thread, you have a list of email addresses and want to see if there are entities already existing for each. Could you make the keys of those entities actually be the email address (or hash thereof) so you can make a slice of keys and run GetMulti? Or if you can't change the keys you have, can you maintain a parallel list of tiny entities just for this purpose?

[John Beckett]

There's no way to restructure the keys, as a single entity can have more than one email address, which is also changeable.  The idea of maintaining a parallel list of tiny entities for this purpose may be a workable solution though, I just need to check whether the tradeoff will be worth it.  Thanks for the suggestion Dave.

On 30 May 2016 at 06:21, Dave Day <d...@golang.org> wrote:

There's no way in the underlying API to perform multiple queries as a single call, so if you do need to use multiple queries then you will have to make multiple calls. It is possible to perform each inside its own goroutine to reduce latency, but that won't reduce the number of calls.

Is it possible to restructure your data to avoid needing to do a query? Reading the linked thread, you have a list of email addresses and want to see if there are entities already existing for each. Could you make the keys of those entities actually be the email address (or hash thereof) so you can make a slice of keys and run GetMulti? Or if you can't change the keys you have, can you maintain a parallel list of tiny entities just for this purpose?

--
You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/VvATa2naW_Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[dele...@gmail.com]

The usual datastore queries based on keys and index.yaml are useful, but don't address all needs. I sometimes leverage the Search API when I need to retrieve entities (or entities fragments) based on custom criteria.
It requires a little plumbing, here is how I do it :
- Whenever an entity is created or updated, create or update a corresponding Document. This can be done in an async delayed task if perf matters.
- Whenever an entity is deleted, delete a corresponding Document. This can be done in an async delayed task if perf matters.
- Design your searchable Document struct so it contains the datastore key, and the fields that you want to search.
- Design your searchable Document struct so that it already contains the subset of fields that you want to display in result list. Then the list wan be displayed directly, not needing a subsequent datastore query.
- If you need the true entities (e.g. to update them), then make a single subsequent datastore query GetMulti.
- The "source of truth" is still the database, i.e. the datastore entities. The searchable documents are just redundant data stored in a search.Index . Consider extra safety code to reindex everything in case the data accidentaly diverges.

This usually works well for my needs. Hope it helps.
Cheers
Valentin

To unsubscribe from this group and all its topics, send an email to google-appengine-go+unsub...@googlegroups.com.