Batch querying the datastore (with Go)

[JohnGB]

I'm using the Go 1.9.37 SDK with App Engine Classic (in case that matters to the answer).

I have some entities of type "user", which have one or more unique email addresses associated with each entity.  Given a list of email addresses, I would like to look up some simple data from the entities which correspond to an email address, and mark the email addresses which don't have a corresponding entity as invalid.  I have the email addresses indexed in the entity.

Is there a way of batch querying the datastore in a single API request to get the first key (there will be only one) which corresponds to each email address?  I was thinking of something similar to the way there can be a batch get (via GetMulti) once one already has the keys?

I would appreciate any suggestions on the most efficient way of doing this, as right now I'm stuck with a lot of single KeysOnly queries (one for each email address), and then a single GetMulti to get all the entities that I want.

[Nick (Cloud Platform Support)]

Hey JohnGB, 

I believe that one KeysOnly query for each email is unavoidable. You could use a Mapreduce or Dataflow pipeline to coordinate the spawning of multiple parallel processes to run those queries, joining them up once they've all completed, or you could just use asynchronous calls, being careful not to overload the individual instance. 

I hope this has been helpful advice. Let me know if you have any further questions or extra information to add, as I'll be glad to assist.

Sincerely,

Nick
Cloud Platform Community Support

[John Beckett]

Hi Nick,

Thanks, that clears up the question for me.  I already have it working with multiple queries, I was just hoping it was possible to have it as a single query without having to create a separate data set specifically for having these queries done quickly.

~ John

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/23x6CBBCp8Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/63de1865-2980-49e4-b49e-17e27abae042%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.