Datastore Query Bundling

[Joshua Smith]

I have a model like this:

class DetailStuff(db.Model):
lots of properties

class TopLevelStuff(db.Model):
lots of properties
detail = db.ReferenceProperty(DetailStuff)

Usually I just deal with the top level stuff when I'm doing broad queries. When drilling down to learn about a particular object, I reference the detail stuff. The magic of the reference property fetches the details, without me having to worry about it.

That works great.

However, I now need to do a broad query that needs the details on every object. If I fetch the TopLevelStuff objects in one query, then when I look at the .detail member of each, it's going to do a single query to the datastore. That's bad, since I'm going to iterate over a whole mess of the top level objects.

I just optimized this by building a list of detail.get_value_for_datastore values, doing a bulk get, and then going back and stuffing the detail data into the top level objects (not as properties, but just attributes of the objects I fetched) so I could do my processing.

That works, but it's a mess.

What I'd like is a way to ask the datastore to go fetch the objects referenced by a LIST of objects as a bulk query. Then I could iterate over the top level objects, and the details would all be there ready to go.

Is there a way to do this now?

If not, perhaps we could ask google to add something like this? Any suggestions on the API?

TopLevelStuff.detail.get_all(list_of_top_level_stuff_objects) or something like that?

-Joshua

[Tim Hoffman]

Hi

Have you looked at Nick Johnsons prefetch_refprops recipes.

I have even modified them so that the caches can be re-used.  This can be a big win.

And it does the db.get on all the reference keys in one go.

http://blog.notdot.net/2010/01/ReferenceProperty-prefetching-in-App-Engine

T

[Joshua Smith]

Thanks for the lead.  That looks pretty good, although he didn't deal with optional properties, and the comments completely trash indentation, which makes it a bit hard to decipher the various commenter's fixes for that.

Hey Nick, you care to revisit this and make a version with *props that's robust to optional fields?

Also, just like the autoretry_datastore_timeouts recipe that everyone had to copy into their code and eventually moved into the SDK, I still think this ought to be built-in to the SDK, since it's clearly something everyone needs to do, and most people probably don't, and it is pretty hard to get exactly right.

-Joshua

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/F5FhkGw60QUJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.