Projection Queries and Objectify

[David Fuelling]

Hey Jeff, 

I noticed in the 1.6.5 appengine SDK release notes (here) there's a section about partial-loading of entity properties via Projection Queries: "You can now perform datastore queries that return a subset of your entity properties with the same performance and cost of a keys-only query. This feature is experimental."

Is this something that is planned for Objectify 4?  Any thoughts as to how (or if) this would work with Hybrid queries? 

Thanks!

david

[Jeff Schnitzer]

I haven't looked too closely at it, mostly because it's something I
haven't come up with a use case for myself and it's hard to create
APIs for unfamiliar features. I'm certainly interested in adding
this. Interested in proposing a first pass at how it would fit into
the API? Maybe just a .project() method on the Query interface?

Adding projection to a query would definitely prevent hybridization.

The big upcoming change that has been occupying a lot of my mental
cycles is the embedded entity feature in SDK 1.6.6. I've been
lobbying for this feature and it's really going to enable a lot of
fantastic new behavior - polymorphic nested classes with any number of
layers of collections. True trees!

Jeff

[Huseyn Guliyev]

Both of these features will be really cool to have in Objectify.

1) Question re: Embedded entity. I thought we already have embedding in OFY. How is the native one different/better?

2) As for projection queries, it is going to be really useful for those who want to get more performance/pay less on big queries. If you have big entities with dozens of fields (almost all real apps do), each time you have to pull all of the data, even if you want a specific property. This costs us a lot of $$, for instance. Being able to specify property name would therefore be very useful.

Here is what they say in GAE documentation:

Most Datastore queries return an entire entity object, but some queries only seek a few properties within an entity. You can query the Datastore for specific properties of an entity using projection queries. These queries provide lower latency and cost (like keys-only queries), but also let you return only the properties you need. 

https://developers.google.com/appengine/docs/python/datastore/queries#Query_Projection 


Thanks.

[Jeff Schnitzer]

On Wed, May 23, 2012 at 6:46 AM, Huseyn Guliyev <hus...@gmail.com> wrote:
> Both of these features will be really cool to have in Objectify.
>
> 1) Question re: Embedded entity. I thought we already have embedding in OFY.
> How is the native one different/better?

In nearly all respects, the EmbeddedEntity approach is far superior.
It will go through the normal load/save code path, so you get
polymorphism, infinitely nested collections, lifecycle methods, etc.
Objectify will support both approaches, but I am going to strongly
encourage people to prefer EmbeddedEntity.

The difference between current @Embed and the EmbeddedEntity (haven't
figured out the relevant annotations yet) are the same as the
difference between NDB's StructuredProperty and
LocalStructuredProperty. In fact, they should be binary-compatible in
the datastore. For example, let's say you have these two classes:

class Address { String city; String state; }
class Location {
@Id Long id;
String name;
Address addy;
}

The current @Embed approach will put something in the datastore with
properties like this:

name = "Here"
addy.city = "San Francisco"
addy.state = "CA"

The embedded entity approach puts something in the datastore that
looks like this:

name = "here"
addy = {
city = "San Francisco"
state = "CA"
}

The downsides of the embedded entity approach are:

* Indexes are not currently supported by the datastore. They might
not be natively supported by the datastore ever. Guido and I have
been discussing how to create a standardized set of synthetic index
properties such that Objectify and NDB are binary-compatible with
datastore contents (ie, you can use Python/NDB instances and
Java/Objectify instances against the same dataset).

* The datastore viewer treats these embedded entity structures as
opaque. Or at least, it did the last time I checked. This will
certainly change at some point as the embedded entity is part of the
"standard" set of datastore values.

There needs to be some significant discussion about how the
annotations surrounding this feature will work. I'm not entirely
certain that @Embed should refer to the 'old' embedding system. I'm
not quite ready to open up this discussion yet - but it will be soon,
because I really need this feature in Voost.

> 2) As for projection queries, it is going to be really useful for those who
> want to get more performance/pay less on big queries. If you have big
> entities with dozens of fields (almost all real apps do), each time you have
> to pull all of the data, even if you want a specific property. This costs us
> a lot of $$, for instance. Being able to specify property name would
> therefore be very useful.
>
> Here is what they say in GAE documentation:
> Most Datastore queries return an entire entity object, but some queries only
> seek a few properties within an entity. You can query the Datastore for
> specific properties of an entity using projection queries. These queries
> provide lower latency and cost (like keys-only queries), but also let you
> return only the properties you need.
> https://developers.google.com/appengine/docs/python/datastore/queries#Query_Projection

Yes, but keep in mind that Projection Queries are not quite like SQL
select statements, and the reason they are "cheap" is not strictly
because they return less data. It's because they are purely
index-walks just like keys-only queries. This imposes some severe
limitations.

Remember that a normal query walks an index BigTable, getting keys,
then issues separate fetches to the data BigTable to get the full data
blob for each one of those keys. Google charges you a read operation
for each value returned (plus an extra read op for the whole query).

A keys-only query skips the fetch to the data BigTable since you have
the key in the index. Google charges you a small op for each value
returned (plus an extra read op for the whole query).

Projection queries recognize that you can return the data in the index
BigTable without needing the extra fetch to the data BigTable. This
means you are only walking the index, doing the equivalent of a
keys-only query. Thus each row returned costs the same as a keys-only
query - a small op.

The downside of this is that you must create and update an appropriate
multi-property index, and you can ONLY select values that are in that
index. Given that you must also have single-property indexes for each
element of a multi-property index, this could easily require a dozen
extra write operations to maintain the projection index. Depending on
the read-to-update ratio of your data, this could potentially cost you
more than simply using a cached hybrid query or even a simple naive
query.

My point is that this is a very specialized tool and really should
only be used for optimizing very particular cases... and only when you
really know what you're doing.

All that said, yes, Objectify will support projection queries.

Jeff

[David Fuelling]

+1 to a .project() method on the Query interface.  It would be awesome if Objectify could help in enforcing some of the rules surrounding this type of query, too.

[Salomon BRYS]

I understand the limitations of such a feature but I also understand its benefits.

I have had this use case very often where I fetch an entire entity just to read ONE property, property that is oftenly indexed for the search I am performing.

A .project(properties, to, project) may be a good solution so, let me add my +1 ;)

[Sathyanarayanan Thilakan]

Hi,

Does objectify support projection queries now ?

Regards,

Sathya

[Jeff Schnitzer]

Not yet. Next item on the hit list.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "objectify-appengine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to objectify-appen...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Peter Rangelov]

Thanks Jeff!

You're on fire

> email to objectify-appengine+unsub...@googlegroups.com.

[Drew Spencer]

Mr Fuelling! I just recognised your photo and realised you are the creator of objectify-utils. I just posted a question about it so maybe you'd like to chime in regarding Ofy5 compatibility?

Drew

[Sathyanarayanan Thilakan]

Hi Jeff,

Is projection query now available in objectify ? 

Regards,

Sathya

--
You received this message because you are subscribed to a topic in the Google Groups "objectify-appengine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/objectify-appengine/uvLIHhHMEM0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to objectify-appen...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Regards,
  Sathya.

[Jeff Schnitzer]

Not yet. It turned out to be somewhat more complicated than I expected and I ran out of time before my next gig started. Next window of opportunity will be a couple weeks. In the mean time, work in progress is on the 'projection' branch.

The nasty issue is that projection query values don't have types; apparently the index is not self-describing, which makes a certain amount of sense. Objectify will have to do a special kind of type mapping using metadata that Objectify does not currently track, and may require changes to the Translator API.

Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appen...@googlegroups.com.

[Sathyanarayanan Thilakan]

Hi Jeff,

Please let me know if you have any clarity on when projection queries will be available in objectify. It will be very useful for the project I am currently working on. 

Thanks again for objectify :)

Regards,

Sathya

[Jeff Schnitzer]

They have been available since 5.0.3.

ofy().load().type(...).project(...)

Jeff

[Sathyanarayanan Thilakan]

Thanks very much, I will try it out.

[Jeff Schnitzer]

Nothing in the wiki docs. But there's nothing particularly complex about it; just call .project() on your query with the field(s) you wish to project. Resulting entities are not session cached.

It's probably worth adding this to the docs, which I'll add to my TODO list when I move them to github.

Jeff

On Fri, Mar 20, 2015 at 3:08 PM, Ben Fisher <bfish...@gmail.com> wrote:

Hi Jeff, is there any documentation on how to properly use projection queries in Objectify?

thanks

Ben

[Ben Fisher]