Batch get based on Keys, performance

[Chister Nordvik]

I am trying to get 40 000 entities from the datastore based on Keys, but that takes up to 1 minute. Why is it so slow using keys, when I can retrieve 40 000 entities with a filter in 6-8 seconds? 

http://stackoverflow.com/questions/15188473/google-app-engine-fastest-way-to-get-a-large-collection-of-objects-from-keys-us

I am sure I am missing something here, but I cannot figure out what :-/ The entity itself isn't that large so are there any other ways to get a large amount of entities from the database using objectify v3?

[Jeff Schnitzer]

Filters use eventual consistency; load-by-key uses strong consistency
by default. Try doing the batch get with
ofy().consistency(Consistency.EVENTUAL).load().key(...) and you should
expect the same performance.

Eventually-consistent fetches will get data from whatever datastore
node happens to be available at the time; strongly-consistency fetches
require some sort of complex paxos handshaking among multiple nodes
that I would have to re-watch the Google I/O HRD video from 2011 to
claim to understand.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "objectify-appengine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to objectify-appen...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Jeff Schnitzer]

BTW Objectify's global caching is not aware of the difference between
strong and weak consistency, so be careful doing an eventually
consistent load-by-key with cached entities. You can put stale data
in the cache that way.

Really the cache needs to be sensitive to the consistency of the
request and "upgrade" eventual to strong results. It's a whole
additional layer of complexity.

Jeff

[Chister Nordvik]

That was just.... magic... My request went from 57 seconds to 12 seconds with that change. Thanks a lot! We don't cache the entities so hope it hasn't any other side-effects :-) 

> email to objectify-appengine+unsub...@googlegroups.com.

[Mat Jaggard]

I'm intrigued by this. Why don't you cache?

To unsubscribe from this group and stop receiving emails from it, send an email to objectify-appen...@googlegroups.com.

[Chister Nordvik]

I didn't get much better performance with caching and with 40 000 records I got the dreaded "out of memory exception" from GAE. That annoyed 40 000 users that didn't get their notifications so I removed caching and it's pretty fast now with that "fix" from Jeff :-)

> email to objectify-appengine+unsubscribe...@googlegroups.com.

> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Jeff Schnitzer]

Generally speaking, if you're iterating lots of records, you probably
are going outside your working set and caching them will actually
decrease cache effectiveness (as well as being very slow to process).
Best to disable caching for big iterations - you can do it for an
individual query with ofy().cache(false).load()...

Jeff

>>>> > email to objectify-appen...@googlegroups.com.

>>>> > For more options, visit https://groups.google.com/groups/opt_out.
>>>> >
>>>> >
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "objectify-appengine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to objectify-appen...@googlegroups.com.

>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "objectify-appengine" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to objectify-appen...@googlegroups.com.