Asynchronous load of relations after constructor (GAE)

[Emanuele Ziglioli]

Here's an idea I've head to speed up complex entity loads, if anybody is interested.

I have a parent class, that is linked to a collection of a child class.

I use json to populate this collection, but that means I have to load the whole collection at the time of serialization, and that's slow and blocking.

What if we used the async() feature to load the collection after the instance is created? that way some if not all the child entities would be ready at the time of serialization.

I think I could achieve that by overriding GarMappingUtils.setIdFromKey()

QueryAsync<Child> children;

@Override setIdFromKey(...){

  super(...);

  children = UserGroupEntity.all().filter( "groupId", id ).async();

}

List<Child> getChildren {

  List<Child> entities = children.fetchKeys();

  Child.batch().get(entities); 

  return entities;

}

It's not that pretty but it should perform better.

I use fetchKeys + batch get in order to use memcache, so that's not entirely non blocking.

It's not pretty, but anything to speed up the slowest database on earth...

E

[Pascal Voitot Dev]

Crazy!!!! You're still using all of that :D

Great to hear that ;)

Pascal

--
You received this message because you are subscribed to the Google Groups "Siena" group.
To unsubscribe from this group and stop receiving emails from it, send an email to siena-discus...@googlegroups.com.
To post to this group, send email to siena-...@googlegroups.com.
Visit this group at http://groups.google.com/group/siena-discuss.
For more options, visit https://groups.google.com/groups/opt_out.

[Emanuele Ziglioli]

Found your old post here

https://groups.google.com/d/topic/siena-discuss/Uhp0sO_966o/discussion

If you see stupid things that I may have misunderstood, don't hesitate. I'm not an expert in asynchronism and I spent lots of time studying GAE doc but this is not always very clear :)

 

It moade me smile, as now you're one of the world asyncronism experts ;-)

I'm gonna play with Future and FutureWrapper (I've discovered it today...) as I spend a lot of time blocking with get() or even batch().

Also UrlFetch on GAE returns a Future, and since I use it, I'd like to see if I can use FutureWrapper as a callback.

http://ikaisays.com/2010/06/29/using-asynchronous-urlfetch-on-java-app-engine/

[Pascal Voitot Dev]

On Tue, Feb 4, 2014 at 9:20 AM, Emanuele Ziglioli <the...@emanueleziglioli.it> wrote:

Found your old post here

https://groups.google.com/d/topic/siena-discuss/Uhp0sO_966o/discussion

If you see stupid things that I may have misunderstood, don't hesitate. I'm not an expert in asynchronism and I spent lots of time studying GAE doc but this is not always very clear :)

 

It moade me smile, as now you're one of the world asyncronism experts ;-)

Yes I know i'm so good (humhumhumhum) XDXDXD

 

I'm gonna play with Future and FutureWrapper (I've discovered it today...) as I spend a lot of time blocking with get() or even batch().

FutureWrapper is a java class?

 

Also UrlFetch on GAE returns a Future, and since I use it, I'd like to see if I can use FutureWrapper as a callback.

http://ikaisays.com/2010/06/29/using-asynchronous-urlfetch-on-java-app-engine/

I must say I don't follow GAE at all for years now... is it good now?

[Emanuele Ziglioli]

FutureWrapper is a java class?

Quoting Pascal Voitot:

The other reason also is that GAE functions return a Future object. But then when you call the Future.get() function on it, it returns a GAE entity and not a Siena Entity.
What we want when we call .get() is to retrieve a Siena Model.
So we need to do the mapping to Siena Model also when calling the .get().
That's why I encapsulate the GAE Future into a SienaFutureWrapper (directly inherited from Google FutureWrapper) which provides the mapping feature. Look at the GaeAsyncPersistenceManager to see how it is used. 
This wrapper is itself encapsulated into a SienaFutureContainer which allows to hide the Java Future Exception problem

It's quite clever actually ;-)

I'm going through your async classes, and see if I can improve performances.

Each call to get() or batch() blocks and spins up new instances.

I can't say GAE is very good from that point of view, it's a real waste of resources.

Perhaps, something really counter-intuitive, like an HttpRequest Thread that keeps pausing and checking if a result is back from a Future would be the most performant solution.

GAE suits us, as we have very few users but lots of (small) data. Costs are low, critical stuff is handled by task queues in a reliable way, mail handling is reliable.

One just needs to work around its limitations: (really) slow datastore, and expensive datastore access.

I love how you can deploy new code over running instances, and having multiple versions or multiple apps at the click of a button.

That's the old GAE for you, it suits one man band operations for few enterprise users.

 

[Pascal Voitot Dev]

On Tue, Feb 4, 2014 at 9:36 AM, Emanuele Ziglioli <the...@emanueleziglioli.it> wrote:

FutureWrapper is a java class?

Quoting Pascal Voitot:

The other reason also is that GAE functions return a Future object. But then when you call the Future.get() function on it, it returns a GAE entity and not a Siena Entity.
What we want when we call .get() is to retrieve a Siena Model.
So we need to do the mapping to Siena Model also when calling the .get().
That's why I encapsulate the GAE Future into a SienaFutureWrapper (directly inherited from Google FutureWrapper) which provides the mapping feature. Look at the GaeAsyncPersistenceManager to see how it is used. 
This wrapper is itself encapsulated into a SienaFutureContainer which allows to hide the Java Future Exception problem

It's quite clever actually ;-)

I'm going through your async classes, and see if I can improve performances.

Each call to get() or batch() blocks and spins up new instances.

I can't say GAE is very good from that point of view, it's a real waste of resources.

Perhaps, something really counter-intuitive, like an HttpRequest Thread that keeps pausing and checking if a result is back from a Future would be the most performant solution.

BTW I've just reminded that Guava provides some Future classes which could help you. I must say Java Future is just a joke... JDK8 will provide some better things but with crazy long stupid names... Anyway, it will be better!

 

GAE suits us, as we have very few users but lots of (small) data. Costs are low, critical stuff is handled by task queues in a reliable way, mail handling is reliable.

One just needs to work around its limitations: (really) slow datastore, and expensive datastore access.

AWS is ~~~everything expensive, that's certainly why GAE still lives (and because Google is behind :))

 

I love how you can deploy new code over running instances, and having multiple versions or multiple apps at the click of a button.

That's the old GAE for you, it suits one man band operations for few enterprise users.

 

--

[Emanuele Ziglioli]

 

BTW I've just reminded that Guava provides some Future classes which could help you. I must say Java Future is just a joke... JDK8 will provide some better things but with crazy long stupid names... Anyway, it will be better!

Yeah, I've come across Guava Futures recently, as I was having some incompatibility issues with the old Google Collect. Need to look into it.

JDK8, looking forward to it. But I don't think GAE's API will change, even if they start supporting Java 8.

As far as I understand the FutureWrapper callback happens in a different thread, so there's no way to return an HttpResponse from there. One still has to call Future.get() from the HttpRequest thread, therefore blocking.

What I think I can try is, say, to build an array of SienaFutures, and maybe Jackson could cleverly call SienaFuture.get() asynchronously, therefore increasing the chances that the Future might actually have returned.

AWS is ~~~everything expensive, that's certainly why GAE still lives (and because Google is behind :))

 

Are you saying AWS is more expensive than GAE?

I've never looked into Amazon, although everyone seems to use it.

All I know is there's nothing like GAE, anything else requires more system maintenance, and I only want to do code.

Although, I think Google has tried just to do too much with GAE. They should have opened up a bit and allowed people for example to control the scheduler the way they like.

[Pascal Voitot Dev]

AWS is expensive, it's clear! And if you don't know exactly the resources you'll need, it's really hard to evaluate the cost until you run your solution. If you can administrate your own server, it's less expensive than any cloud stuff. But if you need lots of servers, naturally, it's different!

 

And AWS is a world in itself... there are so many tools to use to build complex solution... Once you've chosen AWS, I believe it's hard to move to something else...