Another possible improvement: App Engine Modules

[Emanuele Ziglioli]

Hi everyone

Our app, which is based on Vosao, experiences terrible warmup times, so I've ask around for suggestions:

https://plus.google.com/u/0/100986613577956359467/posts/15279TopQ7K

https://plus.google.com/u/0/100986613577956359467/posts/PZDTP8VhPQH

I've worked on many fronts in order to improve performances.
1. changed Siena to always use async api calls. I've stolen an idea from Objectify and now use a Proxy class to hide Future: 

https://github.com/siena/siena/commits/async
https://github.com/Sirtrack/vosao/commits/siena-async

2. I've gone over all my dependencies and tried to remove a number of jars. Sometimes a library just needs a class and import a whole jars. Sometimes, sub-dependencies are not even needed, as they should have a test scope. So, improvements in load time there and jar scanning

3. I've disabled warmup and made sure that servlet are loaded only on demand. Better to suffer a little than a lot all at once.
Vosao includes a significant number of jars...

I've got other ideas, for example, I could split Vosao in two modules, one that is light and customer facing, another one to handle the /cms part.
https://github.com/GoogleCloudPlatform/appengine-modules-sample-java

At the end of the day, making apps run with acceptable performances on GAE/Java is bloody hard.

Emanuele

[Daniel Gerson]

On Wednesday, February 12, 2014 6:02:39 AM UTC+2, Emanuele Ziglioli wrote:

I've got other ideas, for example, I could split Vosao in two modules, one that is light and customer facing, another one to handle the /cms part.
https://github.com/GoogleCloudPlatform/appengine-modules-sample-java

At the end of the day, making apps run with acceptable performances on GAE/Java is bloody hard.

Emanuele

I love your ideas and commitment. I didn't know about AppEngine modules until now, I thought you were talking about Maven modules.

Anyway, I have a suggestion not mentioned so far as I can see.

Why don't you try using Proguard to cut down on the size of the unused dependencies. It should cut out classes that aren't used, and dramatically reduce the dependency footprint. Haven't used it much myself but I know it's used specifically for this functionality.. and I think it's safe to assume that it has switches to tailor that certain stuff must absolutely be there, and not be obfuscated etc. etc.

DMG

[Emanuele Ziglioli]

Hi Daniel,

I don't know about Proguard, will definitely look into it.

I didn't know about Modules either until someone suggested, it's a pretty recent feature.

It definitely adds complexity.

Programming for App Engine is a pain in the ass, especially in Java.

It doesn't really scale very well. Instead it wastes a lot of resources (the instance) waiting for data to come back from the network.

One cannot just program as they would normally do on a server. Everything has to be stripped down to the bare essential.

The Hello World servlet they provide as an example takes 2s to warm up and return. That's the bottom line and if would achieve similar performances, then new instances could spin up and serve requests in a barely acceptable way.

[Daniel Gerson]

Programming for App Engine is a pain in the ass, especially in Java.

It doesn't really scale very well. Instead it wastes a lot of resources (the instance) waiting for data to come back from the network.

One cannot just program as they would normally do on a server. Everything has to be stripped down to the bare essential.

The Hello World servlet they provide as an example takes 2s to warm up and return. That's the bottom line and if would achieve similar performances, then new instances could spin up and serve requests in a barely acceptable way.

 

I understand your frustration, but I think you really mean that the *free* quota doesn't scale very well. From what I've read GAE scales pretty well you have size-able traffic.
I don't think it's expected that a popular/commercial app should have many cold starts, and it fires up new instances only to cope with predictable increases in traffic.

The other model is you pay monthly for a virtual machine or node, which allows you to do all the bits and pieces you need to easily and painlessly scale to the node's limits... and then it becomes difficult.

From an efficiency standpoint, if your VM isn't doing anything, that's a lot of resources wasted... and GAE is priced to encourage users to optimize DB/cache/footprint from day one (as if you'll start scaling fast).

It would be great if they could provide a 'hibernate' function if you could guarantee your app's memory footprint is GC'd to under a particular size, and then keep spawning that copy. They couldn't hibernate you if you're using lots of instance memory. Since the GC is non deterministic from the app's perspective, I don't think that would be easy to do, or detangle it from JVM house-keeping. Would be great though.

Anyway, just thinking out loud.

Check out the proguard results page of bytecode shrinkage. I imagine around 50% shaving may be on the cards (seems to be the average).
http://proguard.sourceforge.net/#results.html

DMG

[Emanuele Ziglioli]

Hi Daniel,

some good suggestions there. I think a GAE instance starts relatively fast considering it's a JVM. There's some magic in the precompilation process, no idea what it does.

If we could "hibernate" an instance that would be great.

The closest to that would be to hack libraries that have a big impact at startup (in our case Restlet and Jaxb), grab the result of their initalization and put into memcache.

Yet another discussion on the GAE forum: https://groups.google.com/forum/#!topic/google-appengine/lqrgtGwnQiM

[Lucio Piccoli]

hi emanuele

the appengine startup times/hardedeadline problem is well documented.
there are many causes but normally due to classpath loading/scanning.
typicallly things like spring/hibernate/jsf due classpath
scanning/loading on startup.
It is a problem with appengine that can limited certain 3rd party packages.

the trunk version of vosao loads fast and easy as there are no clever
3rd party libs to scan.

below is my response on stackover flow.
http://stackoverflow.com/a/10890027/1229529

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Vosao CMS Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vosao-cms-develo...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--
regards

Lucio Piccoli

[Emanuele Ziglioli]

Hi Lucio,

I sent two replies bue they're not showing here

On Tuesday, 18 February 2014 11:46:17 UTC+13, lp wrote:

hi emanuele

the appengine startup times/hardedeadline problem is well documented.
there are many causes but normally due to classpath loading/scanning.
typicallly things like spring/hibernate/jsf due classpath
scanning/loading on startup.
It is a problem with appengine that can limited certain 3rd party packages.

The problem starts with Appengine's libraries themselves:

https://groups.google.com/d/msg/google-appengine/lqrgtGwnQiM/iQa0SdkHye4J

1. if I remove all jar files from the war/WEB-INF/lib, the warmup time is about 2.3 seconds.
2. if I put the only the core app engine sdk jar file in war/WEB-INF/lib, but doesn't reference it, the warmup time is about 2.9 seconds.
3. if I put "DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();" in the jsp file but do nothing eslse, the warmup time is 3.5 seconds
4. if I put a query in the default jsp file, the warmup time is 4.7 seconds.
5. if I convert query result as list, the warmup time is 6 seconds.

 

the trunk version of vosao loads fast and easy as there are no clever
3rd party libs to scan.

Actually Vosao uses Jabsorb (which uses the BeanInfo libraries) does quite a bit of introspection and classpath scanning. Probably in the same order as Jackson.

I'm curious: what's the warmup time for you? Do you pay to keep an instance running or is a cold startup fast enough? It comes to that. We pay to always have an idle instance running but unfortunately often requests are sent to cold instances.

It's not about the money, since it's an enterprise app with few customers but it makes for a poor amateurish experience.

> email to vosao-cms-development+unsub...@googlegroups.com.

[Lucio Piccoli]

Hi emanuele

If your start up time is 4secs then that is awesome.
I haven't checked my times for vosao but it I will report back.

Are your times on a real environment? There is always lots of variability on start up times with same code executing. It is after all a cloud experience.

My spring/jpa app was 30-60secs.
That was all thanks to spring and jpa scanning class path for configuration and the enormous amount of class loading. Even after tuning the config not to scan there was still masssive startup lag. Maybe there are other things involved also.

I only use paid instances on gae otherwise cold lag is a real problem. However I still get startup lag on a new instance but it is manageable with idle instance settings.

Good point about vosao using jabsorb which uses some introspect.

--
regards

Lucio Piccoli

To unsubscribe from this group and stop receiving emails from it, send an email to vosao-cms-develo...@googlegroups.com.

[Lucio Piccoli]

Hi emanuele

my cold start up times on a F1 instance is 10s-12s.

>>> > email to vosao-cms-develo...@googlegroups.com.
>>> > For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>>
>>> --
>>> regards
>>>
>>> Lucio Piccoli
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Vosao CMS Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to vosao-cms-develo...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.

--
regards

Lucio Piccoli