Classloading still way too slow. Any recommended practices? (yes, I am using warmup requests already)

[Per]

After using appstats for weeks now, reducing queries and offloading a
lot to memcache, our application is pretty fast despite the abysmal
performance of the datastore. Once it's been serving for a few
minutes, that is.

But: Very many pages of our application take several seconds (4 to
6s!) to load for the first time, even despite warmup requests and
several other pages having been executing on the same instance
already. Appstats clearly shows that the API calls are not to blame,
and running a local profiler doesn't indicate any problems either.

I added some more classloading (Class.forName()) to the warmup
requests, and it turns out classloading is frequently the cause.
Sometimes it takes 100ms to load 20 large classes (and their anonymous
inner classes). But sometimes it takes 8 seconds! Occasionally it
takes more than 15s, and the warmup request dies!

I'm now considering to let the warmup request call additional
webpages, so that these request warm up the classes instead. However,
given that these new http requests might end up on *other* instances,
it's not really a great option. It will increase probablilty of
classes being loaded, but it's not a guarantee.

So my question: Is there a recommended way of loading classes before
the user tries to get them? Is there a way to "chain" warmup requests
for instance, e.g. a guarantee that the next request will be on the
same instance? That would help a lot.


Cheers,
Per


PS: As much as I love App Engine and the power it brings, I find it
extremely annoying to have to fiddle with unreliable classloading
performance, and queries that take 300ms on average. Maybe I was
overly ambitious in coding a complex application on App Engine, maybe
App engine's goal is really just outsourcing large volume processing
(and not rich user-centered web applications), but the warning signs
are clearly missing then.

IMO, speed and reliability are much more important than anything else,
and as much as I'd love to see SSL support for custom domains,
fulltext search, etc.... I'd really prefer these core problems solved
first. Just my 2 cents.. :)

[Simon Knott]

Are these classes your own, or from third-party libraries? 

Do you deploy your own classes inside a JAR, or just within WEB-INF classes?

[Per]

Hi Simon,

these classes are all our own and they come from web-inf classes. I
double checked that it actually doesn't load anything else, using the -
verbose option locally.

Here's the code snippet showing just the first batch. This can take up
to 8 seconds to load remotely (but also be as fast as 120ms), while it
takes only 20ms to 50ms locally...

try {

Class.forName("com.praisemanager.web.general.panels.comment.view.CommentPanel");

Class.forName("com.praisemanager.web.general.panels.comment.view.CommentListPanel");

Class.forName("com.praisemanager.web.general.panels.comment.view.ViewRepliesAjaxLink");

Class.forName("com.praisemanager.web.general.panels.comment.view.CommentRestrictionsPanel");

Class.forName("com.praisemanager.web.general.panels.comment.ReplyCommentPanel");

Class.forName("com.praisemanager.web.FeedbackRequestPage");
Class.forName("com.praisemanager.web.EditCommentPage");

Class.forName("com.praisemanager.web.general.panels.comment.view.UnveilAnonymousAjaxLink");

Class.forName("com.praisemanager.web.general.panels.characters.PersonTab");
} catch (ClassNotFoundException e) {
e.printStackTrace();
}


Cheers,
Per

[Simon Knott]

Have you tried deploying the classes as a JAR?  I only ask, because I seem to remember someone posting recently who was complaining that file reading on GAE appeared to be slow - maybe it's not the classloading which is actually taking the time, but reading the class files from disk (which seems a bit ridiculous really).

[Per]

Simon, you are my hero!!

I had been deploying the classes and the html raw, e.g. from web-inf/
classes, but it turns out it's 20 to 100 times slower this way!! I
packaged everything up in one neat jar, and now my application flies
again. Even deployment is faster than before.

I guess it wasn't such a problem prior to 1.5.2 because the 3 resident
instances would warm up rather quickly, but the new post 1.5.2 scheme
spins up new instances all the time, so you get this problem all the
time.

Hello Googlers, in case your reading, IMO this belongs in the manual.
The difference between the local devserver and the actual production
system is really massive in this respect, and it's been haunting me
for a while now. A simple bullet-point "please deploy jars only" would
be nice.

Cheers,
Per

[Simon Knott]

Wow, glad it worked so well!  

Time to go and Jar my classes up as well...

[Ikai Lan (Google)]

We suspected this was the case but this is good information. Do you have specific numbers? If not, that's fine. I'll do the measurements myself. If it's significant, it could even be worth building into the deployment tool itself.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

On Fri, Aug 19, 2011 at 1:35 PM, Simon Knott <knott...@gmail.com> wrote:

Wow, glad it worked so well!  

Time to go and Jar my classes up as well...

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/-WlJUmVddaYJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Per]

The classes above took anything from 100ms to 500ms under normal
conditions, and could take up to several seconds when things got
really pear-shaped. Now that everything is jarred up, the classloading
happens in 20ms to 30ms on average, pretty much as it does on the
local devserver. I am sure it will also be slower under big stress,
but still, that's a substantial improvement already.

Adding it to the deployment tool would be great IMO. I was worried
that jarring everything up would make deployment slow (always
deploying a big 1MB jar instead of a couple classfiles only) but to my
surprise it's a lot faster this way. Deployments are fun again! :)

[Jose Montes de Oca]

Hi Peter,

I think we don't make this for you behind the scenes because there is an specific use case where this would not work: For files that you want to "override" that are already in a jar file, this files should remain in the WEB-INF/classes folder.

one alternative would be to modified the jar you are overriding and replace the clase with your own.

Best,

Jose Montes de Oca

[Simon Knott]

Hi,

I suspect the use-case where you want to override a class in another JAR is fairly infrequent, or at least it is in my experience.  

Maybe the deployment tool could just have an option on it to deploy everything as a JAR, if the benefits are really that great, or default it to this scenario and then give people the option of turning it off.

Cheers,

Simon

[de Witte]

Ikai Lan,

I mentioned this before in another thread. Is it possible to redesign the scheduler so that a request never has to wait for an instance to start-up, except if it is the first instance. In this case, a request triggers the startup of a new instance, but is processed by an existing one, unless the startup time is faster than the pending time.

[Per]

Actually, you're right, never mind the "behind the scenes" business.
But if your internal tests confirm that JARs are faster to load,
please add a note to the docs, this would have saved me a lot of time
experimenting with various workarounds.

Cheers,
Per


On Aug 20, 12:21 am, Jose Montes de Oca <jfmontesde...@google.com>
wrote:

[Jeff Schnitzer]

I wonder if this might ameliorate classpath scanning performance
issues, and make it a viable option again.

Jeff

[Simon Knott]

I've just deployed one of my apps with the classes JARed and it's halved the loading request elapsed time for me, although it hasn't really touched the CPU_ms time.

Admittedly it only has 53 classes in it, but a decrease of 5 to 2.5 seconds is a noticeable speed-up which I'm more than happy with!