AppEngine boot times in Ruby, Python, and Java (benchmarks)

[Locke]

Out of curiosity, I decided to implement "hello world" in the two
official, and one unofficial, languages that run on AppEngine. These
apps make no use of the datastore or any services. They just print
"hello world."

The methodology was simply to run the app three times, waiting for
about a minute for AppEngine to unload my app between each request.

All times are in "wall time." Google typically bills you for "CPU
time," which is greater than wall time. The units are seconds.

--- jruby + rack:
15.5
18.0
13.8
average: 15.8 seconds

--- python + google's "webapp" micro-framework:
0.127
0.170
0.079
average: .0.125 seconds

--- java:
0.911
!! canceled after 10 seconds with "Request was aborted after waiting
too long to attempt to service your request." !!
2.4
4.2
average: 2.5 seconds not counting the fatal error

In all three cases, the apps respond in about one tenth those times if
they were already loaded.

And yes, Java performance is all over the map! My simple "hello world"
java app was killed by AppEngine. It was merely running the template
code that came with the java SDK.

My test shows that Python on AppEngine is excellent--some of the
fastest web app execution I've seen on any host anywhere. AppEngine's
Java support, on the other hand, should very much be considered a
"preview release" product. When "hello world" is killed by AppEngine
for taking too long to respond, you know there are some major problems
to work out. Bot both java and jruby, once loaded, run at quite
acceptable speeds. Unfortunately, they are reloaded after mere seconds
without hits, and those reloads can sporadically end in 500s. Based on
these numbers, I would not enable billing or AppEngine Java, nor would
I use it for "production" applications at this time. But AppEngine
Java hasn't been around nearly as long as the Python version, so
perhaps this is understandable.

(the java crash error follows)

01-18 08:50PM 42.121 /helloappengine 500 10085ms 0cpu_ms 0kb Mozilla/
5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.7) Gecko/20091221
Firefox/3.5.7 (.NET CLR 3.5.30729),gzip(gfe)
[myip] - - [18/Jan/2010:20:50:52 -0800] "GET /helloappengine HTTP/1.1"
500 0 - "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.7)
Gecko/20091221 Firefox/3.5.7 (.NET CLR 3.5.30729),gzip(gfe)" "[myapp]"
W 01-18 08:50PM 52.206
Request was aborted after waiting too long to attempt to service your
request. Most likely, this indicates that you have reached your
simultaneous dynamic request limit. This is almost always due to
excessively high latency in your app. Please see
http://code.google.com/appengine/docs/quotas.html for more details.

[Gary B]

The App Engine team recently blogged about loading request performance
in Java: http://googleappengine.blogspot.com/2009/12/request-performance-in-java.html
If your application has a steady stream of traffic, then loading
requests are infrequent and should not contribute significantly to the
cost of running your application.

I also tested loading request times using simple "Hello World"
applications. Here are the times that I found in App Engine CPU ms:

Python 85
Java 525
Clojure 2955

The Java and Clojure applications used precompilation..The source is
available here: http://github.com/garyburd/app-eng-cold-start-test

[Jeff Schnitzer]

...except that even with steady traffic, appengine seems to be
restarting some Java apps all the time. I get a steady 30-60
requests/min and I'm looking at 20 restarts in the last 4 hours.

Jeff

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>
>
>

[Locke]

It's interesting that you have java load times of 525ms. I think the
load time must differ greatly between either servers or times of day
or both.

At any rate, 525ms load times are not something you can rely on in App
Engine, even for "hello world." As I have directly observed, it can
take multiple seconds, or sometimes even more than the 10s "timeout"
limit, to load elementary java apps.

On Jan 19, 2:44 am, Gary B <gary.b...@gmail.com> wrote:
> The App Engine team recently blogged about loading request performance

> in Java:  http://googleappengine.blogspot.com/2009/12/request-performance-in-ja...

[Ikai L (Google)]

Hi,

Thanks for reporting this. I've been suspecting that something is going on with our Java applications due to just the number of reports of aggressive cycling. You've already posted in the other thread - but I'd like to invite any other developers seeing this issue to report their experience here:

http://groups.google.com/group/google-appengine-java/browse_thread/thread/b57e6c4895333fa8/28896258129c8ef3?lnk=gst&q=cold+starts#28896258129c8ef3

It'd also be helpful for those folks who have had an application for some time now when they started noticing the behavior. It could have to do with a specific release or update.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Ikai Lan
Developer Programs Engineer, Google App Engine