app servers compared

[phil.s...@gmail.com]

I heard some talk about startup time of glassfish on the last
episode.

http://agoncal.wordpress.com/2009/12/12/o-application-servers-where-art-thou/

Note tomcat and jetty vs glassfish.

[Casper Bang]

But it's a little bit Apple and Oranges isn't it? Glassfish is an EE
container, we all know how stuff with EE in the name is 2-3 times
slower. But yes, Tomcat and Jetty rocks. :)

/Casper

On Dec 16, 7:56 pm, "phil.swen...@gmail.com" <phil.swen...@gmail.com>
wrote:

> I heard some talk about startup time of glassfish on the last
> episode.
>

> http://agoncal.wordpress.com/2009/12/12/o-application-servers-where-a...

[carl]

It's funny how all the benchmarks compare startup time. That's
important when developing, sure, but when running a production server
does it really matter that much?

And how important is the disk footprint? Although, I do really like
Jetty for making self-contained webapps with the container baked in.
In that usage, the small disk footprint is a big win.

Ram used by the container itself is interesting, but what I'd really
like to see is a comparison of runtime performance of servlets in each
container. Latency, throughput, etc. I wonder which would win for lean
and mean servlet containment? Jetty, Tomcat or Glassfish?

[Casper Bang]

All good points. Startup time with 15-20 applications deployed would
also be more interesting than pure container startup - given how that
can easily take 5+ minutes, a long time in a production environment
which just suffered from PermGenSpace problems and had to be
restarted.

/Casper

[kirk]

Casper Bang wrote:
> All good points. Startup time with 15-20 applications deployed would
> also be more interesting than pure container startup - given how that
> can easily take 5+ minutes, a long time in a production environment
> which just suffered from PermGenSpace problems and had to be
> restarted.
>

anyone running unclustered deserves to suffer a long startup time ;-)

Kirk

> /Casper
>
> On Dec 18, 6:48 am, carl <carl.qu...@gmail.com> wrote:
>
>> It's funny how all the benchmarks compare startup time. That's
>> important when developing, sure, but when running a production server
>> does it really matter that much?
>>
>> And how important is the disk footprint? Although, I do really like
>> Jetty for making self-contained webapps with the container baked in.
>> In that usage, the small disk footprint is a big win.
>>
>> Ram used by the container itself is interesting, but what I'd really
>> like to see is a comparison of runtime performance of servlets in each
>> container. Latency, throughput, etc. I wonder which would win for lean
>> and mean servlet containment? Jetty, Tomcat or Glassfish?
>>
>

> --
>
> You received this message because you are subscribed to the Google Groups "The Java Posse" group.
> To post to this group, send email to java...@googlegroups.com.
> To unsubscribe from this group, send email to javaposse+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/javaposse?hl=en.
>
>
>
>

[Casper Bang]

Even with a cluster, the individual nodes has to start up. Restarting
nodes one by one asynchronously might shield clients from seeing this,
but that does not render it a non-issue.

/Casper

[Brian Leathem]

On 18/12/09 3:44 AM, Casper Bang wrote:
> All good points. Startup time with 15-20 applications deployed would
> also be more interesting than pure container startup - given how that
> can easily take 5+ minutes, a long time in a production environment
> which just suffered from PermGenSpace problems and had to be
> restarted.
>

How *do* you get rid of those PermGenSpace problems (aside from
increasing the PermGenSpace). I get them often and have to completely
restart Glassfish/Netbeans to get back to normal. This can take a
while, as Glassfish doesn't stop nicely when it runs out of PermGen.

I made significant improvements by moving some of my 3rd party Jars into
the app server lib folder, so that those classes wouldn't have to be
re-loaded with every re-deploy. But it still happens.

Maybe with GF v3 finally having support for compile/deploy on save for
maven projects, I'll finally escape this problem.

Brian Leathem

[Casper Bang]

I won't even try to pretend that I understand all the intricate
details of the class loaders, GC and leakage. However you turn and
twist it, it's a design issue somewhere between the JVM and the
application server concept (accidental complexity?). And it's one hell
of a good reason to run your app as an embedded Grizzly or Jetty
instead - much more predictable. Now what I would REALLY love to know,
is how on earth is GAE handling this?!

/Casper

PS: I have never seen something a kin to PermGenSpace on .NET and I
assume this is due to the fact that they took a more KISS attitude (no
custom low-level class loader, only dynamic assembly modules
responsible for their own mess).

[Spencer Uresk]

This is really only a problem that (normally) occurs with lots of
redeploying without restarting the entire app server. I'm speculating
a bit, but I believe with GAE they start a new (separate) instance
when you deploy, which effectively prevents you from running into
PermGen errors.

How errors like this occur is fairly simple - if something outside
your application holds on to a reference to something inside your
application, it will cause the classloader to leak and after enough
redeployments, you will run out of PermGen space. The most common
culprit from what I have seen is a database driver being bundled with
your application, and some libraries can also cause odd issues in this
area. I'm not enough of an expert with classloaders and the like to
know where, exactly, the fault lies - but I'd probably guess that the
complexity of the ClassLoader, as you suggest, is part of the problem.
I'm not sure if this is something that will change at all in Java 7 -
anyone have any idea?

As for how to track down a PermGen error - if you have eliminated all
of the usual suspects, you can start your app server, deploy your app,
undeploy it, and then get a heap dump. In the heap dump, find the
classloader object for that application (in Tomcat and Jetty, it is
an instance of WebAppClassLoader I believe - not sure what it is on
GlassFish) and then find the references to that object. It isn't
always incredibly straightforward or easy, but if you are getting
serious heartburn from PermGen issues, it may be worth a shot.

- Spencer

[Reinier Zwitserloot]

The reason .net doesn't have this problem is due to the different
(perceived) target markets for each.

Java has taken a "let's make it so the server admins like us"
approach, and thus have strictly limited the JVM to take up only as
much memory as you tell it to use. Furthermore, while the JVM should
not eat more than the set amount of memory I give it, it should also
have enough memory to comfortably run my app, lest it slow down to a
crawl. Therefore, if I know my app needs let's say 512m of heap, then
if I tell java it should have no less (and no more!) than 512m heap,
all those 512m need to go to heap. Thus, separate settings for heap,
stack, permgen, etc.

.net takes the consumer approach, which is indeed simpler: If I need
more memory, I'll just take it. I'll keep taking up more memory until
the host system doesn't want to give me any more (which will occur
only after exhausting the entire swap drive as well). If I don't get
an opportunity to garbage collect, this is indeed going to spiral into
a disaster.

There's a nicer solution available which neither fully implements,
though .net is a little closer to this: Be Smarter. There ought to be
a JVM option that says: Use however much memory there is, but be smart
about it. This option should take into consideration total system
memory, and create many soft borders which it won't break unless
there's no other way to continue (e.g. it should start aggressively
garbage collecting before taking more than 10% of the total system's
memory, but if that isn't helping, then break through and set a new
soft border of 20%). For server VMs, you can still use the -Xmx
settings to set up uncrossable memory boundaries. For system response,
a JVM that is in dynamic mode should still consider something like 75%
of total system memory as an unbreakable boundary; it's very likely
there's a memory leak if it ever goes that far, so if 75% isn't
enough, soon 100% won't be enough either, and at that point the app is
slow as molasses (due to constant swapping), and the OS, also starved
for resources, will be so irresponsive it'll be tough to close this
app.

This isn't exactly easy to do; for example, giving each stack a
variable stacksize to accomodate really big stacks is somewhat
complex, as is dynamically growing a stack. You also don't want to
grow a stack too far, as 95% of all stackoverflowexceptions are not an
indicator for a small stack, they are instead an indicator for an
endless recursive loop. You want those to blow up early, and not grind
the system to a halt as the VM marshalls half your swap space and
almost all core memory to create one gigantic stack for the runaway
process to churn for minutes on end before finally crashing.

[Michael Neale]

PermGen space is not required by any JVM, in fact IIRC JRockit never
had it (not sure about IBMs JVM or others).

It seems an out dated concept (that class data is "special" and all
defined well ahead of time to be lazy loaded) which needs to go. But
perhaps if there wasn't permgen space, then class leakage (due to
classes not being GCed until its classloader is GCed) would look like
an ordinary memory leak? (This makes me also think that anonymous
classloading could help as well - which I think was part of
invokedynamic spec, I vaguely remember)