NoClassDefFoundError when stopping net.liftweb:lift-example:2.0-M4 on Tomcat

[Andreas Sewe]

Hi all,

I am currently in the process of assembling a Scala benchmark suite
based on the popular (in VM research circles, anyway) DaCapo benchmark
suite: <http://www.dacapobench.org/>. Lift of course is a natural
candidate for inclusion in such a benchmark suite.

As the DaCapo suite already contains a Tomcat benchmark, which
exercises the Tomcat example webapp, I based my benchmark, which
exercises net.liftweb:lift-example:2.0-M4, on Tomcat as well.
Unfortunately, I am experiencing problems when stopping the lift-
example webapp. And stopping (and re-starting) the webapp is required
as one typically benchmarks many iterations within the same VM
invocation (to allow the JIT to warm up).

The benchmark harness's main loop essentially looks like this:

for (i = 0; i < maxIterations; i++) {
// GET "http://localhost:8080/manager/start?path=/lift-example-2.0-
M4"
//
// Exercise the webapp with a bunch of HTTP requests.
//
// GET "http://localhost:8080/manager/stop?path=/lift-example-2.0-
M4"
}

Ultimately, the problem manifests itself in an NoClassDefFoundError.
It also occurs when using a stand-alone Tomcat 6.0.29; thus, it is not
the harness's fault. Here are the relevant portions of the log file
after first stopping the webapp and then the stand-alone Tomcat
instance:

SEVERE: The web application [/lift-example-2.0-M4] registered the JBDC
driver [org.apache.derby.jdbc.AutoloadedDriver] but failed to
unregister it when the web application was stopped. To prevent a
memory leak, the JDBC Driver has been forcibly unregistered.
14-Oct-2010 1:43:13 PM org.apache.catalina.loader.WebappClassLoader
clearReferencesJdbc
SEVERE: The web application [/lift-example-2.0-M4] registered the JBDC
driver [org.h2.Driver] but failed to unregister it when the web
application was stopped. To prevent a memory leak, the JDBC Driver has
been forcibly unregistered.
14-Oct-2010 1:43:13 PM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/lift-example-2.0-M4] appears to have
started a thread named [ActorPing] but has failed to stop it. This is
very likely to create a memory leak.
14-Oct-2010 1:43:13 PM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: The web application [/lift-example-2.0-M4] created a
ThreadLocal with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@7f3b8a]) and a value of type
[scala.collection.mutable.HashMap] (value [Map()]) but failed to
remove it when the web application was stopped. This is very likely to
create a memory leak.
14-Oct-2010 1:43:16 PM org.apache.catalina.loader.WebappClassLoader
loadClass
INFO: Illegal access: this web application instance has been stopped
already. Could not load net.liftweb.util.ActorPing$$anonfun$schedule
$1$$anonfun$apply$1. The eventual following stack trace is caused by
an error thrown for debugging purposes as well as to attempt to
terminate the thread which caused the illegal access, and has no
functional impact.
java.lang.IllegalStateException
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:
1531)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:
1491)
at net.liftweb.util.ActorPing$$anonfun$schedule
$1.apply(ActorPing.scala:51)
at net.liftweb.util.ActorPing$$anonfun$schedule
$1.apply(ActorPing.scala:51)
at net.liftweb.util.ActorPing$$anon$1.call(ActorPing.scala:61)
at net.liftweb.util.ActorPing$$anon$1.call(ActorPing.scala:60)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
14-Oct-2010 1:43:20 PM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8080
14-Oct-2010 1:43:21 PM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
14-Oct-2010 1:43:21 PM org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080
14-Oct-2010 1:43:21 PM org.apache.catalina.loader.WebappClassLoader
loadClass
INFO: Illegal access: this web application instance has been stopped
already. Could not load org.h2.result.SimpleRow. The eventual
following stack trace is caused by an error thrown for debugging
purposes as well as to attempt to terminate the thread which caused
the illegal access, and has no functional impact.
java.lang.IllegalStateException
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:
1531)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:
1491)
at org.h2.table.Table.getTemplateSimpleRow(Table.java:486)
at org.h2.engine.Database.removeMeta(Database.java:869)
at org.h2.engine.Database.update(Database.java:1510)
at org.h2.schema.Sequence.flush(Sequence.java:120)
at org.h2.schema.Sequence.close(Sequence.java:137)
at org.h2.engine.Database.close(Database.java:1191)
at org.h2.engine.DatabaseCloser.run(DatabaseCloser.java:80)
Exception in thread "Thread-2" java.lang.NoClassDefFoundError: org/h2/
result/SimpleRow
at org.h2.table.Table.getTemplateSimpleRow(Table.java:486)
at org.h2.engine.Database.removeMeta(Database.java:869)
at org.h2.engine.Database.update(Database.java:1510)
at org.h2.schema.Sequence.flush(Sequence.java:120)
at org.h2.schema.Sequence.close(Sequence.java:137)
at org.h2.engine.Database.close(Database.java:1191)
at org.h2.engine.DatabaseCloser.run(DatabaseCloser.java:80)
Caused by: java.lang.ClassNotFoundException: org.h2.result.SimpleRow
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:
1645)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:
1491)
... 7 more

Do you know any solution to this problem? Or is this to be expected?
(After all, lift-example seems to be stuck a version 2.0-M4 for quite
some time.)

Best wishes,

Andreas

[Timothy Perrett]

Can you paste your boot? I dont think you have registered the unload hooks.

Cheers, Tim

> --
> You received this message because you are subscribed to the Google Groups "Lift" group.
> To post to this group, send email to lif...@googlegroups.com.
> To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.
>
>

[Derek Chen-Becker]

Call me crazy, but wouldn't lift-example 2.0 (or even 2.1) be more appropriate than a milestone release? Probably unrelated to this particular error, but still, I would want to test using a GA release.

Derek

[Andreas Sewe]

Hi Tim,

thanks for your quick reply.

> Can you paste your boot? I dont think you have registered the unload hooks.

slow down please. I am just a poor VM researcher; Lift and servlet
containers are not my bread and butter business.

But if you tell me what you are after with a little bit more detail, I
can sure paste it. :-)

Best wishes,

Andreas

[Timothy Perrett]

haha no problem :-)

there should be a file called "Boot.scala" within your project someplace. Can you find it and paste the contents?

Cheers, Tim

[Andreas Sewe]

Hi Tim,

> there should be a file called "Boot.scala" within your project someplace.
> Can you find it and paste the contents?

alas, I have only the WAR (although I read bytecode for a living, that's
probably not too helpful). And the associated POM
<http://www.scala-tools.org/repo-releases/net/liftweb/lift-example/2.0-M4/lift-example-2.0-M4.pom>
ultimately points me to a dead git repository:
<http://github.com/dpp/liftweb/tree/master>. :-(

But it seems like
<http://github.com/lift/lift/tree/master/examples/example/> might be what I
should really be after. I couldn't find a Maven repository serving those
artifacts, though. Do you have any idea where to look for a newer release
of the examples?

Best wishes,

Andreas

[Andreas Sewe]

Hi Derek,

> Call me crazy, but wouldn't lift-example 2.0 (or even 2.1) be more
> appropriate than a milestone release? Probably unrelated to this
> particular error, but still, I would want to test using a GA release.

so would I. But if such a release exists, it hasn't made it to the
scala-tools.org repo yet:
<http://scala-tools.org/repo-releases/net/liftweb/lift-example/>. :-(

That being said, I would be grateful if you can point me to a more
recent release of the Lift demo. (At least the official Lift demo uses a
2.1-SNAPSHOT: <http://demo.liftweb.net/>.)

Best wishes,

Andreas

[Timothy Perrett]

Andreas,

Can I suggest that you don't use the example application for performance testing - it carries a fair bit of weight doing a bunch of other different things that you probably wouldn't do normally in order to demonstrate parts of lift.

You would probably be better off making a small sample app that would be good for this purpose.

Cheers, Tim

[Andreas Sewe]

Hi Tim,

thanks for your patience.

> Can I suggest that you don't use the example application for
> performance testing - it carries a fair bit of weight doing a bunch
> of other different things that you probably wouldn't do normally in
> order to demonstrate parts of lift.

I didn't know that. Can you give an example of its "abnormal" behaviour?

> You would probably be better off making a small sample app that would
> be good for this purpose.

Two problems with this approach: 1) I probably wouldn't produce an app
that would do things normally either and 2) even if I did, other
researchers might still attack the benchmark's design as not being
representative of in-the-wild code.

That being said, demo.liftweb.net was recommended just a couple of
minutes ago on this very list and it does have one appealing property:
It offers functionality very similar to the example webapp in the
already established DaCapo benchmark -- even including things like the
"Number Guessing" game.

But I'll definitely sift through the list's archives to find other demos
as well.

Best wishes,

Andreas

[Timothy Perrett]

Hmm, sorry I am really not familiar with benchmarking so I am not sure what one might be looking to capture.

I was thinking that the demo.liftweb.net app basically does per-service request logging and stuff. I guess its not a huge overhead or anything, but not many apps would do it. Maybe ignore me here :-D

Anyway, lets get back to your error....

When you take the WAR and run it within a normal servlet container, does that work for you?

Cheers, Tim

[Andreas Sewe]

Hi Tim,

> Anyway, lets get back to your error....
>
> When you take the WAR and run it within a normal servlet container, does that work for you?

er, no. As I said earlier, it is not a problem with the harness, merely
with the fact that it tries to stop the webapp before Tomcat itself
shuts down.

1. Grab Tomcat 6.0.29. <http://tomcat.apache.org/download-60.cgi#6.0.29>

2. Dump the example WAR
http://scala-tools.org/repo-releases/net/liftweb/lift-example/2.0-M4/lift-example-2.0-M4.war>
into Tomcat's ./webapps folder.

3. Do a ./bin/startup

4. Use the manager to stop lift-example-2.0-M4. (You may have to add two
lines to ./conf/tomcat-users.xml for this.
)
Do a ./bin/shutdown

And this presents you with lots of mean-looking "SEVERE" errors in your
./logs:

I posted these error messages already in my initial post; all the talk
about benchmarking and iterations was merely there to explain why I
would want to stop the webapp before shutting down Tomcat in the first
place.

I hope this explains things.

Andreas

[David Pollak]

On Thu, Oct 14, 2010 at 7:10 AM, Andreas Sewe <se...@st.informatik.tu-darmstadt.de> wrote:

Hi Tim,

Anyway, lets get back to your error....

When you take the WAR and run it within a normal servlet container, does that work for you?

er, no. As I said earlier, it is not a problem with the harness, merely with the fact that it tries to stop the webapp before Tomcat itself shuts down.

1. Grab Tomcat 6.0.29. <http://tomcat.apache.org/download-60.cgi#6.0.29>

2. Dump the example WAR http://scala-tools.org/repo-releases/net/liftweb/lift-example/2.0-M4/lift-example-2.0-M4.war> into Tomcat's ./webapps folder.

3. Do a ./bin/startup

4. Use the manager to stop lift-example-2.0-M4. (You may have to add two lines to ./conf/tomcat-users.xml for this.
)
Do a ./bin/shutdown

And this presents you with lots of mean-looking "SEVERE" errors in your ./logs:

This is a known issue and a bug in Tomcat.

Lift sits on top of Scala and Scala supports closures.  Closures are implemented under the covers the same way anonymous inner functions are implemented in Java.  Things as syntactically simple as for comprehensions (or even call-by-name invocation) will result in closures... anonymous inner classes and these anonymous inner classes are not loaded until the enclosing method is invoked.

Lift allows for the registration of shutdown hooks.  These shutdown hooks may contain constructs that result in accessing closures... just as accessing a method in Java may cause an anonymous inner class to be instantiated.  Because these classes are not touched until the shutdown hook is accessed, the classloader is accessed and asked for the anonymous inner class.

Certain versions of Tomcat do not allow for loading of classes during the shutdown phase.  This is a bug.  There's nothing in the J/EE or Servlet specs that ban class loading during the shutdown phase of a servlet.
 

Andreas

--
You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im
Surf the harmonics

[Andreas Sewe]

Hi David,

> This is a known issue and a bug in Tomcat.

good to know. My googling didn't turn this issue up, unfortunately. :-(

> Lift sits on top of Scala and Scala supports closures. Closures are
> implemented under the covers the same way anonymous inner functions are
> implemented in Java. Things as syntactically simple as for
> comprehensions (or even call-by-name invocation) will result in
> closures... anonymous inner classes and these anonymous inner classes
> are not loaded until the enclosing method is invoked.

I know. That's what this whole benchmarking effort is about: to
investigate how Java-like Scala behaves at the JVM level -- and whether
the JVM is (performance-wise) aversely affected by the differences.

> Lift allows for the registration of shutdown hooks. These shutdown
> hooks may contain constructs that result in accessing closures... just
> as accessing a method in Java may cause an anonymous inner class to be
> instantiated. Because these classes are not touched until the shutdown
> hook is accessed, the classloader is accessed and asked for the
> anonymous inner class.
>
> Certain versions of Tomcat do not allow for loading of classes during
> the shutdown phase. This is a bug. There's nothing in the J/EE or
> Servlet specs that ban class loading during the shutdown phase of a servlet.

Thanks for the explanation. I have switched to Jetty and am so far happy
with it. (Embedded Jetty is much easier to use from within a benchmark
harness.)

Best wishes,

Andreas