Tomcat/Apache ESME: Shutdown Problems

58 views
Skip to first unread message

Dick Hirsch

unread,
Feb 3, 2010, 1:12:39 PM2/3/10
to Lift
A developer using ESME has a problem.

After he shuts down Tomcat, he sees that there are exceptions in the
log file. Derby is also not getting shutdown correctly. Next time,
when he restarts Tomcat, ESME fails to load and he gets all kinds of
exception in the browser.

Once he manually kils ( with -9) the java process then everything
works fine.

Here is (partial) stack trace from the log file is below.

I haven't seen this problem in jetty.

I was assuming that lift closed the DB connections / sessions
implicitly. Now I'm not so sure.

Does anyone know what the problem might be?

Here is a link to our Boot.scala file.
http://svn.apache.org/viewvc/incubator/esme/trunk/server/src/main/scala/bootstrap/liftweb/Boot.scala?view=markup

Thanks,

Dick


================
SEVERE:
A web application created a ThreadLocal with key of type
[java.lang.ThreadLocal] (value [java.lang.ThreadLocal@13f79f7]) and a
value of type [org.apache.derby.iapi.services.context.ContextManager]
(value [org.apache.derby.iapi.services.context.ContextManager@b52a28])
but failed to remove it when the web application was stopped. To
prevent a memory leak, the ThreadLocal has been forcibly removed.
Feb 3, 2010 7:04:57 AM
org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080
Exception in thread "pool-2-thread-14" java.lang.NullPointerException
at scala.runtime.BoxesRunTime.boxToLong(Unknown Source)
at net.liftweb.http.S$$anonfun$net$liftweb$http$S$$wrapQuery
$1.apply(S.scala:972)
at net.liftweb.http.S$$anonfun$net$liftweb$http$S$$wrapQuery
$1.apply(S.scala:972)
at scala.List.foreach(List.scala:841)
at net.liftweb.http.S$.net$liftweb$http$S$$wrapQuery(S.scala:972)
at net.liftweb.http.S$$anonfun$net$liftweb$http$S$$_nest2InnerInit
$1$$anonfun$apply$25.apply(S.scala:1109)
at net.liftweb.http.S$.net$liftweb$http$S$$doAround(S.scala:906)
at net.liftweb.http.S$$anonfun$net$liftweb$http$S$$doAround$1.apply
(S.scala:907)
at
net.liftweb.mapper.MetaProtoExtendedSession$myWrapper$.apply
(ProtoExtendedSession.scala:80)
at net.liftweb.http.S$.net$liftweb$http$S$$doAround(S.scala:907)

David Pollak

unread,
Feb 3, 2010, 2:30:38 PM2/3/10
to lif...@googlegroups.com
On Wed, Feb 3, 2010 at 10:12 AM, Dick Hirsch <hirsc...@gmail.com> wrote:
A developer using ESME has a problem.

After he shuts down Tomcat, he sees that there are exceptions in the
log file. Derby is also not getting shutdown correctly. Next time,
when he restarts Tomcat, ESME fails to load and he gets all kinds of
exception in the browser.

Once he manually kils ( with -9) the java process then everything
works fine.

Here is (partial) stack trace from the log file is below.

I haven't seen this problem in jetty.

I was assuming that lift closed the DB connections / sessions
implicitly. Now I'm not so sure.

Dick,

Are you sure the Tomcat process is actually shut down?  Just the brief look at the stack trace, it's looking like the JVM process didn't get killed and there's cruft on the local thread.

You might also consider using the new net.liftweb.mapper.StandardDBVendor rather than the roll-your-own DB vendor in Boot.  I'll add a ticket to make sure that StandardDBVendor registers with the container shutdown so it correctly closes all connections.

Thanks,

David


 

--
You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.




--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Surf the harmonics

Dick Hirsch

unread,
Feb 3, 2010, 2:58:14 PM2/3/10
to Lift
David,

Thanks.

Is the new StandardDBVendor just available the 2.0 Snapshot?

Do you have a link to a description of the StandardDBVendor,

D.

On Feb 3, 8:30 pm, David Pollak <feeder.of.the.be...@gmail.com> wrote:


> On Wed, Feb 3, 2010 at 10:12 AM, Dick Hirsch <hirsch.d...@gmail.com> wrote:
> > A developer using ESME has a problem.
>
> > After he shuts down Tomcat, he sees that there are exceptions in the
> > log file. Derby is also not getting shutdown correctly. Next time,
> > when he restarts Tomcat, ESME fails to load and he gets all kinds of
> > exception in the browser.
>
> > Once he manually kils ( with -9) the java process then everything
> > works fine.
>
> > Here is (partial) stack trace from the log file is below.
>
> > I haven't seen this problem in jetty.
>
> > I was assuming that lift closed the DB connections / sessions
> > implicitly. Now I'm not so sure.
>
> Dick,
>
> Are you sure the Tomcat process is actually shut down?  Just the brief look
> at the stack trace, it's looking like the JVM process didn't get killed and
> there's cruft on the local thread.
>
> You might also consider using the new net.liftweb.mapper.StandardDBVendor
> rather than the roll-your-own DB vendor in Boot.  I'll add a ticket to make
> sure that StandardDBVendor registers with the container shutdown so it
> correctly closes all connections.
>
> Thanks,
>
> David
>
>
>
>
>
> > Does anyone know what the problem might be?
>
> > Here is a link to our Boot.scala file.
>

> >http://svn.apache.org/viewvc/incubator/esme/trunk/server/src/main/sca...

> > liftweb+u...@googlegroups.com<liftweb%2Bunsu...@googlegroups.com>


> > .
> > For more options, visit this group at
> >http://groups.google.com/group/liftweb?hl=en.
>
> --
> Lift, the simply functional web frameworkhttp://liftweb.net

> Beginning Scalahttp://www.apress.com/book/view/1430219890

David Pollak

unread,
Feb 3, 2010, 3:06:07 PM2/3/10
to lif...@googlegroups.com
On Wed, Feb 3, 2010 at 11:58 AM, Dick Hirsch <hirsc...@gmail.com> wrote:
David,

Thanks.

Is the new StandardDBVendor just available the 2.0 Snapshot?

It's been around since 1.1-M7 I think.
 

Do you have a link to a description of the StandardDBVendor,


/**
 * The standard DB vendor.
 * @param driverName the name of the database driver
 * @param dbUrl the URL for the JDBC data connection
 * @param dbUser the optional username
 * @param dbPassword the optional db password
 */
class StandardDBVendor(driverName: String,
                       dbUrl: String,
                       dbUser: Box[String],
                       dbPassword: Box[String]) extends ProtoDBVendor {
 
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.




--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890

Timothy Perrett

unread,
Feb 3, 2010, 3:10:33 PM2/3/10
to lif...@googlegroups.com
See: http://is.gd/7Dzv4

Cheers, Tim

> To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.

Dick Hirsch

unread,
Feb 3, 2010, 10:37:16 PM2/3/10
to Lift
I talked to the user and he says that Tomcat didn't shutdown. He had
to kill -9 the process.

We'll move to

On Feb 3, 9:10 pm, Timothy Perrett <timo...@getintheloop.eu> wrote:
> See:http://is.gd/7Dzv4

Thanks for the link. We'll move to the StandardDBVendor in our next
release.

David Pollak

unread,
Feb 3, 2010, 10:46:17 PM2/3/10
to lif...@googlegroups.com
You can also add to Boot.scala (given that the StandardDBVendor is called dbVendor):

LiftRules.unloadHooks.append(() => dbVendor.closeAllConnections_!())

What this will do is close all the DB connections when the context is unloaded.

Additionally, you can switch to H2 which is generally more stable than Derby in my experience.
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890

aw

unread,
Feb 4, 2010, 2:08:27 AM2/4/10
to Lift
I don't think this issue is related to Derby because I am seeing this
problem too -- but I am not using Derby.

What version of Tomcat are you using? I am using 6.0.24 on either
Windows or Solaris 10 and see these errors too, like:

Feb 3, 2010 10:49:22 PM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap


SEVERE: A web application created a ThreadLocal with key of type

[java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1a517bd]) and a
value of type [scala.collection.mutable.HashMap] (value [Map()]) but


failed to remove it when the web application was stopped. To prevent a
memory leak, the ThreadLocal has been forcibly removed.

Feb 3, 2010 10:49:22 PM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap


SEVERE: A web application created a ThreadLocal with key of type

[java.lang.ThreadLocal] (value [java.lang.ThreadLocal@258c74]) and a
value of type [net.liftweb.common.Empty$] (value [Empty]) but failed


to remove it when the web application was stopped. To prevent a memory
leak, the ThreadLocal has been forcibly removed.

Feb 3, 2010 10:49:22 PM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap


SEVERE: A web application created a ThreadLocal with key of type

[java.lang.ThreadLocal] (value [java.lang.ThreadLocal@258c74]) and a
value of type [net.liftweb.common.Empty$] (value [Empty]) but failed


to remove it when the web application was stopped. To prevent a memory
leak, the ThreadLocal has been forcibly removed.

Feb 3, 2010 10:49:22 PM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap


SEVERE: A web application created a ThreadLocal with key of type

[org.slf4j.impl.CopyOnInheritThreadLocal] (value
[org.slf4j.impl.CopyOnInheritThreadLocal@1bb35b]) and a value of type
[null] (value [null]) but failed to remove it when the web application


was stopped. To prevent a memory leak, the ThreadLocal has been
forcibly removed.

Feb 3, 2010 10:49:22 PM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap

Note that while there is a Lift object, there are also lots of other
objects, so I'm not convinced that there is a problem with Lift...

I got the impression that Tomcat added this detection logic in
6.0.24. I don't know how to really fix it. (BTW, I found that I
usually only need to kill, not a kill -9.)

aw

unread,
Feb 4, 2010, 2:33:22 AM2/4/10
to Lift
I just noticed this:

Feb 3, 2010 10:49:23 PM org.apache.coyote.http11.Http11NioProtocol


destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080

Feb 3, 2010 11:19:20 PM org.apache.catalina.loader.WebappClassLoader
loadClass
INFO: Illegal access: this web application instance has been stopped
already. Could not load net.liftweb.http.SessionMaster$$anonfun$1$
$anonfun$apply$11$$anonfun$apply$12. The eventual following stack
trace is caused by an error thrown for debugging purposes as well as
to attempt to terminate the thread which caused the illegal access,
and has no functional impact.
java.lang.IllegalStateException
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:
1370)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:
1329)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
319)
at net.liftweb.http.SessionMaster$$anonfun$1$$anonfun$apply
$11.apply(LiftSession.scala:162)
at net.liftweb.http.SessionMaster$$anonfun$1$$anonfun$apply
$11.apply(LiftSession.scala:159)
at scala.Iterator$class.foreach(Iterator.scala:414)
at scala.Iterator$$anon$7.foreach(Iterator.scala:36)
at net.liftweb.http.SessionMaster$$anonfun
$1.apply(LiftSession.scala:159)
at net.liftweb.http.SessionMaster$$anonfun
$1.apply(LiftSession.scala:136)
at net.liftweb.actor.LiftActor
$class.execTranslate(LiftActor.scala:252)
at net.liftweb.http.SessionMaster
$.execTranslate(LiftSession.scala:91)
at net.liftweb.actor.SpecializedLiftActor$class.net$liftweb
$actor$SpecializedLiftActor$$processMailbox(LiftActor.scala:147)
at net.liftweb.actor.SpecializedLiftActor$$anonfun$3$$anonfun
$apply$1.apply(LiftActor.scala:114)
at net.liftweb.actor.SpecializedLiftActor$$anonfun$3$$anonfun
$apply$1.apply(LiftActor.scala:114)
at net.liftweb.actor.LAScheduler$$anonfun$1$$anon$1$$anon
$2.run(LiftActor.scala:42)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Exception in thread "pool-2-thread-4" java.lang.NoClassDefFoundError:
net/liftweb/http/SessionMaster$$anonfun$1$$anonfun$apply$11$$anonfun
$apply$12
at net.liftweb.http.SessionMaster$$anonfun$1$$anonfun$apply
$11.apply(LiftSession.scala:162)
at net.liftweb.http.SessionMaster$$anonfun$1$$anonfun$apply
$11.apply(LiftSession.scala:159)
at scala.Iterator$class.foreach(Iterator.scala:414)
at scala.Iterator$$anon$7.foreach(Iterator.scala:36)
at net.liftweb.http.SessionMaster$$anonfun
$1.apply(LiftSession.scala:159)
at net.liftweb.http.SessionMaster$$anonfun
$1.apply(LiftSession.scala:136)
at net.liftweb.actor.LiftActor
$class.execTranslate(LiftActor.scala:252)
at net.liftweb.http.SessionMaster
$.execTranslate(LiftSession.scala:91)
at net.liftweb.actor.SpecializedLiftActor$class.net$liftweb
$actor$SpecializedLiftActor$$processMailbox(LiftActor.scala:147)
at net.liftweb.actor.SpecializedLiftActor$$anonfun$3$$anonfun
$apply$1.apply(LiftActor.scala:114)
at net.liftweb.actor.SpecializedLiftActor$$anonfun$3$$anonfun
$apply$1.apply(LiftActor.scala:114)
at net.liftweb.actor.LAScheduler$$anonfun$1$$anon$1$$anon
$2.run(LiftActor.scala:42)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.ClassNotFoundException:
net.liftweb.http.SessionMaster$$anonfun$1$$anonfun$apply$11$$anonfun
$apply$12
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:
1484)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:
1329)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
319)
... 15 more

Maybe there is a shutdown order issue?

David Pollak

unread,
Feb 4, 2010, 4:44:23 PM2/4/10
to lif...@googlegroups.com

Interesting.

I think what's happening is that during the execution of the shutdown hook, a new class is attempting to be loaded.  Why?  Because every function in Scala is represented by a JVM class... and functions are very, very common to define in code (e.g., a for comprehension, etc.)  So, I think that the shutdown hook is trying to instantiate an "anonymous inner class" (in Java parlance), the class is not loaded, so the Classloader attempts to load the class, but the servlet is supposed to be shutting down, so Tomcat's classloader isn't allowing the class to be loaded.  That's just my guess.
 

--
You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.

Reply all
Reply to author
Forward
0 new messages