Unable to shutdown Tomcat server with Memcached sessions open

[sacharya]

I have a tomcat server talking to a cluster of memcached servers. And
I am using the Java SpyMemcached Client.

With connections to the memcached servers open from the Tomcat server,
I cannot gracefully shutdown tomcat using shutdown.sh. I have to
manually kill the tomcat process each time.

Any idea how I can get around this?

[Dustin]

Most likely, you're not running in daemon mode and not sending a
shutdown message to spymemcached.

Depending on how you configure your client (2.4 added
ConnectionFactoryBuilder) you want to have the ConnectionFactory
daemon mode to be true. It's false by default to ensure that it does
all the things it needs to, but it will need a positive shutdown
signal in order to shut down safely.

[sacharya]

I am running it in daemon mode but I wasn't shutting down the
connection after each use. Basically, I set up a pool of few
connections during startup and randomly used which ever connection is
available. That way I didnt have to set up and tear down during each
request.

Is this how I should be doing?

MemcachedClient connection = new MemcachedClient(AddrUtil.getAddresses
(getServers()));
connection.get(key);
connection.shutDown();

[Dustin]

On Oct 6, 12:37 pm, sacharya <sudarshan.acha...@gmail.com> wrote:
> I am running it in daemon mode but I wasn't shutting down the
> connection after each use. Basically, I set up a pool of few
> connections during startup and randomly used which ever connection is
> available. That way I didnt have to set up and tear down during each
> request.
>
> Is this how I should be doing?
>
> MemcachedClient connection = new MemcachedClient(AddrUtil.getAddresses
> (getServers()));
> connection.get(key);
> connection.shutDown();

Try something like this (assuming 2.4rc1):

MemcachedClient connection = new MemcachedClient(
new ConnectionFactoryBuilder().setDaemon(true).build(),
AddrUtil.getAddresses(getServeres()));

In < 2.4, you want to subclass ConnectionFactory and override the
isDaemon() method.

[David Huang]

I think I'm having the same issue as sacharya.
Also this SO question:
http://stackoverflow.com/questions/8112723/how-to-stop-tomcat-from-making-failed-attempts-to-connect-to-local-memcached-ser

I'm using spymemcached 2.8.1, tomcat 6 on linux 2.6.18.2-34-default.
I have a global singleton MemcachedClient instance, it is created and destroyed with a ServletContextListener.

Here is the log:

Jun 28, 2012 10:57:00 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 4251 ms
Jun 28, 2012 10:57:09 AM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8091
Jun 28, 2012 10:57:10 AM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
2012-06-28 10:57:10.890 INFO net.spy.memcached.MemcachedConnection:  Added {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2012-06-28 10:57:10.911 INFO net.spy.memcached.MemcachedConnection:  Added {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
Jun 28, 2012 10:57:11 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/test] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Jun 28, 2012 10:57:11 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load java.util.Set.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:252)
        at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:2030)
Jun 28, 2012 10:57:11 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load java.lang.Integer.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:270)
        at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:2030)
Jun 28, 2012 10:57:11 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load java.nio.channels.SelectionKey.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:274)
        at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:2030)
Jun 28, 2012 10:57:11 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load java.lang.Boolean.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:400)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:275)
        at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:2030)
2012-06-28 10:57:11.170 INFO net.spy.memcached.MemcachedConnection:  Connection state changed for sun.nio.ch.SelectionKeyImpl@1dff2e1b
2012-06-28 10:57:11.173 INFO net.spy.memcached.MemcachedConnection:  Connection state changed for sun.nio.ch.SelectionKeyImpl@2e694f12
2012-06-28 10:57:11.177 INFO net.spy.memcached.MemcachedConnection:  Reconnecting due to failure to connect to {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:407)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:275)
        at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:2030)


Thanks,
David.

[ingenthr]

Hi David,

Are you using daemon threads, as discussed here?

Also, the logging seems to indicate that the thread name is something
other than the thread from spymemcached, which would be something like
"Memcached IO over..." I think there's some other thread blocking
things.

Matt