JBoss Lockups using Spymemcached

[Alastair Surin]

Hey All,

We've been using spymemcached in production for a while and have recently started experiencing problems.  We regularly get timeout exceptions connecting to our memcache servers, although I can confirm (via. telnet) that they are online and responsive from the server in question.  Eventually JBoss itself will deadlock and when I run a thread dump I see that most of the threads are blocked waiting on responses from spymemcached:

Thread 5402: (state = BLOCKED)

 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)

 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20 (Compiled frame)

 - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(int, long) @bci=92 (Interpreted frame)

 - java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(int, long) @bci=25 (Compiled frame)

 - java.util.concurrent.CountDownLatch.await(long, java.util.concurrent.TimeUnit) @bci=10 (Compiled frame)

 - net.spy.memcached.internal.OperationFuture.get(long, java.util.concurrent.TimeUnit) @bci=6, line=160 (Compiled frame)

 - net.spy.memcached.internal.GetFuture.get(long, java.util.concurrent.TimeUnit) @bci=6, line=69 (Compiled frame)

 - net.spy.memcached.MemcachedClient.get(java.lang.String, net.spy.memcached.transcoders.Transcoder) @bci=13, line=1166 (Interpreted frame)

 - net.spy.memcached.MemcachedClient.get(java.lang.String) @bci=6, line=1192 (Compiled frame)

Has anyone encountered anything like this, or does anyone have any advice?  Appreciate your time!