Exception in thread "Memcached IO over" with java.util.ConcurrentModificationException

[Owen Tran]

Hi,

While testing basic failover and recovery on Linux with spymecmached 2.8.1 and couchbase server 1.8, we're getting exceptions in the spy.memcached.MemcachedConnection .handleIO method due to concurrent modification exceptions...

2012-07-02 09:53:44.258 INFO com.couchbase.client.CouchbaseConnection:  Reconnecting {QA sa=cache02.corp.ezrez/10.100.1.64:11210, #Rops=0, #Wops=11, #iq=0, topRop=null, topWop=Cmd: 0 Opaque: 763 Key: 92fe0769.d0d2.4a35.88e0.ca069cd154ae.WEBTIER_EXCEPTION, toWrite=0, interested=0}
2012-07-02 09:53:44.259 INFO com.couchbase.client.CouchbaseConnection:  Connection state changed for sun.nio.ch.SelectionKeyImpl@474a9bee
2012-07-02 09:53:51.085 INFO com.couchbase.client.CouchbaseConnection:  Added {QA sa=/10.100.1.64:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
Exception in thread "Memcached IO over {MemcachedConnection to cache02.corp.ezrez/10.100.1.64:11210 cache01.corp.ezrez/10.100.1.63:11210}" java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
	at java.util.HashMap$KeyIterator.next(HashMap.java:828)
	at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1010)
	at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:248)
	at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:229)

Here's the code from github...

    // see if any connections blew up with large number of timeouts
    for (SelectionKey sk : selector.keys()) {
      MemcachedNode mn = (MemcachedNode) sk.attachment();
      if (mn.getContinuousTimeout() > timeoutExceptionThreshold) {
        getLogger().warn("%s exceeded continuous timeout threshold", sk);
        lostConnection(mn);
      }
    }

The steps to reproduce:

1. Start two nodes
2. Stop node #2
3. Make sure spymemcached is working
4. Start node #2 and rebalance
5. spymemcached throws conccurent modification once it notices node #2 is back

Thanks,
Owen


Here's the logs:

2012-07-02 09:52:44.227 WARN com.couchbase.client.CouchbaseConnection:  Closing, and reopening {QA sa=cache02.corp.ezrez/10.100.1.64:11210, #Rops=0, #Wops=11, #iq=0, topRop=null, topWop=Cmd: 0 Opaque: 763 Key: 92fe0769.d0d2.4a35.88e0.ca069cd154ae.WEBTIER_EXCEPTION, toWrite=0, interested=0}, attempt 14.
2012-07-02 09:53:14.231 INFO com.couchbase.client.CouchbaseConnection:  Reconnecting {QA sa=cache02.corp.ezrez/10.100.1.64:11210, #Rops=0, #Wops=11, #iq=0, topRop=null, topWop=Cmd: 0 Opaque: 763 Key: 92fe0769.d0d2.4a35.88e0.ca069cd154ae.WEBTIER_EXCEPTION, toWrite=0, interested=0}
2012-07-02 09:53:14.231 INFO com.couchbase.client.CouchbaseConnection:  Connection state changed for sun.nio.ch.SelectionKeyImpl@fb95e78
2012-07-02 09:53:14.232 INFO com.couchbase.client.CouchbaseConnection:  Reconnecting due to failure to connect to {QA sa=cache02.corp.ezrez/10.100.1.64:11210, #Rops=0, #Wops=11, #iq=0, topRop=null, topWop=Cmd: 0 Opaque: 763 Key: 92fe0769.d0d2.4a35.88e0.ca069cd154ae.WEBTIER_EXCEPTION, toWrite=0, interested=0}
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
	at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:369)
	at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
	at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:229)
2012-07-02 09:53:14.232 WARN com.couchbase.client.CouchbaseConnection:  Closing, and reopening {QA sa=cache02.corp.ezrez/10.100.1.64:11210, #Rops=0, #Wops=11, #iq=0, topRop=null, topWop=Cmd: 0 Opaque: 763 Key: 92fe0769.d0d2.4a35.88e0.ca069cd154ae.WEBTIER_EXCEPTION, toWrite=0, interested=0}, attempt 15.
2012-07-02 09:53:44.258 INFO com.couchbase.client.CouchbaseConnection:  Reconnecting {QA sa=cache02.corp.ezrez/10.100.1.64:11210, #Rops=0, #Wops=11, #iq=0, topRop=null, topWop=Cmd: 0 Opaque: 763 Key: 92fe0769.d0d2.4a35.88e0.ca069cd154ae.WEBTIER_EXCEPTION, toWrite=0, interested=0}
2012-07-02 09:53:44.259 INFO com.couchbase.client.CouchbaseConnection:  Connection state changed for sun.nio.ch.SelectionKeyImpl@474a9bee
2012-07-02 09:53:51.085 INFO com.couchbase.client.CouchbaseConnection:  Added {QA sa=/10.100.1.64:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
Exception in thread "Memcached IO over {MemcachedConnection to cache02.corp.ezrez/10.100.1.64:11210 cache01.corp.ezrez/10.100.1.63:11210}" java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
	at java.util.HashMap$KeyIterator.next(HashMap.java:828)
	at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1010)
	at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:248)
	at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:229)

// LATER, CONTINUOUS ERRORS

2012-07-02 09:58:32.794 WARN com.couchbase.client.CouchbaseConnection:  Node exepcted to receive data is inactive.  This could be due to afailure within the cluster.  Will check for updated configuration.  Key without a configured node is: null.SESSION_DESCRIPTOR.
Jul 2, 2012 9:58:32 AM com.couchbase.client.CouchbaseConnectionFactory checkConfigUpdate
WARNING: No reconnect required, though check requested. Current config check is 1 out of a threshold of 10.

[ingenthr]

Ouch. We'll have to file an issue and look into this.

I would say though, if you're using Couchbase you really want to be
using the Couchbase Java client, which we've (I'm a Couchbase guy)
just updated this evening. It has special handling for both Couchbase
bucket types and memcached bucket types.

Do use the 1.0.3 release though. It has updates for memcached bucket
types.

Oh.... looking a bit further there, you do seem to be using it.
Actually, this issue is one fixed in the Couchbase 1.0.3 release.

FWIW, this list is really about spymemcached. Either use
couc...@googlegroups.com or the Couchbase forums for discussion of
the Couchbase client.

So... disregard what I said about filling an issue.