Exception during writes

[Ravi]

hi,

we get the below exception when we write to the caceh.

Code:
--------
this.name = name;
List<InetSocketAddress> addList = new
ArrayList<InetSocketAddress>();
for (String ipPort : hostArr) {
String ip = ipPort.split(":")[0];
int port = Integer.parseInt(ipPort.split(":")[1]);
InetSocketAddress add = new InetSocketAddress(ip, port);
addList.add(add);
}
CacheMain.pr("total inetAddresses are:" + addList.size() + " [" +
addList + "]");
client = new MemcachedClient(addList);
map = new CacheMap(client, name);
CacheMain.pr("memcache initialization is complete");

--------
public void put(CacheElement<K, V> element) {
Future<Boolean> future = client.set((String) element.getKey(),
Integer.MAX_VALUE, element.getValue());
try {
future.get();
} catch (InterruptedException e) {
CacheMain.pr(e);
} catch (ExecutionException e) {
CacheMain.pr(e);
}
// CacheMain.pr(map.containsKey(element.getKey()) + "");
}

We need to wait for the write to complete and only then allowed to go
further. The timeout is specified to be very big. Any workarounds to
fix the problem?

Exception:
-------------
jvmNo:4 H:10.1.162.253: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Timed out waiting for operation
at java.util.concurrent.FutureTask
$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
com.fx.core.cache.stat.CacheStress.nThreads(CacheStress.java:239)
at
com.fx.core.cache.stat.ReadWriteDistStress.nIterations(ReadWriteDistStress.java:
73)
at
com.fx.core.cache.stat.PartitionedWriteRandomReadStress.nIterations(PartitionedWriteRandomReadStress.java:
38)
at
com.fx.core.cache.stat.CacheStress.nBatches(CacheStress.java:180)
at
com.fx.core.cache.stat.ReadWriteDistStress.stress(ReadWriteDistStress.java:
54)
at
com.fx.core.cache.stat.PartitionedWriteAndListen.stress(PartitionedWriteAndListen.java:
23)
at com.fx.core.cache.CacheMain$2.run(CacheMain.java:212)
Caused by: java.lang.RuntimeException: Timed out waiting for operation
at
net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:
51)
at com.fx.core.cache.MemCacheImpl.put(MemCacheImpl.java:174)
at com.fx.core.cache.stat.ReadWriteDistStress
$ReadWriteThreadExec.doWrite(ReadWriteDistStress.java:132)
at com.fx.core.cache.stat.ReadWriteDistStress
$ReadWriteThreadExec.nIterations(ReadWriteDistStress.java:116)
at com.fx.core.cache.stat.ReadWriteDistStress
$ReadWriteThreadExec.call(ReadWriteDistStress.java:104)
at com.fx.core.cache.stat.ReadWriteDistStress
$ReadWriteThreadExec.call(ReadWriteDistStress.java:93)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by:
net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out
waiting for operation - failing node: /10.1.162.251:9000
at
net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:
59)
at
net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:
49)
... 10 more

--
ravi

[Matt Ingenthron]

Hi Ravi,

I have a couple of quick questions. Are the memcached servers on a
reasonably local network? Are they, by chance, in a datacenter you
don't really control?

Part of the reason I ask, is we've had some reports of very high
latencies in some co-location or "cloud compute" style datacenters where
the network latency can get to be quite high.

Also, you say the timeout specified is very big. What is it set to?

On spymemcached issue 124, I was able to demonstrate that some cases
where the latency is high for an operation, it can be due to the
operation not being scheduled on the OS.

Thanks,

- Matt

[Ravi]

they are in the same lan. The timeout is set to Integer.max_value.

We are doing stress runs.. and have abt 40 threads in one jvm who do
writes. We have 8 such jvms (clients) and 2 cache servers who hold the
data.

We are using Memcacehe Daemon (java version of memcache server).

let
me know if more data is required.

--
ravi