Timed out waiting for operation on cache get

[Calvin Chen]

Anybody here know why this is happening?  I'm running Simple Spring Memcached 3.6 with a spymemcached client(2.11.7) against AWS Elasticache nodes, issue doesn't seem to be related to connectivity as I'm able to nc -v all 3 of the cluster nodes from where the application is deployed.

Issue also seems to be hitting just 1 of the 3 Elasticache nodes, none of the other 2 nodes are ever redistributed to on these operation failures.  The SpyMemcachedConfiguration class I'm using defines the following values,

 <prop key="memcached.consistentHashing">true</prop>

 <prop key="memcached.failureMode">Redistribute</prop>

 <prop key="memcached.hashingAlgorithm">KETAMA_HASH</prop>

 <prop key="memcached.locatorType">CONSISTENT</prop>

 <prop key="memcached.maxReconnectDelay">10000</prop>

 <prop key="memcached.timeoutExceptionThreshold">1998</prop>

 <prop key="memcached.operationTimeout">10000</prop>

 

Exception thrown,

net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: com-test-common.cache.amazonaws.com/172.18.22.75:11211

at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:167) ~[spymemcached-2.11.7.jar:2.11.7]

at net.spy.memcached.internal.GetFuture.get(GetFuture.java:69) ~[spymemcached-2.11.7.jar:2.11.7]

at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1230) ~[spymemcached-2.11.7.jar:2.11.7]

at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1257) ~[spymemcached-2.11.7.jar:2.11.7]

at com.google.code.ssm.providers.spymemcached.MemcacheClientWrapper.get(MemcacheClientWrapper.java:147) ~[spymemcached-provider-3.6.0.jar:na]

at com.google.code.ssm.CacheImpl.get(CacheImpl.java:276) ~[simple-spring-memcached-3.6.0.jar:na]

at com.google.code.ssm.CacheImpl.get(CacheImpl.java:106) ~[simple-spring-memcached-3.6.0.jar:na]

at com.google.code.ssm.aop.SingleReadCacheAdvice.cache(SingleReadCacheAdvice.java:67) ~[simple-spring-memcached-3.6.0.jar:na]

at com.google.code.ssm.aop.ReadThroughSingleCacheAdvice.cacheGetSingle(ReadThroughSingleCacheAdvice.java:53) [simple-spring-memcached-3.6.0.jar:na]

at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source) ~[na:na]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]

at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_80]

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE]

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE]

at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE]

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE]

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE]

[Calvin Chen]

Redirecting to the same post I have in the SSM group,

https://groups.google.com/forum/#!topic/simple-spring-memecached/CsNdv-8Rhb4