voldemort.store.InsufficientOperationalNodesException: Get operation interrupted!

[rajp...@audiencescience.com]

We have a 5 node voldemort cluster that we use with BDB. We have
about 12 GB of data in each node and cache size of 17 GB , Java Max
Heap is 24 GB.

The issue we're running into is the readers of data (servlet) throw
the following exception. The web app is deployed in Tomcat (max
threads=100) , voldemort client config is default maxConn = 6 ,
maxThread=5.
The traffic to Tomcat is approx 150 per second.


voldemort.store.InsufficientOperationalNodesException: Get operation
interrupted!
at voldemort.store.routed.RoutedStore.get(RoutedStore.java:
494)
at voldemort.store.routed.RoutedStore.get(RoutedStore.java:
454)
at voldemort.store.routed.RoutedStore.get(RoutedStore.java:72)
at voldemort.store.DelegatingStore.get(DelegatingStore.java:
59)
at
voldemort.store.stats.StatTrackingStore.get(StatTrackingStore.java:64)
at
voldemort.store.serialized.SerializingStore.get(SerializingStore.java:
79)
at voldemort.store.DelegatingStore.get(DelegatingStore.java:
59)
at
voldemort.store.versioned.InconsistencyResolvingStore.get(InconsistencyResolvingStore.java:
50)
at
voldemort.client.DefaultStoreClient.get(DefaultStoreClient.java:121)
at
voldemort.client.DefaultStoreClient.getValue(DefaultStoreClient.java:
111)
at
com.audiencescience.service.user.AbstractUserDataService.getAsiUid(AbstractUserDataService.java:
202)
at
com.audiencescience.service.user.AbstractUserDataService.getUser(AbstractUserDataService.java:
170)
at
com.audiencescience.apollo.bidder.service.user.UserDataRequester.getUserRecordByGoogleKey(UserDataRequester.java:
31)
at
com.audiencescience.apollo.bidder.service.user.UserDataRequester.getUserData(UserDataRequester.java:
54)
at com.audiencescience.apollo.bidder.service.DecisionService
$1.call(DecisionService.java:81)
at com.audiencescience.apollo.bidder.service.DecisionService
$1.call(DecisionService.java:76)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:
1238)
at java.util.concurrent.FutureTask
$Sync.innerGet(FutureTask.java:227)
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
at
java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:
251)
at voldemort.store.routed.RoutedStore.get(RoutedStore.java:
492)
... 20 more

[Kirk True]

Hi,

Is it one node in particular that's getting the error?

If you're seeing ~150 requests/second at the application tier, and
assuming that the keys are distributed evenly about the 5 nodes in the
cluster, that's ~30 requests/second to each of the nodes in the
Voldemort cluster which is about as light of load as it gets.

It also looks like your data set is fitting completely in RAM. Gets
should be less than 5 milliseconds unless something insane is happening.

Can you tell us a little bit more about your configuration, request data
size, etc.?

Thanks,
Kirk