Unreachable Store Exception & Failure while checking out socket error
871 views
Skip to first unread message
Mani
Nov 8, 2010, 5:05:56 AM11/8/10
to project-voldemort
Hi Everyone,
I am trying to do the load testing for my project to make sure that it
can be deployabale on production environment. I am using only one node
with 4 stores and bdb persistence to store the data. While doing the
load testing, come across the below exceptions:
13:02:33 WARN [AbstractStoreClientFactory]: Failed to bootstrap from
tcp://localhost:6666
voldemort.store.UnreachableStoreException: Failure while checking out
socket for localhost:6666(vp1):
at voldemort.store.socket.SocketPool.checkout(SocketPool.java:
92)
at voldemort.store.socket.SocketStore.get(SocketStore.java:
121)
at voldemort.store.socket.SocketStore.get(SocketStore.java:47)
at
voldemort.store.serialized.SerializingStore.get(SerializingStore.java:
79)
at
voldemort.client.AbstractStoreClientFactory.bootstrapMetadata(AbstractStoreClientFactory.java:
253)
at
voldemort.client.AbstractStoreClientFactory.bootstrapMetadataWithRetries(AbstractStoreClientFactory.java:
221)
at
voldemort.client.AbstractStoreClientFactory.getRawStore(AbstractStoreClientFactory.java:
131)
at
voldemort.client.DefaultStoreClient.bootStrap(DefaultStoreClient.java:
80)
at
voldemort.client.DefaultStoreClient.<init>(DefaultStoreClient.java:74)
at
voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:
124)
at
voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:
118)
at
com.netcore.mammcwt.store.StoreWrapper.<init>(StoreWrapper.java:44)
at
com.netcore.mammcwt.cache.SingleInitializationCache.getCommDetailsStoreClient(SingleInitializationCache.java:
147)
at com.netcore.mammcwt.store.StoreDAO.<init>(StoreDAO.java:42)
at
com.netcore.mammcwt.dlrstatushandler.SMSDlrStatusHandlerTask.run(SMSDlrStatusHandlerTask.java:
67)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:
218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:
258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:
317)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at
voldemort.store.socket.SocketResourceFactory.negotiateProtocol(SocketResourceFactory.java:
102)
at
voldemort.store.socket.SocketResourceFactory.create(SocketResourceFactory.java:
89)
at
voldemort.store.socket.SocketResourceFactory.create(SocketResourceFactory.java:
41)
at
voldemort.utils.pool.KeyedResourcePool.attemptGrow(KeyedResourcePool.java:
153)
at
voldemort.utils.pool.KeyedResourcePool.checkoutOrCreateResource(KeyedResourcePool.java:
133)
at
voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:
106)
at voldemort.store.socket.SocketPool.checkout(SocketPool.java:
87)
... 17 more
13:02:33 WARN [SMSDlrStatusHandlerTask]: No available boostrap
servers found!
voldemort.client.BootstrapFailureException: No available boostrap
servers found!
at
voldemort.client.AbstractStoreClientFactory.bootstrapMetadataWithRetries(AbstractStoreClientFactory.java:
236)
at
voldemort.client.AbstractStoreClientFactory.getRawStore(AbstractStoreClientFactory.java:
131)
at
voldemort.client.DefaultStoreClient.bootStrap(DefaultStoreClient.java:
80)
at
voldemort.client.DefaultStoreClient.<init>(DefaultStoreClient.java:74)
at
voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:
124)
at
voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:
118)
at
com.netcore.mammcwt.store.StoreWrapper.<init>(StoreWrapper.java:44)
at
com.netcore.mammcwt.cache.SingleInitializationCache.getCommDetailsStoreClient(SingleInitializationCache.java:
147)
at com.netcore.mammcwt.store.StoreDAO.<init>(StoreDAO.java:42)
at
com.netcore.mammcwt.dlrstatushandler.SMSDlrStatusHandlerTask.run(SMSDlrStatusHandlerTask.java:
67)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Configuration details:
node.id=0
max.threads=100
http.enable=true
socket.enable=true
enable.readonly.engine=true
file.fetcher.class=voldemort.store.readonly.fetcher.HdfsFetcher
Can anyone please help me to fix this issue?
Thanks,
Mani
I am trying to do the load testing for my project to make sure that it
can be deployabale on production environment. I am using only one node
with 4 stores and bdb persistence to store the data. While doing the
load testing, come across the below exceptions:
13:02:33 WARN [AbstractStoreClientFactory]: Failed to bootstrap from
tcp://localhost:6666
voldemort.store.UnreachableStoreException: Failure while checking out
socket for localhost:6666(vp1):
at voldemort.store.socket.SocketPool.checkout(SocketPool.java:
92)
at voldemort.store.socket.SocketStore.get(SocketStore.java:
121)
at voldemort.store.socket.SocketStore.get(SocketStore.java:47)
at
voldemort.store.serialized.SerializingStore.get(SerializingStore.java:
79)
at
voldemort.client.AbstractStoreClientFactory.bootstrapMetadata(AbstractStoreClientFactory.java:
253)
at
voldemort.client.AbstractStoreClientFactory.bootstrapMetadataWithRetries(AbstractStoreClientFactory.java:
221)
at
voldemort.client.AbstractStoreClientFactory.getRawStore(AbstractStoreClientFactory.java:
131)
at
voldemort.client.DefaultStoreClient.bootStrap(DefaultStoreClient.java:
80)
at
voldemort.client.DefaultStoreClient.<init>(DefaultStoreClient.java:74)
at
voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:
124)
at
voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:
118)
at
com.netcore.mammcwt.store.StoreWrapper.<init>(StoreWrapper.java:44)
at
com.netcore.mammcwt.cache.SingleInitializationCache.getCommDetailsStoreClient(SingleInitializationCache.java:
147)
at com.netcore.mammcwt.store.StoreDAO.<init>(StoreDAO.java:42)
at
com.netcore.mammcwt.dlrstatushandler.SMSDlrStatusHandlerTask.run(SMSDlrStatusHandlerTask.java:
67)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:
218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:
258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:
317)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at
voldemort.store.socket.SocketResourceFactory.negotiateProtocol(SocketResourceFactory.java:
102)
at
voldemort.store.socket.SocketResourceFactory.create(SocketResourceFactory.java:
89)
at
voldemort.store.socket.SocketResourceFactory.create(SocketResourceFactory.java:
41)
at
voldemort.utils.pool.KeyedResourcePool.attemptGrow(KeyedResourcePool.java:
153)
at
voldemort.utils.pool.KeyedResourcePool.checkoutOrCreateResource(KeyedResourcePool.java:
133)
at
voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:
106)
at voldemort.store.socket.SocketPool.checkout(SocketPool.java:
87)
... 17 more
13:02:33 WARN [SMSDlrStatusHandlerTask]: No available boostrap
servers found!
voldemort.client.BootstrapFailureException: No available boostrap
servers found!
at
voldemort.client.AbstractStoreClientFactory.bootstrapMetadataWithRetries(AbstractStoreClientFactory.java:
236)
at
voldemort.client.AbstractStoreClientFactory.getRawStore(AbstractStoreClientFactory.java:
131)
at
voldemort.client.DefaultStoreClient.bootStrap(DefaultStoreClient.java:
80)
at
voldemort.client.DefaultStoreClient.<init>(DefaultStoreClient.java:74)
at
voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:
124)
at
voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:
118)
at
com.netcore.mammcwt.store.StoreWrapper.<init>(StoreWrapper.java:44)
at
com.netcore.mammcwt.cache.SingleInitializationCache.getCommDetailsStoreClient(SingleInitializationCache.java:
147)
at com.netcore.mammcwt.store.StoreDAO.<init>(StoreDAO.java:42)
at
com.netcore.mammcwt.dlrstatushandler.SMSDlrStatusHandlerTask.run(SMSDlrStatusHandlerTask.java:
67)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Configuration details:
node.id=0
max.threads=100
http.enable=true
socket.enable=true
enable.readonly.engine=true
file.fetcher.class=voldemort.store.readonly.fetcher.HdfsFetcher
Can anyone please help me to fix this issue?
Thanks,
Mani
Kirk True
Nov 9, 2010, 10:08:31 AM11/9/10
to project-...@googlegroups.com, Mani
Hi Mani,
This looks like a pretty low-level error. Are you getting any messages
in the server logs?
Thanks,
Kirk
Mani
Nov 10, 2010, 2:00:27 AM11/10/10
to project-voldemort
Hi Kirk,
After modifying the configuration, am not getting the below
exception.
Since I started using Voldemort recently, can you suggest me the
optimal configuration parameters?
At sometimes, I get the following errors:
1. Too many open connections. 500 of 500 threads in use, denying
connections..
2. 1 reads required, but 0 succeeded.
voldemort.store.InsufficientOperationalNodesException: 1 reads
required, but 0 succeeded.
3. voldemort.store.UnreachableStoreException: Failure while checking
unavailable
16:40:06 INFO [AbstractStoreClientFactory]: Node0 partitionList:[0,
1] has been marked as unavailable, destroying socket pool
Thanks,
Mani
After modifying the configuration, am not getting the below
exception.
Since I started using Voldemort recently, can you suggest me the
optimal configuration parameters?
At sometimes, I get the following errors:
1. Too many open connections. 500 of 500 threads in use, denying
connections..
2. 1 reads required, but 0 succeeded.
voldemort.store.InsufficientOperationalNodesException: 1 reads
required, but 0 succeeded.
3. voldemort.store.UnreachableStoreException: Failure while checking
out socket for localhost:6666(vp1):
4. 16:40:06 INFO [BannagePeriodFailureDetector]: Node 0 now
unavailable
16:40:06 INFO [AbstractStoreClientFactory]: Node0 partitionList:[0,
1] has been marked as unavailable, destroying socket pool
Thanks,
Mani
_Mike
Nov 10, 2010, 1:28:46 PM11/10/10
to project-voldemort
I had a similar problem with unreachable store exceptions. It turns
out that I mistakenly assumed that the StoreClientFactory was caching/
reusing the connections, and the server nodes reached their limit on
threads. If you cache your client connections, you might have better
luck.
On Tue, Nov 10, 2010 at 2:10 am, Mani <maniraj...@gmail.com> wrote:
> In my case, connection will get created and stored in the single
> initialization cache while starting the server. After that, for every
>incoming request, makes use of connection from the cache only.
> Is it correct?
That is the general idea, but you could possibly cache a few per
client.
It all depends on your client usage needs.
_Mike
out that I mistakenly assumed that the StoreClientFactory was caching/
reusing the connections, and the server nodes reached their limit on
threads. If you cache your client connections, you might have better
luck.
On Tue, Nov 10, 2010 at 2:10 am, Mani <maniraj...@gmail.com> wrote:
> In my case, connection will get created and stored in the single
> initialization cache while starting the server. After that, for every
>incoming request, makes use of connection from the cache only.
> Is it correct?
That is the general idea, but you could possibly cache a few per
client.
It all depends on your client usage needs.
_Mike
Mani
Nov 11, 2010, 8:15:43 AM11/11/10
to project-voldemort
Kirk, Mike,
I have performed load testing on my app. Here are the details about
machines used:
Machine 1 (Voldemort):
----------------------
Machine configuration:
Dual Core CPU
4GB RAM
Voldemort Configuration:
node.id=0
max.threads=5000
############### DB options ######################
http.enable=true
socket.enable=true
# BDB
bdb.write.transactions=false
bdb.flush.transactions=false
bdb.cache.size=1G
storage.configs=voldemort.store.bdb.BdbStorageConfiguration,
voldemort.store.readonly.ReadOnlyStorageConfiguration
Machine 2 (my app):
-------------------
Machine configuration:
Dual Core CPU
4GB RAM
Load test observations:
App running on this machine connects to voldemort in machine 1 to do
the operations.
I am using Jmeter to do the load test. No. of threads is 300 and loop
count is forever in Jmeter load test settings.
After running the test for 15 minutes, getting the following errors in
my app logs:
1. 17:55:52 WARN [BannagePeriodFailureDetector]: Node 0 set as
unavailable
105)
at voldemort.store.routed.RoutedStore
$1.execute(RoutedStore.java:79)
at voldemort.store.routed.RoutedStore
$GetCallable.call(RoutedStore.java:900)
at voldemort.store.routed.RoutedStore
$GetCallable.call(RoutedStore.java:880)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.concurrent.TimeoutException: Could not acquire
resource in 500 ms.
at
voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:
104)
at voldemort.store.socket.SocketPool.checkout(SocketPool.java:
87)
Then,
17:55:52 WARN [DlrStatusHandlerTask]: 1 reads required, but 0
562)
at voldemort.store.routed.RoutedStore.get(RoutedStore.java:
454)
at voldemort.store.routed.RoutedStore.get(RoutedStore.java:72)
at voldemort.store.DelegatingStore.get(DelegatingStore.java:
59)
at
voldemort.store.stats.StatTrackingStore.get(StatTrackingStore.java:64)
at
voldemort.store.serialized.SerializingStore.get(SerializingStore.java:
79)
at voldemort.store.DelegatingStore.get(DelegatingStore.java:
59)
at
voldemort.store.versioned.InconsistencyResolvingStore.get(InconsistencyResolvingStore.java:
50)
at
voldemort.client.DefaultStoreClient.get(DefaultStoreClient.java:121)
at
voldemort.client.DefaultStoreClient.get(DefaultStoreClient.java:154)
at
com.netcore.mammcwt.store.StoreWrapper.get(StoreWrapper.java:55)
at
com.netcore.mammcwt.store.StoreDAO.getCommunicationId(StoreDAO.java:
343)
at
com.netcore.mammcwt.dlrstatushandler.DlrStatusHandlerTask.run(DlrStatusHandlerTask.java:
67)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: voldemort.store.UnreachableStoreException: Failure while
checking out socket for 192.168.5.30:6666(vp1):
105)
at voldemort.store.routed.RoutedStore
$1.execute(RoutedStore.java:79)
at voldemort.store.routed.RoutedStore
$GetCallable.call(RoutedStore.java:900)
at voldemort.store.routed.RoutedStore
$GetCallable.call(RoutedStore.java:880)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.util.concurrent.TimeoutException: Could not acquire
resource in 500 ms.
at
voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:
104)
at voldemort.store.socket.SocketPool.checkout(SocketPool.java:
87)
... 11 more
Then,
18:00:30 INFO [DefaultStoreClient]: bootstrapping metadata.
18:00:30 WARN [DlrStatusHandlerTask]: org.jdom.JDOMException: Could
not load default SAX parser: org.apache.xerces.parsers.SAXParser: SAX2
driver class org.apache.xerces.parsers.SAXParser not found:
org.apache.xerces.parsers.SAXParser
voldemort.xml.MappingException: org.jdom.JDOMException: Could not load
default SAX parser: org.apache.xerces.parsers.SAXParser: SAX2 driver
class org.apache.xerces.parsers.SAXParser not found:
org.apache.xerces.parsers.SAXParser
at voldemort.xml.ClusterMapper.readCluster(ClusterMapper.java:
106)
at
voldemort.client.AbstractStoreClientFactory.getFailureDetector(AbstractStoreClientFactory.java:
201)
at
voldemort.client.AbstractStoreClientFactory.getRawStore(AbstractStoreClientFactory.java:
154)
181)
at com.netcore.mammcwt.store.StoreDAO.<init>(StoreDAO.java:46)
at
com.netcore.mammcwt.dlrstatushandler.DlrStatusHandlerTask.run(DlrStatusHandlerTask.java:
65)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.jdom.JDOMException: Could not load default SAX parser:
org.apache.xerces.parsers.SAXParser: SAX2 driver class
org.apache.xerces.parsers.SAXParser not found:
org.apache.xerces.parsers.SAXParser
at org.jdom.input.SAXBuilder.createParser(SAXBuilder.java:618)
at org.jdom.input.SAXBuilder.build(SAXBuilder.java:460)
at org.jdom.input.SAXBuilder.build(SAXBuilder.java:888)
at voldemort.xml.ClusterMapper.readCluster(ClusterMapper.java:
93)
... 13 more
Caused by: java.lang.ClassNotFoundException:
org.apache.xerces.parsers.SAXParser
at
org.xml.sax.helpers.XMLReaderFactory.loadClass(XMLReaderFactory.java:
189)
at
org.xml.sax.helpers.XMLReaderFactory.createXMLReader(XMLReaderFactory.java:
180)
at org.jdom.input.SAXBuilder.createParser(SAXBuilder.java:610)
... 16 more
Caused by: java.lang.ClassNotFoundException:
org.apache.xerces.parsers.SAXParser
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:
301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at
org.xml.sax.helpers.NewInstance.newInstance(NewInstance.java:49)
at org.xml.sax
My use case is, the app should support 300 req/sec. Can you please
help me in tuning parameters?
Thanks,
Mani
I have performed load testing on my app. Here are the details about
machines used:
Machine 1 (Voldemort):
----------------------
Machine configuration:
Dual Core CPU
4GB RAM
Voldemort Configuration:
node.id=0
max.threads=5000
############### DB options ######################
http.enable=true
socket.enable=true
# BDB
bdb.write.transactions=false
bdb.flush.transactions=false
bdb.cache.size=1G
storage.configs=voldemort.store.bdb.BdbStorageConfiguration,
voldemort.store.readonly.ReadOnlyStorageConfiguration
Machine 2 (my app):
-------------------
Machine configuration:
Dual Core CPU
4GB RAM
Load test observations:
App running on this machine connects to voldemort in machine 1 to do
the operations.
I am using Jmeter to do the load test. No. of threads is 300 and loop
count is forever in Jmeter load test settings.
After running the test for 15 minutes, getting the following errors in
my app logs:
1. 17:55:52 WARN [BannagePeriodFailureDetector]: Node 0 set as
unavailable
voldemort.store.UnreachableStoreException: Failure while checking out
socket for 192.168.5.30:6666(vp1):
at voldemort.store.socket.SocketPool.checkout(SocketPool.java:
92)
at voldemort.store.socket.SocketStore.get(SocketStore.java:
121)
at voldemort.store.socket.SocketStore.get(SocketStore.java:47)
at voldemort.store.logging.LoggingStore.get(LoggingStore.java:
92)
at voldemort.store.socket.SocketStore.get(SocketStore.java:
121)
at voldemort.store.socket.SocketStore.get(SocketStore.java:47)
105)
at voldemort.store.routed.RoutedStore
$1.execute(RoutedStore.java:79)
at voldemort.store.routed.RoutedStore
$GetCallable.call(RoutedStore.java:900)
at voldemort.store.routed.RoutedStore
$GetCallable.call(RoutedStore.java:880)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.concurrent.TimeoutException: Could not acquire
resource in 500 ms.
at
voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:
104)
at voldemort.store.socket.SocketPool.checkout(SocketPool.java:
87)
Then,
17:55:52 WARN [DlrStatusHandlerTask]: 1 reads required, but 0
succeeded.
voldemort.store.InsufficientOperationalNodesException: 1 reads
required, but 0 succeeded.
at voldemort.store.routed.RoutedStore.get(RoutedStore.java:
voldemort.store.InsufficientOperationalNodesException: 1 reads
required, but 0 succeeded.
562)
at voldemort.store.routed.RoutedStore.get(RoutedStore.java:
454)
at voldemort.store.routed.RoutedStore.get(RoutedStore.java:72)
at voldemort.store.DelegatingStore.get(DelegatingStore.java:
59)
at
voldemort.store.stats.StatTrackingStore.get(StatTrackingStore.java:64)
at
voldemort.store.serialized.SerializingStore.get(SerializingStore.java:
79)
at voldemort.store.DelegatingStore.get(DelegatingStore.java:
59)
at
voldemort.store.versioned.InconsistencyResolvingStore.get(InconsistencyResolvingStore.java:
50)
at
voldemort.client.DefaultStoreClient.get(DefaultStoreClient.java:121)
at
voldemort.client.DefaultStoreClient.get(DefaultStoreClient.java:154)
at
com.netcore.mammcwt.store.StoreWrapper.get(StoreWrapper.java:55)
at
com.netcore.mammcwt.store.StoreDAO.getCommunicationId(StoreDAO.java:
343)
at
com.netcore.mammcwt.dlrstatushandler.DlrStatusHandlerTask.run(DlrStatusHandlerTask.java:
67)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: voldemort.store.UnreachableStoreException: Failure while
checking out socket for 192.168.5.30:6666(vp1):
at voldemort.store.socket.SocketPool.checkout(SocketPool.java:
92)
at voldemort.store.socket.SocketStore.get(SocketStore.java:
121)
at voldemort.store.socket.SocketStore.get(SocketStore.java:47)
at voldemort.store.logging.LoggingStore.get(LoggingStore.java:
92)
at voldemort.store.socket.SocketStore.get(SocketStore.java:
121)
at voldemort.store.socket.SocketStore.get(SocketStore.java:47)
105)
at voldemort.store.routed.RoutedStore
$1.execute(RoutedStore.java:79)
at voldemort.store.routed.RoutedStore
$GetCallable.call(RoutedStore.java:900)
at voldemort.store.routed.RoutedStore
$GetCallable.call(RoutedStore.java:880)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.util.concurrent.TimeoutException: Could not acquire
resource in 500 ms.
at
voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:
104)
at voldemort.store.socket.SocketPool.checkout(SocketPool.java:
87)
... 11 more
Then,
18:00:30 INFO [DefaultStoreClient]: bootstrapping metadata.
18:00:30 WARN [DlrStatusHandlerTask]: org.jdom.JDOMException: Could
not load default SAX parser: org.apache.xerces.parsers.SAXParser: SAX2
driver class org.apache.xerces.parsers.SAXParser not found:
org.apache.xerces.parsers.SAXParser
voldemort.xml.MappingException: org.jdom.JDOMException: Could not load
default SAX parser: org.apache.xerces.parsers.SAXParser: SAX2 driver
class org.apache.xerces.parsers.SAXParser not found:
org.apache.xerces.parsers.SAXParser
at voldemort.xml.ClusterMapper.readCluster(ClusterMapper.java:
106)
at
voldemort.client.AbstractStoreClientFactory.getFailureDetector(AbstractStoreClientFactory.java:
201)
at
voldemort.client.AbstractStoreClientFactory.getRawStore(AbstractStoreClientFactory.java:
154)
at
voldemort.client.DefaultStoreClient.bootStrap(DefaultStoreClient.java:
80)
at
voldemort.client.DefaultStoreClient.<init>(DefaultStoreClient.java:74)
at
voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:
124)
at
voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:
118)
at
com.netcore.mammcwt.store.StoreWrapper.<init>(StoreWrapper.java:44)
at
com.netcore.mammcwt.cache.SingleInitializationCache.getVendorAckIdCommIdMapStoreClient(SingleInitializationCache.java:
voldemort.client.DefaultStoreClient.bootStrap(DefaultStoreClient.java:
80)
at
voldemort.client.DefaultStoreClient.<init>(DefaultStoreClient.java:74)
at
voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:
124)
at
voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:
118)
at
com.netcore.mammcwt.store.StoreWrapper.<init>(StoreWrapper.java:44)
at
181)
at com.netcore.mammcwt.store.StoreDAO.<init>(StoreDAO.java:46)
at
com.netcore.mammcwt.dlrstatushandler.DlrStatusHandlerTask.run(DlrStatusHandlerTask.java:
65)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.jdom.JDOMException: Could not load default SAX parser:
org.apache.xerces.parsers.SAXParser: SAX2 driver class
org.apache.xerces.parsers.SAXParser not found:
org.apache.xerces.parsers.SAXParser
at org.jdom.input.SAXBuilder.createParser(SAXBuilder.java:618)
at org.jdom.input.SAXBuilder.build(SAXBuilder.java:460)
at org.jdom.input.SAXBuilder.build(SAXBuilder.java:888)
at voldemort.xml.ClusterMapper.readCluster(ClusterMapper.java:
93)
... 13 more
Caused by: java.lang.ClassNotFoundException:
org.apache.xerces.parsers.SAXParser
at
org.xml.sax.helpers.XMLReaderFactory.loadClass(XMLReaderFactory.java:
189)
at
org.xml.sax.helpers.XMLReaderFactory.createXMLReader(XMLReaderFactory.java:
180)
at org.jdom.input.SAXBuilder.createParser(SAXBuilder.java:610)
... 16 more
Caused by: java.lang.ClassNotFoundException:
org.apache.xerces.parsers.SAXParser
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:
301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at
org.xml.sax.helpers.NewInstance.newInstance(NewInstance.java:49)
at org.xml.sax
My use case is, the app should support 300 req/sec. Can you please
help me in tuning parameters?
Thanks,
Mani
gxm
Nov 12, 2010, 12:05:27 PM11/12/10
to project-voldemort
Hi Mani,
I'm certain that the first error caused the second.
The root cause of the first error is the socket pool does not return a
socket within the specified timeout period of 500 ms.
if you are using the default client settings, there may be other
settings you want to adjust as well, such as ClientConfig's
setMaxThreads (default 5) & setMaxConnectionsPerNode (default 6),
since you are using 300 application threads, while Voldemort's client
inside your application is only using 5 threads.
However, if you only need to support 300 requests per second, you
probably don't need 300 threads, as most requests should take much
less than a full second.
I recommend starting off with just a single thread, and increasing the
threads from there. A single thread lets you verify that your basic
application is working. Then go to 5 threads, 10 threads, 20, 50,
etc. I'd be surprised if you need that many to achieve your goal.
I'm not sure about the SAX parser issue, however I suspect it could be
related to the first. I wouldn't bother with it until you address the
first issue.
> ...
>
> read more »
I'm certain that the first error caused the second.
The root cause of the first error is the socket pool does not return a
socket within the specified timeout period of 500 ms.
"Caused by: java.util.concurrent.TimeoutException: Could not acquire
resource in 500 ms."
You could increase that timeout to a longer period of time, however,
resource in 500 ms."
if you are using the default client settings, there may be other
settings you want to adjust as well, such as ClientConfig's
setMaxThreads (default 5) & setMaxConnectionsPerNode (default 6),
since you are using 300 application threads, while Voldemort's client
inside your application is only using 5 threads.
However, if you only need to support 300 requests per second, you
probably don't need 300 threads, as most requests should take much
less than a full second.
I recommend starting off with just a single thread, and increasing the
threads from there. A single thread lets you verify that your basic
application is working. Then go to 5 threads, 10 threads, 20, 50,
etc. I'd be surprised if you need that many to achieve your goal.
I'm not sure about the SAX parser issue, however I suspect it could be
related to the first. I wouldn't bother with it until you address the
first issue.
>
> read more »
Manikandan R
Nov 12, 2010, 12:30:35 PM11/12/10
to project-...@googlegroups.com
@gxm,
Thanks a lot for your detailed explanation.
I was assuming that my code makes use of same connection from single initialization cache (created while starting the app server) for every new incoming request. But it was not. For every new request, app server is creating new connection to the voldemort to perform the operation.
I have fixed this. Connection gets created while starting the app server and it will be in cache. I ran the load tests of hitting the app server with request continously. Also, I have set the max.threads=100. Since 4 stores are there, 4 connection gets created while starting the app server. For every new incoming request, it just makes use of the same connection from the cache.
Since the app server is making use of the same connection, it takes very long time to complete each request. But it was not throwing any exception mentioned in the previous mail. SInce all the get, put ops in request processing makes use of the same connection,
1. Is it correct way of doing? Why is taking long time to complete each request? How to solve this? Should I use pooling of max 100 (25 threads for each store)?
2. Will be there any concurrency issues?
Also, after 2 hrs,
21:52:02 WARN [CommHandlerTask]: Get operation interrupted!
voldemort.store.InsufficientOperationalNodesException: Get operation interrupted!
at voldemort.store.routed.RoutedStore.get(RoutedStore.java:494)
at voldemort.store.routed.RoutedStore.getVersions(RoutedStore.java:869)
at voldemort.store.routed.RoutedStore.getVersions(RoutedStore.java:72)
at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:85)
at voldemort.store.serialized.SerializingStore.getVersions(SerializingStore.java:114)
at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:85)
at voldemort.client.DefaultStoreClient.getVersions(DefaultStoreClient.java:134)
at voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:180)
at com.netcore.mammcwt.store.StoreWrapper.put(StoreWrapper.java:69)
at com.netcore.mammcwt.store.StoreDAO.insertCommDetails(StoreDAO.java:177)
at com.netcore.mammcwt.store.StoreDAO.commHandler(StoreDAO.java:162)
at com.netcore.mammcwt.commhandler.CommHandlerTask.run(CommHandlerTask.java:106)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1238)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
at java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:251)
at voldemort.store.routed.RoutedStore.get(RoutedStore.java:492)
... 14 more
voldemort.store.InsufficientOperationalNodesException: Get operation interrupted!
at voldemort.store.routed.RoutedStore.get(RoutedStore.java:494)
at voldemort.store.routed.RoutedStore.getVersions(RoutedStore.java:869)
at voldemort.store.routed.RoutedStore.getVersions(RoutedStore.java:72)
at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:85)
at voldemort.store.serialized.SerializingStore.getVersions(SerializingStore.java:114)
at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:85)
at voldemort.client.DefaultStoreClient.getVersions(DefaultStoreClient.java:134)
at voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:180)
at com.netcore.mammcwt.store.StoreWrapper.put(StoreWrapper.java:69)
at com.netcore.mammcwt.store.StoreDAO.insertCommDetails(StoreDAO.java:177)
at com.netcore.mammcwt.store.StoreDAO.commHandler(StoreDAO.java:162)
at com.netcore.mammcwt.commhandler.CommHandlerTask.run(CommHandlerTask.java:106)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1238)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
at java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:251)
at voldemort.store.routed.RoutedStore.get(RoutedStore.java:492)
... 14 more
Can you please help me in this regard?
Thanks,
Mani
--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To post to this group, send email to project-...@googlegroups.com.
To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.
gxm
Nov 12, 2010, 5:07:54 PM11/12/10
to project-voldemort
Mani,
If you dig into the stack trace of the error above, you will see that
the failure came about because a get took more than 15 seconds.
I suspect each request is taking a long time because you don't have
enough sockets in the pool to service all of your client threads. The
above code will address that.
>I have set the max.threads=100
Based on the format, I assume you are referring to the
server.properties config file that lives on the server?
You need to explicitly set the thread settings used by the client.
To do this, you need code like this on your client:
ClientConfig clientConfig = new ClientConfig();
int maxThreads = 100;
clientConfig.setMaxThreads(maxThreads);
clientConfig.setMaxConnectionsPerNode(maxThreads);
clientConfig.setBootstrapUrls(url);
StoreClientFactory factory = new
SocketStoreClientFactory(clientConfig);
Once you get this up and running, you can use JMX with Voldemort to
verify settings and check internal timing stats.
Thread dumps are also a great way to verify how many threads your
client is using, and to see what state the threads are in over time.
Cheers,
Greg
> voldemort.store.serialized.SerializingStore.getVersions(SerializingStore.ja va:114)
> ...
>
> read more »
If you dig into the stack trace of the error above, you will see that
the failure came about because a get took more than 15 seconds.
I suspect each request is taking a long time because you don't have
enough sockets in the pool to service all of your client threads. The
above code will address that.
>I have set the max.threads=100
Based on the format, I assume you are referring to the
server.properties config file that lives on the server?
You need to explicitly set the thread settings used by the client.
To do this, you need code like this on your client:
ClientConfig clientConfig = new ClientConfig();
int maxThreads = 100;
clientConfig.setMaxThreads(maxThreads);
clientConfig.setMaxConnectionsPerNode(maxThreads);
clientConfig.setBootstrapUrls(url);
StoreClientFactory factory = new
SocketStoreClientFactory(clientConfig);
Once you get this up and running, you can use JMX with Voldemort to
verify settings and check internal timing stats.
Thread dumps are also a great way to verify how many threads your
client is using, and to see what state the threads are in over time.
Cheers,
Greg
> at
> voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:85)
> at
> voldemort.client.DefaultStoreClient.getVersions(DefaultStoreClient.java:134 )
> at
> voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:180)
> at com.netcore.mammcwt.store.StoreWrapper.put(StoreWrapper.java:69)
> at
> com.netcore.mammcwt.store.StoreDAO.insertCommDetails(StoreDAO.java:177)
> at com.netcore.mammcwt.store.StoreDAO.commHandler(StoreDAO.java:162)
> at
> com.netcore.mammcwt.commhandler.CommHandlerTask.run(CommHandlerTask.java:10 6)
> voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:85)
> at
> voldemort.client.DefaultStoreClient.getVersions(DefaultStoreClient.java:134 )
> at
> voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:180)
> at com.netcore.mammcwt.store.StoreWrapper.put(StoreWrapper.java:69)
> at
> com.netcore.mammcwt.store.StoreDAO.insertCommDetails(StoreDAO.java:177)
> at com.netcore.mammcwt.store.StoreDAO.commHandler(StoreDAO.java:162)
> at
> com.netcore.mammcwt.commhandler.CommHandlerTask.run(CommHandlerTask.java:10 6)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j ava:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j ava:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos (AbstractQueuedSynchronizer.java:1238)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.InterruptedException
> at
> at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
> at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at
> java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorServ ice.java:251)
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
> at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at
>
> read more »
Kirk True
Nov 12, 2010, 10:48:51 PM11/12/10
to project-...@googlegroups.com, gxm
Hi all,
What version of Voldemort are you using? The client side has change a good deal lately and should obviate the need for tons of client-side threads.
300 client-side threads is a bit over the top as is 5000 server threads.
Some questions:
For small (say, 1k values) 300 requests per second is well within reason. Honestly I think the problem lies with overestimating the number of threads (on both the client and server) that are needed.
Kirk
What version of Voldemort are you using? The client side has change a good deal lately and should obviate the need for tons of client-side threads.
300 client-side threads is a bit over the top as is 5000 server threads.
Some questions:
- How much data is on the Voldemort node (in GBs) in comparison to your heap size and BDB cache?
- What's the average size of the data in the request? Is it on the order of a few KBs or are we talking MBs in each request?
- Are the requests you're making to data in the cache or are you testing cold data accesses?
For small (say, 1k values) 300 requests per second is well within reason. Honestly I think the problem lies with overestimating the number of threads (on both the client and server) that are needed.
Kirk
Mani
Nov 18, 2010, 2:11:01 AM11/18/10
to project-voldemort
Thanks Greg, Kirk for your support.
As Greg explained, I just added the code in the client side to create
a pool of connections while starting the server itself. For every
incoming request, it just makes of connections from the pool.
I have configured max thread as 100, clientConfig.setMaxThreads(100)
and max.threads=500 at the server (server.properties). It is working
fine. No issues.
Thanks a lot.
Have few questions..
1. When should we increase the BDB cache size? Is it by checking the
size of the data directory -
data/bdb ?
2. How one can know that Voldemort is retrieving the value from cache
or not?
Thanks,
Mani
On Nov 13, 8:48 am, Kirk True <k...@mustardgrain.com> wrote:
> Hi all,
>
> What version of Voldemort are you using? The client side has change a
> good deal lately and should obviate the need for tons of client-side
> threads.
>
> 300 client-side threads is a bit over the top as is 5000 server threads.
>
> Some questions:
>
> 1. How much data is on the Voldemort node (in GBs) in comparison to
> >>> setMaxThreads (default 5)& setMaxConnectionsPerNode (default 6),
> ...
>
> read more »
As Greg explained, I just added the code in the client side to create
a pool of connections while starting the server itself. For every
incoming request, it just makes of connections from the pool.
I have configured max thread as 100, clientConfig.setMaxThreads(100)
and max.threads=500 at the server (server.properties). It is working
fine. No issues.
Thanks a lot.
Have few questions..
1. When should we increase the BDB cache size? Is it by checking the
size of the data directory -
data/bdb ?
2. How one can know that Voldemort is retrieving the value from cache
or not?
Thanks,
Mani
On Nov 13, 8:48 am, Kirk True <k...@mustardgrain.com> wrote:
> Hi all,
>
> What version of Voldemort are you using? The client side has change a
> good deal lately and should obviate the need for tons of client-side
> threads.
>
> 300 client-side threads is a bit over the top as is 5000 server threads.
>
> Some questions:
>
> your heap size and BDB cache?
> 2. What's the average size of the data in the request? Is it on the
> order of a few KBs or are we talking MBs in each request?
> 3. Are the requests you're making to data in the cache or are you
>
> read more »
Enrico Donelli
Apr 11, 2011, 3:34:01 AM4/11/11
to project-...@googlegroups.com
Hi _Mike
when you write "cache your client connections", do you mean to cache the StoreClient?
Thanks in advance!
Enrico
when you write "cache your client connections", do you mean to cache the StoreClient?
Thanks in advance!
Enrico
Reply all
Reply to author
Forward
0 new messages