Voldemort client intermittently not able to fetch data from server ?

[ams...@gmail.com]

Hi,

I have one Voldemort server running with default configuration , except my stores.xml looks like

 <stores>

  <store>

    <name>stringStore</name>

    <persistence>bdb</persistence>

    <routing>client</routing>

    <hinted-handoff-strategy>proximity-handoff</hinted-handoff-strategy>

    <replication-factor>1</replication-factor>

    <required-reads>1</required-reads>

    <preferred-reads>1</preferred-reads> 

    <required-writes>1</required-writes>

    <preferred-writes>1</preferred-writes>

    <key-serializer>

      <type>string</type>

    </key-serializer>

    <value-serializer>

      <type>string</type>

    </value-serializer>

    <retention-days>1</retention-days>

  </store>

  <store>

    <name>binaryStore</name>

    <persistence>bdb</persistence>

    <routing>client</routing>

    <hinted-handoff-strategy>proximity-handoff</hinted-handoff-strategy>

    <replication-factor>1</replication-factor>

    <required-reads>1</required-reads>

    <preferred-reads>1</preferred-reads> 

    <required-writes>1</required-writes>

    <preferred-writes>1</preferred-writes>

    <key-serializer>

      <type>string</type>

    </key-serializer>

    <value-serializer>

      <type>identity</type>

    </value-serializer>

    <retention-days>1</retention-days>

  </store>

<store>

    <name>listStore</name>

    <persistence>bdb</persistence>

    <routing>client</routing>

    <hinted-handoff-strategy>proximity-handoff</hinted-handoff-strategy>

    <replication-factor>1</replication-factor>

    <required-reads>1</required-reads>

    <preferred-reads>1</preferred-reads> 

    <required-writes>1</required-writes>

    <preferred-writes>1</preferred-writes>

    <key-serializer>

      <type>string</type>

    </key-serializer>

    <value-serializer>

     <type>json</type>

<schema-info>["string"]</schema-info> 

   </value-serializer>

    <retention-days>1</retention-days>

  </store>

</stores>

when i continuously send get request , it intermittently gives me this error

voldemort.store.InsufficientOperationalNodesException: 1 get versionss required, but only 0 succeeded Original replication set :[0] Known failed nodes before operation :[] Estimated live nodes in preference list :[0] New failed nodes during operation :[0]

at voldemort.store.routed.action.PerformSerialRequests.execute(PerformSerialRequests.java:142)

at voldemort.store.routed.Pipeline.execute(Pipeline.java:214)

at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:530)

at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:1)

at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86)

at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86)

at voldemort.store.serialized.SerializingStore.getVersions(SerializingStore.java:144)

at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:86)

at voldemort.client.DefaultStoreClient.getVersions(DefaultStoreClient.java:163)

at voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:213)

at VolClient.main(VolClient.java:44)

Caused by: voldemort.store.UnreachableStoreException: Could not connect to http://10.40.94.15:8081/stores for stringStore

at voldemort.store.http.HttpStore.executeRequest(HttpStore.java:178)

at voldemort.store.http.HttpStore.getVersions(HttpStore.java:203)

at voldemort.store.http.HttpStore.getVersions(HttpStore.java:1)

at voldemort.store.nonblockingstore.ThreadPoolBasedNonblockingStoreImpl$3.request(ThreadPoolBasedNonblockingStoreImpl.java:84)

at voldemort.store.nonblockingstore.ThreadPoolBasedNonblockingStoreImpl$3.request(ThreadPoolBasedNonblockingStoreImpl.java:1)

at voldemort.store.nonblockingstore.ThreadPoolBasedNonblockingStoreImpl$6.run(ThreadPoolBasedNonblockingStoreImpl.java:129)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to 10.40.94.15:8081 timed out

at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:125)

at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)

at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)

at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)

at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:573)

at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)

at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)

at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)

at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)

at voldemort.store.http.HttpStore.executeRequest(HttpStore.java:166)

... 11 more

Why i get this kind of error when Voldemort server is running ?

2.  When i kill my client application i get debug logs on voldemort server side as 

java.io.IOException: Connection reset by peer

at sun.nio.ch.FileDispatcher.read0(Native Method)

at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)

at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)

at sun.nio.ch.IOUtil.read(IOUtil.java:175)

at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)

at org.mortbay.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:131)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)

at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)

[14:39:21,843 org.mortbay.log] DEBUG EXCEPTION   [394718586@VoldemortHttp-6]

 java.io.IOException: Connection reset by peer

at sun.nio.ch.FileDispatcher.read0(Native Method)

at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)

at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)

at sun.nio.ch.IOUtil.read(IOUtil.java:175)

at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)

at org.mortbay.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:131)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)

at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:5

But when i disconnect network i get No logs on voldemort Server side ,

Why the behavior in both the cases is different ?

Regards,

ams

[Chinmay Soman]

Hi,

For 1) Try switching to a SocketStoreClientFactory, as described here => http://www.project-voldemort.com/voldemort/quickstart.html

2) When you terminate a Java process, the OS will take care of closing the sockets. That's why you see the connection reset messages on the server.

However, when you disable network (the ethernet port for eg), there is no way for the client to send a RST to the server (they will eventually be terminated by the OS). In fact, even on the client side, if there are no active requests, the sockets probably wont be terminated right away.

[amstm07]

Thanks Chinmay 

I have tried 1st one and it helped and reduced the frequency of occurrence of that error almost NIL.

I understood your 2nd reason.

Thanks,

ams

--
You received this message because you are subscribed to a topic in the Google Groups "project-voldemort" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/project-voldemort/xCnya1MR2hc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to project-voldem...@googlegroups.com.
Visit this group at http://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/groups/opt_out.