remote connection failure via http/tcp

87 views
Skip to first unread message

zhen...@tune.com

unread,
Feb 6, 2015, 4:48:41 PM2/6/15
to project-...@googlegroups.com
Hi,
I'm testing on a scala client for voldemort. The client is on a isolated spark cluster and part of the code is as follows:

    println("==============String store example=================");

    // In production environment, the StoreClient instantiation should be done using factory pattern
    // through a Framework such as Spring
    val bootstrapUrl = "http://DNSForNode0FromVoldemortCluster:8081" #for http connection
#  val bootstrapUrl = "tcp://DNSForNode0FromVoldemortCluster:6666" #for tcp connection

    val factory = new HttpStoreClientFactory(new ClientConfig().setBootstrapUrls(bootstrapUrl))

    val resolver = new ArbitraryInconsistencyResolver[Versioned[String]]

    val client = new DefaultStoreClient[String, String]("test", resolver, factory, 3)

    // put initial value
    println("Putting an initial value")
    client.put("some_key", "initial value")

    // get the value
    println("Getting Initial value")
    var versioned = client.get("some_key")


And the voldemort cluster is a three node cluster with http-port 8081,8081,8082 respectively, and socket-port 6666,6668,6670 respectively.

When I want to connect via tcp, the error message is:

==============String store example=================
Exception in thread "main" voldemort.client.BootstrapFailureException: No available bootstrap servers found!
at voldemort.client.AbstractStoreClientFactory.bootstrapMetadataWithRetries(AbstractStoreClientFactory.java:546)
at voldemort.client.AbstractStoreClientFactory.getRawStore(AbstractStoreClientFactory.java:284)
at voldemort.client.AbstractStoreClientFactory.getRawStore(AbstractStoreClientFactory.java:262)
at voldemort.client.DefaultStoreClient.bootStrap(DefaultStoreClient.java:99)
at voldemort.client.DefaultStoreClient.<init>(DefaultStoreClient.java:83)
at voldemortTest$.main(voldemortInOut.scala:34)
at voldemortTest.main(voldemortInOut.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

And when I switch to http, the error says:

==============String store example=================
Putting an initial value
Exception in thread "main" voldemort.store.InsufficientOperationalNodesException: 2 get versionss required, but only 1 succeeded Original replication set :[0, 1] Known failed nodes before operation :[] Estimated live nodes in preference list :[0, 1] New failed nodes during operation :[1]
at voldemort.store.routed.action.PerformSerialRequests.execute(PerformSerialRequests.java:132)
at voldemort.store.routed.Pipeline.execute(Pipeline.java:212)
at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:601)
at voldemort.store.routed.PipelineRoutedStore.getVersions(PipelineRoutedStore.java:81)
at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:90)
at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:90)
at voldemort.store.stats.StatTrackingStore.getVersions(StatTrackingStore.java:67)
at voldemort.store.stats.StatTrackingStore.getVersions(StatTrackingStore.java:40)
at voldemort.store.serialized.SerializingStore.getVersions(SerializingStore.java:147)
at voldemort.store.DelegatingStore.getVersions(DelegatingStore.java:90)
at voldemort.client.DefaultStoreClient.getVersions(DefaultStoreClient.java:172)
at voldemort.client.DefaultStoreClient.getVersionWithResolution(DefaultStoreClient.java:342)
at voldemort.client.DefaultStoreClient.getVersionForPut(DefaultStoreClient.java:357)
at voldemort.client.DefaultStoreClient.put(DefaultStoreClient.java:222)
at voldemortTest$.main(voldemortInOut.scala:38)
at voldemortTest.main(voldemortInOut.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: voldemort.store.UnreachableStoreException: Could not connect to http://ec2-54-146-216-7.compute-1.amazonaws.com:8082/stores for test
at voldemort.store.http.HttpStore.executeRequest(HttpStore.java:179)
at voldemort.store.http.HttpStore.getVersions(HttpStore.java:195)
at voldemort.store.http.HttpStore.getVersions(HttpStore.java:50)
at voldemort.store.nonblockingstore.ThreadPoolBasedNonblockingStoreImpl$3.request(ThreadPoolBasedNonblockingStoreImpl.java:84)
at voldemort.store.nonblockingstore.ThreadPoolBasedNonblockingStoreImpl$3.request(ThreadPoolBasedNonblockingStoreImpl.java:81)
at voldemort.store.nonblockingstore.ThreadPoolBasedNonblockingStoreImpl$6.run(ThreadPoolBasedNonblockingStoreImpl.java:129)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to ec2-54-146-216-7.compute-1.amazonaws.com:8082 timed out
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:129)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:151)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at voldemort.store.http.HttpStore.executeRequest(HttpStore.java:167)
... 10 more

Are there any ways to work around this issue? Please suggest, thanks for your time.

Best,
Zhenzhen

Arunachalam

unread,
Feb 6, 2015, 5:35:50 PM2/6/15
to project-...@googlegroups.com
I haven't worked much with http and could have some rough edges.

How are you bringing up a 3 node cluster ? Can you post your cluster.xml ?

From the call stack it seems like it fetched the raw cluster.xml and when it tried to connect to the host specified it failed.

Thanks,
Arun.

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at http://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

Arunachalam

unread,
Feb 6, 2015, 5:36:45 PM2/6/15
to project-...@googlegroups.com
Also don't try writing your own program in the first place, play around with the shell to isolate the issues and when the shell succeeds proceed with your own program as it could save time ( in figuring out the configuration issue and the program logic issue)

Thanks,
Arun.

zhen...@tune.com

unread,
Feb 6, 2015, 5:56:27 PM2/6/15
to project-...@googlegroups.com
for tcp, I use SocketStoreClientFactory

And I think it may be amazon ec2 cluster security group issue. The client and the cluster belong to distinct security groups. When I move the scala client to one node of the clusters and try to connect to another node as an entry for the voldemort, it works.

The cluster.xml is as follows:
<cluster>
  <name>mycluster</name>
  <server>
    <id>0</id>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>
    <partitions>0, 1</partitions>
  </server>
  <server>
    <id>1</id>
    <http-port>8082</http-port>
    <socket-port>6668</socket-port>
    <admin-port>6669</admin-port>
    <partitions>2, 3</partitions>
  </server>
  <server>
      <id>2</id>
      <host>ec2DNS3.compute-1.amazonaws.com</host>
      <http-port>8083</http-port>
      <socket-port>6670</socket-port>
      <admin-port>6671</admin-port>
      <partitions>4, 5</partitions>
  </server>
</cluster>

Thanks!

Arunachalam

unread,
Feb 6, 2015, 5:59:46 PM2/6/15
to project-...@googlegroups.com
Cool.
 
If you think security issue, you can try to just telnet to the port even before launching the voldemort client.

Thanks,
Arun.

zhen...@tune.com

unread,
Feb 6, 2015, 6:24:01 PM2/6/15
to project-...@googlegroups.com
Thank you so much for the advice:)
Reply all
Reply to author
Forward
0 new messages