Adding node to a cluster

[Robert Koletka]

Hi,

I have a cluster of 3 nodes servicing requests from an application.

The application performs random reads and writes and prints out the list of known hosts in the cluster with the cluster.getKnownPoolHosts(true) method every 30 seconds.

Now if I add a new node to the cluster, the application carries on printing out the initial three nodes without ever printing out the fourth and the new node does not get selected to send requests to.

Surely as a new node gets added the cluster Hector should start randomly selecting it and sending it requests to be handled.

I have set setAutoDiscoverHosts to true and the delay to 30 seconds.

Is there something that I am missing here?

Thanks

Rob

[Patricio Echagüe]

you need to set auto discovery in cassandraHostConfigurator

Sent from my Android

[Robert Koletka]

Hi Patricio,

I do set auto discovery to true and the auto discovery delay to 30 seconds.

So my understanding is that at start up and every 30 seconds thereafter hector should have an updated list of nodes to send requests to, this however does not happen.

Even when i provide the seed node IP of a 3 node cluster, the other two nodes do not get discovered.

Rob

[Robert Koletka]

Im getting the following error in the log files

ERROR Hector.me.prettyprint.cassandra.connection.NodeAutoDiscoverService-1 me.prettyprint.cassandra.connection.NodeAutoDiscoverService - Discovery Service failed attempt to connect CassandraHost

java.lang.NullPointerException

        at me.prettyprint.cassandra.connection.NodeAutoDiscoverService.discoverNodes(NodeAutoDiscoverService.java:102)

        at me.prettyprint.cassandra.connection.NodeAutoDiscoverService.doAddNodes(NodeAutoDiscoverService.java:68)

        at me.prettyprint.cassandra.connection.NodeAutoDiscoverService$QueryRing.run(NodeAutoDiscoverService.java:59)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:662)

[Patricio Echagüe]

Interesting. I'll look at it shortly.

What hector version are you using?

Sent from my Android

[Robert Koletka]

Im using hector 1.0.2

Thanks

Rob

[Patricio Echagüe]

Can you upgrade to -3 ? We improved the auto discovery service among other things

Sent from my Android

[Patricio Echagüe]

also, are you using Spring by any chance ?

2012/3/6 Patricio Echagüe <patr...@gmail.com>

[robert....@gmail.com]

No I'm not using spring, I will try version 1.0.3 at work tomorrow and see if that sorts out the issue.


Rob

Sent via my BlackBerry from Vodacom - let your email find you!

---

From: Patricio Echagüe <patr...@gmail.com>

Sender: hector...@googlegroups.com

Date: Tue, 6 Mar 2012 08:34:39 -0800

To: <hector...@googlegroups.com>

ReplyTo: hector...@googlegroups.com

Subject: Re: Adding node to a cluster

[Robert Koletka]

I have tried using 1.0.3 and I am still getting the same exception.

[Patricio Echagüe]

can you paste the code where you initialize chc and the cluster ?

[Robert Koletka]

Unfortunately i dont have my code on me.
Could you possibly post an example of what you think it should look like.
I have followed the start up guide on github.

i have put log entries into NodeAutoDiscoverService class, every time the thread executes it does find the other nodes when i output
it seems like the null pointer exception is thrown when the

tokenRange.getEndpoint_details()is called on line 102.
if i place a log statement for tokenRange.toString() it does however output the IP's of the other nodes.

[Patricio Echagüe]

the guide in hector-client.org is correct. I'm going to ask you for some help debugging the issue. Is it possible for you to  debug it by attaching eclipse to it ?

Just curious about what object reference is null in that class.

Also, if there is any chance you can create a test case where you can reproduce it, that would be ideal.

[robert....@gmail.com]

Okay, I will try get all that info for you tomorrow. I can debug using netbeans.

Sent via my BlackBerry from Vodacom - let your email find you!

---

From: Patricio Echagüe <patr...@gmail.com>

Sender: hector...@googlegroups.com

Date: Wed, 7 Mar 2012 09:01:04 -0800

To: <hector...@googlegroups.com>

ReplyTo: hector...@googlegroups.com

Subject: Re: Adding node to a cluster

[Robert Koletka]

Hi Patricio

The tokenRange.getEndpoint_details() command in the NodeAutoDiscoveryService is returning null which in turn is causing the null pointer exception.

Any idea of why this could be happening?

Rob

[Robert Koletka]

I have found a non ideal solution/fix.

Ive modified the NodeAutoDiscoveryService by doing the following.

for(KeyspaceDefinition keyspaceDefinition: cluster.describeKeyspaces()) {

        if (!keyspaceDefinition.getName().equals(Keyspace.KEYSPACE_SYSTEM)) {

          List<TokenRange> tokenRanges = cluster.describeRing(keyspaceDefinition.getName());

                for(TokenRange tr : tokenRanges)

                {

                    List<String> endPoints = tr.getEndpoints();

                    for(String s : endPoints)

                    {

                        CassandraHost foundHost = new CassandraHost(s, cassandraHostConfigurator.getPort());

                        foundHosts.add(foundHost);

                    }

                    

                }

//          for (TokenRange tokenRange : tokenRanges) {

//              for (EndpointDetails endPointDetail : tokenRange.getEndpoint_details()) {

//                // Check if we are allowed to include this Data Center.

//                if (dataCenterValidator.validate(endPointDetail.getDatacenter())) {

//                  // Maybe add this host if it's a new host.

//                  CassandraHost foundHost = new CassandraHost(endPointDetail.getHost(), cassandraHostConfigurator.getPort());

//                  if ( !existingHosts.contains(foundHost) ) {

//                    log.info("Found a node we don't know about {} for TokenRange {}", foundHost, tokenRange);

//                    foundHosts.add(foundHost);

//                  }

//                }

//              }

//          }

          break;

        }

      }

This is a temporary fix but it works. This however does not include the data center check.

[Patricio Echagüe]

Robert, could you attach the code where you initialize your cluster?

Sent from my Android

[Patricio Echagüe]

Also, are you using simple snitch?

Sent from my Android

[robert....@gmail.com]

Yes I am using simple snitch. Sorry I forgot to add the init code. Will do it tomorrow.

Sent via my BlackBerry from Vodacom - let your email find you!

---

From: Patricio Echagüe <patr...@gmail.com>

Sender: hector...@googlegroups.com

Date: Thu, 8 Mar 2012 08:33:49 -0800

To: <hector...@googlegroups.com>

ReplyTo: hector...@googlegroups.com

Subject: Re: Adding node to a cluster

[Patricio Echagüe]

What cassandra version are you guys using?

Sent from my Android

[robert....@gmail.com]

I think its version 0.8.5

Sent via my BlackBerry from Vodacom - let your email find you!

---

From: Patricio Echagüe <patr...@gmail.com>

Sender: hector...@googlegroups.com

Date: Thu, 8 Mar 2012 08:42:42 -0800

[Patricio Echagüe]

There is no getEndpointDetails() in TokenRange for that version of cassandra apparently. I just checked out the code for that version.

[robert....@gmail.com]

So I should be using 1.0 or higher then?

Sent via my BlackBerry from Vodacom - let your email find you!

---

From: Patricio Echagüe <patr...@gmail.com>

Sender: hector...@googlegroups.com

Date: Thu, 8 Mar 2012 08:53:22 -0800

[Patricio Echagüe]

if you can , yes. We use 1.0.6 currently at my job.

[Robert Koletka]

Hi Patricio,

Everything seems to be working correctly now that I have upgraded to version 1.0.8.

Thanks for the help.

Rob

[Patricio Echagüe]

Np.Glad it worked

Sent from my Android