Network transfer to one node twice as others

[Anishek Agarwal]

Hello,

We have a cluster with 3 nodes each having default(256) tokens with cassandra 2.0.14 running. 

We are using javax driver 2.1.5 with tokenawarepolicy(dcaware). I am observing a curious case as described below please help me understand why this is happening.

We have a separate client machine running about 50 threads and continuously writing to random partitions. On the network using iftop i can see that everytime one of the nodes will have twice the traffic than the other two. This is also the case when i try to do the reads.

ex:

node1 : 4mb / s

node2: 2mb / s 

node3: 2mb / s 

model : partition_key, value

Is this normal ? Anything i can do so it traffic is evenly distributed. 

RF: 3 and CL: 1 

Please let me know if any other details are required. 

thanks

anishek

[Olivier Michallat]

Hi,

At first sight this is unexpected, but there are lots of factors that may influence the results:

Are you sure that your data is evenly distributed? Is this generated data or does it come from another source? Does each write/read have approximately the same amount of data?

How are you sending your queries? Prepared or simple statements? execute or executeAsync?

Are the machines hosting Cassandra identical? On the same subnetwork?

Which partitioner are you using?

Also when you say "everytime one of the nodes will have twice the traffic", is that always the same node, or if you start your test twice do you see the node changing?

--

Olivier Michallat

Driver & tools engineer, DataStax

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

[Anishek Agarwal]

On Thu, Apr 23, 2015 at 7:21 PM, Olivier Michallat <olivier....@datastax.com> wrote:

Hi,

At first sight this is unexpected, but there are lots of factors that may influence the results:

Are you sure that your data is evenly distributed? Is this generated data or does it come from another source? Does each write/read have approximately the same amount of data?

 

We generate the data via our scripts with all threads getting a non overlapping section of the total keyspace (1million). There is only one value column which also has fixed width text in it so read/writes read the same amount of data. 

 

How are you sending your queries? Prepared or simple statements? execute or executeAsync?

 

Using simple statements, execute is used. 

Are the machines hosting Cassandra identical? On the same subnetwork?

same subnetwork , all having cassandra 2.0.14 version. 

Which partitioner are you using?

 org.apache.cassandra.dht.Murmur3Partitioner 

Also when you say "everytime one of the nodes will have twice the traffic", is that always the same node, or if you start your test twice do you see the node changing?

yeh it moves to different nodes at different tests.

[Olivier Michallat]

I can't think of any explanation from the driver's side. If you use simple calls to session.execute without calling setRoutingKey and setKeyspace on the statement object, token awareness does not kick in so the policy delegates to the dcaware one, which is round-robin.

Any chance that you could share your test scripts?

--

Olivier Michallat

Driver & tools engineer, DataStax

[Anishek Agarwal]

Hey Olivier,

the tests are at https://github.com/anishek/cassandra-example.git class used is CassandraBitTables. 

to run use  "mvn -Dcontact.points="" -Dkey.space= -Dnum.threads= -Dtest=CassandraBitTables#insertData test"

where contact.points == comma separated list of ip:port

key.space= number of partition keys 

num.threads= number of client  threads to use to write/read data.

i thought just using the keyspace name when creating the session via cluster object would be enough.  

Also i have RF: 3 and only 3 node cluster i thought reads should not be affected with the above(also Querybuilder does not seem to provided setting the metadata when selecting) as for writes they might get affected but if the default is round-robin it still does not explain why the network usage from my client to one of nodes is twice than the other two.

I will modify it and try to run them with the routing key set for writes to see if there is any difference. 

Thanks

anishek

[Anishek Agarwal]

I thought i update this thread with the solution, so it might be helpful for others.

It was a problem with the way i had configured the nodes. The servers i had were having 2 ethernet cards one with public and other with private IP. Since i am new to cassandra i had configured the rpc_address to 0.0.0.0. Now when i gave two server public IP's as contact points for my test somehow the client logs sated that one of them was connected over the public ip and the other nodes were over Private IP.  The iftop logs though showed all the traffic moving over private IP. I dont know why this is happening. 

However i changed the rpc_address for now to be public IP for testing and its working fine with the traffic now equally loaded on across the three nodes. 

Thanks

Anishek

[Olivier Michallat]

Great, thanks for posting the solution!

--

Olivier Michallat

Driver & tools engineer, DataStax