node connections

[Stephen Jones]

I've got the most out of the box set up for my Cassandra Cluster. Now that we're starting to scale with the system, it seems that one of the hosts in the cluster is getting more connections than any of the other hosts. I believe this to be the rudimentary cause to the slow connection issue I'm seeing. When trying to connect to the pool, it takes ~2:30s to connect, sometimes. Thoughts on how to better spread the connections around? 

Thanks in advance for the help! 

-Stephen

[Tyler Hobbs]

How many nodes are you passing to the ConnectionPool constructor, and what pool_size are you using?  Making sure pool_size (which defaults to 5) is a multiple of the number of nodes you pass in will help to ensure a more even number of connections to each node.

--
You received this message because you are subscribed to the Google Groups "pycassa-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pycassa-discu...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Tyler Hobbs
DataStax

[Stephen Jones]

I am passing a list of 6 nodes - with a pool size of 5. After making the change to the correct pool_size (6) - connection speeds have improved. Now that OpsCenter has been installed, I've noticed there are spikes in the write request, while my networked hosts are writing out data at a consistent rate. Are there any suggestions on how to better optimize the system to alleviate spikes in write requests demonstrated below. 

I've also noticed that there is a read balance issue that i'm seeing. Any thoughts on how to better connect to the pool and get a better balance for reads and writes. 

​​

Below are my standard ways of connecting to the pool/cluster. Any thoughts oh how to better balance this system. 

HOSTS=['cassaws002', 'cassaws003', 'cassaws004', 'cassaws005', 'cassaws006', 'cassaws007']

self.pool = ConnectionPool(POOL_NAME,server_list=HOSTS,timeout=30,pool_size=6)

self.cf = ColumnFamily(self.pool,cf_name)

Thanks for your help!

Stephen 

- S

--
You received this message because you are subscribed to a topic in the Google Groups "pycassa-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pycassa-discuss/MwShVZ_EtbI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pycassa-discu...@googlegroups.com.

[Tyler Hobbs]

The spikes in write requests are probably due to OpsCenter itself.  It stores a new chunk of metric data in the cluster every minute.

As for your read imbalance, I'm not as sure.  OpsCenter is also reading its metric data from your cluster (when you use the UI).  I believe it should automatically balance requests across multiple nodes, but you may want to try adding multiple nodes to your OpsCenter config file (the one that's specifically for that cluster).  I think you can do something similar through the UI, but I looked in a while, so I'm not sure.

[Tyler Hobbs]

Ah, and one more note on your config: a 30s timeout is *really* high, and will probably cause you nothing but pain in the event of failures.  I suggest keeping it at or below 1s so that operations can be quickly retried against another node if one node fails.

--

Tyler Hobbs
DataStax

[Stephen Jones]

Reason for the high timeout time is due to the really high loads on the hosts. It takes a while for it to connect and respond so I've had to turn it up. In regards to the high loads, is there anyway to reduce the loads. Each of my hosts have a CPU load of 20+. I've reduce the data set down as slim as possible. I'm assuming the computation for compacting is what causing the high loads. Is there anyway to compact less at the sacrifice of data size? Is the only solution to add more nodes to the cluster? 

Thank you for you guidance Tyler. 

- S

[Tyler Hobbs]

On Fri, Dec 5, 2014 at 12:35 PM, Stephen Jones <sbjo...@gmail.com> wrote:

Reason for the high timeout time is due to the really high loads on the hosts. It takes a while for it to connect and respond so I've had to turn it up. In regards to the high loads, is there anyway to reduce the loads. Each of my hosts have a CPU load of 20+. I've reduce the data set down as slim as possible. I'm assuming the computation for compacting is what causing the high loads. Is there anyway to compact less at the sacrifice of data size? Is the only solution to add more nodes to the cluster? 

If you're using LeveledCompactionStrategy, switching to SizeTiered would lower the compaction impact.  You can also throttle compaction throughput if your compaction load is spiky, but if it's constantly high, your nodes will just fall behind on compaction, and you'll end up spending more I/O on reads.

Adding more nodes is probably your best option if your data size is large and you're fairly sure compaction is what's causing the load.

--

Tyler Hobbs
DataStax