Multiple data center performance

Showing 1-9 of 9 messages
Multiple data center performance Daning Wang 6/7/13 9:49 PM
We have deployed multi-center but got performance issue. When the nodes on other center are up, the read response time from clients is 4 or 5 times higher. when we take those nodes down, the response time becomes normal(compare to the time before we changed to multi-center).

We have high volume on the cluster, the consistency level is one for read. so my understanding is most of traffic between data center should be read repair. but seems that could not create much delay.

What could cause the problem? how to debug this?

Here is the keyspace,

[default@dsat] describe dsat;
Keyspace: dsat:
  Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
  Durable Writes: true
    Options: [dc2:1, dc1:3]
  Column Families:
    ColumnFamily: categorization_cache
 

Ring

Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address           Load       Tokens  Owns (effective)  Host ID                               Rack
UN  xx.xx.xx..111       59.2 GB    256     37.5%             4d6ed8d6-870d-4963-8844-08268607757e  rac1
DN  xx.xx.xx..121       99.63 GB   256     37.5%             9d0d56ce-baf6-4440-a233-ad6f1d564602  rac1
UN  xx.xx.xx..120       66.32 GB   256     37.5%             0fd912fb-3187-462b-8c8a-7d223751b649  rac1
UN  xx.xx.xx..118       63.61 GB   256     37.5%             3c6e6862-ab14-4a8c-9593-49631645349d  rac1
UN  xx.xx.xx..117       68.16 GB   256     37.5%             ee6cdf23-d5e4-4998-a2db-f6c0ce41035a  rac1
UN  xx.xx.xx..116       32.41 GB   256     37.5%             f783eeef-1c51-4f91-ab7c-a60669816770  rac1
UN  xx.xx.xx..115       64.24 GB   256     37.5%             e75105fb-b330-4f40-aa4f-8e6e11838e37  rac1
UN  xx.xx.xx..112       61.32 GB   256     37.5%             2547ee54-88dd-4994-a1ad-d9ba367ed11f  rac1
Datacenter: dc2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address           Load       Tokens  Owns (effective)  Host ID                               Rack
DN  xx.xx.xx.199    58.39 GB   256     50.0%             6954754a-e9df-4b3c-aca7-146b938515d8  rac1
DN  xx.xx.xx..61      33.79 GB   256     50.0%             91b8d510-966a-4f2d-a666-d7edbe986a1c  rac1


Thank you in advance,

Daning

Re: Multiple data center performance srmore 6/8/13 6:53 AM
I am seeing the similar behavior, in my case I have 2 nodes in each datacenter and one node always has high latency (equal to the latency between the two datacenters). When one of the datacenters is shutdown the latency drops.

I am curious to know whether anyone else has these issues and if yes how did to get around it.

Thanks !
Re: Multiple data center performance Daning Wang 6/11/13 2:34 PM
It is counter caused the problem. counter will replicate to all replicas during write regardless the consistency level. 

In our case. we don't need to sync the counter across the center. so moving counter to new keyspace and all the replica in one center solved problem.

There is option replicate_on_write on table. If you turn that off for counter might have better performance. but you are on high risk to lose data and create inconsistency. I did not try this option.

Daning
Re: Multiple data center performance Alain RODRIGUEZ 6/12/13 4:56 AM
"counter will replicate to all replicas during write regardless the consistency level"

I that the normal behavior or a bug ?


2013/6/11 Daning Wang <dan...@netseer.com>

Re: Multiple data center performance Sylvain Lebresne 6/12/13 6:54 AM
It is the normal behavior, but that's true of any update, not only of counters.

The consistency level does *not* influence which replica are written to. Cassandra always write to all replicas. The consistency level only decides how replica acknowledgement are waited for.

--
Sylvain
Re: Multiple data center performance Alain RODRIGUEZ 6/12/13 7:31 AM
"The consistency level does *not* influence which replica are written to. Cassandra always write to all replicas."

I am aware of this.

My understanding of what Daning said was that client waited for all nodes to acknowledge the write before considering it successful (inducing extra latency) for counters column families while using multiDC, "regardless the consistency level"  I might have misunderstood because of Daning's words about CL.

Is there something special of this kind regarding counters over multiDC ?

Thank you anyway Sylvain


2013/6/12 Sylvain Lebresne <syl...@datastax.com>

Re: Multiple data center performance Sylvain Lebresne 6/12/13 7:48 AM

Is there something special of this kind regarding counters over multiDC ?

No. Counters behave exactly as other writes as far the consistency level is concerned.
Technically, the counter write path is different from the normal write path in the sense that a counter write
will be written to one replica first and then written to the rest of the replicas in a second time (with a local
read on the first replica in between, which is why counter writes are slower than normal ones). But,
outside of the obvious performance impact, this has no impact on the behavior observed from a
client point of view. The consistency level has the exact same meaning in particular (though one
small difference is that counters don't support CL.ANY).

--
Sylvain

Re: Multiple data center performance Alain RODRIGUEZ 6/12/13 7:53 AM
Crystal clear, we use a lot of counters and I am always happy to learn this kind of things.

Thanks a lot.

Alain


2013/6/12 Sylvain Lebresne <syl...@datastax.com>

Re: Multiple data center performance Daning Wang 6/12/13 2:20 PM
Sorry for the confusion. 

Sylvain -  Do you think what could cause the client higher latency in multiDC(CL=one for read and write) ? clients only connect to nodes in the same DC. we did see the performance greatly improved after changing the replication factor for counters, but still slower than other DC is shutdown.


Thanks,

Daning