Cassandra multi dc replication nodetool rebuild issue + schema mismatch

[Utkarsh Sharma]

Team,
I have a Cassandra cluster of 6 nodes and I've added a new datacenter to the existing setup of 6 nodes. I’ve followed all the steps but getting the below error when I run nodetool rebuild on the new dc’s nodes.

nodetool rebuild -- datacenter1
nodetool: Unable to find sufficient sources for streaming range (389537640310009811,390720100939923035] in keyspace system_distributed
See 'nodetool help' or 'nodetool help <command>'.

Nodetool status

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address    Load   Tokens   Owns (effective) Host ID               Rack
UN 172.21.201.205 1.75 GiB 256     16.5%      ff0accd4-c33a-4984-967f-3ec763fe5414 rack1
UN 172.21.201.45 1.55 GiB 256     17.0%      d3ac5afa-d561-43ee-89e2-db1d20c59b38 rack1
UN 172.21.201.44 2.37 GiB 256     17.0%      73d8e6c6-0aa3-4a91-80fc-8c7068c78a64 rack1
UN 172.21.201.207 1020.15 MiB 256     16.0%      5751ea7d-b339-43b3-bcfe-89fcbc60dea0 rack1
UN 172.21.201.46 1.64 GiB 256     17.0%      1c1afbfc-6a4b-40f0-a4c3-1eaa543eb2d5 rack1
UN 172.21.201.206 1.13 GiB 256     17.2%      b11bfef9-e708-45cc-9ab3-e52983834096 rack1
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address    Load   Tokens   Owns (effective) Host ID               Rack
UN 10.41.6.155  983.91 KiB 256     17.3%      bf7244bb-70dc-4d91-8131-cbe4886f09e7 rack1
UN 10.41.6.157  946.36 KiB 256     15.5%      5499e7cc-db23-4163-8f0c-8f437f61bd6f rack1
UN 10.41.6.156  1.14 MiB 256     15.3%      f27e94a6-7e1c-4177-9f88-36d821a7808d rack1
UN 10.41.6.159  659.3 KiB 256     17.3%      453e97df-5b83-4798-9e5e-a13bbb33acee rack1
UN 10.41.6.158  909.51 KiB 256     18.2%      a4bc046a-e2ef-4fd4-9ab7-18be642a4d5a rack1
UN 10.41.6.160  1.08 MiB 256     15.5%      267cf9d0-cd55-4186-a737-998443125b19 rack1

node tool describe cluster status
#nodetool describecluster

Cluster Information:
   Name: Recommendation
   Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch
   DynamicEndPointSnitch: enabled
   Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
   Schema versions:
     ea63e099-37c5-3d7b-9ace-32f4c833653d: [10.41.6.155, 10.41.6.157, 10.41.6.156, 10.41.6.159, 10.41.6.158, 10.41.6.160]     fd507b64-3070-3ffd-8217-f45f2f188dfc: [172.21.201.205, 172.21.201.45, 172.21.201.44, 172.21.201.207, 172.21.201.206, 172.21.201.46]

Can someone quickly help me fix this issue?

[Utkarsh Sharma]

I've managed to fix this issue by changing the replication factor of the system_distributed keyspace to NetworkTopologyStrategy  to include both DCS on the new dc1.
But as I've already made this change on datacenter1, I was expecting that it will be automatically be replicated on the new dc1 too. 

The issue is not fixed yet, coz the nodetool rebuild command is now running but it replicated only 1mb of data from old dc to new dc. 
Also, there is still a different schema version for old and new dc nodes

#nodetool describecluster

Cluster Information:
   Name: Recommendation
   Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch
   DynamicEndPointSnitch: enabled
   Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
   Schema versions:
     ea63e099-37c5-3d7b-9ace-32f4c833653d: [10.41.6.155, 10.41.6.157, 10.41.6.156, 10.41.6.159, 10.41.6.158, 10.41.6.160]     fd507b64-3070-3ffd-8217-f45f2f188dfc: [172.21.201.205, 172.21.201.45, 172.21.201.44, 172.21.201.207, 172.21.201.206, 172.21.201.46]

I am unable to replicate the data from the old dc to the new dc yet. 

[Utkarsh Sharma]

Found the issue and managed to fix it. Both the DC were running with different Cassandra versions. So I had to downgrade the Cassandra version on the new DC to avoid any disruption. 

Thanks & Regards

Utkarsh Sharma DevOps Lead + 91 9168615342

--
Vous recevez ce message, car vous êtes abonné au groupe Google Groupes "cassandra-unit-users".
Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le concernant, envoyez un e-mail à l'adresse cassandra-unit-u...@googlegroups.com.
Cette discussion peut être lue sur le Web à l'adresse https://groups.google.com/d/msgid/cassandra-unit-users/849d9eec-9209-4ee0-a985-14a4d0cec9fbn%40googlegroups.com.