Force nodes from one DC to lead the repair?

[Max C.]

Hi Everyone,

We have 2 DCs (Europe & US), with 3 nodes each (RF 3). We’re using Reaper’s DATACENTER_AWARE option. Looking at the repair log, it appears that for each token range there is one node across the global cluster that is selected to receive the merkle tree for the range; and then all replicas are synced from there. I see further evidence of this by the fact that there is exactly 1 reaper segment (not 1 segment per DC) for each token range.

1) Do I have this correct?

2) Our DC2 has the same hardware as DC1, but is almost completely idle. Instead of having reaper choose a random DC for each segment, Would there be a significant performance advantage for the DC1 nodes to have reaper *always* choose nodes in DC2 to lead the repair? If so, how do I configure that?

Thanks for for the info, and to TLP for maintaining this crucial part of the C* ecosystem!

- Max

Sample log below — IPs, table name, etc are masked.

INFO [AntiEntropyStage:1] 2019-10-23 01:00:15,507 RepairSession.java:176 - [repair #X] Received merkle tree for TABLE1 from /DC1_NODE2
INFO [AntiEntropyStage:1] 2019-10-23 01:00:15,507 RepairJob.java:270 - Validating /DC1_NODE3
INFO [AntiEntropyStage:1] 2019-10-23 01:00:15,786 RepairSession.java:176 - [repair #X] Received merkle tree for TABLE1 from /DC1_NODE3
INFO [AntiEntropyStage:1] 2019-10-23 01:00:15,786 RepairJob.java:270 - Validating /DC1_NODE1
INFO [AntiEntropyStage:1] 2019-10-23 01:00:16,120 RepairSession.java:176 - [repair #X] Received merkle tree for TABLE1 from /DC1_NODE1
INFO [AntiEntropyStage:1] 2019-10-23 01:00:16,434 RepairSession.java:176 - [repair #X] Received merkle tree for TABLE1 from /DC2_NODE1
INFO [AntiEntropyStage:1] 2019-10-23 01:00:16,434 RepairJob.java:270 - Validating /DC2_NODE3
INFO [AntiEntropyStage:1] 2019-10-23 01:00:17,888 RepairSession.java:176 - [repair #X] Received merkle tree for TABLE1 from /DC2_NODE3
INFO [AntiEntropyStage:1] 2019-10-23 01:00:17,888 RepairJob.java:270 - Validating /DC2_NODE2
INFO [AntiEntropyStage:1] 2019-10-23 01:00:19,327 RepairSession.java:176 - [repair #X] Received merkle tree for TABLE1 from /DC2_NODE2
INFO [RepairJobTask:19] 2019-10-23 01:00:19,332 SyncTask.java:66 - [repair #X] Endpoints /DC2_NODE1 and /DC2_NODE2 are consistent for TABLE1
INFO [RepairJobTask:13] 2019-10-23 01:00:19,332 SyncTask.java:66 - [repair #X] Endpoints /DC2_NODE1 and /DC2_NODE3 are consistent for TABLE1
INFO [RepairJobTask:17] 2019-10-23 01:00:19,332 SyncTask.java:66 - [repair #X] Endpoints /DC2_NODE3 and /DC2_NODE2 are consistent for TABLE1
INFO [RepairJobTask:9] 2019-10-23 01:00:19,336 SyncTask.java:66 - [repair #X] Endpoints /DC2_NODE2 and /DC1_NODE3 are consistent for TABLE1
INFO [RepairJobTask:16] 2019-10-23 01:00:19,336 SyncTask.java:66 - [repair #X] Endpoints /DC2_NODE1 and /DC1_NODE3 are consistent for TABLE1
INFO [RepairJobTask:7] 2019-10-23 01:00:19,336 SyncTask.java:66 - [repair #X] Endpoints /DC2_NODE3 and /DC1_NODE3 are consistent for TABLE1
INFO [RepairJobTask:19] 2019-10-23 01:00:19,337 SyncTask.java:66 - [repair #X] Endpoints /DC1_NODE3 and /DC1_NODE1 are consistent for TABLE1
INFO [RepairJobTask:14] 2019-10-23 01:00:19,337 SyncTask.java:66 - [repair #X] Endpoints /DC2_NODE1 and /DC1_NODE1 are consistent for TABLE1
INFO [RepairJobTask:6] 2019-10-23 01:00:19,337 SyncTask.java:66 - [repair #X] Endpoints /DC2_NODE3 and /DC1_NODE1 are consistent for TABLE1
INFO [RepairJobTask:13] 2019-10-23 01:00:19,337 SyncTask.java:66 - [repair #X] Endpoints /DC1_NODE2 and /DC1_NODE1 are consistent for TABLE1
INFO [RepairJobTask:12] 2019-10-23 01:00:19,337 SyncTask.java:66 - [repair #X] Endpoints /DC2_NODE3 and /DC1_NODE2 are consistent for TABLE1
INFO [RepairJobTask:17] 2019-10-23 01:00:19,337 SyncTask.java:66 - [repair #X] Endpoints /DC1_NODE2 and /DC1_NODE3 are consistent for TABLE1
INFO [RepairJobTask:11] 2019-10-23 01:00:19,337 SyncTask.java:66 - [repair #X] Endpoints /DC2_NODE2 and /DC1_NODE2 are consistent for TABLE1
INFO [RepairJobTask:8] 2019-10-23 01:00:19,337 SyncTask.java:66 - [repair #X] Endpoints /DC2_NODE1 and /DC1_NODE2 are consistent for TABLE1
INFO [RepairJobTask:20] 2019-10-23 01:00:19,338 SyncTask.java:66 - [repair #X] Endpoints /DC2_NODE2 and /DC1_NODE1 are consistent for TABLE1
INFO [RepairJobTask:8] 2019-10-23 01:00:19,340 RepairJob.java:143 - [repair #X] TABLE1 is fully synced