Sample cluster configuration interacts poorly with rebalancing, bug 246
Freeman, Tim
<cluster>...
<server>
<id>0</id>...
<partitions>0,1,2,3</partitions>
</server>
<server>
<id>1</id>...
<partitions>4,5,6,7</partitions>
</server>
</cluster>
Based on my reading of ConsistentRoutingStrategy, this seems broken. A key will get hashed to a partition 0 through 7 randomly. Let's think about a key K that hashes to 1. The replication factor is 2. The first copy of K is in partition 1 on node 0. ConsistentRoutingStrategy will go find the next partition after partition 1 that is not on node 0 to store the second copy of K. The first one is partition 4 on node 1. In general, partition 4 will get the secondary copy for keys hashing to 0,1,2, or 3. Partition 0 will get the secondary copy for keys hashing to 4,5,6, or 7. Partitions 1,2,3,5,6, and 7 will only get the first copies of keys, so they'll be about 1/5 the size of partitions 0 and 4.
If we rebalance, things get worse. Suppose we add a node number 2, and move partition 2 to it, leaving all the other partitions where they were. I suppose rebalance should copy partition 2 from node 0 to node 2. Before the rebalance, K will be stored in partitions 1 and 4. After the rebalance, the second copy for K will be in partition 2 on node 2. However, that second copy is not there, it's in partition 4, where it may linger forever. Furthermore our reliability is broken. Suppose that after the rebalance and before any subsequent writes to K, node 0 goes down. An attempt to read K will look in partition 1 on node 0 (which is down) and partition 2 on node 2 (which doesn't have it), so it will fail to find any good copies even though only one node went down.
If this is correct, one possible fix is to require that, if the replication factor is N, then every N consecutive partitions are on different nodes, both initially and for any configuration reached by rebalancing. If this is the right thing to do, then:
* the example quoted above should be fixed to conform to this rule,
* the rule should be checked in the initial configuration and before enacting any proposed rebalance, and
* the code in ConsistentRoutingStrategy.getReplicatingPartitionList should just look at the first numReplicas partitions rather than potentially looking at all of them.
I filed this as bug 246 at http://code.google.com/p/project-voldemort/issues/detail?id=246. We can close it if it's based on a misunderstanding. I haven't tried running the scenario yet where I predict above that rebalancing will break. I will update the bug if it breaks as I expect, or close the bug and post a retraction here if I'm wrong.
Tim Freeman
Email: tim.f...@hp.com
Desk in Palo Alto: (650) 857-2581
Home: (408) 774-1298
Cell: (408) 348-7536
bhupesh bansal
The configuration example is kept simple for ease of understanding, you are right that for this example all replication for partitions
0,1,2,3 will go to partition 4 and 4,5,6,7 will go to 0.
Voldemort comes with a cluster.xml generating script which randomizes the partition allocation to avoid this problem. In any case
rebalancing will keep number of copies consistent across two nodes. If you asked to move partition 2 --> node 1 Rebalancing will make
sure that for keeping replication level consistent as 2 it will keep a copy of 2 data on node-0 itself.
Hope this helps.
Best
Bhupesh
--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To post to this group, send email to project-...@googlegroups.com.
To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.