Comparison with Calvin?

[Luke Lu]

http://cs-www.cs.yale.edu/homes/dna/papers/calvin-sigmod12.pdf

Thanks in advance.

__Luke

[Maysam Yabandeh]

Hi Luke,

Calvin partitions the transactions among sequencers to be assigned with deterministic execution orders which has to be respected by all the next executers (schedulers). Partitioning is the key here to scale to 500K TPS. It has it own drawbacks as well. Normally, the partitioned nodes has to communicate to decide on a consistent order, which obviously would be inefficient. Calvin relies on accurate clocks to circumvent this problem: every sequencer sends its assigned orders to all schedulers every 10 ms and no scheduler progresses until it receives the batch from all sequencers. To allow the system to progress despite failure of a sequencer, the sequencer is replicated using Paxos. I guess the fail-over is still not implemented in Calvin, and I am not sure if a quick, efficient fail-over mechanism would go with the current approach, but we need to wait and see.

Omid implements an optimistic concurrency control mechanism and does not fix a pre-execution order for transactions: the order is implicitly determined only if two transactions conflict. In Omid the clients run the transactions (vs. Calvin where the system does), which allows running arbitrary transactions. In Calvin the rows that will be accessed by a transaction must be specified prior to execution (there are tricks to circumvent that but it seems that it would not go with arbitrary transactions). Similarly to the sequencer in Calvin, the status oracle (the conflict detector) in Omid is a bottleneck: on a dual-core machines it scales to 120K TPS (2 modified rows/txn) and 80K TPS (8 modified rows/txn). We have designed a partitioning scheme for Omid which allows it to scale to 1+ million TPS. However, since the current delivered performance is good enough for lots of potential users, we decided to postpone the implementation of the partitioning scheme.

Cheers
- Maysam Yabandeh

[Luke Lu]

Thanks for the reply, Maysam!

Very informative. It'll be great to see the partitioning design though.

[Maysam Yabandeh]

Hi Luke,

I put a proof-of-concept implementation of the partitioning design on the megaOmid branch of Omid. The preliminary results shows scalability to 750K TPS. More details are at:
https://github.com/yahoo/omid/wiki/MegaOmid

Cheers
- Maysam Yabandeh