Raft implementation on Vert.x

[Jordan Halterman (kuujo)]

CopyCat

I know some of you already know about this project, but I don't think I've ever mentioned it in the group and I know there has been some interest in it if only from an academic standpoint.

CopyCat is a framework for building highly-consistent, fault-tolerant replicated state machines. The project actually started out as a an implementation of the Raft consensus algorithm on the Vert.x event bus. However, I've recently refactored it into an abstract framework. CopyCat is now simply a pure implementation of the Raft algorithm that features a pluggable messaging layer. Additionally, it includes extensions for developing systems on the Vert.x event bus or using Vert.x TCP/HTTP clients and servers, which is why I'm still mentioning it here.

For those who are unfamiliar with Raft, it's a consensus algorithm for managing a replicated log - in the same family as Paxos. Raft uses leader election to maintain log consistency and coordinate state changes across a cluster. CopyCat now supports all the features of the Raft algorithm - many of which are quite complex - including local snapshots, snapshot replication, and cluster membership changes. The framework supports custom protocols and endpoints as service providers, allowing users to build replicated state machines over a variety of transports (TCP, event bus, other messaging systems, etc) and with custom user-facing interfaces (TCP, HTTP, REST, etc). I suppose using Vert.x's EventBusBridge, one could even develop a client-side Javascript application backed by a highly-consistent, fault-tolerant replicated state machine. But I digress.

For a more in-depth explanation of how CopyCat works see the lots and lots of documentation on the project page and for an example see the fault-tolerant key-value store tutorial. Note, however, that the key-value store is only used as an example because simple and easily understood; there are plenty of really good open source key-value stores that you should be using instead of creating your own :-)

Happy hacking!

Jordan Halterman

Disclaimer: This project is primarily just an academic exercise. Consensus algorithms are notoriously difficult to implement, and I make no guarantees as to the reliability, usability, safety, or efficiency of this particular implementation. However, with some time and contributions I have no doubt it can become a stable and useful framework.

[Sharat Koya]

Hi, since you posted this it seems that both vertx and copycat have moved on. Do you know if there is a implementation sample of raft using vertx available?  

thanks.

[Jordan Halterman]

Copycat is still designed in a manner that would allow it to be easily implemented on the Vert.x event bus. There's a project that uses Atomix as a cluster manager for Vert.x: http://github.com/atomix/atomix-vertx

Atomix is a high-level API for operating on replicated state machines via Copycat's Raft implementation. So, my answer to you is that it depends on your use case. Copycat can be run on the Vert.x event bus simply by implementing a custom Transport that uses the event bus. For example, here is Copycat's NettyTransport: https://github.com/atomix/catalyst/tree/master/netty/src/main/java/io/atomix/catalyst/transport The same can certainly be implemented on the Vert.x event bus, or you can just use atomix-vertx if you really just want the consistency embedded in your cluster via normal Vert.x distributed data structures.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/48e2c04d-af74-441b-9dd6-a73ea1740079%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jordan Halterman]

But TBH there are some issues with implementing Raft on top of the Vert.x event bus simply because the event bus is a higher level of abstraction than is needed by Raft to ensure safety. The Vert.x event bus is inherently location agnostic. When you send a message to address "foo," Vert.x defines where "foo" is and the application should not care. But Raft must care. If there are two listeners at "foo" then messages can be split between them. Raft relies on static addresses to track the state of each node in the cluster, and that's why Copycat/Atomix requires a unique Address for each replica. It wouldn't make sense for a server to move machines in Raft since it's largely tied to the on-disk state. That's why Raft has been implemented at a lower level (the cluster manager) instead.

If you want to use Raft in Vert.x you should use the Atomix cluster manager and keep a reference to the Atomix instance. You can use it to create custom replicated state machines or operate on the existing resources like locks, queues, maps, sets, membership groups, messaging, etc.

I'm always available to answer more questions! Feel free to hop in the Atomix chat room if you prefer as well.

[Sharat Koya]

Hi, thanks for that. My plan was to use it to help keep different vertx clusters in sync for slower changing data. With in a cluster I am happy with vertx (hazelcast) taking care of things but we have to deal with a WAN and wanted to keep slower changing data structures in sync across the clusters. Effectively a cluster is a node rather than a verticle within a JVM. If I could get a raft node running in the cluster using vertx's HA then the clusters would talk to each other.

Any pointers would be great.