Roadmap ideas for JGroups Raft 2.0

[José Bolina]

Hey, everyone.

I've created a blog post with some ideas I've been playing with, which I would like to evolve and release on version 2.0.0. This is the post: https://jabolina.github.io/posts/jgroups-raft-2-0-x-roadmap/

I would very much appreciate any comments and feedback.

Cheers,

[Bela Ban]

Hi Jose

my feedback below

* Do you have a branch in which to see the changes? v2/rehat-red-tape?
* why isn;'t
* Config: withJGroupsConfig() <-> withConfiguration()? Plus a builder inside a builder? Configuration.create()? IMO this is too complicated, why not:

.withJGroupsConfig("test-raft.xml") // The name of the cluster the nodes we'll connect to. .withClusterName("replicated-hash-map") // Register the serialization context so ProtoStream can serialize the objects. .registerSerializationContextInitializer(new ReplicatedHashMapSerializationContextImpl()) // We use a custom configuration to set the raft ID and members. // This is utilized by JGroups when creating the stack to replace the properties. .withID("X"
.withMembers("A,B,C,X")
.build()

* ReadCommand / WriteCommand: hmm, are we back into megamorphic call dispatching? :-) Too complicated IMO, and what's the rationale for returning completion stages ? CFs? I prefer blocking calls with vthreads... in your example, you also don't use the return value (CFs)...

* Hmm... annotations everywhere... :-)

* All in all, I'm not sure what the new APi improves... I kind of find apply() -> submit() and Read/WriteOperation a useful change of semantics, and perhaps going from byte[] arrays to operations, but I'm not a big fan of operations...

I probably have to write a small example myself to see how to use the new API.

Don't misunderstand my feedback, but here's a bit of devil's advocate: somehow I feel that there's a version 2 coming and you have to make some API changes to warrant the new major version increment... :-)

Let's chat after I've built my demo using 2.0.

Other feedback, folks?

[José Bolina]

On Mon, May 5, 2025 at 8:19 AM Bela Ban <bel...@gmail.com> wrote:

Hi Jose

my feedback below

* Do you have a branch in which to see the changes? v2/rehat-red-tape?

Currently, I only have it in my fork (https://github.com/jabolina/jgroups-raft/tree/v2/new-api). I am using a JGroups SNAPSHOT version to have the constructor with `Properties` parameter. :(

 

* why isn;'t
* Config: withJGroupsConfig() <-> withConfiguration()? Plus a builder inside a builder? Configuration.create()? IMO this is too complicated, why not:

.withJGroupsConfig("test-raft.xml") // The name of the cluster the nodes we'll connect to. .withClusterName("replicated-hash-map") // Register the serialization context so ProtoStream can serialize the objects. .registerSerializationContextInitializer(new ReplicatedHashMapSerializationContextImpl()) // We use a custom configuration to set the raft ID and members. // This is utilized by JGroups when creating the stack to replace the properties. .withID("X"
.withMembers("A,B,C,X")
.build()

I guess this might be a naming problem, but the `Configuration` interface I was using there is really superfluous and we could drop it. A better name for it would be `SystemProperties`, since it is just a wrapper for `Properties`.

In the example, I am replacing the system properties for `raft_id` and `raft_members`, but it could replace any system property in the JGroups configuration.

<TCP bind_addr="${jgroups.bind.address}" ... />  could utilize something like `Configuration.create("jgroups.bind.address", "127.0.0.1").build().

This way, the user could use the XML as it is today, and only change programmatically if needed. But yeah, I think maybe exposing something specific only to RAFT might be better. I'll experiment with it.

 

* ReadCommand / WriteCommand: hmm, are we back into megamorphic call dispatching? :-) Too complicated IMO, and what's the rationale for returning completion stages ? CFs? I prefer blocking calls with vthreads... in your example, you also don't use the return value (CFs)...

Oh, please no more megamorphic calls!! This enters a bit in the internal implementation. The ReadCommand/WriteCommand are the interfaces exposed to the user, and they are just markers. Internally, these interfaces are sealed and have only a single concrete implementation.

They only serve to select the proper submit method (a read or a write), so we can implement optimizations for read operations down the line. We could achieve the same goal by exposing a single Command interface and creating a `read`/`write` method to submit the command, though it wouldn't catch a write operation submitted in the read method by mistake.

Regarding the return of CFs. I thought of having the sync and async counterpart for the methods since these seem simple to implement.

The sync version would receive an extra int and TimeUnit for timeout.

 

* Hmm... annotations everywhere... :-)

Yeah 😭 I am torn, not sure if there is another approach which makes it so simple as adding an annotation.

 

* All in all, I'm not sure what the new APi improves... I kind of find apply() -> submit() and Read/WriteOperation a useful change of semantics, and perhaps going from byte[] arrays to operations, but I'm not a big fan of operations...

I probably have to write a small example myself to see how to use the new API.

Don't misunderstand my feedback, but here's a bit of devil's advocate: somehow I feel that there's a version 2 coming and you have to make some API changes to warrant the new major version increment... :-)

Let's chat after I've built my demo using 2.0.

Thanks, let me know if you have problems with something. But the idea with these additions is just to streamline and simplify the boilerplate needed.

I BIG thing I haven't written in the blog, though:

The new API **IS NOT**  a replacement for the existing API with `RaftHandle` and implementing the `StateMachine` interface without the annotations.

They can live together without problems. Users should be able to choose what fits their needs.

 

Other feedback, folks?

--

José Bolina

[Martin Tauber]

Hi everyone,

here are my two cents :)

(1) I like the idea of the Cluster Monitoring and Managing functions I think they also could be a great help debugging the cluster.

(2) For the state machine I have implemented a powerfull but loosly coupled implemantation in my environment to make it easy to use. It exports two methods register(id, object) and execute(id, methodname, args).

with the register you register an object to the state machine. The object must be serializable and will be written and read from the state files. The execute method will search for the object in the registry and find the appropriate method of the methods class and execute it. The result must be serializable as well of course. This allows me to quickly and object and functionality to my state machine - but is is loosly coupled to typos in the name wrong arguments etc. will create errors ...

Just as an idea ...

Kind Regards

Martin

On Friday, May 2, 2025 at 11:03:05 PM UTC+2 jbo...@redhat.com wrote:

[Bela Ban]

Jose and I discussed this over a phone call. This is a recap, to keep
the discussion in the community.

I used his branch to implement a distributed register demo (basically a
long) that is updated by multiple threads in a number of nodes. Each
thread randomly applies an addition, subtraction, multiplcation or
division with a random number to the register. When stopped, every node
must have exactly the same register. This demo will be added to the 2.0
branch.

We're still trying to reduce the complexity of 2.0 further, e.g. by
removing the registration annotation. An idea is to the have a class
implement the state machine without implementing `StateMachine`, e.g.

@StateMachine
public class Register { // could possibly be changed to a record
    @State(id=1,version=1)
    protected long number;

    @ReadOperation
    public long get() {return number;}

    @WriteOperation
    public void set(long num) {this.number=num;}

    @WriteOperation
    public boolean compareAndSwap(long expected, long new_val) {...}

    // note that apply(), readContentsFrom(), writeContentsTo() are gone
}


This way, we could actually send operations defined in the state machine
in JRaft.submit(). Jose is thinking more about this, but please feel
free to pitch in an submit feedback. The more ideas, the better!

An interesting topic was on which JDK to baseline 2.0. Currently
protostream requires 17. I suggested 21, which is an LTS and has virtual
threads which are a great feature to have. If vthreads are available, we
could even use a blocking API (no CompletionStages) without downsides...

However, baselining on 17 or 21 means that some folks will not be able
to use it for the time being.
Feedback? Remain on JDK 11? Go to 17/21?

Cheers

--
Bela Ban | http://www.jgroups.org

[Martin Tauber]

I like the idea of annotation everywhere ;) But be aware that you would need to include some lib to scan your classed for the annotations - if you don't want to write that yourself ...  I am using org.reflections in my project.

my thoughts here:

The major problem i had to overcome was the serialization ... I am using jackson to serialize and deserialize the object of the state methods. I am packing the the request into a message object that will contain the serialized arguments of the method as well as the method name and the object identifier to identify the object that should be used for the method call. the result of the method is then also packed into a result message in the apply(). For the deserialization you need to carry the class information of the args and the result to be able to deserialize them - hope this makes sense.

Kind Regards

Martin

[Bela Ban]

On 07.05.2025 10:44, Martin Tauber wrote:

I like the idea of annotation everywhere ;) But be aware that you would need to include some lib to scan your classed for the annotations - if you don't want to write that yourself ...  I am using org.reflections in my project.

The experimental 2.0 branch [1] created by Jose uses protostream (Infinispan, but no deps on Infinispan itself).

my thoughts here:

The major problem i had to overcome was the serialization ... I am using jackson to serialize and deserialize the object of the state methods. I am packing the the request into a message object that will contain the serialized arguments of the method as well as the method name and the object identifier to identify the object that should be used for the method call. the result of the method is then also packed into a result message in the apply(). For the deserialization you need to carry the class information of the args and the result to be able to deserialize them - hope this makes sense.

I guess atm Jose uses protostream and annotations to generate marshalling/unmarshalling code... but this is still wip and I guess that's why we want to discuss it here at the earliest possible time.

We're still trying to wrap our heads around how to reconcile calling JRaft.submit(operation) and 'operation' in the class annotated with @StateMachine... this needs to be as simple as possible, otherwise there's no need to use annotations/protostream over application-based handling of byte[] arrays.

[1] https://github.com/jabolina/jgroups-raft/v2/new-api

--
You received this message because you are subscribed to the Google Groups "jgroups-raft" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jgroups-raft...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/jgroups-raft/96c9fd6c-a9b7-407a-a560-dabf37dc5c8en%40googlegroups.com.