Question regarding commit return

[mc lin]

Hi raft-dev

Here is a paper [1] which discusses about improve raft by return to client when committed. And it make sense. As we all know, raft will return to client only data are applied. I am very curious about how raft make decision to return only after applied. Is that possible raft will consider return after committed in the future?

Thanks you for your answer!

reference:

[1]Wang, Y., Wang, Z., Chai, Y., & Wang, X. (2021, April). Rethink the Linearizability Constraints of Raft for Distributed Key-Value Stores. In 2021 IEEE 37th International Conference on Data Engineering (ICDE) (pp. 1877-1882). IEEE.

[Oren Eini (Ayende Rahien)]

Primary issue is that returning after commit vs. apply means that you can't return to the client the new state.

Consider a simpler calculator.

A client sends: Inc 5 command

You _can_ return to the client as soon as you commit to the log, sure. But then you won't be able to tell it what the _result_ of the operation is.

If you don't _need_ to, sure. But in most cases, that is important.

To give a more realistic example. CreateUniqueUserName.

You usually need the _result_ of the operation to do anything meaningful, not just recording this.

--
You received this message because you are subscribed to the Google Groups "raft-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raft-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/raft-dev/5f31fad6-98b8-460d-bb7e-6a2f52f7b2a9n%40googlegroups.com.

--

Oren Eini CEO   /   Hibernating Rhinos LTD Mobile:  972-52-548-6969 Sales:  sa...@ravendb.net Skype:  ayenderahien Support:  sup...@ravendb.net

[Jinkun Geng]

Just made a quick view of the ICDE paper.

IIUC,

The commit return is somewhat to simply return "commit without execution result". EPaxos and Fast Paxos can also do such things, but such optimization is argued in John's NSDI'21 "EPaxos Revisited" paper that "it is not globally sufficient".

The Immediate Read optimization is to read from memory before dumping to stable storage. It can be correct if we are using some diskless recovery procedure. But based on Raft recovery, I am not sure whether it can maintain correctness: The client read from the leader's memory, and then got the KV result which have not been persisted to the disk. Then the leader fails, and the recovers immediately, and it fetches the data from stable storage, which do not contain this new KV. Then the other replicas also fail one by one, and recover from the stable storage, without containing the new KV. Finally, every replica recovers but nobody contains the new KV, but the new KV has already been read by the clients.  [Seems there are some corner cases.

[mc lin]

Thanks for you clarify!

As the commit return flow which discuss in the paper, Return immediately after committed we can also tell the updated value of a key, since a write operation is to simply insert, update, or delete the value of a key.

Is that the _result_ you mentioned here are not just the value?

Thanks

[mc lin]

Thanks! gjk...

This is a Great point, I thinks this paper not explore the raft recovery part yet. But if a new KV log was committed by leader, I believe that leader should be able to fetch that new KV after leader recovers. Since that log was committed which means more than half of peers have persisted  that log.

[Oren Eini (Ayende Rahien)]

How can you tell what the value would be?

Assume that you have a compare & swap option.

You are saying something like: CAS(Key, ExpectedOld, NewValue)

The queue of yet uncommitted entries is large (network issue).

You enqueue the command, but you can't commit or apply it yet.

Then we start committing. The commit stage is 1000, but the apply is at 500.

What can you return? You still have to _run_ the command, no?

To view this discussion on the web visit https://groups.google.com/d/msgid/raft-dev/fa558a52-4b25-4975-a0fa-aea79d175491n%40googlegroups.com.

[Jinkun Geng]

I guess the paper is saying "just return the commit acknowledgment", "I have committed your requests and your result will be known later".

[mc lin]

I see. Thanks for your patience!