Regarding persistence of commitIndex

[Sen]

I was reading this thread here where the persistence of commitIndex was discussed. As far as I understand, an event is considered committed if and only if it is replicated to at least a majority of the servers.

However, this is discovered by the current leader only when it receives all the acknowledgements from the followers. This means that once an entry is committed, it can't vanish. As this is the case, shouldn't the committed index be persisted.

Not persisting it doesn't cause any problems because an `AppendEntries` RPC by the leader would settle it and if no client request does happen, then the leader can commit a no-op (as I read in the paper and here).

However, if we did persist it then a node could start up and apply up to what was globally agreed as the highest committed index as far as it knows. Now, even if this node had crashed, it's commit index is still correct because the current global commit index is >= this crashed node's commit index.

In such a case, the node wouldn't have to wait for a message from the leader to know how far it has to commit. Otherwise it has to remain idle for that period.

What do you think? Or am I missing something?

[Sen]

One argument against keeping `commitIndex` volatile would be that the heartbeat duration is small so the nodes would be idle for hopefully a small while in which case it would offer no benefit.

[A. Jesse Jiryu Davis]

To make a value persistent means writing it to disk and waiting for an fsync to complete. This takes an order of magnitude longer than an in-memory update. So if we required the leader to persist the commitIndex each time it advances, it would make latency several times longer for updates, and it would probably hurt throughput as well.

The Raft paper is a little vague about when a node must persist a change to its state. Certainly it must persist the latest term it's seen and the last vote it granted. The paper implies that a node persists new log entries when processing an AppendEntries RPC, but I think it's ok to only append them in-memory.

--
You received this message because you are subscribed to the Google Groups "raft-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raft-dev+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/raft-dev/94d445e1-d999-45c1-be53-f64934e5d3e7n%40googlegroups.com.

[dr-dr xp]

Yes. Persisting the commit index offers clear benefits: upon startup, the server can immediately restore to the latest committed state. This eliminates the need to implement additional barriers that would otherwise be required to prevent clients from reading obsolete data from the state machine during recovery.

OpenRaft provides this feature already:

https://github.com/databendlabs/openraft/blob/50cb1cc3aa7e561361d018b8be17a93b7c57771a/openraft/src/storage/v2/raft_log_storage.rs#L70-L79