commited questions

[Paul Moore]

A

seems like the word committed is used in a few different senses. In a previous thread I saw the phrase 'leader tells the followers to commit'.

This seems to not be the same as  'present in the log of majority of servers' - which is what I thought committed means.

B

should I be writing 'committed' in log entries? Or does cluster startup automatically work out the last committed entry (I know I could work through the logic but I just wanted to ask what the consensus is)

[Kijana Woodard]

When people write 'leader tells the followers to commit', I assume they mean that the leader updates its commit index for the next heartbeat. From that, a follower knows it can apply the log entry to its state machine.

--
You received this message because you are subscribed to the Google Groups "raft-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raft-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oren Eini (Ayende Rahien)]

There are two meaning for this word.

When an entry is present on a majority of the nodes, it is considered committed. 

That is, at some point in the future, we are guaranteed that it would be applied to the state machine.

Note that it it possible that an entry is committed without anyone in the cluster actually being aware of that yet.

Once a leader get enough confirmation from nodes, it knows that an entry has been commited, then apply it to its state machine.

At that point it tells all the other nodes that it has confirmed that this entry is committed, so it can be applied to their state machines.

Usually it is short handed to "leader tell them to commit"

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

[Diego Ongaro]

Let me reiterate John's recent warning: "Careful: the fact that a log entry is replicated on a majority of servers does not guarantee its commitment. This is a very subtle point about Raft. A log entry is only committed if (a) it has been replicated on a majority of the servers *and* (b) all of those replicas were created by the leader that initially received [created] the entry. In other words, if a leader is completing the replication of the log entry from an earlier term (whose leader crashed before finishing its replication), this entry is not committed until the leader has also replicated an entry from its own term."

[Paul Moore]

BTW - I just read the comments about lastApplied being as persistent as the state machine. Ie in real life it has to be persistent. This is not stated in the raft paper. The raft paper clearly says that laspApplied is volatile. This should be fixed as its very misleading

[Diego Ongaro]

That's harsh, Paul. Why do you think lastApplied has to be persistent? If the state machine is kept in-memory, there's no use to persisting lastApplied. That's how LogCabin operates, for example.

-Diego

[Oren Eini (Ayende Rahien)]

lastApplied is as persistent as the state machine.

In our impl, it is the responsibility of the state machine to deal with that

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

[Li Xiang]

> If the state machine is kept in-memory, there's no use to persisting lastApplied.

Even if the state is kept on disk, you do not have to persistent lastApplied.
We need to make sure reapply the same commands will not affect the
state and in some systems it is possible.

[Oren Eini (Ayende Rahien)]

It is actually pretty important to keep lastApplied.

a) Perf - avoid reapplying a lot of commands.

b) Complexity - making all operation idempotent is _hard_. Especially when they have side effects.

Typical example, assume that we use Raft to build key/val store, but we need to track the number of times a key has changed. 

Trying to do that properly with idempotency is really hard.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

[Paul Moore]

Sorry if it sounds harsh not intended to be. But the doc (which BTW is great) clearly says lastIndex is volatile, whereas it is in fact as persistent as the state machine. For log cabin thats volatile, for persistent databases thats persistent. Had me confused for sometime

[Diego Ongaro]

Ah, I see the issue now: Figure 2 of the In Search of an Understandable Consensus Algorithm paper lists lastApplied as volatile but does not mention whether the state machine is volatile or not. We were (still are) focused on in-memory state machines when we wrote the paper, as that's what I had implemented in LogCabin. I've made a note of this issue and will try to publish errata/updates including it someday. Sorry for the confusion and thanks for your persistence ;)

-Diego